Facts

Philip Kucheryavy, Software Engineer in the Operations Team

• He is 24 and has a beard
• In love with Linux and Python
• Paranoiac
• Has no diploma of higher education
• http://github.com/Friz-zy

Dmitry Trofimov, Head of Front-End Design for UI of WoT

• Interested in coding since he was 14
• At 17, he wrote a simple GUI for MS-DOS
• In addition to World of Tanks, his favorite games include Heroes of Might and Magic III
• Big fan of Java
• Has a number of diplomas and certificates in programming

Dmitry Ovchinnikov, Web Developer

• He came to web development through Perl
• Has three diplomas of education
• Hates wires and recognizes only Bluetooth/Wi-Fi
• After seven years with Gentoo, switched to OS X and has no regrets
• Hides that he knows PHP

Numbers

• Over 3,400 employees work in the offices of Wargaming all over the world
• More than 15 games released by WG since the company was founded in 1998
• 100,000,000+ users registered in projects of Wargaming
• In 50 countries around the world, World of Tanks Blitz reached Top 3 applications in App Store and was downloaded more than 5,000,000 times

Technology stack

Philip Kucheryavy

 

Our server is driven by BigWorld engine, which is written in C++ and Python. All that is critical for speed is written in C++, the rest is done in Python.

We have a lot of hardware, a huge Zoo. There is no way to go without automation here. To automate things, we use Ansible, Fabric and Puppet. And we monitor with Zabbix.

We are looking into fashionable virtualization solutions like Docker but still do not want to let it into production. Currently, we are trying to transfer the local instances to such virtualization, as it provides major savings. Now they are also hosted by the virtual machines, but these are more “serious” solutions that are not so specialized, such as VMware and others.

A portion of everything is written in Erlang, such the transactions. But these are is only the external additions to the engine, subsystems acting as individual crutches, like the message delivery and communication. The engine is not involved in this.

We have a very particular development process, a lot of tools that are very customized to our needs. That is, we use open source, try to contribute, hold all sorts of meetups (I personally love to participate in them). At the same time, we have some very particular things that should not be put in public. Not because it’s a big secret, but simply because no one would need them, except for us.

We have our own hardware. We personally select the necessary hardware, buy it and then operate it. Of course, the cloud is great, but it is expensive and not always convenient.

If suddenly there is some sort of “explosion” in the load, we always have a back-up capacity on cold start. It is a sort of fallback option to quickly bring up another cluster in case of load jump.

When you fully own the hardware, you can realistically assess the situation. Let’s say, we have played for a long time with the hyper trading and still are trying to evaluate its pros and cons. In some situations, it only interfered with the work and we deliberately disabled it. By having access to the hardware, we can personally update the BIOS firmware and see what happens there.

In general, if a need arises, we can rearrange everything and bring the stack up within a day or less. Naturally, we try to load balance the hardware by projects, and when we need to shift the hardware somewhere else, it is not a problem, as it can be done quite quickly.

We handle the scaling mostly with RabbitMQ. In addition, BigWorld has its own connection technology in clustering (between machines and between clusters). A Russian server today is nine peripherals and a center. BigWorld has its own technology for connection of peripheral with the center, and its own protocol. The “Rabbit” is used for connection between the BigWorld and the web, separately, as a stack.

Our QA have access to the magic button “make me feel good”. That is, at any time, when they need a certain version of an environment or project, they press this button, and strange as it may seem, they start to feel good. I believe that everyone should seek to automate his work as much as possible.

Mostly, we have monthly development cycles. At the end, we roll out the production. More often, it happens locally. We roll out the production under manual control. This is partly automated and partly not.

We deploy directly from source, no packages. As a matter of fact, we work with large amounts of code, and nobody wants to compile each time the packages of many gigabytes. Let alone when we just need to make a patch. In addition, the delivery is provided simultaneously to many servers. This immediately fills the channel and resources, which at the same time also experience the load from the players. In other words, everything must be done very carefully.

Dmitry Trofimov

 

What is BigWorld on the client side? Ironically, it is the same Python as on the server. In our company, BigWorld is connected to technology that allows to integrate Flash into the game client. There’s a virtual machine, a proprietary implementation from AutoDesk. There is also a GFX player that runs Flash inside the client.

Yes, we use Flash as the UI for Tanks.

It is believed that Flash is dying. Let me say that this is nonsense. The trend is that Flash becomes a highly specialized technology. Yes, all sorts of Flash games on the Web, most likely, will die. But in such major projects as “Tanks”, “Ships”, and so on, Flash has proven to be a very popular technology. It is perfectly suited to meet the challenges in terms of performance and in some other areas… like adding buttons and that sort of thing.

Why Flash? First, when we took up Flash, there were no alternatives like HTML5. Secondly, Flash has a native support for HTML and we do some stuff by using HTML markup. Actually, I would argue whether HTML5 is faster or provides higher performance. At least for our client, the analysis revealed that HTML5 would provide less performance than Flash.

Switching to HTML5 would mean retraining a vast team of Flash developers for a new technology, which cannot be called an optimal solution. And, of course, don’t forget that, in this case, everything would have to be rewritten. This is not something that we want now.

Personally, I would not count on HTML5 at this stage of its development, given that it doesn’t stand very well in the market. It has competitors that are advancing quite actively, including Mozilla, Unity, they are now a very close relationship to develop this technology. As a result, HTML5 might step aside altogether.

By the way, we tried not to use Flash by making UI with available alternatives, such as Python components. As it turned out later, these were just terrible crutches, we constantly lacked something. Besides, what was available couldn’t deliver what we needed. We began to think what to do about it and started to write, on our own, various components. As soon as it came to complex animations, effects and other things, it was clear that we need a very serious editor to define various visual things, animate them and make them “alive”. Flash was the only truly powerful technology that was able to handle it all. Since then, we keep using it.

Even though we are looking for an alternative to Flash, so far it perfectly handles the current tasks. For now, we are not planning any internal implementation.

In the Flash, we use all sort of open source things, like GreenSock, ported to standard Java libraries in order to work with data structures.

The only thing is that, to some extent, finding flash developers has become a real problem. The fact is that our demands in terms of Flash have grown considerably, while almost all of the most experienced guys are either already working in our team or are engaged by similar companies. It has become really hard to find a good and experienced professional in the market.

Philip Kucheryavy

Of course, Flash is not the only client technology. For different platforms, we use different technology. So, for example, we craft a separate client for Xbox; for the mobile version, we also have a separate engine, in the development of which is actively involved our studio. We have many different technologies.

Data storage

Dmitry Ovchinnikov

 

On the Web, we use a Python, Django, memcached and MySQL stack, they all work well.

Amazing as it may see, we don’t store user data but transfer them further: we just authenticate them and transfer. It is difficult for me to estimate what is going on out there behind this “large stone wall”, but as far as I remember, there’s MySQL. Or more precisely, Percona.

Philip Kucheryavy

We are still experimenting with different MySQL forks. Basically, everything depends on the hardware. All data related to BigWorld and the web rely on MySQL. The web has its own databases, and Big World – its own, but they are able to communicate with each other through RabbitMQ.

We are using Percona and MySQL. In addition, we look at them on different operating systems. I know for sure that we are not engaged in super optimization, our solutions are mostly out of the box.

RabbitMQ can be easily clusterized, and it fits our tasks. Historically, we are actively using it. For deployment, we are developing various proprietary tools (for logging and other goodies).

We pay attention to modern and trendy stacks, such as Kibana and Logstash. For them, we are considering, for example, ZeroMQ. But the cluster needs a centralized solution, where ZeroMQ doesn’t fit in, as it is decentralized. RabbitMQ can be better integrated in our scheme.

We have our own Hadoop cluster. The data guys are working with it for statistics and related things. We collect these data not for the players but for ourselves. We aggregate various statistics, we analyze them, and then our analysts make calculations.

We also use NoSQL databases, but not for production. In our deployment, we use MongoDB to store statistics and information: We use it to collect statistics from servers and process them further by our proprietary tools. Roughly speaking, it’s just a JSON data storage. This is not production, it’s just a self-help.

Challenges and fails

Philip Kucheryavy

There was not a single project where we managed to completely avoid fails. Personally, my first fail occurred on a test server. This server was also involved in production, but it was intended not for the players but for ourselves, to run the tests. It was my very first deployment in the company, and at the time, I was with the company for only three months. I rolled out the code, we tested it, and it turned out that I provided a completely wrong code. I had to rework it and ask QA to check everything again. I was ashamed. This situation made me remember very well that it is worth to double-check everything and to be more attentive.

Often, the fails are caused by something that is beyond our control. Sometimes, it is the equipment that fails, and we have to respond quickly. Sometimes, the technical issues arise not in our company but in external providers.

There was once a situation when the support service received many complaints that the players couldn’t play, everything was lost, we would all die, nothing was working. We tried to figure out what happened, but no one would come forward and recognize its fault. The suspicion fell on one of the backbones in Russia. Our guys from support team simply called all their friends who have ever worked with providers and found out that, indeed, there had been a backbone failure at a certain provider. They just didn’t want to recognize it.

Most of the problems with providers arise in China. Of course, the load there is high enough, but the problems arise simply because of the connection with China. The particular aspects of Chinese Internet are such that a link between two neighboring sites may be worse than between me and China. And we should also not forget about all their mess with firewalls, all technical details take a very long time to agree.

China is actively filtering and cutting everything. Everyone who works with China and its providers must certify everything that goes out. They are actively looking for ports and cut everything that comes in excess. We had to agree with them even the ports.

For beginner game makers (and not only for them)

Dmitry Trofimov

When starting your own project, you need to understand what is your initial budget. If your initial budget is less than a million dollars, forget about any Flash. Flash can be really useful only in large projects. If the project is small and you want immediately pop up and stay afloat, it is better to choose a simpler technology, perhaps, by starting with Unity. You can even write something with no frills and compile open source libraries with C++.

We often say: “This is GameDev, baby”. GameDev market is very rough. The rate of technological change here is simply fantastic, and no one is protected from anything.

Sometimes you have to support a legacy solution just because it’s plain cheaper. Even if we adopt some ultra relevant and promising technology to work on a project, it is possible that it will be very difficult and expensive to change this technology, when needed. So, the choice is simple: either to accept and live with it while supporting old technology or rewrite everything globally, change your personnel and retrain. This is expensive and often doesn’t make sense.

Philip Kucheryavy

Proceed round by round. To make sure your product pays off as quickly as possible, it should be done based on your own knowledge. If you know something and like it, there is no reason to change the technology and try get a quick insight into something else just because it is cooler and that technology will be supported for another N years.

The scheme is simple: craft a product and get money. Hire developers, craft an even cooler product and get more money. But crafting something Great all the time… Well, one day it might not work out. Not everyone is able to give birth to Minecraft.

All depends on resources, such as money, time and people. If a start-up needs to limit itself to the maximum extent, then follow the advice described in all textbooks on business – squeeze out the most from what you already have and try to shove the rest on someone else.

Delegate responsibilities. Let’s say, if the company has no resources, no good administrators for hardware, why bother with it at all? It is easier for such company to go to the cloud, rise quickly, earn money, hire someone and only then start thinking about the hardware

Dmitry Trofimov

Different approaches and attitudes to technology represent different business models. For example, you can spend at once a lot of money, craft some ultra abstract system (which, by the way, will be very expensive to support). But, at the same time, you may even not be able to enter the market, because the start was too long. On the other hand, you can downsize, hardcode and shitcode, then agonize about providing support but, at the same time, enter the market and quickly implement some features. And get money.

About myself

Philip Kucheryavy

In Wargaming, I am working already for more than two and a half years. I was hired as a junior straight from university, where I studied radiophysics. In the end, I didn’t finish my studies and instead opted for the work.

How did I get here? After the third year of studies, I was actively looking for what to do. At that time, I had an advanced level knowledge of Linux and Python. I found a vacancy in WG, but missed this opportunity. I was invited for a security job in the bank and there were some other small vacancies, but I waited for a month, and there was another vacancy in WG. I came for an interview and, in the end, stayed here, which I don’t regret at all.

I love to read very much and absorb huge amount of information. Among web resources, I read Habrahabr and Hacker Magazine..

At my spare time, I craft things in Python on GitHub. But these are home-grown designs mostly aimed at self-training. I try to deliberately go into the areas where I have no experience but which are interesting for me.

At work, I also write in Python, but these are mostly some small bells and whistles designed to help, automate something. At home, I’m trying to get involved with other things, such as asynchronous programming.

For me, the work is a hobby that pays good money. How else to describe it? WG is a team where I come to talk about my topics of interest. This is a community, which organizes itself by interests.

From time to time, I speak at meetups, and this is really interesting. These events gather different people, for example, from Minsk. They don’t know each other, and try to team up and get acquainted. It’s great when you come back, you see the same person for the third time, and you are already communicating perfectly, sharing contacts. This brings people together.

Dmitry Trofimov

I am involved in coding since I was 14. I graduated from college as a technician/programmer. Immediately after the studies, I went to work as developer. I was hired under the same scheme that we have already discussed. I had no experience in Java, but I was hired for training as a Java developer. I worked in this position for about a year, but then there was a crisis and I ended up in GameDev. At that time, RamStudio, a company, was looking for Flash developers, I passed the interview and learned there to use Flash technology, which I initially didn’t know. Next, I decided to move to a more serious project and, after some time, got into Wargaming.

After a while, I got “powerlevelled”, learned the ropes of all necessary technologies and figured out specific features of the World of Tanks. Now, in my spare time, I am not involved with Flash anymore, I became interested in mobile apps development, as it is very interestingly overlapping with Java.

For me, the work is also a hobby that I am paid for. I’ve been involved in this hobby for more than ten years now, and sometimes I think it’s just a great luck and things like that usually don’t happen. I can say one thing, do what you like, and everything will work out for you.

Dmitry Ovchinnikov

Hah, everyone is such a good sport, everyone is learning something. But I love to sit on my couch and play with PlayStation :)* I love to eat well and my wild dream is to rewrite the Rails on Python. I have a very tender love for Rails and, since now the development goes in Python, I miss them sometimes.

I have a deep sympathy for the world of Ruby, its conventions, agreements, approaches. I want to slowly drag all this… well, maybe to my desktop. But I like the very idea that I am doing this it. Even though it is unlikely that I will ever show it to anyone 🙂

I came to WG for a position of Python developer but, before than, I actually never saw the Python. I have been a long time in the web development. I started by using Perl, and then I had various relationship with Ruby, PHP, and so on. So when Wargaming offered me to try a job with Python, I thought, why not?

I’ve seen many companies. I worked in outsourcing companies, in product companies and now I am in GameDev. I can say that, in my opinion, Wargaming is the best gathering of professionals that you can find in the industry. In addition, they all are very active, funny and interesting people that you can communicate with every day.

In Wargaming, there is constantly some non-work related activity, including those organized by WG employees (sports events, technical events, meetups). The company doesn’t stand on the sidelines. It helps in every way and supports such self-organization.

We are holding monthly Python Meetups in Minsk with the support of Wargaming in an effort to widen the community of developers.

It’s very difficult to convey how you feel being a part of the Wargaming. You can view the photo reports, videos from our events, but until you come here, you will not understand it.

How to get into Wargaming

Philip Kucheryavy

I can’t say that, in order to work with WG, you need to know specifically about this and that. Personally, I believe that, in life, you need to find something that you like and do it. There is no need to focus on a specific company. When I was looking for a job, I searched for positions that would “fit me” and I knew that I wanted to get involved with Python. I’ve been searching by using this criterion and, at the time, there were job offerings just from three companies, including Wargaming.

In life, you need to find something that is your own and experience fun from it, follow this path and you will be hired wherever you go. When I went for the interview in WG, my future boss said that, after looking at my resume, he didn’t want to interview me, but HR insisted, because I had a very strong letter of motivation. In the end, the boss admitted that he did not regret accepting to interview me because my eyes were shining with interest and this was exactly the reason why I have been hired. I’m just a fan of my work.

Dmitry Trofimov

If you’re a fan of your work, you can find a job that you like and it is not necessary to sharpen your skills for a specific company. However, as the leader of a rather big team, I would mention one more skill, that you cannot absolutely do without.

Learn to communicate with people and find a common language. This is probably even more important than being a cool technical expert. Human language is imperfect. Sometimes, there are misunderstandings, you should be able to resolve them, and convey your ideas to your partner. This is especially important in such a big company as WG.

Dmitry Ovchinnikov

My example shows that a lack of experience with a particular technology is not an obstacle. There is a great need in having a focus, general experience and understanding of what is going on. As for a particular language… It’s not a rocket science, and it comes pretty quickly.

First, you get a general knowledge of how everything works, the general structure, architecture. As for the language to write on… in about five years, it becomes not so important.

You need more passion in what you do. The motto of our company is “Globally with passion.” We really need this passion. We like people who care a lot about what they do; people who are interested in doing something; who not just “go to their job” but try to become a part of larger Wargaming that we create together.