The beginning of Nginx. Igor Sysoev tells the story of the renowned web server

On December 12, 2019, a surprise search was conducted in the Moscow office of Nginx, Inc. Igor Ippolitov, an engineer at Nginx, was the first to inform the public of it in his Twitter. The original tweet was removed shortly after the publication (Ippolitov was ‘kindly asked’ to do so), but other users have saved it and published photos of the search warrant.

According to this document, the search was conducted in connection to a criminal case instituted under Article 146 of the Criminal Code of the Russian Federation, part 3, paragraphs “b” and “c” (violation of copyright and neighboring rights committed by a group of persons by previous concert or by an organized group on an especially large scale). Not only may the developers and founders of Nginx lose their creation, but also be jailed for up to 6 years.

The plaintiff, Rambler, claims that some “unidentified persons” have created not later than October 4, 2004, during working hours and as per company management’s instructions, a “computer program called Nginx”, then published it on the Internet “with the intention to violate copyright rights”, and started its distribution claiming that Igor Sysoev, a former Rambler employee and the founder and developer of Nginx, has exclusive rights to it.

“We found out that the exclusive right of Rambler Internet Holding to the Nginx web server have been violated by third parties. In view of this, Rambler Internet Holdings assigned its right to file claims and lawsuits in relation to the violation of the intellectual property rights to Nginx to Lynwood Investments CY Ltd. that possesses sufficient competencies to restore the justice in the rights ownership issue,” – Kommersant.ru reports citing the Rambler Press Service.

According to Kommersant.ru, Lynwood Investments is affiliated with Rambler Group co-owner Alexander Mamut: the businessman used to own British book retailer Waterstones via this company.

Rambler Group estimates its losses at $821.6 thousand as of 2011.

As a reminder, in 2011, Igor Sysoev left Rambler and founded Nginx, Inc. In addition to open-source software, this company offers commercial products as well. Currently, Nginx is one of the most popular web servers worldwide; nearly a quarter of all websites on the Internet use it.

By 2018, Nginx revenues have reached $26 million; in March 2019; F5 Networks, a world leader in multicloud services, purchased it for $670 million. The project development team, including its founders Igor Sysoev and Maxim Konovalov, continues working on Nginx as a part of F5.

Back in 2011, Stepan Ilyin, then-Editor-in-Chief of Xakep.ru, has interviewed Igor Sysoev and asked whether Rambler may lay any claims in relation to the intellectual property rights and whether Igor retains the rights to this software package. Sysoev responded that everything is OK: he started developing Nginx prior to taking a job with Rambler, and the product has been released under a BSD license – i.e. as open-source software – from the very beginning.

– Igor, can you tell about your education, how you became a programmer, and what brought you to computers?

I was born in a small town in Kazakhstan. My father was with the military. When I was some one year old, he was transferred to Alma-Ata (currently Almaty), and I lived there until I turned 18. In 1987, I graduated from the high school and went to Moscow to apply to the Bauman Moscow State Technical University. However, I wasn’t accepted in the first year. So, I returned to Alma-Ata and started working as a laboratory technician in the Institute for Continuing Education of the Ministry of Geology of the USSR. The institute was equipped with old computers, Iskra-226, and I started programming on them in BASIC.

During that period, Radio Magazine has published a series of articles describing how to build a Radio-86RK computer. Thanks to them, I got a good understanding of the computer internal structure and work principles. My first encounter with computers occurred a little bit earlier: in the high school, I attended a computer club in the Young Pioneer Palace; it was equipped with Yamaha MSX machines. I remember that in my first program, I had mixed up 1 and l. As a result, it did not work…

– Do you remember your first program that was used by other people?

My first big program created for public use was the AV antivirus. I wrote it in 1989-1990. It was written in the assembly language; so, the size of the assembly code was some 100 KB. The program was able to detect a few viruses; it had an embedded database storing signatures of viruses known in the USSR in that period. There were ten viruses at maximum: Marijuana, Sophia, Vienna, etc. This was my first program, I had distributed it in binaries (at that time, I did not publish the source code). The program has spread throughout the country, it was even installed on several factories. There was feedback as well: people were sending me viruses on diskettes. I had supported this antivirus for some time, but by 1992, I lost interest in it, and the program died.

In 1994, I graduated from the university. A year earlier, I started working as a system administrator in an oil trade company. I had worked there for almost 7 years and decided to leave in April 2000. At that time, NASDAQ has crashed, the dot-com bubble collapsed, and I decided to switch to the Internet. I had worked in XXL.RU Internet shop for six months and I vividly remember that on November 13, 2000, I joined Rambler.

– What was your job in Rambler?

I was a system administrator. In addition to this work, I started writing programs in my spare time. I would like to note that programming was not a part of my job. But I had time and desire; my first project was an adaptation of the patch compressing the Apache output. By that time, the name mod_gzip was already used; so, I named my variant mod_deflate; it worked with Apache 1.3.

Then I was asked to deal with the mod_proxy module. I looked at it and decided that it would be easier to write everything from scratch rather than adapt somebody else’s code. This is how the mod_accel module (a reverse proxying module and set of patches for Apache) was created. This happened in spring 2001 as well.

– So, you were creating all these modules for Rambler and concurrently posted them online in open access?

In most cases, yes. In fact, mod_deflate originated from a patch written by Dmitrii Khrustalev when he was working for RBK Group. That patch was the basis, so I probably wrote only a half of the code.

In fall, 2001, I decided to write a simpler and more powerful web server than Apache. Similar servers already existed in that period, but all of them were unable to proxy – they could serve only static content. These servers had a common defect: they worked in a single thread and, accordingly, it was impossible to scale them, for instance, on a dual-core computer.

By that time, I had accrued an extensive experience with Apache – both as a system administrator and programmer. The two modules written by me have given me additional knowledge: I had to review the Apache source code and figure out how it was working. Therefore, many things have migrated to Nginx from Apache ideologically. Not its code, but only its ideology: the entire Nginx code was written from scratch.

However, I did not like some aspects in Apache. For instance, you could inadvertently create a very difficult-to-maintain configuration. Just imagine: the website develops, new functions are added, and ultimately, it becomes impossible to maintain such a construction. You have to add something and ask yourself: “What would break this time because of this addition?”. I tried to avoid such things in Nginx. Its development started in spring 2002.

– How quickly your developments became known to people not involved with Rambler? How was the project evolving?

In 2003, my developments became known outside Rambler; furthermore, several sites started using Nginx. The first one was Rate.ee, an Estonian dating site that still operates. By the way, this is one of the most crowded sites in Estonia. Then mamba.ru and zvuki.ru started using Nginx to distribute MP3 files.

In the beginning of 2004, Rambler launched the foto.rambler.ru service. My colleague, Oleg Bunin, asked me to put the finishing touch on the Nginx request proxying functionality to be able to use it at its full capacity, including the implementation of the Rambler’s photo service. Up until that moment, the project was mostly an academic one: I was steadily writing the code, but this process could be never-ending, and it might never be put in production. So, I have finished the proxying function on an urgent basis. In early 2004, the version supporting proxying was released, and the foto.rambler.ru service was launched based on Nginx.

On October 4, 2004 – it was a Sputnik launch anniversary – I released the first public version: 0.1.0.

– Currently, the Nginx project is growing very fast, but how it was in the beginning?

Now it grows fast enough indeed. In the beginning, the growth was more humble. For obvious reasons, in the first year, Nginx was gaining popularity mostly in Russia. Later, it became known abroad, and some enthusiasts started using it at their own risk. In that period, an English-language mailing list was created, and third-party resources started writing about Nginx. The users were sending me more and more wishes and remarks, I was making corrections, and the product popularity was steadily growing. Now the project grows very fast; this is the reason why a company was created. Alone, I wouldn’t be able to handle it.

– Are you saying that there was no promotion, and the project ‘made itself’?

From my side, there was no purposive promotion. Perhaps, the best promotion is a good product? The main reason behind its growth is that Nginx was actually “working”, and its users shared their positive experience with other administrators – i.e. it was the word-of-mouth marketing. In my opinion, the popularity of Nginx is determined by several factors. First, this is an effective and free software enabling to save hardware resources and money. Second, it really works well.

– But there are similar products as well, for instance, lighttpd.

In fact, there are a few more reasons: Nginx supports a bizarre combination of important features making it possible to create an efficient web infrastructure. I was gradually adding such features, and they made Nginx an ultimate tool. Concurrently, Nginx is not overloaded with unnecessary functions and still remains a pretty compact project. In addition, its module structure enables many companies and third-party developers to create their own extensions for the Nginx core. Overall, Nginx has become a sort of a web platform.

Speaking of lighttpd (lighty), at some point, it was more popular than Nginx and better known in the world. Its author is German developer Jan Kneschke. The reason behind its higher popularity was that for the Western world, Russia is an unknown country with balalaikas, bears, and snow – while Germany is a part of Europe. Of course, Jan Kneschke has a better knowledge of the English language, and the state of his English-language documentation was better.

By the way, the FastCGI protocol got a second wind thanks to lighttpd. Up until 2000-2001, it was an exotic thing; everybody had used Apache built-in interpreters: PHP, Perl, and Python. But because it was impossible to execute PHP code inside lighttpd, FastCGI became a solution and got a second wind. Back in 2000, people were saying: “Why do we need FastCGI? We have mod_php, and everything works fine with it.”

– What are the main Nginx application cases these days?

Heavily loaded sites primarily use it for proxying. Nginx is installed as the front-end, and it proxies back-end applications either via HTTP or via FastCGI or WSGI. A standard approach involves its combined usage with Apache; for instance, at my previous job, Nginx had worked that way for a long time; and they switched to FastCGI only a couple of years ago. Interestingly, in this case, the statistics shows that Nginx has appeared, while Apache disappeared. However, in reality, both elements are used: Nginx is just a component of the proxy system visible from the outside.

– Can you explain in a simple way: why in the world the requests have to proxied?

In other words, why are people using Apache in combination with Nginx? One might think: why use an extra component? Apache is convenient in situations when some app has to be executed, for instance, with mod_php. Now imagine that the PHP can generate, let’s say, 100 responses per second, while each response is 100 KB in size. Not all clients have a broadband connection; 10 years ago, modems were widespread, while mobile Internet is very popular these days. After all, somebody may have a bad provider or a slow tariff.

So, we have a response 100 KB in size, while the effective transmission speed towards the client is 80 Kbit/s (i.e. 10 KB/s). This means that the response will be transmitted to the client for 10 seconds. All this time, while the client is slowly downloading the response, Apache and PHP ‘consume’ 10-20 MB of memory per client. In other words, instead of doing tasks that can be performed quickly, Apache is waiting until the slow clients download their responses. This requires tons of memory and processor horsepower.

When we put Nginx between the clients and Apache, everything starts running faster. Nginx undertakes the entire response at the maximum speed, thus releasing Apache, and then slowly transmits it to the clients without using much memory. Nginx consumes not much memory or processor horsepower because it uses a different web server architecture: a nonblocking one based on asynchronous event processing. It allows processing thousands of connections in the framework of a single process (unlike Apache, where each connection is processed in a separate process or thread – HackMag).

In addition, we can transfer all static files from the back-end; this is a simple task, and Nginx performs it with the maximum efficiency. Depending on the memory and network connection with the server, Nginx may transmit tens of thousands of such static files per second.

– Let’s go back to typical scenarios.

The first scenario: we are just accelerating; this can be implemented even for a single site. We have Apache; then we put Nginx in front of it – and voila! A miracle happened! People do so and then write on Habr.com: “Wow! It’s working!” The second scenario: we are proxying, but have many back-ends. In other words, we can effectively scale the entire system horizontally – provided that the app allows that. In this situation, Nginx acts as a load balancer.

One of the shortcomings of the current implementation is the lack of various balancing policies. However, the people use the product, it works, while we intend to add new functions. What else? Another scenario: many people do not like Apache for various reasons. They want only Nginx and don’t want to install Apache. In that case, all their scripts work through FastCGI for PHP or through WSGI for Python.

Take, for instance, WordPress.com: they started using Nginx as a load balancer a long time ago, while their web server was commercial LiteSpeed. This year, they have completely migrated to Nginx; now their PHP works in the FastCGI mode.

Another standard scenario: Nginx distributes static content, including MP3, FLV-, MPEG4, pictures etc.

– Let’s talk about security. Had you discovered any severe vulnerabilities in Nginx since its release?

There were various vulnerabilities. However, none of them allowed the attacker to get remote access of execute code. The attacker could crash working processes, but not execute a malicious code. What is the main principle of an exploit? We have dropped something onto the server, and it is added to the server’s stack. The server works, makes a rollback, and encounters our code.

So, to implement an exploit, you must know where the process stack is. Normally, if you have Debian/Ubuntu and a binary file, you may reconstruct a crash situation on your PC, try to locate this stack, and create an exploit. To counter this threat, developers started randomizing the address space; for instance, this feature is implemented in modern versions of Windows.

– ASLR?

Exactly. Randomization. Our stack was here, and now it is there. Accordingly, it is impossible to guess: you have captured a packet but cannot figure out where is its stack now. Nginx is simpler in this aspect: there are virtually no data that can be read in the stack on the client’s side. There are a very few cases where this mechanism is used, but the code in these regions is ‘bulletproof’ enough. Nginx stores the data received from clients in the heap memory and allocates the memory using malloc.

Accordingly, if you write more data there, you won’t get on the stack pointer. This randomization has been in Nginx from the very beginning. Overall, it is very difficult, if not impossible, to write a workable exploit. In addition, processes handling the requests do not work on behalf of the root.

Of course, we had issued security advisories; they can be found on the website. In my opinion, we must react to error messages adequately, calmly, and professionally. For instance, trying to conceal a bug after its publication is useless; it just discredits the project.

– How many people were involved in the project initially and how many are involved in its development now?

For a long time, I was alone; I single-handedly wrote nearly the entire code. Four years ago, Maksim Dunin started helping me. In addition to the two of us, various people were sending us patches as the project developed. Many users just send letters describing a problem or express their wishes. They say: “Here is an error; it can be fixed this way.” And we are doing our best.

Currently, the project involves Ruslan Ermilov who handles all the documentation. He fulfills several tasks, including translation of the current documentation from Russian into English, information updates, and documentation adaptation with the purpose to make it clear and straightforward for users reading it for the first time. This is a common problem: the author writes documentation keeping in mind some context. For the author, many things are obvious; but as a result, some important details are omitted in the documentation. We are actively combating this problem: Ruslan views Nginx ‘from the outside’, he has a ‘fresh look’, and his writing is clear for everybody. In addition, Ruslan has an extensive experience in the development and documenting of complex software projects.

– Now, can we discuss Nginx, Inc. and why you decided to create a business?

I think, sometime in 2008, I received the first letter from an investor; I don’t remember now who it was. In the last two years, I received some ten such letters. The people wanted to do something with Nginx and establish a company. But I had refused because I am not really a businessman. However, ultimately, I realized that I cannot continue the project development alone, my capacity was not sufficient for everything.

It took me some time to understand how and with whom I wand to create a company ‘around’ Nginx. In fact, I rarely change my life; for instance, prior to Rambler, I had worked in the same company for seven years. Then I worked in Rambler for ten years. Changes make me uncomfortable. Still, this spring, I made a final decision to found a company in order to facilitate further project development. In part, this decision was inspired by Serguei Beloussov, the founder of Parallels and Runa Capital. After we talked, the idea to create a company became much closer to me.

– Serguei knows how to convince people, doesn’t he?

Serguei is a very interesting person; he is always a pleasure to discuss business and other matters with. Serguei is also a dominant boss; I think he influences plenty of decisions in his companies. As the owner, he likes to control everything and be directly involved in the business.

Overall, the negotiations with investors, signing the terms of the deal, etc. are tiresome things. First, because there are many boring details and huge volumes of legal paperwork in English. It is hard to read these documents even in Russian, not to mention English. Negotiating everything, finding a consensus: we want this, while they want that… Psychologically, this is difficult to me. But then the investors give your business a boost, and everything becomes easier.

– You had worked for Rambler and concurrently developed Nginx. Rambler did not have any rights to it? This is a delicate matter. How did you manage to retain the rights to the project?

Yes, this matter is delicate. Of course, it is of interest not only to you, and we have thoroughly elaborated it. According to the Russian legislation, the company owns what was created in the framework of your job duties or under a separate contract. In other words, there must be a contract with the person, and this contract must stipulate that some software product has to be developed. I was a system administrator in Rambler and developed the product in my spare time. From the very beginning, the product has been released under a BSD license – i.e. as open-source software. Rambler started using Nginx when its main functionality was already completed. Furthermore, the first to use Nginx were websites Rate.ee and zvuki.ru, not Rambler.

– Who else are employed with Nginx, Inc.?

We have Sergei Budnevich. He is a system administrator and maintains the corporate infrastructure. Our infrastructure is not that large, but it still exists. We have mailing lists, a main server, automated assembly, package testing, error tracking, etc. Sergei is of great help in these aspects. Right now, we are planning to release packages for several more Linux distributions: CentOS and Ubuntu. Sergei is responsible for automation of various processes involving the development, testing, and support. There are two more people: Andrei Alekseev (he does the marketing) and Maksim Konovalov (he is the boss who makes the company operating).

– What is your official position in the company?

Officially, I am its technical director. I am not really good in managing people; my primary focuses are the architecture of future projects and transfer of the developments to ‘the team’. It is difficult to delegate work; but the company was created specifically to enhance the development process and the product; so, I am now trying to teach myself how to do this. My colleagues handle organizational issues, communication with clients, marketing, liaison with partners, staff recruitment, etc. Of course, we encounter various issues; it is not that easy to learn how to communicate at different levels. In fact, all of us are involved in the company affairs because the staff is not big, while the amount of work is huge.

– It was difficult for you to delegate work because it seemed that others would do it poorly, and it is easier to do everything by yourself, right?

Yes, my approach was: I better do this myself because I will do it better, or because it is too long to explain what to do, or because it is psychologically difficult to say: “Do this.” For a number of reasons, it was difficult for me to delegate powers personally. Now, as a technical director, I am mostly responsible for the architecture and development quality.

– Thank you very much for the interview! It must be admitted that you have mastered the art of delegation as you directed us to Maksim Konovalov with all business-related questions.

By the way, this is my first interview. I agreed only because we have established a company. This spring, people from another IT magazine had asked me, and I told them: “Sorry, I don’t like, don’t want, and don’t know how to do this.”

– Thanks again! Maksim, in your negotiations with investors, had you presented to them a formal business plan? What are you going to make money on?

Funds have invested money in Nginx because they consider it a very promising product. Of course, a detailed business plan was important, but American investors make decisions not only based on a business plan stipulating how much we will earn next year to the accuracy of a few dimes. For them, it was important that Nginx is very popular and it is a finished, existing product.

Speaking of our money-making strategies: first of all, we want to establish the right balance between the free and commercial functionality. We want to achieve what a number of other companies failed to fully achieve in the past. There are several examples of businesses specializing in the development of open-source products who failed to maintain the right balance. As a result, they had to get rid of some features in their products, ask ridiculous money for them, etc. Ultimately, the users got upset, and the development of such projects stopped.

– So, you want to create a separate commercial product and find a balance between the open-source and commercial products?

We don’t want to create a separate commercial product; we want to make commercial extensions to our main open-source product. It will develop; new features wanted by the community will appear in it. The money we received will help us to advance the product development to a new level. Igor does not write the entire code alone anymore; the team-based development process is currently under construction. We hire people in Russia, the engineering team will stay in Moscow.

Overall, the open-source product is our primary focus, and it will remain in the future.

On the other hand, we are aware that many of our clients – big, medium, and even small companies – have been using Nginx for a long time. They have built their businesses with it, and are grateful to us. When we meet, they are saying: “The product is great, thank you very much! But it lacks some features we need. Can you implement them? We are ready to pay you for that.” Based on such conversations, we get an understanding of what we can sell without upsetting the supporters of the open-source product and discrediting the project as a whole. We collect such requests and compare them with wishes received from the user community. We look for intersections; if we realize that some functions are needed by everybody, not only by some company, we include these functions in the free version.

Some companies even tell us: “We can pay you for these features to expedite their implementation. We want them to be included in the open-source version, we don’t want these features to be exclusive and/or commercial.” This is called “sponsored development.”

At this point, we believe that commercial extensions will be mostly developed for major Nginx-based projects. For instance, commercial extensions may facilitate the handling of thousands of instances, add extended performance monitoring, additional functionality for hosting, cloud, and CDN infrastructures, etc.

– So, you are focused on the product, right? Don’t you want to sell separately your services, including deployment and consulting?

The point is that the company is small. It should remain small; we don’t want our staff to grow to several hundred people. We actively collaborate with partners, system integrators, software and hardware vendors; we actively look for channels that could be used for work via our partners. Consulting services will be provided partially through the partners and partially through us. Unfortunately, we are unable to render consulting and technical support services to all users.

– What can ordinary users expect in the foreseeable future? Are you planning any new features?

The release notes and code history for the last three months clearly show that since the company creation, we have significantly intensified the development and bug fixing processes. We have integrated plenty of revisions and new options. For instance, MP4 streaming has been added – Igor was asked to implement it for several years. The work is ongoing; the functionality is developing.

– Is it true that Igor Sysoev is the principal shareholder of the company, while the rest of the shares (i.e. the minority stake) belong to investors?

Yes, Igor is the principal shareholder. In total, there are three company founders. And of course, there is a group of investors; they jointly own a certain portfolio of shares. By the way, technically, the receipt of funds from investors is a simple procedure. Securities are issued in accordance with the respective legislation, and investors buy these securities for a certain amount of money. You get this sum and use it to fuel the company development. This is how it works in Nginx, Inc.