It was at this point that I realized that the performance of the user-land code in general (including NGINX) could be much better if it wasn't for the legacy network stack in the kernel, which does a bad job at some things such as slow performance accept()ing new sockets, and slower performance because every packet read has to be another system call unnecessarily copying memory; why not use shared memory? That's when I started to get interested in C10M... :-)
I also forgot to mention about performance. Whether you code * foo or the easier to comprehend and safer(?) foo[i] then the compiler still does an awesome job optimizing. However, it's much easier to assert() if variable i is in range rather than *foo. And it's also easier to read a verbose log variable i (usually a 'human-readable' integer) than to read a verbose log pointer (a 'non-human-readable' long hex number). At the time we wrote a lot of network daemons for the cloud and performance tested against other freely available code to ensure that we weren't just re-inventing the wheel. NGINX seemed to be the next fastest, but our 'dumbed down' version of C ran about twice as fast as NGINX according to various benchmarks at the time. Looking back, I think that's because we performance tested our code from the first lines of production code, and there was no chance for any type of even puppy fat to creep in. Think: 'Look after the cents and the dollars look after themselves' :-) Plus NGINX also has to handle the generic case, whereas we only needed to handle a specific subset of HTTP / HTTPS.
We do agree that the experienced developers are necessary along side the less experienced developers. At least when the project is getting off the ground and the less experienced developers are still learning the ropes.
Thanks! And I have not seen any teams doing this kind of thing either.
Another tidbit: The developer pairs were responsible for writing both the production code and associated automated tests for the production code. We had no 'QA' / test developers. All code would be reviewed by a third developer prior to check-in. However, at one stage we tried developing all the test code in a high level scripting language with the idea that it would be faster to write the tests and need less lines of test source code. However, because we did several projects like this, we noticed that there was no advantage to writing tests in a scripting language. The ratio of production C source lines to test source lines was about the same whether the test source code was written in C or a scripting language. Further, there was an advantage to writing the tests in C because they ran much faster. We had some tens of thousands of tests and all of them could compile and run in under two minutes total, and that includes compiling three version of the sources and running the tests on each one; production, debug, and code coverage builds. Because the entire test cycle was so fast then developers could do 'merciless refactoring'.
"I have never professionally audited a C project and not found a vulnerability."
Just out of interest: How many C projects have you audited? And have you ever looked for a relationship between the number of vulnerabilities and the percentage code coverage from automated tests?
Not all C developers need be a top 0.001% C developer to write high performance and rock solid code. Why?
I'm a very experienced C programmer and one day my boss came to me and said that the sales guys had already sold a non-existing client side module to a house hold name appliance manufacturer. The deal was inked and it had to be ready in only 3 months. Even worse, it had to run in the Unix kernel of the appliance and therefore be rock solid so as to not take the whole appliance down. It also had to be ultra high performance because the appliance was ultra high performance and very expensive. Now the really bad news: I had a team made up of 3 more experienced C developers (including myself) and 3 very un-experienced C developers. We also estimated that in order to code all the functionality it would take at least 4 months. So we added on another 4 less experienced C developers (the office didn't have a lot of C developers). The project was completed in time, a success, and almost no bugs were found, and yet many developers without much C experience worked on the project. How?
(a) No dynamic memory allocation was used at run-time and therefore we never had to worry about memory leaks.
(b) Very, very few pointers were used. Instead mainly arrays. And not just C developers understand array syntax, e.g. myarray[i].member = 1 :-) Therefore we never had to worry about invalid pointers.
(c) Source changes could only be committed together with automated tests resulting in 100% code coverage and after peer review. This meant that most bugs were discovered immediately after being created but before being checked in to the source repository. We achieved 100% code coverage with an approx. 1:1 ratio of production C source code to test C source code.
(d) All code was written using pair programming.
(e) Automated performance tests were ran on each code commit to immediately spot any new code causing a performance problem.
(f) All code was written from scratch to C89 standards for embedding in the kernel. About a dozen interface functions were identified which allowed us to develop the code in isolation from the appliance, and not have to learn the appliance etc.
(g) There was a debug version of the code littered with assert()s and very verbose logging. Therefore, we never needed to use a traditional debugger. The verbose logging allowed us to debug the multi-core code. Regular developers were not allowed to use mutexes etc themselves in source code. Instead, generic higher level constructs were used to achieve multi-core. My impression is that debugging via sophisticated log files is faster than using a debugger.
(h) We automated the process of the Makefile so that developers could create new source files and/or libraries on-the-fly without having to understand make voodoo. C header files were also auto generated to increase developer productivity.
(i) Naming conventions for folders, files, and C source code were enforced programmatically and by reviews. In this way it was easier for developers to name things and comprehend the code of others.
In essence, we created a kind of 'dumbed down' version of C which was approaching being as easy to code in as a high level scripting language. Developers found themselves empowered to write a lot of code very quickly because they could rely on the automated testing to ensure that they hadn't inadvertently broken something, even other parts of the code that they had little idea about. This only worked well because there was a clear architecture and code skeleton. The rest was like 'painting by numbers' for the majority of developers who had little experience with C.
The same team went on to develop more C software using the same technique and with great success.
Reason #4: Another problem is that the network kernel was never designed to do internet on the mass scale desired today. Companies like whatsapp devoted lots of time to getting e.g. 2M concurrent TCP connections (considered good) running on a single box, mainly because of the greedy overhead and design of the legacy network kernel. Whereas, in theory it should be possible to have 10M or more concurrent TCP connections on modern average hardware. So from this POV then the legacy network kernel is the bloated memory greedy mess that Java is to software development. See http://c10m.robertgraham.com/p/manifesto.html
Looks interesting. I'll take a look. You might also be interested in mTCP (https://github.com/eunyoung14/mtcp), or possibly adding mTCP functionality to packet bricks?
The title of the article does not mention CloudFlare; only bypassing. The fact that the CloudFlare architecture pushes a higher bandwidth of packets into the network kernel and bypasses the rest does not make it a good technique or to be recommended. If you are primarily interested in the best performance with a single NIC solution then I believe it is suboptimal. Why? You are asking the CPU to do two different types of work; optimized and unoptimized. Because of cache line pollution then the "unoptimized" work via the network kernel will pollute the other work. I may be wrong but I would bet you'd get better performance by separating your CloudFlare specific workload onto two boxes, each with one NIC. In this scenario then no cache line pollution can occur. Of course, these two boxes might not be easily possible within three existing CloudFlare architecture. But this has nothing to do with the general idea of packets bypassing the kernel. After the bypass you want the CPU to process those packets in the most efficient way...
If you are only interested in pushing infrequently used ssh packets into the kernel for e.g. low bandwidth health monitoring -- while all other packets bypass there kernel -- then why would this be considered a "toy app"? Surely it's a useful technique because it allows netmap to be used on very many cheap dedicated hosts for rent where only one NIC is available and you have no control over the hardware, or?
This is the inaccurate sentence: "Snabbswitch, DPDK and netmap take over the whole network card, not allowing any traffic on that NIC to reach the kernel." Obviously with netmap traffic to the NIC may reach the kernel...
The article appears to be inaccurate. Why? AFAIK it's possible to make use of netmap on a box with a single nic. I tried this out for myself about 2 years ago on a VMware virtual machine. How does it work? A user land packet filter can "forward" certain packets on to the kernel -- e.g. ssh packets in my case -- while others stick around in shared memory for kernel bypass. This means that I can ssh to the box and the ssh packets flow via the kernel, while the rest of the packets bypass the kernel, but all packets floor over the same nic. Nice :-)
Lines 18 and 32 of the pseudo clearly show a "while(true)" which blocks... so how can this be called "non-blocking"? Isn't looping until we get lucky pretty much like a "spin lock"? And because the number of loops is not deterministic then surely starvation can occur? For example, if all the threads try to push or pull at the same time to the head or tail of the double linked list?
FYI I also tried changing the mix so that 10% of the ops were del/put and 90% of the ops were gets. Unfortunately the ops per second went down to between 0.1 and 0.2 million :-(
FYI I experimented a little bit with the perf test and instead of using consecutive 32bit ints as keys then I used randomized 64bit ints as keys. I had to reduce the number of keys to 60 million otherwise it ran out of map space :-( Anyway, the resulting LMDB data file ended up as 1.9GB which makes an average of 34 bytes per key,value pair. Even bigger than before. I guess B+trees are only inherently more efficient than hash tables when a significant number of keys are very similar?
Here's a link [1] which tests LMDB on a Rackspace server with 16 vCPUs. To make the test fairer to ShardHashFile then the LMDB data file is stored in /dev/shm so that the disk does not get in the way of the test. The test first inserts 70 million keys (tried but failed to insert 100 million keys; how to do that?) using 16 processes (one for each CPU), then it reads the 70 million keys again using the 16 processes, then it updates 2% of the keys while reading the other 98% of the keys again, again using the 16 processes.
Read performance without any writing does seem excellent at 6.x million reads per second across the 16 processes. However, insert speed is very slow at only 0.1 million inserts per second. But then LMBD does not claim to be fast at writing. Unfortunately the mix 2% update, 98% read brings the read performance down from 6.x million ops per second to only 0.7 million ops per second. So LMDB seems like an excellent solution if one hardly ever wants to insert.
I would also be very happy if anybody can find ways to optimize the test since I could not find a tutorial on programming with the LMDB API. For example, is it necessary to always use a txn when putting and getting? I also couldn't figure out how to insert 100 million keys because in order to make the map size big enough then mdb_env_set_mapsize() always complained when giving it super large values. Is this a limitation of LMDB or how else to increase the map size so that 100 million keys (or more) can be mapped? And as a side question: Inserting the keys is so slow: Is there a faster way to initially insert all the keys in order to speed up the performance test?
I was also surprised at how big the LMDB data.mdb file gets with 70 million keys & values. The keys & values are 4 byte + 4 byte, so 8 bytes each. However, the data.mdb file ended up as 1.8GB which works out to about 27.6 bytes per key,value pair... which does not seem that good compared to a hash table, or?