Google Compute Engine MPI Latency(blog.rescale.com)
blog.rescale.com
Google Compute Engine MPI Latency
http://blog.rescale.com/mpi-latency-on-google-compute-engine/
2 comments
This isn't that interesting or surprising. 100-200 microseconds is the latency of all ethernet that I have ever seen. Infiniband or other high performance networks can achieve about 10-100x lower latency, but are very expensive. Infiniband switches and cards can be double the cost of a cluster.
One of the things that Google is also working on is PCIe switching, and using that to get rid of having to the ethernet encapsulation and conversion. This allows for significantly lower latencies <5 μs.
That sounds awesome. Do you have any links to describe this research? I would be interested in the technology if it stands to lower the cost of low latency networks.
This is slightly off-topic, but...
There's plenty of references to physical Infiniband cards being allocated to a guest VM. What I haven't seen though is a hypervisor-level Infiniband vswitch. I wonder if that could help with some of the latency issues; for example, inter-VM memory access could be done via host-level shared memory. Ideally the vswitch could handle inter-host links too, saving the need for manual assignment of cards to VMs.
There's plenty of references to physical Infiniband cards being allocated to a guest VM. What I haven't seen though is a hypervisor-level Infiniband vswitch. I wonder if that could help with some of the latency issues; for example, inter-VM memory access could be done via host-level shared memory. Ideally the vswitch could handle inter-host links too, saving the need for manual assignment of cards to VMs.
Mellanox did some experiments a few years ago that showed that DMAing to the NIC and back was actually faster than going through dom0 in software. Things may have changed since then, although it's probably irrelevant since the people who want to use RDMA and the people who want to use VMs don't overlap much.
Just for comparison, on non-virtualized 10GbE I think we got ~30 us. I would guess the extra latency in the cloud is caused by the hypervisor.
Also, last I checked 16 Gbps Infiniband was cheaper than 10 Gbps Ethernet but people don't want to use verbs.
Also, last I checked 16 Gbps Infiniband was cheaper than 10 Gbps Ethernet but people don't want to use verbs.
To me this is where the "software defined networking" type of virtualization can really make an impact.
The network performance of a known cluster of virtualized instances could be extremely quick if you just lie, and say your packet went through a network, when really you just pass a pointer in the hypervisor..
I assume this has already been done, but at almost 200 microseconds, you know it hasn't been done in these experiments.
The network performance of a known cluster of virtualized instances could be extremely quick if you just lie, and say your packet went through a network, when really you just pass a pointer in the hypervisor..
I assume this has already been done, but at almost 200 microseconds, you know it hasn't been done in these experiments.
That's an interesting hack for MPI, but I don't think it would be worth the complexity of implementing it. There are two possible use cases here:
One: You're testing multiple VMs on one box to simulate a network. This is a sane use case, but a bit of overhead from network latency isn't going to kill you. It definitely isn't worth the effort of messing around with faking your network.
Two: You're trying to wring whatever performance you can out of a single box. In this case, you can ditch the VMs, and just run your MPI nodes as threads or processes on the local machine, using far faster and more efficient transports than TCP.
Also, in this specific case, note that the instances are not necessarily on the same machine.
One: You're testing multiple VMs on one box to simulate a network. This is a sane use case, but a bit of overhead from network latency isn't going to kill you. It definitely isn't worth the effort of messing around with faking your network.
Two: You're trying to wring whatever performance you can out of a single box. In this case, you can ditch the VMs, and just run your MPI nodes as threads or processes on the local machine, using far faster and more efficient transports than TCP.
Also, in this specific case, note that the instances are not necessarily on the same machine.
That is only the case for networking inside of a single physical machine. Most HPC MPI use cases span many machines, for which this is typical latency of ethernet.
In cases where I've used OpenMPI it was spanning machines as well as being multiple processes on the same box. The goal was making that interchangeable (in my use case).
I imagine google's compute engine isn't all on separate machines, but utilizes VMs heavily.. although it's probably a bad idea to put one customer's VM's on all the same box.
That's a long way of saying "of course you're right, I guess my thought doesn't contribute as much as I thought" :)
I imagine google's compute engine isn't all on separate machines, but utilizes VMs heavily.. although it's probably a bad idea to put one customer's VM's on all the same box.
That's a long way of saying "of course you're right, I guess my thought doesn't contribute as much as I thought" :)
You are also right of course. Intra-machine latency is quite important. Many problems can be decomposed in to smaller parallel parts that can be done per machine.