Benchmarking High Performance I/O with SSD for Cassandra on AWS(techblog.netflix.com)

79 pointsby riffraffvor 14 Jahren13 comments

techblog.netflix.com

Benchmarking High Performance I/O with SSD for Cassandra on AWS

http://techblog.netflix.com/2012/07/benchmarking-high-performance-io-with.html

2 comments

halaylivor 14 Jahren

> Using sixty threads to write 2GB files at once using 64KB writes, results in 1099MBytes/s at 0.06ms service time.

It's not a good idea to scale disk writes using threads when you can use aio. You've introduced unneeded scheduling and locking penalty from threads. Additionally, such threads will be accessing and modifying kernel structures frequently and that's going to cause additional locking inside the kernel. They'll all be modifying same resources and structures while writing.

papercrunchervor 14 Jahren

Using async write I/O wouldn't allow them to test the capabilities of the underlying hardware as writes would return immediately.

Also, sixty threads on such a powerful machine is not really a big deal. Even though 0.06ms service time seems small, it is large enough such that you won't have more than a handful of threads competing for CPU resources (they'd waiting for some I/O completion)

wmfvor 14 Jahren

You certainly get notified when AIO O_DIRECT writes complete, so it is possible to benchmark AIO. For example, using fio.

adriancovor 14 Jahren

The point you should take away from this benchmark is that a 3 year old Centos 5 kernel with no tuning, no filesystem tuning and default mdadm options did 100K IOPS or 1GByte/sec pretty much regardless of the benchmark options in use. It just needed enough concurrency to load up the disk. There are many other options on iozone for using different variants of posix IO and threading. I showed the basic options and the mmap option. So with careful tuning you could do IOPS more efficiently, but what I found was very accessible out-of-the-box performance.

__alexsvor 14 Jahren

> It's not a good idea to scale disk writes using threads when you can use aio

Last I checked (at least a couple of years ago admittedly) glibcs POSIX aio_* implementation was implemented using threads despite kernel support for it.

wmfvor 14 Jahren

But there is a (nearly undocumented) native Linux AIO API.

Xorlevvor 14 Jahren

And that's better?

wmfvor 14 Jahren

It is better in the sense that it is actual AIO. Whether you want AIO so badly that you're willing to suffer the lack of docs is a case-by-case decision.

mvgooglervor 14 Jahren

Yes. The io_submit based async io API is much better, especially if you are doing lots of iops.

adriancovor 14 Jahren

for example the mmap read test used /opt/iozone/bin/iozone -w -s 2G -r 4k -i 2 -B -t 60 -F test0 test1 test2 test3 test4 test5 test6 test7 test8 test9 test10 test11 test12 test13 test14 test15 test16 test17 test18 test19 test20 test21 test22 test23 test24 test25 test26 test27 test28 test29 test30 test31 test32 test33 test34 test35 test36 test37 test38 test39 test40 test41 test42 test43 test44 test45 test46 test47 test48 test49 test50 test51 test52 test53 test54 test55 test56 test57 test58 test59

spleeyahvor 14 Jahren

    TL;DR

    What follows is a more detailed explanation of the benchmark configuration and results. TL;DR is short for "too long; don't read". If you get all the way to the end and understand it, you get a prize...

I believe it's "Too Long; Didn't Read"

jmreidvor 14 Jahren

Ya, they've got it backwards. I was a little confused when the TL;DR was longer than the first part.

robert_nsuvor 14 Jahren

I sure they meant "too long; don't read" to fall in line with their "prize" of coming to interview if you could make sense of the explanations and benchmarks.