Iosnoop For Linux(brendangregg.com)
brendangregg.com
Iosnoop For Linux
http://www.brendangregg.com/blog/2014-07-16/iosnoop-for-linux.html
8 comments
I just added an experimental execsnoop to the repo as well: https://github.com/brendangregg/perf-tools .
Great tool !
It is interesting because we are working to extract the same kind of information but with LTTng for low intrusiveness and remote analysis of production servers.
If you are interested, you can have a look at the following repository :
https://github.com/jdesfossez/lttng-analyses
All I see in COMM is ksoftirqd, btrfs-transaction or md127_raid. No real programs names :(
Those of the OS X persuasion can find similar functionality in fs_usage.
https://developer.apple.com/library/mac/documentation/Darwin...
https://developer.apple.com/library/mac/documentation/Darwin...
He also has a slide about OSX performance:
http://www.slideshare.net/brendangregg/analyzing-os-x-system...
http://www.slideshare.net/brendangregg/analyzing-os-x-system...
fs_usage is a useful tool; the original version of iosnoop(1m) has also been available since Dtrace was added to OS X 10.5.
There's sc_usage too for cumulative display of all system calls.
Very nice. Thanks Brendan!
Looks like a great tool - now I wish I had the storage to test it on...
If it's not reliable as claimed by the article, then does it really serve the purpose? There might be a code-path in an innocent little app that calls brk() often and then gets tainted as causing IO.
Aside, thanks for reminding me about blktrace. I had forgotten the name last week when I needed the tool.
Aside, thanks for reminding me about blktrace. I had forgotten the name last week when I needed the tool.
You can use it to study block device I/O details, confirming latency outliers, and analyzing device, size, and offset patterns. It's the COMM and PID columns that aren't reliable -- they show what's on-CPU at the time of block device issue, which is usually the origin PID, but sometimes not. This has always been the case for iosnoop (all versions). I'm explaining this caveat better today than I have in the past.
As for the origin PID... I usually trace that from a different layer of the stack (vfs, file system, or syscall). But this approach has different caveats: the overhead is much higher (more frequent requests), and if I want to associate it with the block I/O, then many solutions are going to require debuginfo, and/or a different tracer. Also, what if a process buffers minutes of disk I/O then exits, and then that PID is reused by a different process calling disk I/O? It's a remote possibility, but I just wanted to mention another problem in this area.
As for the origin PID... I usually trace that from a different layer of the stack (vfs, file system, or syscall). But this approach has different caveats: the overhead is much higher (more frequent requests), and if I want to associate it with the block I/O, then many solutions are going to require debuginfo, and/or a different tracer. Also, what if a process buffers minutes of disk I/O then exits, and then that PID is reused by a different process calling disk I/O? It's a remote possibility, but I just wanted to mention another problem in this area.