Linux Ftrace TCP Retransmit Tracing(brendangregg.com)
brendangregg.com
Linux Ftrace TCP Retransmit Tracing
http://www.brendangregg.com/blog/2014-09-06/linux-ftrace-tcp-retransmit-tracing.html
2 comments
It's borderline. While this is a hack, ftrace and kprobes are built-in to the kernel, and are immediately available on all our production Ubuntu and CentOS instances. And /proc/net/tcp is usually still valid for long-running sessions with retransmits. I'll install and use SystemTap when there's no other practical way.
I wrote some other TCP scripts using ftrace, and the caching of /proc/net/tcp approach didn't cut it (due to short-lived sessions, which were too often missing by the time /proc/net/tcp could be read), and I'll need to rewrite them in SystemTap (or something else). Those make much better examples of why SystemTap. I'll publish them when I get a chance.
I wrote some other TCP scripts using ftrace, and the caching of /proc/net/tcp approach didn't cut it (due to short-lived sessions, which were too often missing by the time /proc/net/tcp could be read), and I'll need to rewrite them in SystemTap (or something else). Those make much better examples of why SystemTap. I'll publish them when I get a chance.
I do like your method in that it doesn't require anything additional installed, that's a big advantage.
Sometimes SystemTap isn't an option because of various restrictions, such as long-winded change control or simply not allowing compilers on production systems. Plus debuginfo packages just seem to confuse people.
Sometimes SystemTap isn't an option because of various restrictions, such as long-winded change control or simply not allowing compilers on production systems. Plus debuginfo packages just seem to confuse people.
What distribution do you use with SystemTap, and do you get it from your distro's repositories or externally? I've spent a few hours trying to get a 'hello world' working on a couple occasions on CentOS and Ubuntu, and never really gotten anywhere. Although I have gotten kernel panics.
EL 5 and 6, just run "yum install systemtap" and "stap-prep"
Here's a basic example from my scripts folder: http://pastie.org/private/zggvir0zdflcixnvleximq
When a kernel module calls request_module(), this prints out the backtrace so you can see what calls it and why. Example output from a "service iptables restart" is included. If you want some homework, add the name of the module being requested to the printf, at a glance it should be accessible via $name
Here's a basic example from my scripts folder: http://pastie.org/private/zggvir0zdflcixnvleximq
When a kernel module calls request_module(), this prints out the backtrace so you can see what calls it and why. Example output from a "service iptables restart" is included. If you want some homework, add the name of the module being requested to the printf, at a glance it should be accessible via $name
What's the performance overheads of SystemTap? I've looked at it, and used it for poking at things under a microscope on a VM, but I've never used it on a production system. Could I instrument production code with it?
It really depends what you are doing and how you're doing it.
If you only pull a few variables out and aggregate their reporting into one printf, then your overhead is close to one printf. For something like tcp_retransmit_skb that's usually not even noticeable.
However if you're performing frequent live calculations on an oft-called function, like every single tcp transmit, like you can definitely introduce too much overhead and negatively affect the system.
I'm far from a SystemTap expert, but some low-hanging fruit is to look at what you're doing and see if you can do it fewer times, or see if there's a quicker way or a built-in way to do it already.
If you only pull a few variables out and aggregate their reporting into one printf, then your overhead is close to one printf. For something like tcp_retransmit_skb that's usually not even noticeable.
However if you're performing frequent live calculations on an oft-called function, like every single tcp transmit, like you can definitely introduce too much overhead and negatively affect the system.
I'm far from a SystemTap expert, but some low-hanging fruit is to look at what you're doing and see if you can do it fewer times, or see if there's a quicker way or a built-in way to do it already.
get it
I was gonna say, I have done exactly this using SystemTap.
Looking at your script, stap has a lot more "batteries included" so such a script ends up being a lot shorter.
Even with a custom guru-mode function to pull more data out the skb, such as window size or retransmission timer, it's easy to do this in under 30 lines using SystemTap.
SystemTap also has the advantage of being able to modify data in the running kernel, not just read it. I haven't used this a great deal, but some other guys at work have used it to reproduce corruption bugs which would otherwise take months to re-appear.