I don't blog, and therefore have never written about the multiple times DTrace helped me and saved me a lot of time.
It also made my job fun again as a sysadmin.
Here are couple of examples I remember, from my days at a large financial institution.
We're planning a samba upgrade on a server with ~700 clients.
Non regression tests show the performance of copying large files has degraded by 30%
Dtrace, aggregating on smbd stacks, helped us figure out the root cause in half an hour. This was reported and eventually properly patched upstream.
We started to dtrace most large batch jobs or apps, and found lots of easy performance fixes. In house apps launching thousands of 'grep", "cut", etc.. with a LD_LIBRARY_PATH set to search over 50 directories OVER NFS. haha, libc's not there, not there.. ! ohhh there it is.
Identifying stupid behavior in vendor apps for which we had no source. (but could complain about and sometimes get a fix)
NIS groups (yeah Im old) slowing down NFS a lot, etc well, I guess I used it with profit at least once a week.
You could probably do some of what DTrace does with other tools, but: it was safe in production, it had little overhead, and it was fun :-)
Ah! I also used it to teach our interns a little about the OS internals. fbt probes with flowindent, watch them go "ahhhhhh that's what happens!".
My current job is all about linux and I wish everyday for something half as good.
So yeah, "it made my life better" is an understatement.
It also made my job fun again as a sysadmin.
Here are couple of examples I remember, from my days at a large financial institution.
We're planning a samba upgrade on a server with ~700 clients. Non regression tests show the performance of copying large files has degraded by 30% Dtrace, aggregating on smbd stacks, helped us figure out the root cause in half an hour. This was reported and eventually properly patched upstream.
We started to dtrace most large batch jobs or apps, and found lots of easy performance fixes. In house apps launching thousands of 'grep", "cut", etc.. with a LD_LIBRARY_PATH set to search over 50 directories OVER NFS. haha, libc's not there, not there.. ! ohhh there it is.
Identifying stupid behavior in vendor apps for which we had no source. (but could complain about and sometimes get a fix)
NIS groups (yeah Im old) slowing down NFS a lot, etc well, I guess I used it with profit at least once a week.
You could probably do some of what DTrace does with other tools, but: it was safe in production, it had little overhead, and it was fun :-)
Ah! I also used it to teach our interns a little about the OS internals. fbt probes with flowindent, watch them go "ahhhhhh that's what happens!".
My current job is all about linux and I wish everyday for something half as good. So yeah, "it made my life better" is an understatement.