Rreverrse Debugging(huonw.github.io)
huonw.github.io
Rreverrse Debugging
http://huonw.github.io/blog/2015/10/rreverse-debugging/
15 comments
Windows has had this capability for at least 8 years with "time travel tracing" (or iDNA tracing), which produces a dump file that can be traversed back and forward in WinDBG or Visual Studio. Very useful for figuring out what caused unexpected state in complex integration scenarios with timing-dependent bugs. http://www.thewindowsclub.com/microsoft-time-travel-tracing-...
Yeah I love GDB. It's so mighty. The interesting that you can script your debuging and it's just mindblowing. It helped me much in debuging net communication. It was tough, but GDB let much! And It's for C/C++ too.
A commercial product covering similar territory is UndoDB http://undo-software.com/undodb/
> I’ve never been a huge user of debuggers. Being able to diagnose my code line-by-line, statement-by-statement sounded theoretically good to me, but I’ve never really “clicked” with it in practice.
Years ago, I would've been floored by a comment like this. But lately I find myself using them a little less often. These days, the codebases I interact with leverage more and more small-scope unit tests. This usually means that an unexpected test failure requires much less imagination to explain.
But debuggers remain a terribly simple way to identify the cause of a segfault/bus error. Even with code I've never seen -- if armed with just a stack trace I can identify a system misconfiguration or a workaround to avoid the error.
Years ago, I would've been floored by a comment like this. But lately I find myself using them a little less often. These days, the codebases I interact with leverage more and more small-scope unit tests. This usually means that an unexpected test failure requires much less imagination to explain.
But debuggers remain a terribly simple way to identify the cause of a segfault/bus error. Even with code I've never seen -- if armed with just a stack trace I can identify a system misconfiguration or a workaround to avoid the error.
I definitely use a debugger for getting a backtrace from a hard crash, but it is (well, was) pretty much the only case where I'd reach for it as the first choice.
Have you ever used a good visual debugger? I find that command line debuggers require you to keep too much state in your head at once, and even if you have a great memory and this isn't a problem, they still require you to query the new state after each step.
Sadly, most of the visual debuggers on linux don't meet my minimum standards for "good" (crashy, slow, typically have trouble inspecting certain types (mysteriously, since they're GDB frontends and GDB has no issue with it), etc...), and successive update to MSVC makes it's debugger worse and worse (although it's still the best I'm aware of at the moment)...
Sadly, most of the visual debuggers on linux don't meet my minimum standards for "good" (crashy, slow, typically have trouble inspecting certain types (mysteriously, since they're GDB frontends and GDB has no issue with it), etc...), and successive update to MSVC makes it's debugger worse and worse (although it's still the best I'm aware of at the moment)...
I used visual debuggers in college, Eclipse and Java. But then I found Ruby, and switched almost entirely from debugging to unit tests. Yes, they're not entirely the same thing, but I had found that using a debugger meant I didn't write tests. Once I found and fixed the problem, done.
Once I started writing tons of tests, I found I didn't need the debugger as often. When your methods are less than ten lines, they're much easier to reason about, and a method becomes a pretty decent 'unit' to check out, no need to step through.
However, this also probably has to do with my relative skill levels and the languages and tools involved too. Ruby debugging has gotten better, but only in the last couple of years. And I didn't really know how to write tests nearly as well when I was doing Java.
Once I started writing tons of tests, I found I didn't need the debugger as often. When your methods are less than ten lines, they're much easier to reason about, and a method becomes a pretty decent 'unit' to check out, no need to step through.
However, this also probably has to do with my relative skill levels and the languages and tools involved too. Ruby debugging has gotten better, but only in the last couple of years. And I didn't really know how to write tests nearly as well when I was doing Java.
I work in an area where unit testing is, generally speaking, more effort than its worth (game programming -- the state space is generally too huge for unit tests to be really practical). So I can't really comment on the effectiveness of them, other than to say that yes, they are different.
Typically, instead of unit tests, I write state tracking and visualization code. This goes a long way towards helping track down bugs and keeping me out of the debugger, but honestly, a lot of that is just an indication of debugging tools being crap in general.
Typically, instead of unit tests, I write state tracking and visualization code. This goes a long way towards helping track down bugs and keeping me out of the debugger, but honestly, a lot of that is just an indication of debugging tools being crap in general.
I've always find having to type commands in very distracting - makes it harder for me to think through things. When there's some kind of command grammar, and few prompts, and you have to remember to press Return, it's like trying to count while somebody shouts random numbers in my ear. Hovering the mouse and pressing individual keys is about all I can manage :)
Moving on from my own personal intellectual deficiencies, one nice thing about Visual Studio (that everybody should copy, in my view) is that running your program under the debugger is the default. So any time you see anything unusual happen, or there's a crash, you can start debugging straight away.
This is also good when combined with automated tests. Get your tests to stop the debugger at the point of failure (for VC++, something like "if(IsDebuggerPresent()){__debugbreak();}" in your test macro(s) will do the trick) just before they abort or throw or whatever. Then when you have a failing test, you have all the information you'd have normally - and the option of some post-mortem examination. Sometimes, you won't need it, but sometimes, you will. The latter situations are always a bit more stressful and I think it makes sense to optimise for that.
Moving on from my own personal intellectual deficiencies, one nice thing about Visual Studio (that everybody should copy, in my view) is that running your program under the debugger is the default. So any time you see anything unusual happen, or there's a crash, you can start debugging straight away.
This is also good when combined with automated tests. Get your tests to stop the debugger at the point of failure (for VC++, something like "if(IsDebuggerPresent()){__debugbreak();}" in your test macro(s) will do the trick) just before they abort or throw or whatever. Then when you have a failing test, you have all the information you'd have normally - and the option of some post-mortem examination. Sometimes, you won't need it, but sometimes, you will. The latter situations are always a bit more stressful and I think it makes sense to optimise for that.
So agree here. I have only used GDB command line for c++, pdb for python and the Chrome Dev Tools js debugger. GDB was a nightmare when I was in college, new to programming in general, not a clue what I was doing...
Then I got into js development, the Chrome one was night and day different. I could actually see what was happening, it displays where it is stopping every time. You can see that picture you need to paint in your head on the screen without trying to remember which letter to press, which bits of code you have to store in RAM in your brain. I have many complaints about it still, but it works pretty well for most of the things I need it for.
Then I was shown pdb, which is very much like GDB from what I remember of it. Now armed with my knowledge of how debuggers work from Chrome, I am much more confident in pdb, but it's still a pain in the ass compared to something more "visual".
Then I got into js development, the Chrome one was night and day different. I could actually see what was happening, it displays where it is stopping every time. You can see that picture you need to paint in your head on the screen without trying to remember which letter to press, which bits of code you have to store in RAM in your brain. I have many complaints about it still, but it works pretty well for most of the things I need it for.
Then I was shown pdb, which is very much like GDB from what I remember of it. Now armed with my knowledge of how debuggers work from Chrome, I am much more confident in pdb, but it's still a pain in the ass compared to something more "visual".
Definitely agree for stack traces, but what about other debugger features like breakpoints, step-by-step execution, or memory introspection? When working with a large, multi-threaded system, I have found these features more trouble than they're worth.
EDIT: Grammar.
EDIT: Grammar.
Is there a good GUI wrapper for gdb or lldb?
Ive tried using gdb with Emacs (via GUD[1]) if that counts, but it never worked out for me in the sense that I felt I got anything extra in return for the investment.
Maybe there's some special things I forgot to do, but out of the box GUD definitely felt (too) minimal for me.
Lately I've used LLDB when working on the .NET framework, and that seems like quite a reasonable thing to work with too at this point. I'd love some emacs-magic for this too :)
[1] http://www.emacswiki.org/emacs/GrandUnifiedDebugger
Maybe there's some special things I forgot to do, but out of the box GUD definitely felt (too) minimal for me.
Lately I've used LLDB when working on the .NET framework, and that seems like quite a reasonable thing to work with too at this point. I'd love some emacs-magic for this too :)
[1] http://www.emacswiki.org/emacs/GrandUnifiedDebugger
I generally use cgdb (or gdb -tui) but for some complex tasks, I use the ddd gui frontend. It's not a pretty gui and the usability is not great but the visualization it provides is excellent. I've used it for complex data structures and debugging numerical algorithms. Getting it to display structured data as you step through the code can be super helpful.
That being said, I hope that lldb will catch on and there will be some decent front ends for it. The GDB-MI (machine interface) used in debugger frontends is quite limited and clumsy. The lldb debugger should be a lot easier to embed in a project as a library (as opposed to using pipes and gdb-mi).
That being said, I hope that lldb will catch on and there will be some decent front ends for it. The GDB-MI (machine interface) used in debugger frontends is quite limited and clumsy. The lldb debugger should be a lot easier to embed in a project as a library (as opposed to using pipes and gdb-mi).
Qt Creator's gdb integration handles the basics well (I haven't used it with lldb, but lldb supports the gdb MI protocol so it should at very least "work").
Qt Creator is great for Qt. KDevelop and Emacs gud-gdb also provide nice vanilla frontends.
"Imagine"... I've been doing it in Visual Studio for years.
Step 1. Find problematic code
Step 2. Step back to before problematic statement
Step 3. Change problematic data value and/or code to hypothesized good one (while debugger is active and paused)
Step 4. Continue execution and observe if output is as desired
Step 5. Write test case.
Easy peasy. I'm glad to see the Rust community doing something similar, as the benefits of ease of development are not to be discounted, and Rust seems like a pretty nifty language.
Step 1. Find problematic code
Step 2. Step back to before problematic statement
Step 3. Change problematic data value and/or code to hypothesized good one (while debugger is active and paused)
Step 4. Continue execution and observe if output is as desired
Step 5. Write test case.
Easy peasy. I'm glad to see the Rust community doing something similar, as the benefits of ease of development are not to be discounted, and Rust seems like a pretty nifty language.
That seems to describe memory editing plus "edit and continue", which is distinct from record&replay and reverse execution.
Also, rr has nothing to do with Rust in particular, although this article shows how to use it with Rust.
Also, rr has nothing to do with Rust in particular, although this article shows how to use it with Rust.
> "That seems to describe memory editing plus "edit and continue", which is distinct from record&replay and reverse execution."
Sounds more like IntelliTrace to me (if you're not familiar with it, it's worth mentioning it's only available in Visual Studio Ultimate).
https://msdn.microsoft.com/en-us/library/dd572114.aspx
Sounds more like IntelliTrace to me (if you're not familiar with it, it's worth mentioning it's only available in Visual Studio Ultimate).
https://msdn.microsoft.com/en-us/library/dd572114.aspx
Step 2 was explicitly stepping backwards i thought. The rest was just a continuation of what the ability gives you and what I often do after stepping backwards to observe the execution.
rr works for C++/C code too?
To expand on the monosyllabic sibling answer, rr is completely language agnostic because it works at the level of machine code, (ab)using platform features like performance counters to do its thing. It should basically work for every x86 and x86-64 program you can debug with gdb, no matter which language(s) it was compiled with.
Yes.
This sounds exciting so I thought I'd look it up: http://stackoverflow.com/questions/26431737/can-i-get-revers...
"Visual Studio (2010, 2012, 2013 Ultimate only .. does not support C++"
:(
"Visual Studio (2010, 2012, 2013 Ultimate only .. does not support C++"
:(
Yeah, for sure; as I say in "Removing the rose-coloured glasses", rr is definitely not the first tool to do this, it's just the first one I've been lucky enough to use. I've not done much development on Windows, so haven't had an opportunity to use/enjoy Visual Studio.
Also, I'm sure there's a large chunk of people like me, who don't use debuggers much and have no idea what a good one is capable of.
Also, I'm sure there's a large chunk of people like me, who don't use debuggers much and have no idea what a good one is capable of.
Too bad this sometimes moves your focus from the problem you are solving and makes you an output matching machine.
HAH! True, but it's just one tool in a vast array of problem solving tools.
Elm has a "time traveling debugger", http://debug.elm-lang.org/
I'm eager to give it a try in a real project.
I'm eager to give it a try in a real project.
IntelliJ's debugger can "drop frame", after which you can re-run a function.
For those interested, Java has a number of such "time-travelling" debuggers (I've used one to great effect):
* TOD: http://pleiad.cl/tod/index.html
* Chronon (commercial): http://chrononsystems.com/
* Jive: http://www.cse.buffalo.edu/jive/
* Whyline (research): http://www.cs.cmu.edu/~NatProg/whyline-java.html
* Omniscient Debugger (discontinued?): http://www.lambdacs.com/debugger/
* TOD: http://pleiad.cl/tod/index.html
* Chronon (commercial): http://chrononsystems.com/
* Jive: http://www.cse.buffalo.edu/jive/
* Whyline (research): http://www.cs.cmu.edu/~NatProg/whyline-java.html
* Omniscient Debugger (discontinued?): http://www.lambdacs.com/debugger/
Thanks for this list!
And even without: an IDE like NetBeans (and probably any major Java IDE) can revert to a certain stack frame and even apply code changes on the fly - up to a certain degree of freedom only, but okayish for normal programmer life, where you need this only rarely if you have many smaller scope unit tests in place.
And even without: an IDE like NetBeans (and probably any major Java IDE) can revert to a certain stack frame and even apply code changes on the fly - up to a certain degree of freedom only, but okayish for normal programmer life, where you need this only rarely if you have many smaller scope unit tests in place.
I just saw a presentation that describes the rr vm/recorder in more detail. I am very impressed (though i haven't got the chance to use/test it).
https://youtu.be/H4iNuufAe_8
pure science fiction - instead of recording the change produced by each instruction they are recording what changed between system calls (result of system call is recorded) and scheduling points.
They do record the result of each system call and somehow manage to count the number of instructions per scheduled thread (they can only do that on new intel processors - counting the number of branches not instructions as instruction counter is not reliable. They are doing their own scheduling since the VM is all running in a single thread. It is also very Linux specific - for general case they use ptrace to record the system calls and their result, but tracing of some operating system calls is optimized by injecting stuff into the kernel (!)) - that's the reason why the recorded data is of minimal size.
I wonder how they are dealing with epoll - here the result is passed via shared memory and not via the system call interface. Still i guess they are lucky that there is no kqueue like interface on Linux - with kqueue it would have been even harder to track when the event result comes in.
https://youtu.be/H4iNuufAe_8
pure science fiction - instead of recording the change produced by each instruction they are recording what changed between system calls (result of system call is recorded) and scheduling points.
They do record the result of each system call and somehow manage to count the number of instructions per scheduled thread (they can only do that on new intel processors - counting the number of branches not instructions as instruction counter is not reliable. They are doing their own scheduling since the VM is all running in a single thread. It is also very Linux specific - for general case they use ptrace to record the system calls and their result, but tracing of some operating system calls is optimized by injecting stuff into the kernel (!)) - that's the reason why the recorded data is of minimal size.
I wonder how they are dealing with epoll - here the result is passed via shared memory and not via the system call interface. Still i guess they are lucky that there is no kqueue like interface on Linux - with kqueue it would have been even harder to track when the event result comes in.
This might be impractical, but is it possible to record gdb traces using a different processor, as long as that processor has access to the RAM the program is running in?
For example, could you record gdb traces with a GPU in a PC with hUMA?
http://www.tomshardware.com/news/AMD-HSA-hUMA-APU,22324.html
That could allow for gdb traces to be recorded constantly without a CPU performance hit (traces could be truncated if/when the size grew too large).
For example, could you record gdb traces with a GPU in a PC with hUMA?
http://www.tomshardware.com/news/AMD-HSA-hUMA-APU,22324.html
That could allow for gdb traces to be recorded constantly without a CPU performance hit (traces could be truncated if/when the size grew too large).
Oh man, this would have made my life troubleshooting a font layout issue in LibreOffice one hell of a lot easier :-(
My master's work was in the context of debugging. I spent a great deal of time reading descriptions of ancient debugging systems - most far in advance of the capabilities of the present day. The height of the art was reached in the early 90s - sophisticated logic structures over your program state were being held over your program state to automatically trigger actions - reverse debuggers were done back in the '70s as I recall; I believe the last major advance was done by an Italian chap in '92. Even SLIME and Common Lisp were quite well passed up by these capabilities. Most of those advances took place in the Lisp and Prolog world. I infer that this was due to the symbolic execution capabilities which the Fortran/Algol families have disavowed.
So it's always nice seeing advances in the industrial art come out, but a little wistful, knowing what once was.
So it's always nice seeing advances in the industrial art come out, but a little wistful, knowing what once was.
I would really like reading more about this.
The real interesting thing with rr is how it's done at the bare metal with very low overhead, using deep hardware-level tricks involving the performance counters on modern x86 CPUs. It's relatively easy to do this when you have a VM, but doing it on native code running directly on the CPU is significantly more interesting.