Inspecting coredumps like it's 2021(nixos.mayflower.consulting)
nixos.mayflower.consulting
Inspecting coredumps like it's 2021
https://nixos.mayflower.consulting/blog/2021/09/06/coredumpctl/
10 comments
Was anybody else disappointed that it's still just gdb for the debugger? I've thought for a long time that something better must be out there, especially a debugger that better handles multiple threads in an application, but every time I need to debug something I'm back in the 80s.
What would be better? How to improve the handling of multiple threads?
Every time I have tried a different debugger, I have always ended up coming running back to GDB. My only gripes with it are basically gripes with DWARF.
Every time I have tried a different debugger, I have always ended up coming running back to GDB. My only gripes with it are basically gripes with DWARF.
I still use dbx from Sun Studio. The now-oracle connection taints it, but it was from good old Sun. It was always better than gdb for threaded debugging.
Haven't checked if it is still available but I have it installed from long ago.
Haven't checked if it is still available but I have it installed from long ago.
I don't get what it's missing other than a GUI. I can view stacktraces, view memory in various formats, create new commands on the fly or in .gdbinit... what are people missing?
Well I miss a GUI.
I used to use DDD as a GUI frontend to GDB.
https://www.gnu.org/software/ddd/
https://www.gnu.org/software/ddd/
I tried to love ddd for quite a few months. It's definitely a GUI, of the same era as Xfig. That's about the nicest thing I can say about it; I prefer the CLI.
[deleted]
What about lldb? Has anybody had extensive experience with both to say anything about the actual debugging experience? Does it offer anything more than just "infrastructure"?
Same basic concept, though it had some nicer representations of non-C language things OOB when I tried it.
Really, my kingdom for a debugger that is as accessible as the one from IntelliJ.
The CLion front end to gdb/lldb is very good. Not perfect, but that's mainly do the oddities of interacting with the underlying debugger.
https://www.jetbrains.com/help/clion/debugging-code.html
https://www.jetbrains.com/help/clion/debugging-code.html
There is something better out there. https://pernos.co/
Pernosco is certainly 2021... Debugging as a service...
Did anyone try it? It looks like it a web frontend for rr, but the idea seems alien to me, especially the pricing part. A subscription for a number of submissions per month?! If there is anything you can't really plan, it is bugs, one month, you may need only one or two, the next it will be 50 in a day.
Did anyone try it? It looks like it a web frontend for rr, but the idea seems alien to me, especially the pricing part. A subscription for a number of submissions per month?! If there is anything you can't really plan, it is bugs, one month, you may need only one or two, the next it will be 50 in a day.
I’ve tried it, and it’s amazing.
It’s more than merely a front–end for rr, because with rr you are still inspecting a single moment in time in your program. You can rewind to past moments any time you want, but that fundamental limitation remains. With Pernosco you are really querying a database that contains all the information about all the states your program was ever in. It’s a superpower that you can purchase.
I think the price is generally worth it, if you have at least a couple bugs to use it on every month. If you have more than 5 per month but don’t want to go up a tier, I believe you can pay “a la carte” just for the extras. If you have enough people, and you can afford the hardware to run it, I really recommend the on–premises installation.
It’s more than merely a front–end for rr, because with rr you are still inspecting a single moment in time in your program. You can rewind to past moments any time you want, but that fundamental limitation remains. With Pernosco you are really querying a database that contains all the information about all the states your program was ever in. It’s a superpower that you can purchase.
I think the price is generally worth it, if you have at least a couple bugs to use it on every month. If you have more than 5 per month but don’t want to go up a tier, I believe you can pay “a la carte” just for the extras. If you have enough people, and you can afford the hardware to run it, I really recommend the on–premises installation.
I see a lot of sudo in there, for debugging a program presumably not running as root. Is it possible with this tooling to debug core dumps as a normal user?
Systemd's core dump utility stuffs the files in /var/lib/systemd/coredump with 640 permissions and ownership of root:root regardless of who owned the process.
It might be paranoia on someone's part, but I agree that it causes a lot of unnecessary privilege escalation for people who want to actually fix the problem.
It might be paranoia on someone's part, but I agree that it causes a lot of unnecessary privilege escalation for people who want to actually fix the problem.
> Systemd's core dump utility stuffs the files in /var/lib/systemd/coredump with 640 permissions and ownership of root:root regardless of who owned the process.
This isn't true for +ACL builds of systemd, look at the output of `getfacl /var/lib/systemd/coredump/$foo` and you'll see.
This isn't true for +ACL builds of systemd, look at the output of `getfacl /var/lib/systemd/coredump/$foo` and you'll see.
.. but you can just run coredumpctl gdb or coredumpctl dump <the dump> > /tmo/YOLO.dump
This isn't true, at least on Fedora. When my program crashes I run "coredumpctl gdb" without sudo and it works fine - I use this all the time.
(It could be that Fedora has done some special configuration and that it's not like this out of the box for upstream systemd)
(It could be that Fedora has done some special configuration and that it's not like this out of the box for upstream systemd)
I'm not at my laptop right now, but from https://www.freedesktop.org/software/systemd/man/coredumpctl...
It looks like coredumpctl might be suid? Or that the service is, and coredumpctl gets access through the service - so there's a certain elevation going on either way?
It looks like coredumpctl might be suid? Or that the service is, and coredumpctl gets access through the service - so there's a certain elevation going on either way?
It's not. It sets ACLs so that the user that owns the process may read the coredump, but typical ls doesn't show that.
ls -l will note the presence of ACLs with a "+" next to the permissions. I typically `ls -l` or just have that aliased to ll if it isn't already… (but YMMV)
sudo should not be required if the coredumped process was running as your user, but this will only be true if systemd was built with +ACL (look at `systemctl --version` output).
Coredumps can contain sensitive information about privileged processes, so it's rational to not have it world-readable.
I agree, that seems like a very bad idea.
I probably missed what exactly in this article is "like it's 2021"? Does a reference to Prometheus or systemd makes something more "modern"?
Visual studios support for minidumps is top notch. You can load it, there’s a symbol server if you’re dealing with a binary not built locally, and then you can fire up the parallel stacks to visually see what all the threads are doing.
Coredumps and command line analysis with gdb by comparison leave much to be desired.
Coredumps and command line analysis with gdb by comparison leave much to be desired.
Yeah, my thoughts too…
Inspecting coredumps with command line gdb, feels like title should be ”like it’s 2001”. Because that’s when I learned gdb :P
Inspecting coredumps with command line gdb, feels like title should be ”like it’s 2001”. Because that’s when I learned gdb :P
Linux was created in 1991, and its use is growing more than ever. When something was created doesn't really say anything about modern day usage.
GDB is still in wide use today and shows no sign of becoming history. If anything, its easy scripting features continues to prompt new tools based on GDB to pop up.
GDB is still in wide use today and shows no sign of becoming history. If anything, its easy scripting features continues to prompt new tools based on GDB to pop up.
It would be more correct to compare gdb to windbg.
Clion has some support for core dumps, yet it is lagging behind msvs, ofc, but is slowly catching up.
Clion has some support for core dumps, yet it is lagging behind msvs, ofc, but is slowly catching up.
So I guess it's still using gdb like it's 1999.
One day we may get a vim or even emacs mode for gdb, who knows? Until then, I will continue to "enjoy" interacting with the data of the program under debugger in the ed-style, like a True Programmer™.
The stories have it that back in the days some people could read actual core dumps with the tips of their fingers à la Braille, but surely that's just an urban myth.
The stories have it that back in the days some people could read actual core dumps with the tips of their fingers à la Braille, but surely that's just an urban myth.
GDB has a GUI (TUI). Several, in fact...
Indeed, and coredumpctl can be told to pass `--tui` to the gdb invocation with `-A --tui`
Also you can use another debugger of your choice with the `--debugger` flag.
Also you can use another debugger of your choice with the `--debugger` flag.
I've tried gdb --tui once, and it managed to mangle my xterm's terminal geometry into something non-Euclidean, so thanks but no thanks.
Emacs already has the GUD front end mode for GDB.
vim has `:help terminal-debug`.
gdb has changed a lot since 01999.
[deleted]
Visual Studio 2022 has support for debugging Linux binaries running under WSL2: https://devblogs.microsoft.com/cppblog/build-and-debug-c-wit...
Does Nix really not have debug symbols available for most packages? That's astonishing....
Nix is the package manager. NixOS is the OS.
It's very common to strip debug symbols from packages. For Ubuntu, it's even a different repository! I fail to see why that's astonishing.
It's very common to strip debug symbols from packages. For Ubuntu, it's even a different repository! I fail to see why that's astonishing.
Splitting debug symbols into separate packages is expected. Splitting them into a separate repository is a bit surprising but no big deal.
The problem with NixOS is that it doesn’t ship debug symbols at all.
Nix does at least make it easy to rebuild a package with debug symbols enabled, but:
- Some packages take a very long time to rebuild.
- IIUC, the standard builds are done without debug info enabled at all, rather than building with it enabled and then deleting it. Usually, whether debug info is enabled shouldn’t affect any other aspects of code generation, so rebuilding with debug info should produce a symbol-rich binary that’s compatible with existing core dumps for the original binary. But this is not always the case.
- If the package you want debug info for is a library, Nix will want to rebuild not just the library itself but everything that (transitively) depends on it. This is really a more general problem with Nix: it has zero notion of ABI compatibility. But always building with debug info would work around the problem.
The problem with NixOS is that it doesn’t ship debug symbols at all.
Nix does at least make it easy to rebuild a package with debug symbols enabled, but:
- Some packages take a very long time to rebuild.
- IIUC, the standard builds are done without debug info enabled at all, rather than building with it enabled and then deleting it. Usually, whether debug info is enabled shouldn’t affect any other aspects of code generation, so rebuilding with debug info should produce a symbol-rich binary that’s compatible with existing core dumps for the original binary. But this is not always the case.
- If the package you want debug info for is a library, Nix will want to rebuild not just the library itself but everything that (transitively) depends on it. This is really a more general problem with Nix: it has zero notion of ABI compatibility. But always building with debug info would work around the problem.
People can be astonished by anything if they didn't ask the right questions before spouting answers :)
One feature of Nix is to be able to rebuild something the same way on different systems. The article suggests that adding the debug symbols is a simple flag in the build expression.
Based on that, you shouldn't need the debug symbols in advance, though ymmv, when trying to debug an issue in container A on dev machine B.
Based on that, you shouldn't need the debug symbols in advance, though ymmv, when trying to debug an issue in container A on dev machine B.
I think standard practice is to strip the binary before release.
i love the attempt here but isn't 2021 all about containers? does k8s even generate coredumps? lol. what a joke.
K8s can generate coredumps if you configure it to, and they can actually still be useful in debugging certain classes of problems! However, at the multi-thousand person company I work at, there is like one guy concerned with core dumps.
Last time I tried to get coredumps working in a container (maybe four years ago) I got nowhere (it wasn't for a C/C++ application configuring everything to generate a core dump and then store it somewhere retrievable was too much).
Coredumps are a crucial source of information when investigating crashes and containerization doesn't invalidate any of that. Sure application restarts could minimize the impact of issues, but investigating them still is and should be standard practice. Furthermore, just because something isn't easy to do with k8s doesn't mean that it should be laughed at or scorned upon. I guess it's things like this that's causing articles to pop up that are somewhat skeptical of k8s.
[deleted]
This brings up a question I have had, is it not possible to put the debug info in a separate file that gdb could load instead of in the binary itself? If you could do this, you could then easily have xyz-dbg packages that only contained these files, or even an distro repository of these files that the debugger could download as needed.
This is exactly what most distributions do these days.
The debug symbol packages are usually called -dbgsym (Ubuntu, Debian, ...) or -debuginfo (openSUSE, Fedora).
This has been done automatically for almost a decade in most distros' packaging systems. GDB usually tells you what are the names of the debug packages you need to install for whatever you are trying to debug.
The "distro repository of files that the debugger can download as needed" is called a debuginfod URL. E.g., for Debian : https://debuginfod.debian.net/ , openSUSE : https://debuginfod.opensuse.org/ , etc. This is a more recent development and distros are just starting to turn it on by default. But if enabled you'll basically see GDB start downloading symbol files automatically ala Windows when you run it. It even downloads the source code as it goes.
Personally I think the latter is a security risk (since it sends to a server the hash/build-id of everything you try to debug), but I can't argue it's not convenient.
The debug symbol packages are usually called -dbgsym (Ubuntu, Debian, ...) or -debuginfo (openSUSE, Fedora).
This has been done automatically for almost a decade in most distros' packaging systems. GDB usually tells you what are the names of the debug packages you need to install for whatever you are trying to debug.
The "distro repository of files that the debugger can download as needed" is called a debuginfod URL. E.g., for Debian : https://debuginfod.debian.net/ , openSUSE : https://debuginfod.opensuse.org/ , etc. This is a more recent development and distros are just starting to turn it on by default. But if enabled you'll basically see GDB start downloading symbol files automatically ala Windows when you run it. It even downloads the source code as it goes.
Personally I think the latter is a security risk (since it sends to a server the hash/build-id of everything you try to debug), but I can't argue it's not convenient.
Yes. You can separate the debug info from an object file with objcopy:
For more info, here's a link:
https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Fil...
You can also use gcc's -gsplit-dwarf command line option when compiling to produce a .dwo file containing your debug information:
https://gcc.gnu.org/wiki/DebugFission
objcopy --only-keep-debug foo.o foo.o.debug
objcopy --strip-debug --add-gnu-debuglink=foo.o.debug foo.o
For more info, here's a link:
https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Fil...
You can also use gcc's -gsplit-dwarf command line option when compiling to produce a .dwo file containing your debug information:
https://gcc.gnu.org/wiki/DebugFission
I wanted to ask you if the "-gsplit-dwarf" is a 100% complete alternative to strip the symbols as done previously. I ask you because some modern projects[1] still prefer to compile with symbols and strip afterward.
[1] https://github.com/dotnet/runtime/blob/main/eng/native/funct...
If you are in control of your build, ideally you want to embed build IDs and use a symbol server. Here is an older article from Bruce Dawson on this (unfortunately way underdeveloped) area of Linux:
https://randomascii.wordpress.com/2013/02/20/symbols-on-linu...
https://randomascii.wordpress.com/2013/02/20/symbols-on-linu...