LTTng: An open-source tracing framework for Linux(lttng.org)

68 pointsby thezorgvor 12 Jahren15 comments

LTTng: An open-source tracing framework for Linux

http://lttng.org/

5 comments

dmixvor 12 Jahren

Wow, open source sites are getting nice designs these days. Is there a business behind this project or is it via donation?

The documentation is excellent.

cbabvor 12 Jahren

Thanks, we worked quite hard on this new redesign!

edit: Yes, EfficiOS [1] is the open source business behind the LTTng project :).

[1] - https://www.efficios.com/

dmixvor 12 Jahren

Some UX feedback after attempting to install it on Fedora:

1) Clicked Fedora on install docs

2) Installed lttng-tools and lttng-ust RPMs

3) Clicked Getting Started in docs. Ran "lttng list --kernel".

3a) Got an error saying I need to run the systemd service.

4) Started systemd service (didn't see this mentioned in the docs?)

5) Reran "lttng list --kernel".... error saying I'm missing a module

5a) Reread docs, realized I need to build LTTng-modules. Which led me to find out I need to also build liburcu

6) Installed liburcu but got an error building lttng-modules https://gist.github.com/dmix/d2d4c3c1b823e82486bf (I could submit this as a bug report?)

So I gave up to attempt the install another time.

cbabvor 12 Jahren

Thanks for reporting your issues on Fedora.

4) We will definitely fix that omission.

5a) liburcu is a dependency of the lttng-ust and lttng-tools. The dependency should be accounted for in the respective Fedora packages.

6) Yes please submit a bug for this at [1]. We will investigate that failure to build on Fedora.

[1] - http://bugs.lttng.org/projects/lttng-modules

mtdewcmuvor 12 Jahren

But the website looks excellent.

jpfrvor 12 Jahren

LTTng is closely related to liburcu (now on http://urcu.so/).

It provides lockless datastructures for screaming fast concurrency on multicore machines. You will be pleasantly surprised by the concurrent hash-map implementation that comes with it [1].

[1] here's the paper for background: https://www.usenix.org/conference/usenixatc11/resizable-scal...

wrlvor 12 Jahren

In the event LGPL is too restrictive, there's also concurrencykit (http://concurrencykit.org/), which provides similar mechanisms (both for concurrent memory reclamation and various datastructures built on top).

jordighvor 12 Jahren

Here is a pretty cool presentation about it, in French:

https://www.youtube.com/watch?v=9XokHJadkq4

The slides are in English, if that helps:

http://lttng.org/files/papers/presentations/lttng20tracingfo...

cbabvor 12 Jahren

You can also find all the presentations/publications that we did through the years here [1].

[1] - http://lttng.org/publications/

rasurvor 12 Jahren

So how does this compare to somerthing like Sysdig, for example?

mvladvor 12 Jahren

"Excluding proprietary solutions, a few competing software tracers exist for Linux. ftrace is the de facto function tracer of the Linux kernel. strace is able to record all system calls made by a user process. SystemTap is a Linux kernel and user space tracer which uses custom user scripts to produce plain text traces. sysdig also uses scripts, written in Lua, to trace and analyze the Linux kernel.

The main distinctive features of LTTng is that it produces correlated kernel and user space traces, as well as doing so with the lowest overhead amongst other solutions. It produces trace files in the CTF format, an optimized file format for production and analyses of multi-gigabyte data. LTTng is the result of close to 10 years of active development by a community of passionate developers. It is currently available on all major desktop and embedded Linux distributions."

Further down you'll discover that your application(s) are linked with a library provided by the project.

Most of them rely on kprobes [1], or like ftrace on [2]. But there is also oprofile [3], and almighty perf [4], which rely on CPU extensions to implement tracing, (also implemented and by the HW manufacturer as well -- but in a OS-agnostic way).

[1] http://lxr.free-electrons.com/source/Documentation/kprobes.t...

[2] http://lxr.freeelectrons.com/source/Documentation/filesystem...

[3] http://oprofile.sourceforge.net/doc/internals/index.html

[4] https://perf.wiki.kernel.org/index.php/Main_Page

hitlin37vor 12 Jahren

how different is this from oprofile?

abusquevor 12 Jahren

Well, I've never used OProfile myself, so please take this answer with a grain of salt, but from a quick perusal of their website, and as the project's name seems to suggest, OProfile is a sampling profiler, which usually means it will collect data from counters at given intervals of time, producing a number of statistics over time. This generally implies a somewhat significant overhead, and consequent performance drop.

LTTng, on the other hand, is a tracer, which means it collects events from the kernel (using the built-in tracepoint facilities) as they happen. It's also possible to trace userspace apps (with lttng-ust), or define your own tracepoints. This has the benefit of being much more detailed, and also has a much smaller overhead.

You might be interested in reading the "What is Tracing?" section of the lttng docs [0], which does a far better job at explaining this than I do.

[0] - http://lttng.org/docs/#doc-nuts-and-bolts

hitlin37vor 12 Jahren

thanks for explaining. it sure looks interesting. i'm going to read now.

krakensdenvor 12 Jahren

oprofile has been abandoned, and no longer works on recent distributions. 'perf', I think, is the usual replacement, though the tooling is not as nice.