In fact, many data teams are guilty of overlooking critical questions like “Are we actually monitoring the data?” after deploying multiple pipelines to production. They might celebrate the success of the first pipeline and feel confident about deploying more. Still, they need to consider the health and robustness of their ETL pipeline for long-term production use. This lack of foresight can lead to significant problems down the line and undermine trust in the data sets produced by the pipeline.
I’ve challenged this idea in the form of writing a Prometheus exporter for process trees started on Linux. One may wonder what processes are started on Linux machines and if things are expected. Generally, it’s difficult to see if the process is intended to be run or not. This utility aims at making low-overhead monitoring of every process launch to remove noisy parts of process trees. Events are provided through Linux kernel Process Events Connector . This small utility is the attempt to mine useful information about process trees in a concise and low-overhead method, running a Rust application in the user space. The other alternative could be built on top of the Extended Berkeley Packet Filter (eBPF) technology, but it’s kernel-space arcane magic. Why not? Well, better next time. eBPF programs have to be verified to not crash the kernel. Trying that in Rust is probably better with the bcc crate.