Show HN: Watch what files any Linux process accesses(github.com)

131 pointsby spieglt6 anni fa37 comments

Show HN: Watch what files any Linux process accesses

https://github.com/spieglt/whatfiles

12 comments

Not taking anything away from the worth of this tool but if you do happen to find yourself needing to quickly inspect which files a process has open you can do so using the /proc file system:

    ls -l /proc/$PID/fd/

Additionally you can also use the /proc file system to display where the cursor is in those files by outputting the contents of

    /proc/$PID/fdinfo/$FD

which is handy if you have a long running process but forgot to pipe it into `pv` (or any other long running ingest that lacks a progress UI)

(Both tricks are Linux only)

simcop23876 anni fa

Pv supports this directly with -d pid, http://manpages.ubuntu.com/manpages/bionic/man1/pv.1.html

hnlmorg6 anni fa

I learn something new every day. Thank you

simcop23876 anni fa

Yea it's really nice for making a progress bar for things like cp, mv, etc.

im3w1l6 anni fa

The main drawback of this is that the program may access a file too quickly for a polling strategy to notice.

Wowfunhappy6 anni fa

On Mac, there's also lsof. (Also on Linux? I always assumed it was just a standard Unix tool, but maybe not?)

    lsof -c [process name]
    lsof -p [pid]

hnlmorg6 anni fa

It is available for Linux however it's not always part of the base install of every distribution (but a worthwhile install none-the-less).

Not taking anything away from lsof as I do use it myself but I do also think there is also merit in learning the /proc file system too because it helps illuminate some of the not-so-hidden magic behind Linux. Even if you then still find lsof or similar become your preferred utility.

thebruce87m6 anni fa

Admittedly my Linux skills are basic so I’m not sure how it compares, but I like the utility “progress” for monitoring things: https://github.com/Xfennec/progress/blob/master/README.md

Edit: By monitoring things, I mean like finding how far through a dd is after you have started it already.

PuffinBlue6 anni fa

Similarly, and perhaps obviously to some, you can use that to check how many files that process has open in total:

ls -al /proc/$PID/fd/ | wc -l

Useful for actually knowing what that specific process has open and is respecting your Max Open Files settings.

I'm looking at you Influx.

bostonsre6 anni fa

biotop and biolatency surface similar info. they come with a ton of other ridiculously awesome tools in BCC tools. they are a set of python wrapper scripts that run eBPF programs. using eBPF generally has a really low impact on performance when compared with other tools that do similar work.

https://github.com/iovisor/bcc

microtonal6 anni fa

bcc also has opensnoop, which is really nice for seeing which files are being opened:

https://github.com/iovisor/bcc/blob/master/tools/opensnoop_e...

ravinder_sbu6 anni fa

How is this different from using something like,

`strace -e trace=file`

I see that you are using ptrace to monitor a process. That is also used by strace. Is there something else your application does that strace does not (In relation to files)?

capableweb6 anni fa

From the README:

> Isn't this just a reimplementation of strace -fe trace=creat,open,openat,unlink,unlinkat ./program?

> Yes. Though it aims to be simpler and more user friendly.

Doctor_Fegg6 anni fa

For macOS, fs_usage does the same job. I find it invaluable to find out what process is churning the disk (usually mds...).

MCOfficer6 anni fa

Just a heads up (read: shameless plug), there's an AUR package:

https://aur.archlinux.org/packages/whatfiles-git/

spieglt6 anni fa

Wow, thank you very much! I've thought about trying to get a few of my projects into distribution repos but was somewhat intimidated by the process.

MCOfficer6 anni fa

You're welcome - but please check out #1/#2 so i can remove that ugly patch ;)

spieglt6 anni fa

Oh cool, was that you also? I will after work today.

MCOfficer6 anni fa

nope, someone else. I just stumbled over it and it's currently part of the package. Thanks for your effort ^^

spieglt6 anni fa

Patch merged.

bavell6 anni fa

This is one of the reasons I love Arch so much :) Thank you!

atrudeau6 anni fa

For doing the opposite - what processes access a given file - I like to use Audit (https://wiki.archlinux.org/index.php/Audit_framework#Audit_f...).

em5006 anni fa

This looks very similar to fatrace, which is already in the standard ubuntu and fedora repos.

edit: fatrace is system-wide, whereas the current tools monitors a specific process

http://manpages.ubuntu.com/manpages/trusty/man1/fatrace.1.ht...

https://piware.de/2012/02/fatrace-report-system-wide-file-ac...

unhammer6 anni fa

Lots more such tools at https://jvns.ca/debugging-zine.pdf (opensnoop-bpfcc and strace would be the most like this one)

Erwin6 anni fa

BTW, if you are using strace for this, check out the -y option recently added to strace. It will print the filename next to each file descriptor like this:

     read(3</proc/filesystems>, "", 1024)    = 0

Another interesting new strace option is -k which does a stack dump after each syscall. this can be useful to find out what part of the application, like some obscure lib, does weird system calls in your app.

st0le6 anni fa

IMO ProcMon on Windows is its equivalent. Not Process Explorer.

pjmlp6 anni fa

Since Windows Vista there is a mini version of it via "Open Resource Monitor" on the process explorer.

https://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-12...

ToFab1236 anni fa

There was news 2 years ago that MS was porting the sysinternals tool to linux. Did that ever happen?

https://mspoweruser.com/microsoft-working-on-sysinternals-fo...

fractalf6 anni fa

Ah, the system internal tools is one thing I really miss after switching to Linux on all my pcs. This port would be so awesome

pjmlp6 anni fa

Azure Insights provides similar kind of information.

Maybe that is where the porting effort went.

ToFab1236 anni fa

Could well be. If I recall correctly the motivation behind the porting was to give their engineers a unified set of tooling for troubleshooting windows and Linux (on Azure). I just assumed that the porting efforts would result in a similar set of tools to use locally on the box.

spieglt6 anni fa

You're right, thanks. Fixed.

pjmlp6 anni fa

You can also get the same information via Resource Monitor, available by default since Vista.

https://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-12...

dkdk82836 anni fa

Any reason why this is better than audit? I read README but i’m still not clear.

tyingq6 anni fa

Auditd is system wide (as would be inotify or fanotify based solutions).

This traces file events of a single process. Strace can be coaxed into something similar.

amelius6 anni fa

Can it be invoked recursively?

Because strace on Linux still fails with:

    strace: ptrace(PTRACE_TRACEME, ...): Operation not permitted

in those cases :(

spieglt6 anni fa

I have strace'd whatfiles, in fact that was a very useful way to debug a couple things, so maybe? I have not been able to attach to the same process with both whatfiles and strace, however.