A small primer on xargs(bitops.io)

48 pointsby bitops14 yıl önce19 comments

http://bitops.io/blog/1336893229/xargs

21 comments

DHowett14 yıl önce

"In particular, the presence of \0 in your data will throw xargs for a loop (no pun intended). In this case, it is necessary to use the -0 option. This is particularly relevant if you are using the find command and supplying the -print0 option."

This glosses over the fact that those two things are intended to be used together. Delimiting filenames with a \0 is superior to delimiting them with spaces or newlines, as filenames cannot usually contain ASCII NUL; xargs and find can therefore be used to remove heterogeneous lists of files.

Additionally, the example given may better be worded as "find . -name '* junk* -exec rm '{}' \;" or even "find . -name '* junk* | xargs rm". (as given it strikes me as a useless use of grep and is additionally fragile if ls is aliased to "ls --color=always", spewing ANSI into

EDIT: Please excuse the additional spaces in the find command arguments, they are a formatting avoidance feature.

darklajid14 yıl önce

If we're ripping this sample apart, I'd like to point out that

  rm *junk*

in this case would've done the same, without spawning lots of useless stuff.

xargs is cool. The example fails to show a decent use case. For me a line like that would indicate that someone is _not_ familiar with his tools. Similar to

  cat somefile | grep something

instead of

  grep something somefile

Pipes and processes aren't endangered or (in practice) limited, but we still don't need to be wasteful?

babarock14 yıl önce

Furthermore, parsing the output of `ls` is a terrible practice[1] that makes for an even poorer sample.

A little further in the article, the author even suggests running

    `ls | grep junk`

(instead of the much simpler `ls junk*`. Do I need to point that out?).

I also know some people are going to retort that wasting a process or a pipe is not that bad. This discussion has been had over and over again, probably ever since Unix was invented.

In any case, the poor example chosen, as well as the flaky commands used, seriously hinder the credibility of the article.

[1]: http://mywiki.wooledge.org/ParsingLs

emmelaich14 yıl önce

They don't do the quite the same thing.

   ls | grep junk

is more similar to

   ls -d *junk*

You can argue the first is simpler. You can also argue more successfully that it is more general.

edit: I acknowledge the points in your link. Advocates for the unix "bunch of small tools" philosophy tend to overlook its weak spots.

asdfaoeu14 yıl önce

If you still want to use xargs and ls perhaps avoiding \0's (perhaps so you can easily grep on the data) you can use "ls -Q | xargs rm". The advantage of that is you can still easily use grep on the data as it is still newline delimited but you won't have problems with spaces.

Also ls is never aliased to "ls --color=always" unless you've done that more likely is its aliased to "ls --color=auto" or similar which will work normally.

robertskmiles14 yıl önce

> more likely is its aliased to "ls --color=auto" or similar which will work normally.

My ls (which I haven't much reconfigured) behaves very differently depending on whether it's outputting to a shell or a pipe or redirect. Not just colours, but columning and filetype indicators as well.

[deleted]14 yıl önce

lubomir14 yıl önce

Instead of -exec rm {} \; you could use -delete option as well.

babarock14 yıl önce

This option is not portable at all. For instance, it won't work on Solaris' find. I strongly advise against the use of non portable options.

exDM6914 yıl önce

It may be a good idea when writing scripts that have to be portable.

But there's quite a lot of nice non-standard options in the GNU userspace tools, and it would be a shame not to use them when available.

kjetil14 yıl önce

Little known fact: the -0/-print0 idea is due to DJB.

gizzlon14 yıl önce

"What xargs is doing is taking each line of input and applying the supplied argument or command to the line of input"

Actually it does not run command once for each line of input. This confused me for some time until I read the man page:

"The command line for command is built up until it reaches a system-defined limit (unless the -n and -L options are used). The specified command will be invoked as many times as necessary to use up the list of input items. In general, there will be many fewer invocations of command than there were items in the input. This will normally have significant performance benefits. Some commands can usefully be executed in parallel too; see the -P option"

Think you're looking for "-n 1" if you want that behavior..

leed25d14 yıl önce

I was going to respond on this very point myself. This blog entry is just plain wrong wrt the way that xargs works.

slashclee14 yıl önce

xargs is so 20th-century. parallel[0] is way more awesome (especially with all the multi-core systems everybody has now). Cool bonus: all the things you learn for xargs work in parallel too!

[0]: http://www.gnu.org/software/parallel/

wwwhizz14 yıl önce

I Was wondering what the difference with running parallel jobs using xargs -P was, but that's covered in their man-page: http://www.gnu.org/software/parallel/man.html#differences_be...

knewter14 yıl önce

A billion times, this. I just learned of GNU Parallel about 3 months ago and wish everyone knew about it. Have seen it used to stunning effect with a comically large pipeline (OCR patents) across a comically large number of EC2 instances. Very awesome indeed.

webreac14 yıl önce

I try to use perl instead of sed, awk, bash each time the needs are beyond basic usage. This applies also sometimes for xargs for example with the following command. find . -name junk\* | perl -nle unlink

My favorite usage of xargs if for parallel operations: cat hostlist | xargs -P 0 -n 1 -I host rsh host gzip -9 log/*.log

djoe4814 yıl önce

ls | grep junk | xargs rm # is dangerous due to The separator problem https://en.wikipedia.org/wiki/Xargs#The_separator_problem.

rcthompson14 yıl önce

ls | grep junk | xargs trash-put

https://github.com/andreafrancia/trash-cli/#readme

[deleted]14 yıl önce

bitops14 yıl önce

OP here - thanks all for the comments - so yes, it looks like I don't know xargs as well as I thought I did. I'll update the post to be more accurate.

As for the points about a bad example, I am fully aware that it is extremely contrived. The point was to not confuse anyone with commands they don't already know. I'm wary of posting examples that necessitate people learning 4-5 new commands when their focus is learning just one.