Tag Archives: file names

xargs: How To Control and Use Command Line Arguments

I am trying to use xargs command using shell pipes and not able to understand how to control and use command line arguments. For example I’d like to find out all *.c file located in 100s of sub-directories and move them to another directory called ~/old.src. How do I use command line args with xargs to achieve the same?

xargs command is designed to construct argument lists and invoke other utility. xargs reads items from the standard input or pipes, delimited by blanks or newlines, and executes the command one or more times with any initial-arguments followed by items read from standard input. Blank lines on the standard input are ignored.

xargs is more safer and easy to use

xargs functionality can be achived using the backquote feature of shell. But, it offers more options. It can deal with blanks or special characters in file names easily. It is often used with find, grep and other commands.

xargs examples

For example following example will print 1 2 3 4 using xargs (echo command is default)

$ echo 1 2 3 4 | xargs echo

OR

$ echo 1 2 3 4 | xargs

You can force xargs to use at most max-args arguments per command line. For example following will use first two argument per command:

$ echo 1 2 3 4 | xargs -n 2

Find all .bak files in or below the current directory and delete them.

$ find . -name "*.bak" -type f -print | xargs /bin/rm -f

{} as the argument list marker

{} is the default argument list marker. You need to use {} this with various command which take more than two arguments at a time. For example mv command need to know the file name. The following will find all .bak files in or below the current directory and move them to ~/.old.files directory:

$ find . -name "*.bak" -print0 | xargs -0 -I {} mv {} ~/old.files

You can rename {} to something else. In the following example {} is renamed as file. This is more readable as compare to previous example:

$ find . -name "*.bak" -print0 | xargs -0 -I file mv file ~/old.files

Where,

  1. -0 If there are blank spaces or characters (including newlines) many commands will not work. This option take cares of file names with blank space.
  2. -I Replace occurrences of replace-str in the initial-arguments with names read from standard input. Also, unquoted blanks do not terminate input items; instead the separator is the newline character.

Dealing file names with blank spaces and newline

The following will work incorrectly if there are any filenames containing newlines or spaces (it will find out all .mp3 file located in current directory and play them using mplayer):

$ find . -iname "*.mp3" -print | xargs mplayer

To get rid of this problem use -0 option:

$ find . -iname "*.mp3" -print0 | xargs -0 -I mp3file mplayer mp3file

To find out all *.c file located in 100s of subdirectories and move them to another directory called ~/old.src, use:

$ find /path/to/dir -iname "*.c" -print0 | xargs -0 -I file mv file ~/old.src

Avoiding errors and resource hungry problems with xargs and find combo

To copy all media files to another location called /bakup/iscsi, you can use cp as follows:

$ cp -r -v -p /share/media/mp3/ /backup/iscsi/mp3

However, cp command may fail if an error occurs such as if the number of files is too large for the cp command to handle. xargs in combination with find can handle such operation nicely. xargs is more resource efficient and will not halt with an error:

$ find /share/media/mp3/ -type f -name "*.mp3" -print0 | xargs -0 -r -I file cp -v -p file --target-directory=/bakup/iscsi/mp3

Please note that all of the above commands are tested with GNU/xargs version. BSD and UNIX xargs command may not have options such as -r. Please refer to your local xargs man page for further info:

man xargs

Sysadmin To Be Resources

Juan February 17, 2009 at 7:40 pm

You can also do the following:

find /share/media/mp3/ -type f -name '*.mp3' -print -exec cp -v -p {} --target-directory=/bakup/iscsi/mp3 ";"

Reply

Ole Tange July 24, 2010 at 12:44 pm

If you know the file names do not contain \n and you have GNU Parallel http://www.gnu.org/software/parallel/ installed, then you can do:

find /share/media/mp3/ -type f -name ‘*.mp3′ -print | parallel -X cp -v -p {} /bakup/iscsi/mp3

It is somewhat shorter and will not spawn a cp for each copy. GNU Parallel also uses \n as its default separator which is helpful if you are using tools like head or tail.

Watch the intro video to GNU Parallel at http://www.youtube.com/watch?v=OpaiGYxkSuQ

Reply

sathiya February 18, 2009 at 11:04 am

nice reference to get to know about xargs.

Reply

dennytech February 18, 2009 at 1:11 pm

Pretty good. I’m a newbie and struggle but I got most of this. Thanks. My screen displayed zero as lower case o so the comment beginning “Where,” threw me for a minute. I searched the page for -(lower case o) and it wasn’t found, however, I did find -(numeral zero). Did “-print0″ refer to the blanks/newline problem or is it something else?

Reply

Vivek Gite February 18, 2009 at 2:50 pm

It is zero (0) and note small o letter.

The -print0 option prints ( displays ) the full file name on the screen, followed by a null character (instead of the newline character that -print uses). This allows file names that contain newlines or other types of white space to be correctly interpreted by programs that process the find output. This option corresponds to the -0 option of xargs.

HTH

Reply

yoander February 18, 2009 at 9:37 pm

Juan:
It’s true, you can get the same results using find -exec option but using xargs is more efficient

Reply

anil February 19, 2009 at 3:10 am

Dear
i am trying to find and delete some files but it gives error

~]# find . -name “abc*” xargs /bin/rm -f
 find: paths must precede expression
 Usage: find [path...] [expression]
~]ls abc abc1.c xargs /bin/rm abc
 ls: xargs: No such file or directory
 abc abc abc1.c /bin/rm

Can you suggest where i am going wrong.

Thanks
anil

Reply

Vivek Gite February 19, 2009 at 6:46 am

Try
find . -name "abc*" -print0| xargs /bin/rm -f

Reply

Mel February 24, 2009 at 4:38 am

Note that –target-directory is a GNU extension to cp(1).

Also, at least on FreeBSD:

find . -type f -exec grep -l '/bin/bash' {} +

is exactly the same as:

find . -type f -print0 |xargs -0 grep -l '/bin/bash'

The subtle difference being a terminating +, rather then a \;.

Last but not least, cp without shell globs doesn’t have a limit to the number of files it can handle. The problem comes when the shell converts the globs on the command line to actual arguments.

Reply

commbot March 24, 2009 at 11:42 pm

Thanks for your very useful and informative blog. I learn a lot from reading your tips and its has been a great deal of help for me on numerous occasions.

Using find xargs to change the UID or GID of files and directories fails if there is a space in the folder name. I tried two versions of the same command to accomplish this, with differing results. This is on FreeBSD 7.1

find -X / -group oldGID -print0 | xargs chgrp newGID
will work on filenames but not directories

find / -group oldGID -print0 | xargs -0 chgrp newGID
works for both

Reply

commbot March 24, 2009 at 11:49 pm

a mistake, first command should be:
find -X / -group oldGID -print | xargs chgrp newGID

Reply

Asrafi May 11, 2010 at 9:14 am

Dear Friends
I am facing a peculiar problem while trying to run a code with xargs. It runs successfully on cygwin, but can’t work on ubuntu. I’ve checked the xargs versions on both the platform, they are same. Can’t understand the problem really.

xargs -r -a input.txt -L1 code.exe > output.txt

This command simply take the inputs from input file on a line by line basis and after executing with the exe file writes the output in the output file.
in ubuntu it gives error
“xargs: tcas.exe: no such file or directory” .

Reply

hileon May 19, 2010 at 4:49 am

it seems this line :

$ find . -name “*.bak” -print0 | xargs -0 -I mv {} ~/old.files

should be:

$ find . -name “*.bak” -print0 | xargs -0 -I{} mv {} ~/old.files

nice post, very helpful to me.

Reply

max July 18, 2010 at 6:29 am

how do I pipe the john the ripper to hydra using xargs?
I tried but it doesn’t seem to work.

Reply

Nigel July 29, 2010 at 11:53 am

I have a bunch of script that cp files/dir from 1 place to another. The problem is that destination already has the same file albeit an older ones. I want cp to to replace them all without asking for user input. I try this on my fedora machine -
cp -rf /my/source/* /my/dest/ | xargs echo “yes”
cp still ends up needing me to enter “y” for this line -
cp: overwrite /my/dest/file1 ?

Is there some way to automate this copy/replace, without any user input?

thanks,

Reply

Vivek Gite July 29, 2010 at 12:49 pm
\cp -rf /my/source/* /my/dest/

OR

/bin/cp -rf /my/source/* /my/dest/

More info here

Reply

Nigel July 30, 2010 at 1:45 am

Thanks Vivek.

I should have been more clear. I need to run the scripts as root. And root’s bashrc alias cp to cp -i. Any calls to cp/rm/mv becomes interactive. I know that removing the alias will make cp work as usual. But its a shared machine, and i dont want to modify anything on it.

I need a scriptable way to echo yes to get around this. Any idea?

Reply

Vivek Gite July 30, 2010 at 2:47 am

FYI, when you use \cp or /bin/cp syntax you do not make any changes to root aliases or anything else. So they are safe to call from your script. But if you must, than use:

yes | cp -rf /my/source/* /my/dest/

Reply

Nigel July 30, 2010 at 3:57 am

Thanks Vivek, this was just what i was looking for.

Reply

Ole Tange September 1, 2010 at 3:00 pm

Give full path to cp:

/bin/cp -rf /my/source/* /my/dest/

Reply

Philippe Petrinko August 31, 2010 at 10:26 am

@Vivek,
There is 1 typo in “argument list marker” first example:
Just after the -I parameter, there should be “{}”
(unless this, the command fails)

Reply

felipe1982 September 1, 2010 at 1:36 pm

I’m trying to copy thousands of files to my Flash Drive using
`find /source-dir/ -type f -print0 | xargs -n 1000 cp -t /media/USB`
But this doesn’t stop at 1000 files. It continues. How can I force a maximum number of files, and quit when finished?
–felipe

Reply

Vivek Gite September 1, 2010 at 1:57 pm

No it will not work. If you pass 1000 arguments at a time then it will only work. You may need to use the for loop with counter using bash or any other scripting language :

#!/bin/bash
c=0
for f in /source-dir/*
do
  [ $c -eq 1000 ] && break
   /bin/cp -f "$f" /media/USB
  (( c++ ))
done

HTH

Reply

Ole Tange September 1, 2010 at 2:57 pm

If you have GNU Parallel http://www.gnu.org/software/parallel/ installed you can do this:

find /source-dir -type f | head -n 1000 | parallel cp {} /media/USB

It will work even if you have spaces and quotes in your filenames.

Watch the intro video for GNU Parallel to learn more:
http://www.youtube.com/watch?v=OpaiGYxkSuQ

Reply

Philippe Petrinko September 2, 2010 at 12:13 pm

@ felipe1982

I am sure you care for the resquest you posted,
By the way, would you give us some feedback,
for instance, did you try the code I suggested ?

I have tested it all right, so it should work for you.
Let us know.

Reply

Philippe Petrinko September 1, 2010 at 2:24 pm

Nice idea, Vivek.

Let me try this one-liner:

find /sourcedir -type f -print|head -n1000|while read; do echo -e -n "$REPLY\000"; done|xargs -0 -I {} cp "{}" /dest_dir

Limit is given by the head parameter,
but you need to rebuild a zero terminated list of arguments in order to allow xargs to use -0
(Zero terminated strings)

Reply

Vivek Gite September 1, 2010 at 5:31 pm

Philippe,

Done!

Also, I’m working on “preview button” mod. This will sort out this kind of issues. It is one of the most requested feature here. So by this weekend it should be up and running.

Reply

Philippe Petrinko September 1, 2010 at 7:01 pm

@Vivek
:-O You ‘ve got problems with WordPress too. ;-)

The backslash is well positionned,
but there is only one zero after it, there should be three zeroes !
[again feel free to delete this post]

Reply

Vivek Gite September 2, 2010 at 11:03 am

Dam, wordpress always try to filter too much from comments to avoid html / js based attacks. BTW, I liked your idea.

Reply

Philippe Petrinko September 2, 2010 at 12:16 pm

@ Vivek

Still, you chapter ” {} as the argument list marker ”
still misses “{}” after the -I selector
into the first code example.

Would you at last modify your code ? ;-)

Reply

ryan December 3, 2010 at 9:13 pm

Uses available procs on a machine:
find / -type f -print0 | xargs -r0 -P$(nproc) -n10 md5sum # Process files in parallel over available processors

Reply

avatar_007 September 22, 2011 at 5:25 pm

xargs some how still dose not work for me
I do the following instead.
cat file_list.txt | awk ‘{print “cp “$0″ copyfiles_dir/”}’ | sh

Reply

Gabriel Glusgold September 29, 2011 at 2:09 pm

How could I use xargs for removing several lines containing something?, example:

file1: contains
line1: eeee
line2: aaaa

file2: contains
line1: aaaa
line2: sssss
line3: nnnnn

then, The final objetive is make a command line using xarg like this:

grep -il “aaaa” file* | xarg grep -v “aaaa” _____here i dont know how is the followwing, I need something that eliminates “aaaa” from each file that find the “aaaa” line.

thanks in advance

Reply

Philippe Petrinko September 29, 2011 at 9:01 pm

Hi Gabriel.

Actually, there are many ways to achieve this task.

1) Let’s start by the simpliest one:

sed -i “/aaa/d” file*

Ok, this does not use [xargs]

2) Using [xargs]

grep -il ccc file*|xargs -I{} sed -i “/ccc/d” {}

Let us know some feedback.

– Philippe

Reply

Piyush February 7, 2012 at 1:57 pm

Hi all,
I wish to copy all files with extension jpg to a particular directory. To this effect, the command below will suffice

ls -tr | grep *.jpg| xargs -n1 -i cp {} ${tobe_copied_path1}/

Next, I wish to rename all ‘:’ in the file names to ‘.’ Will it be possible to do it using the same command line? I was trying something along the lines
ls -tr | grep *.jpg | xargs -n1 -i cp {} ${tobe_copied_path1}/`echo \{\} | sed ‘s/:/./g’`
but could not get through.
Any pointers on this one?
Thanks in advance.

Reply

Philippe Petrinko February 8, 2012 at 9:30 am

Hi Piyush,

First, use [find] to select files, not [ls]. Behave, try to follow Vivek’s advice! ;-)

Second, try to use bash shell variable pattern substitution
syntax is ${parameter/pattern/string}.

You’ll find what you need by reading the fantastic manual: type :

man bash

And look for pattern substitution. This will do the trick.
– Philippe

Reply

Philippe Petrinko February 8, 2012 at 11:45 am

Hi again, Piyush.

At this very moment, I did not find a trick with xargs to rename on the fly.
So, my solution is actually this one-liner command.
Assuming that source directory is named “from”, and destination is named “to”, you can tailor to your needs following compound command line:

fromdir=from; todir=to; find ${fromdir} -type f -iname “*.jpg” -print|while read; do f=${REPLY//:/.}; f=$(basename ${f}); cp $REPLY ${todir}/$f; done

Please let us know you feedback which would most certainly be instructive.

May be Vivek could find a solution using [xargs]?

– Philippe

Reply

Philippe Petrinko February 8, 2012 at 12:59 pm

At last, I got this working:
Trick is to build a command line with [echo] that is passed to bash through a pipe.
Again, you just have to change [f] and [t] variables with source and destination directories. That’s all folks!

export f=from; export t=to; find ${f} -type f -iname “*.jpg” -print0|xargs –null -n 1 -I{} echo ‘f1={}; f2=$(basename ${f1}); f2=${f2//:/.}; echo cp “${f1}” $t/”${f2}”‘|bash

Hope this helps.

Know a better way anyone?

(I used to do a similar trick some time ago, but do not remember yet… I think that was with some system call – almost similar)

– Philippe

Reply

Dinesh February 13, 2012 at 11:33 am

It’s really nice … Thank you

Reply