execute command on a defined range of directories

Question

Is it possible to 'partition' a directory listing - say in to blocks of some number, to perform different actions on each 'range'?

For example, lets say I have the following directories inside another folder:

$ ls test
abc/
def/
hij/
klm/
nop/
qrs/
tuv/
wxy/
zzz/

And I want to perform some action on the first 3 directories, another on the second 3, and so on.

My thought was that perhaps a loop across numbers would work, based on the output of something like ls | nl (but before anyone mentions it, I know parsing ls is a no-no!)

This obviously doesn't work, but illustrates my point I hope:

for dir in `ls | nl`; 
do
    do-something-with ${dir{1..3}} # Where $dir has taken on some numerical value linked to the folder)
    do-something-with ${dir{4..6}} 
# And so on...
    do-something-with ${dir{n-3..n}} 
done

The folders I intend to actually do this on, can be worked on in any order (i.e. the final splits can be totally arbitrary), but they have no logical naming consistency - by which I mean they can't be organised sensibly alphabetically or numerically based on any key within the directory name themselves.

Answer 1

There's no reason to do either of the following mistakes for this situation:

1. Use non-portable GNU extensions (such as xargs -0)
2. Parse the filenames as a stream of text only, pretending that they can't contain newlines.

You can handle this portably without much difficulty at all:

set -- */
while [ "$#" -gt 0 ]; do
  case "$#" in
    1)
      my-command "$1"
      shift 1
      ;;
    2)
      my-command "$1" "$2"
      shift 2
      ;;
    *)
      my-command "$1" "$2" "$3"
      shift 3
      ;;
  esac
done

Really this is more verbose than necessary, but it is readable this way.

For a different type of partitioning, where you want to split all directories into exactly three different methods of handling (but don't need to handle them at the same time), you could do it like so:

x=1
for d in *; do
  [ ! -d "$d" ] && continue
  case "$x" in
    1)
      # do stuff with "$d"
      ;;
    2)
      # do stuff with "$d"
      ;;
    3)
      # do stuff with "$d"
      ;;
  esac
  x=$(($x+1))
  [ "$x" -eq 4 ] && x=1
done

Answer 2

With printf/xargs:

printf "%s\0" */ | xargs -0 -n3 echo do-something

• printf prints the current directory contents null delimited. The slash after the * matches only directories, not files.
• xargs reads the input null delimited -0
  • -n3 splits the input in 3 parts.

---

The output (with your example directories):

do-something abc/ def/ hij/
do-something klm/ nop/ qrs/
do-something tuv/ wxy/ zzz/

Answer 3

The -n argument to xargs can be sometimes used for this. If you want to run jobs in parallel, the -P option can be used as well.

/tmp/t$ ls -1
dir1
dir2
dir3
dir4

/tmp/t$ ls -1 |xargs -n 3 echo runcmd
runcmd dir1 dir2 dir3
runcmd dir4

Don't worry too much about parsing the output of ls|nl. If you need to worry about that kind of thing, you can use find .. -print0 and the -0 argument to xargs. (assuming both are available).

Answer 4

If I could modify Otheus's answer little bit, it would be like this

for d in ``ls -1 | xargs -n 3 echo` ; do (cd "$d" && somecommand); done

and so on with other directories too.

Answer 5

Since Wildcard's solution is the one I eventually used, I've marked his answer as accepted, but if it's useful for people stumbling across this in the future, below is my code for distributing 330 directories amongst 11 other directories evenly.

Perhaps worth pointing out that it seemed pretty damn fast too!

#!/bin/bash

numdirs=11

for ((i=1; i<=$numdirs; i++))
do
    mkdir -p ~/simulation_groups/group_${i}
done

x=1

for d in * ; do
  [ ! -d "$d" ] && continue
  case "$x" in
1)
  mv $d ~/simulation_groups/group_${x}
  ;;
2)
  mv $d ~/simulation_groups/group_${x}
  ;;
3)
  mv $d ~/simulation_groups/group_${x}
  ;;
4)
  mv $d ~/simulation_groups/group_${x}
  ;;
5)
  mv $d ~/simulation_groups/group_${x}
  ;;
6)
  mv $d ~/simulation_groups/group_${x}
  ;;
7)
  mv $d ~/simulation_groups/group_${x}
  ;;
8)
  mv $d ~/simulation_groups/group_${x}
  ;;
9)
  mv $d ~/simulation_groups/group_${x}
  ;;
10)
  mv $d ~/simulation_groups/group_${x}
  ;;
11)
  mv $d ~/simulation_groups/group_${x}
  ;;
  esac
  x=$(($x+1))
  [ "$x" -eq $numdirs+1 ] && x=1
done