Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-)

Loops in Shell

(Lecture Notes for Professor Nikolai Bezroukov Shell Programming Class)

Materials of this lecture notes combine copyrighted materials from Learning the Korn Shell (A Nutshell Handbook) by Bill Rosenblatt (first edition, 1993) and examples from Advanced Bash-Scripting Guide.

This lecture notes were created by the author for his Unix shell course in FDU. As they include copyrighted materials they should be used strictly for educational purposes.

• Introduction
   

• While and until loops

  • Breaking out of a Loop

  • Continuing to the next iteration

  • Examples of while Loops from Advanced BASH Programming Guide

  • Examples of until Loops from Advanced BASH Programming Guide

• For loop ("old" for loop in shell is different from C for loop; in bash 3.2 C-style for loop is available)

  • Old for loop

    • Examples of for Loops from Advanced BASH programming Guide

  • New (C-style) for loop

  • Examples

   

Introduction

A loop is a block of code that iterates (repeats) a list of commands as long as the loop control condition is true.

Borne shell introduced several types of loops, that were pretty innovative and powerful for its time. Now they looks completely archaic and syntax looks like a perversion but we have what we have. In you cannot overcome allergy please use Perl or other more modern scripting language. Until loop is implemented incorrectly in all shells since Borne shell.

Most current version of shells (bash and ksh93) support just three original types of loop from Born shell (with its unique Algol-68 style syntax).  So the progress here stopped in late seventieth of the last century.  Some technical decisions look very strange or arbitrary or both.  You need to be patient and avoid crying "What a crap you are teaching us !!!"  at the lectures ;-). Humans are very adaptable and get used to things even if they look very unnatural and obscure at the beginning, especially if they are well paid for those skills :-). 

while  loop

Bash provides while loops but not until loop. There is also a construct called until loop but it was implemented incorrectly and has the test at the top, unlike  analogous constructs in Pascal (while/do and repeat/until) and C (while and do/until). This is legacy of Borne shell and it is amazing how such incorrect solution survived for so long (the notion of feature deprecation was unknown in 70th and 80th; it was unknown by POSIX committee (committee to defeat alliance of Sun and AT&T ;-). It is still unknown in shell designer world in 2008).

NOTE: In shell the until condition is checked at the top of the loop, not at the bottom as it is in analogous constructs in C and Pascal.

The result is that you can convert any until into a while by simply negating the condition. Therefore we will ignore the existence of until throughout the rest of course.

The syntax for while is:

while condition
do
    statements...
done

while and until are actually most useful when conditions are simple. In the last they often were used to imitate C-style for loop, for example:

For example

i=1
while (( $i < 10 )); do
	echo i=$i
	let i++
done

As an example let's discuss an implementation of  a simplified version of the shell's built-in whence command.

By "simplified," we mean that we will implement only the part that checks all of the directories in your PATH for the command you give as argument (we won't implement checking for aliases, built-in commands, etc.).

We can do this by picking off the directories in PATH one by one, using one of the shell's pattern-matching operators, and seeing if there is a file with the given name in the directory that you have permission to execute. Here is the code:

mypath=$PATH:
while [[ -n $mypath ]]; do
    mydir=${mypath%%:*}
    if [[ -x $mydir/$1 && ! -d $mydir/$1 ]]; then
        print "$mydir/$1"
        return
    fi
    mypath=${mypath#*:}  
done
return 1

The only things that might need commentary here are regular expressions:

mypath=${mypath#*:}
mydir=${mypath%%:*}

The first of these uses another shell string operator: this one deletes the shortest match to the pattern given from the front of the string. By now, this type of operator should be familiar. This line deletes the front directory from $path and assigns the result back to path. The second line is the same as before the while: it finds the (new) front directory in $path and assigns it to dir. This sets up the loop for another iteration.

Thus, the code loops through all of the directories in PATH. It exits when it finds a matching executable file or when it has "eaten up" the entire PATH. If no matching executable file is found, it prints nothing and exits with an error status.

We can enhance this script a bit by taking advantage of the UNIX utility file(1). file examines files given as arguments and determines what type they are, based on the file's magic number and various heuristics (educated guesses). A magic number is a field in the header of an executable file that the linker sets to identify what type of executable it is.

If filename is an executable program (compiled from C or some other language), then typing file filename produces output similar to this:

filename: ELF 32-bit LSB executable 80386 Version 1

However, if filename is not an executable program, it will examine the first few lines and try to guess what kind of information the file contains. If the file contains text (as opposed to binary data), file will look for indications that it is English, shell commands, C, FORTRAN, troff(1) input, and various other things. file is wrong sometimes, but it is mostly correct.

We can just substitute file for print to print a more informative message in our script:

path=$PATH
dir=${path%%:*}
while [[ -n $path ]]; do
    if [[ -x $dir/$1 && ! -d $dir/$1 ]]; then
	  file $dir/$1
	  return
    fi
    path=${path#*:}
    dir=${path%%:*}
done
return 1

Assume that fred is an executable file in the directory /usr/bin, and that bob is a shell script in /usr/local/bin. Then typing file fred produces this output:

/usr/bin/fred: ELF 32-bit LSB executable 80386 Version 1

And typing file bob has this result:

/usr/local/bin/bob: commands text

Before we end this chapter, we have two final notes. First, notice that the statement dir=${path%%:*} appears in two places, before the start of the loop and as the last statement in the loop's body. Some diehard C hackers are offended by this Pascal-like coding technique. Certain features of the C language allow programmers to create loops of the form:

while iterative-step; condition; do
    ...
done

This is the same as the form of the script above: the iterative-step runs just before the condition each time around the loop.

We can write our script this way:

path=$PATH
while dir=${path%%:*}; [[ -n $path ]]; do
    if [[ -x $dir/$1 && ! -d $dir/$1 ]]; then
	file $dir/$1
	return
    fi
    path=${path#*:}
done
return 1

Although this example doesn't show great programming style, it does make the code smaller-hence its popularity with C programmers. Make sure you understand that our script is functionally identical to the previous script.

Breaking out of a Loop

To break out of a loop, the command word break is used. Command execution will continue with the first command line found after the end of the loop (after the word "done").

If loops have been nested, it is possible to break out of the current and any previous, nested, loops by following the word break with the number of loops to break out of.

while true
do
command1

  for variable in one two three four five
  do
  echo "$variable"
    if [[ "$variable" = "four" ]]
    then
       break
    fi
  done

commandn   # <<<<<break to this line <<<<
done
another_command_line
 

while true
do
command1

  for variable in one two three four five
  do
    echo "$variable"
    if [[ "$variable" = "four" ]]
    then
    break 2
    fi
  done

commandn
done
another_command_line   # <<<<break to this line <<<<
 
 

Continuing to the next iteration

for variable in one two three four five
do
echo "$variable"
   for othervar in a b c d
   do                         # <<<<continue to this line
   echo "$variable $othervar"
    if [[ "$othervar" = "c" ]]
    then
    continue
    fi
   command_in_nested_loop
   done
done

continue can have argument which can be used for nested loop and specify the numbe of levels to continue the next operation.

for variable in one two three four five
do      # <<<<continue to this line
echo "$variable"
 for othervar in a b c d e
 do
 echo "$variable $othervar"
  if [[ "$othervar" = "c" ]]
  then
  continue 2
  fi
 command_in_nested_loop
 done
done

Examples of while Loops
 from Advanced BASH Programming Guide

#!/bin/bash

var0=0
LIMIT=10

while [ "$var0" -lt "$LIMIT" ]
do
  echo -n "$var0 "        # -n suppresses newline.
  #             ^           Space, to separate printed out numbers.

  var0=`expr $var0 + 1`   # var0=$(($var0+1))  also works.
                          # var0=$((var0 + 1)) also works.
                          # let "var0 += 1"    also works.
done                      # Various other methods also work.

echo

exit 0

 Another while loop

#!/bin/bash

echo

while [ "$var1" != "end" ]     # while test "$var1" != "end"
do                             # also works.
  echo "Input variable #1 (end to exit) "
  read var1                    # Not 'read $var1' (why?).
  echo "variable #1 = $var1"   # Need quotes because of "#".
  # If input is 'end', echoes it here.
  # Does not test for termination condition until top of loop.
  echo
done  

exit 0

A while loop may have multiple conditions. Only the final condition determines when the loop terminates. This necessitates a slightly different loop syntax, however.

 while loop with multiple conditions

#!/bin/bash

var1=unset
previous=$var1

while echo "previous-variable = $previous"
      echo
      previous=$var1
      [ "$var1" != end ] # Keeps track of what $var1 was previously.
      # Four conditions on "while", but only last one controls loop.
      # The *last* exit status is the one that counts.
do
echo "Input variable #1 (end to exit) "
  read var1
  echo "variable #1 = $var1"
done  

# Try to figure out how this all works.
# It's a wee bit tricky.

exit 0

As with a for loop, a while loop may employ C-like syntax by using the double parentheses construct (see also Example 9-29).

C-like syntax in a while loop

#!/bin/bash
# wh-loopc.sh: Count to 10 in a "while" loop.

LIMIT=10
a=1

while [ "$a" -le $LIMIT ]
do
  echo -n "$a "
  let "a+=1"
done           # No surprises, so far.

echo; echo

# +=================================================================+

# Now, repeat with C-like syntax.

((a = 1))      # a=1
# Double parentheses permit space when setting a variable, as in C.

while (( a <= LIMIT ))   # Double parentheses, and no "$" preceding variables.
do
  echo -n "$a "
  ((a += 1))   # let "a+=1"
  # Yes, indeed.
  # Double parentheses permit incrementing a variable with C-like syntax.
done

echo

# Now, C programmers can feel right at home in Bash.

exit 0

 

A while loop may have its stdin redirected to a file by a < at its end.

A while loop may have its stdin supplied by a pipe.

Examples of until Loops from Advanced BASH Programming Guide

#!/bin/bash

END_CONDITION=end

until [ "$var1" = "$END_CONDITION" ]
# Tests condition here, at top of loop.
do
  echo "Input variable #1 "
  echo "($END_CONDITION to exit)"
  read var1
  echo "variable #1 = $var1"
  echo
done  

exit 0

For loops

In bash there are two types of for loops: "old" and "new" (C-ctyle). The latter is much better and unless you need to do something for which old for loop is especailly sutable you should aviod using old syntax. Forget about silly noice about compatibility. Bash is now standard de-facto and is availble of nots operating systems. It is still not availbale on HP-UX 11 but that's the problem of HP. Many people including me do not consider HP-UX to a  legitimate Unix ;-). Anyway even they provide bash with binaries availbale for all HP-UX version you can find.  It is easier to install it on all systems that struggle with compatibility questions.

"Old" for loop

This type of for look is a bastard child of the initial versions of shell when the key requirement was small size of executable. Like many bad design decisions it survived in old subsequent version. The for loop allows you to repeat a section of code a fixed number of times. During each time through the code (known as an iteration), a special variable called a loop variable is set to a different value; this way each iteration can do something slightly different.

The for loop used in Born shell is different from for look used in C. The main difference is that the shell's for loop doesn't let you specify a number interactions and counter. It iterated thought the list which can be supplied in several different ways:

• as a list of arguments to the script or function.

• as a list of files from the current or some other directory

• As the explicit list of values.

The syntax for the for loop in shell is close to foreach loop in Perl:

for name [in list]
do
    statements that can use $name...
done

The list is a list of names. (If in list is omitted, the list defaults to "$@", i.e., the quoted list of command-line arguments, but you should always supply the in list for the sake of clarity.)

You can generate sequnces programmtically:

        #!/bin/bash
        for i in `seq 1 10`;
        do
           echo $i
        done

This way you can generate sequnce of floating-point numbers, for example :

	for flc in $(seq 1.0 .01 1.1)
	do
	     echo $flc; 
	done

As an example let's create a script that prints the type of each file:

for filename in "$@" ; do
    finfo $filename
    print
done

Next, we define function finfo:

function finfo {
    if [[ ! -a $1 ]]; then
        print "file $1 does not exist."
        return 1
    fi
    file $1
}

The complete script consists of the for loop code and the above function, in either order; good programming style dictates that the function definition should go first.

Here is another programming task that requires usage of for loop

New (C-style) for loop

In bash (since version 2.04) new form of  for syntax, one that looks a lot like C Language, but with double parentheses, was introduced:

	$ for (( i=0 ; i < 10 ; i++ )) ; do echo $i ; done

Its more general form can be described as:

	for (( expr1 ; expr2 ; expr3 )) ; do list ; done

The use of double parentheses indicates that expressions can use syntax of ((..)) construct.  several iteration counters can be used. For example:

	for (( i=0, j=0 ; i < 10 ; i++, j++ ))
	do
	    echo $((i*j))
	done

That for loop initializes two variables (i and j), then has a more complex second expression adding the two together before doing the less-than comparison. The comma operator is used again in the third expression to increment both variables.

Examples of for loops
from Advanced BASH programming Guide

#!/bin/bash
# Listing the planets.

for planet in Mercury Venus Earth Mars Jupiter Saturn Uranus Neptune Pluto
do
  echo $planet  # Each planet on a separate line.
done

echo

for planet in "Mercury Venus Earth Mars Jupiter Saturn Uranus Neptune Pluto"
# All planets on same line.
# Entire 'list' enclosed in quotes creates a single variable.
do
  echo $planet
done

exit 0

 for loop with two parameters in each [list] element

#!/bin/bash
# Planets revisited.

# Associate the name of each planet with its distance from the sun.

for planet in "Mercury 36" "Venus 67" "Earth 93"  "Mars 142" "Jupiter 483"
do
  set -- $planet  # Parses variable "planet" and sets positional parameters.
  # the "--" prevents nasty surprises if $planet is null or begins with a dash.

  # May need to save original positional parameters, since they get overwritten.
  # One way of doing this is to use an array,
  #        original_params=("$@")

  echo "$1		$2,000,000 miles from the sun"
  #-------two  tabs---concatenate zeroes onto parameter $2
done

# (Thanks, S.C., for additional clarification.)

exit 0

A variable may supply the [list] in a for loop.

 Fileinfo: operating on a file list contained in a variable

#!/bin/bash
# fileinfo.sh

FILES="/usr/sbin/privatepw
/usr/sbin/pwck
/usr/sbin/go500gw
/usr/bin/fakefile
/sbin/mkreiserfs
/sbin/ypbind"     # List of files you are curious about.
                  # Threw in a dummy file, /usr/bin/fakefile.

echo

for file in $FILES
do

  if [ ! -e "$file" ]       # Check if file exists.
  then
    echo "$file does not exist."; echo
    continue                # On to next.
   fi

  ls -l $file | awk '{ print $9 "         file size: " $5 }'  # Print 2 fields.
  whatis `basename $file`   # File info.
  echo
done  

exit 0

The [list] in a for loop may contain filename globbing, that is, using wildcards for filename expansion.

Operating on files with a for loop

#!/bin/bash
# list-glob.sh: Generating [list] in a for-loop using "globbing".

echo

for file in *
do
  ls -l "$file"  # Lists all files in $PWD (current directory).
  # Recall that the wild card character "*" matches every filename,
  # however, in "globbing", it doesn't match dot-files.

  # If the pattern matches no file, it is expanded to itself.
  # To prevent this, set the nullglob option
  # (shopt -s nullglob).
  # Thanks, S.C.
done

echo; echo

for file in [jx]*
do
  rm -f $file    # Removes only files beginning with "j" or "x" in $PWD.
  echo "Removed file \"$file\"".
done

echo

exit 0

Omitting the in [list] part of a for loop causes the loop to operate on $@, the list of arguments given on the command line to the script. A particularly clever illustration of this is Example A-17.

 Missing in [list] in a for loop

#!/bin/bash

#  Invoke this script both with and without arguments,
#+ and see what happens.

for a
do
 echo -n "$a "
done

#  The 'in list' missing, therefore the loop operates on '$@'
#+ (command-line argument list, including whitespace).

echo

exit 0

It is possible to use command substitution to generate the [list] in a for loop.

Generating the [list] in a for loop with command substitution

#!/bin/bash
#  for-loopcmd.sh: for-loop with [list]
#+ generated by command substitution.

NUMBERS="9 7 3 8 37.53"

for number in `echo $NUMBERS`  # for number in 9 7 3 8 37.53
do
  echo -n "$number "
done

echo 
exit 0

This is a somewhat more complex example of using command substitution to create the [list].

A grep replacement for binary files

#!/bin/bash
# bin-grep.sh: Locates matching strings in a binary file.

# A "grep" replacement for binary files.
# Similar effect to "grep -a"

E_BADARGS=65
E_NOFILE=66

if [ $# -ne 2 ]
then
  echo "Usage: `basename $0` search_string filename"
  exit $E_BADARGS
fi

if [ ! -f "$2" ]
then
  echo "File \"$2\" does not exist."
  exit $E_NOFILE
fi 

IFS="\n"         # Per suggestion of Paulo Marcel Coelho Aragao.
for word in $( strings "$2" | grep "$1" )
# The "strings" command lists strings in binary files.
# Output then piped to "grep", which tests for desired string.
do
  echo $word
done

# As S.C. points out, lines 23 - 29 could be replaced with the simpler
#    strings "$2" | grep "$1" | tr -s "$IFS" '[\n*]'


# Try something like  "./bin-grep.sh mem /bin/ls"  to exercise this script.

exit 0

More of the same.

 Listing all users on the system

#!/bin/bash
# userlist.sh

PASSWORD_FILE=/etc/passwd
n=1           # User number

for name in $(awk 'BEGIN{FS=":"}{print $1}' < "$PASSWORD_FILE" )
# Field separator = :    ^^^^^^
# Print first field              ^^^^^^^^
# Get input from password file               ^^^^^^^^^^^^^^^^^
do
  echo "USER #$n = $name"
  let "n += 1"
done  


# USER #1 = root
# USER #2 = bin
# USER #3 = daemon
# ...
# USER #30 = bozo

exit 0

A final example of the [list] resulting from command substitution.

 Checking all the binaries in a directory for authorship

#!/bin/bash
# findstring.sh:
# Find a particular string in binaries in a specified directory.

directory=/usr/bin/
fstring="Free Software Foundation"  # See which files come from the FSF.

for file in $( find $directory -type f -name '*' | sort )
do
  strings -f $file | grep "$fstring" | sed -e "s%$directory%%"
  #  In the "sed" expression,
  #+ it is necessary to substitute for the normal "/" delimiter
  #+ because "/" happens to be one of the characters filtered out.
  #  Failure to do so gives an error message (try it).
done  

exit 0

#  Exercise (easy):
#  ---------------
#  Convert this script to taking command-line parameters
#+ for $directory and $fstring.

The output of a for loop may be piped to a command or commands.

 Listing the symbolic links in a directory

#!/bin/bash
# symlinks.sh: Lists symbolic links in a directory.


directory=${1-`pwd`}
#  Defaults to current working directory,
#+ if not otherwise specified.
#  Equivalent to code block below.
# ----------------------------------------------------------
# ARGS=1                 # Expect one command-line argument.
#
# if [ $# -ne "$ARGS" ]  # If not 1 arg...
# then
#   directory=`pwd`      # current working directory
# else
#   directory=$1
# fi
# ----------------------------------------------------------

echo "symbolic links in directory \"$directory\""

for file in "$( find $directory -type l )"   # -type l = symbolic links
do
  echo "$file"
done | sort                                  # Otherwise file list is unsorted.
#  Strictly speaking, a loop isn't really necessary here,
#+ since the output of the "find" command is expanded into a single word.
#  However, it's easy to understand and illustrative this way.

#  As Dominik 'Aeneas' Schnitzer points out,
#+ failing to quote  $( find $directory -type l )
#+ will choke on filenames with embedded whitespace.
#  Even this will only pick up the first field of each argument.

exit 0


# Jean Helou proposes the following alternative:

echo "symbolic links in directory \"$directory\""
# Backup of the current IFS. One can never be too cautious.
OLDIFS=$IFS
IFS=:

for file in $(find $directory -type l -printf "%p$IFS")
do     #                              ^^^^^^^^^^^^^^^^
       echo "$file"
       done|sort

The stdout of a loop may be redirected to a file, as this slight modification to the previous example shows.

 Symbolic links in a directory, saved to a file

#!/bin/bash
# symlinks.sh: Lists symbolic links in a directory.

OUTFILE=symlinks.list                         # save file

directory=${1-`pwd`}
#  Defaults to current working directory,
#+ if not otherwise specified.


echo "symbolic links in directory \"$directory\"" > "$OUTFILE"
echo "---------------------------" >> "$OUTFILE"

for file in "$( find $directory -type l )"    # -type l = symbolic links
do
  echo "$file"
done | sort >> "$OUTFILE"                     # stdout of loop
#           ^^^^^^^^^^^^^                       redirected to save file.

exit 0

There is an alternative syntax to a for loop that will look very familiar to C programmers. This requires double parentheses.

 A C-like for loop

#!/bin/bash
# Two ways to count up to 10.

echo

# Standard syntax.
for a in 1 2 3 4 5 6 7 8 9 10
do
  echo -n "$a "
done  

echo; echo

# +==========================================+

# Now, let's do the same, using C-like syntax.

LIMIT=10

for ((a=1; a <= LIMIT ; a++))  # Double parentheses, and "LIMIT" with no "$".
do
  echo -n "$a "
done                           # A construct borrowed from 'ksh93'.

echo; echo

# +=========================================================================+

# Let's use the C "comma operator" to increment two variables simultaneously.

for ((a=1, b=1; a <= LIMIT ; a++, b++))  # The comma chains together operations.
do
  echo -n "$a-$b "
done

echo; echo

exit 0

---

Now, a for-loop used in a "real-life" context.

Old News ;-)

Bash by example, Part 2

The standard "for" loop in bash is pretty idiosyncratic

OK, we've covered conditionals, now it's time to explore bash looping constructs. We'll start with the standard "for" loop. Here's a basic example:

#!/usr/bin/env bash

for x in one two three four
do
    echo number $x
done

output:
number one
number two 
number three 
number four

What exactly happened? The "for x" part of our "for" loop defined a new environment variable (also called a loop control variable) called "$x", which was successively set to the values "one", "two", "three", and "four". After each assignment, the body of the loop (the code between the "do" ... "done") was executed once. In the body, we referred to the loop control variable "$x" using standard variable expansion syntax, like any other environment variable. Also notice that "for" loops always accept some kind of word list after the "in" statement. In this case we specified four English words, but the word list can also refer to file(s) on disk or even file wildcards. Look at the following example, which demonstrates how to use standard shell wildcards:

#!/usr/bin/env bash

for myfile in /etc/r*
do
    if [ -d "$myfile" ] 
    then
      echo "$myfile (dir)"
    else
      echo "$myfile"
    fi
done

output:

/etc/rc.d (dir)
/etc/resolv.conf
/etc/resolv.conf~
/etc/rpc                  

The above code looped over each file in /etc that began with an "r". To do this, bash first took our wildcard /etc/r* and expanded it, replacing it with the string /etc/rc.d /etc/resolv.conf /etc/resolv.conf~ /etc/rpc before executing the loop. Once inside the loop, the "-d" conditional operator was used to perform two different actions, depending on whether myfile was a directory or not. If it was, a " (dir)" was appended to the output line.

We can also use multiple wildcards and even environment variables in the word list:

for x in /etc/r??? /var/lo* /home/drobbins/mystuff/* /tmp/${MYPATH}/*
do
    cp $x /mnt/mydir
done

Bash will perform wildcard and variable expansion in all the right places, and potentially create a very long word list.

While all of our wildcard expansion examples have used absolute paths, you can also use relative paths, as follows:

for x in ../* mystuff/*
do
    echo $x is a silly file
done

In the above example, bash performs wildcard expansion relative to the current working directory, just like when you use relative paths on the command line. Play around with wildcard expansion a bit. You'll notice that if you use absolute paths in your wildcard, bash will expand the wildcard to a list of absolute paths. Otherwise, bash will use relative paths in the subsequent word list. If you simply refer to files in the current working directory (for example, if you type "for x in *"), the resultant list of files will not be prefixed with any path information. Remember that preceding path information can be stripped using the "basename" executable, as follows:

for x in /var/log/*
do
    echo `basename $x` is a file living in /var/log
done

Of course, it's often handy to perform loops that operate on a script's command-line arguments. Here's an example of how to use the "$@" variable, introduced at the beginning of this article:

#!/usr/bin/env bash

for thing in "$@"
do
    echo you typed ${thing}.
done

output:

$ allargs hello there you silly
you typed hello.
you typed there.
you typed you.
you typed silly.

Recommended Links

Loops Korn Shell Reference

UNIX Shell Script Tutorials & Reference

Loop Control (Advanced Bash-Scripting Guide)

Last modified: June 02, 2008