Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-)

Pipes and Coroutines in Shell

Korn shell and its derivates like ksh93 and zsh  ( possesses an interesting and unique feature: the ability to pipe a file into a loop. This is effectively an implementation of simple coroutines although shell has also more general form (See Learning Korn Shell by Bill Rosenblat and Arnold Robbins). In bash this capability is limited as bash does not run the last stage of the pipe as the current process... The developers probably think they can reinvent the wheel but got a square..  Googling for "bash pipe subprocesses order" shows the extent of the problem, but I couldn't find what the bash developers' official stand

Let's assume that we need to find all files that contain string "19%" which is a typical for printing commands like "19%2d"

cd/ /usr/bin
ls | while read file
do
    echo $file
    string $file | grep '19%'
done

Here we use the ls command to generate the list of the file names and this list it piped into a loop. In a loop we echo command and then run strings piped to grep looking for suspicious format strings.

In another example from O'Reilly "Learning Korn Shell" (first edition). Here we will pipe awk output into the loop. This is a  function that, given a pathname as argument, prints its equivalent in tilde notation if possible:

function tildize {
    if [[ $1 = $HOME* ]]; then
        print "\~/${1#$HOME}"
        return 0
    fi
    awk '{FS=":"; print $1, $6}' /etc/passwd | 
        while read user homedir; do
            if [[ $homedir != / && $1 = ${homedir}?(/*) ]]; then
                print "\~$user/${1#$homedir}"
                return 0
            fi
        done
    print "$1"
    return 1
}

Loop can also serve as a source to input for the pipe. For example

{ while read line'?adc> '; do
      print "$(alg2rpn $line)"
  done 
} | dc

As an example; assume that you want to go through all C files of a directory and, if they are readable to you, convert the filenames to contain lowercase letters only (this example may be a little contrived). We can do it it in slightly different ways.

The first script calls tr inside the the for-loop:

#!/bin/sh
for x in *.c
do
  [ -r $x ] && echo $x | tr 'A-Z' 'a-z'
done

#!/bin/sh
for x in *.c
do
  [ -r $x ] && echo $x 
done | tr 'A-Z' 'a-z'

Old News ;-)

pipe viewer. 

Can be inserted into any normal pipeline between two processes to give a visual indication of how quickly data is passing through, how long it has taken, how near to completion it is, and an estimate of how long it will be until completion. It has precompiled Solaris binary (Solaris binary )

The Linux and Unix Menagerie Using Bash To Feed Command Output To A While Loop Without Using Pipes!

But here's a really neat trick for getting this to work in bash 2.x. If you change your program to be structured like so:

while read line
do
        echo $line
done < <(ls -1d *)

Your outcome will result in success!! You've got the command output and you didn't have to use a pipe to feed it to the while loop!

NOTE: The two most important things to remember about doing this are that:

1. The space between the first < and second < is mandatory! Although, it should be noted that, between the two <'s, you can have as many spaces as you want. You can even use a tab between the two <'s, they just can't be directly connected.

2. The command, from which you want to use output as fodder for the while loop, needs to be run in a subshell (generally placed between parentheses, just like the ones surrounding this sentence) and the left parenthesis must immediately follow the second <, with "no" space in between!

We've already looked at what happens if you ignore rule number 1 and use << instead of < <. If you ignore rule number 2, you'll get:

./program: line 4: syntax error near unexpected token `<'
./program: line 4: `done < < (ls -1d *)'

And here's the "even better part" - In bash 3.x, you don't have to worry about all that spacing anymore, as they've added a new feature which does the same thing (or is it really just an old feature dressed up to make it seem fabulous? ;) In bash 3.x, you can use the triple-< operator. Actually, I believe the <<< syntax is referred to as a "here string," but that's purely academic. They could call it "fudge," as long as it works ;)

So, in bash 3.x, you could write a while loop that takes input from a command without using a pipe like so:

while read line
do
 echo hi $line
done <<< `ls -1d *`

NOTE: The space between the <<< and your backticked (or otherwise extrapolated) command output is not necessary and you can have as much space as the shell can stand between those two parts of the "here string." Of course, the three <'s need to be all clumped together with no space in between them.

bash Cookbook Reader - Contributions browse

pipeline of commands
We discuss in the book (in the chapter on common mistakes) the fact that a pipeline of commands runs those commands in subshells. The result (or dilema) is that what happens in those subshells (e.g. counting something) is lost to the parent shell script unless the parent captures output from the pipeline, but that isn't always easy or desirable.

The bash man page describes a feature of bash called "Process Substitution" that lets you substitute the output of a pipeline of commands (actually a list of commands) using <(list) as the syntax.

But notice how the feature is described:

The process list is run with its input or output connected to a FIFO or some file in /dev/fd. The name of this file is passed as an argument to the current command as the result of the expansion.

 

The <(...) is going to be replaced with the name of a fifo. So if you wrote:
 

wc <(some commands)

the result would be:

wc fifo

that is, the fifo filename is passed to the command. That's fine for commands like wc that can accept a filename. But what about a builtin like while?

It turns out that you can add the redirect from the fifo, but the space between the two less-than signs is crucial to distinguish it from "<<", the "here document" syntax.

So you can write:

  while read a b c
  do
    ...
  done < <(pipeline of commands)

Internal Commands and Builtins

Piping output to a read, using echo to set variables will fail.

Yet, piping the output of cat seems to work.

cat file1 file2 | while read line do echo $line done

However, as Bjön Eriksson shows:

Example 14-8. Problems reading from a pipe

#!/bin/sh # readpipe.sh # This example contributed by Bjon Eriksson. last="(null)" cat $0 | while read line do echo "{$line}" last=$line done printf "\nAll done, last:$last\n" exit 0 # End of code. # (Partial) output of script follows. # The 'echo' supplies extra brackets. ############################################# ./readpipe.sh {#!/bin/sh} {last="(null)"} {cat $0 |} {while read line} {do} {echo "{$line}"} {last=$line} {done} {printf "nAll done, last:$lastn"} All done, last:(null) The variable (last) is set within the subshell but unset outside.

The gendiff script, usually found in /usr/bin on many Linux distros, pipes the output of find to a while read construct.

find $1 \( -name "*$2" -o -name ".*$2" \) -print | while read f; do . . .

 

It is possible to paste text into the input field of a read. See Example A-39.

ksh vs bash setting variable in piped loops are lost - Ubuntu Forums

ksh vs bash: setting variable in piped loops are lost

---

I'm a long time unix (not linux) programmer, mainly shell scripts.
Unix does not know about bash, but rather ksh or sh.

So I often build stuff like this:
 

Code:

n=0
du | sort -n | while read size dir
do
  if [ "$size" -gt 100000 ]
  then
    n=$((n+1))
  fi
done
echo "Found $n too big files"

the question is not how to reformulate this differently using awk or perl.
The question is:

in ksh this script returns the correct value
in bash it always returns 0

This is because in ksh the last command in the pipe runs in the current process, whereas in bash the first command runs in the current proces. Result: the modified variables are lost.

I'm still puzzled that his is the case and that nobody really cares about.
Maybe I'm missing the point and there is some simple environment variable or other setting to change to enable ksh compatible

Recommended Links

Shell Programming

Shell Script Pearls - Google Book Search

My Favorite bash Tips and Tricks

Last modified: August 07, 2008