|
Softpanorama |
May the source be with you, but remember the KISS principle ;-)
|
Tee is a very interesting Unix command. It is a very important Unix command as it permit constuction of complex pipes. The idea is to split pipe so that you write to the standard output (most commonly the next stage of the pipe) and to a file (or named pipe) simultaneously.
Tee is also very useful in complex pipes debugging: you place the tee command anywhere in a pipe command to divert a copy of the standard input (of tee) to disk and analyze correctness of this stage. A special variant of the tee for the shell is called script and permits duplicating all input commands submitted to a shell into a file.
The name is also very apt: while it is taken from the world of plumbing it perfectly describes the purpose.
The typical usage is to split recorded some stage of a complex pipe either for debugging or for further processing. In the latter case this is the only way complex pipes can be implemented in Unix. Target in tee can be not only file but a named pipe.
A typical example of tee usage would be
sort somefile.txt | tee sorted_file.txt | uniq -c | head 12 > top12.txt
More complex example would be processing records of proxy usage that contain code 403 to extract champions, top 12 users who got the most of rejections:
# gzip -d -c all_code403.gz | cut -d ' ' -f 1 | sort -k 1 | uniq -c | sort -r \
| tee ./d$1/all_users | head -12 > ./d$1/all_champs
here we first write all the users sorted in reverse order to the disk and then generated the top 12 using head. Un this case standard output of tee is piped to head
You can use the tee command to debug complex pipes that for some reason do not work as expected. It can be placed anywhere in a pipe to check what the output looks like at a certain point. you can also save some pipes stages output for future usage if it represents the result of the stage that takes a lot of time. Such situation often arise of you process proxy or HTTP server logs using pies.
The other interesting usage of tee is to split pipe into several substreams using named pipes, for example
mkfifo pipe1 pipe2 # create
named pipes
echo ***
gzip -d -c all$1.gz | tee pipe1 | grep '" 200' | tee pipe2 >(gzip > all$1ok.gz)
| cut -d '"' -f 2 | cut -d '/' -f 3 | tr '[:upper:]' '[:lower:]' | sort
| uniq -c | sort -r > ./d$1/oksites$1 & \
<pipe1 cut -d ' ' -f 1 | grep '" 403 ' | sort | uniq -c | sort -r > ./d$1/all$1xuser
& \
<pipe2 cut -d ' ' -f 1 | sort | uniq -c | sort -r | tee ./d$1/all$1users
| head -12 > ./d$1/all$1champ
Following is the general format of the tee command.
tee [ -ai ] file_list
Options
The following options may be used to control how tee functions.
| -a | Appends the output to an existing file. If the file does not exist it will be created. Normally an existing file would be overwritten. |
| -i | Ignore interrupts. If you press the Delete key, tee ignores the interrupt signal sent to it and continues processing. |
Arguments
The following argument may be passed to the tee command.
| file_list | One or more files where tee writes copies of the input. |
Type nawk -F: '{ print $1}' /etc/passwd | sort | tee users | lp and press Return. Notice the message returned by lp. You now have a printout on the default printer. The output is also stored in the file named users.
5.2.2.3. Writing to output and files simultaneouslyYou can use the tee command to copy input to standard output and one or more output files in one move. Using the -a option to tee results in appending input to the file(s). This command is useful if you want to both see and save output. The > and >> operators do not allow to perform both actions simultaneously.
This tool is usually called on through a pipe (|), as demonstrated in the example below:
Cool commands
- The following command saves stdout and stderr to the files "out.txt" and "err.txt", respectively.
- The following command appends stdout and stderr to the files "out.txt" and "err.txt", respectively.
- The following command functions similar to the above two commands, but also copies stdout and stderr to the files "stdout.txt" and "stderr.txt", respectively.
Note: Lines that end in a backslash are continued on the next line. Any such lines should be keyed in as one complete line. The lines are too long to fit on the web page without formatting them this way.
- Cool Tar Command
Note: Lines that end in a backslash are continued on the next line. Any such lines should be keyed in as one complete line. The lines are too long to fit on the web page without formatting them this way.
- Cool Find Command - looks in each file for searchstring
[root@server /root]# find . -type f -exec grep -i \ searchstring \{\} --with-filename \;
Note: Lines that end in a backslash are continued on the next line. Any such lines should be keyed in as one complete line. The lines are too long to fit on the web page without formatting them this way.
- Sample Test Script
- For loop demonstrating stderr and stdout
echo Standard Out >stdout.txt echo Standard Error >stderr.txt for X in bzImage modules modules_install; do make $X; done 1>>stdout.txt 2>>stderr.txt
- Redirection in Linux
0 = stdin
1 = stdout
2 = stderr
- Using the tee command to save stdout to tee.txt. stdout is still displayed on the screen.
- Using the tee command to append stdout to tee.txt. stdout is still displayed on the screen.
- Using the script command to capture both stderr and stdout
- Notes about pipe "|"
- Lets consider a channel one of stdout or stderr.
- The pipe can only carry one channel, namely stdout.
- When you have a command line which utilizes the pipe command, stderr gets sent to the display while stdout gets sent through the pipe on to the other awaiting commands.
- Swapping stdout and stderr will allow stdout to be sent to the display while stderr can be sent through the pipe.
- Pipe by itself will only take stdout. If you swap stdout and stderr before you get to the pipe then the pipe will take stdout (which is now stderr).
- stdout and stderr can be combined and sent through the pipe; however, you have now combined both stderr and stdout and there is no way to separate them.
- using a pipe in a subshell "()" causes stdout to go through the pipe and causes stderr to pass through the subshell back to the display. This provides a way to grab stdout and stderr. Basically you could get both stdout and stderr out of a subshell.
- Order of redirecting
- Method 1
#The above command causes the original stderr to go to stdout,
# and causes the original stdout to go to outfile.txt.
#Notice at the time that fd (file descriptor) 2 is redefined, fd 1 still pointed to stdout; therefore, fd 2 will continue to point to stdout independently of what later happens to fd 1. Basically fd 2 copies the item (or address) that fd 1 is pointing to.
(Corrected an incorrect statement - Thanks to Morten for bringing it to my attention.)
- Method 2 #The following command causes both the original stdout and stderr to go to outfile.txt.
#Notice that 1=outfile.txt when 2 is redefined, so 2 goes to outfile.txt too (the two
# channels are combined).
Capturing stderr with tee. Swapping stderr and stdout
- The full command
The following section will walk through the theory behind this command. Once you have a general understanding of this theory, you should be able to easily regenerate this entire command without notes.
- The walk through
- Contents of the ./cmd script
- Results from running the ./cmd script
Although you see both lines printed on the screen, behind the scenes one actually went to stdout and the other went to stderr. If you were to do a pipe, only stdout goes through the pipe. Normally this is the desired effect.
- Capturing stdout
The following will capture a copy of stdout and save it to a file called "stdout.txt"
stdout goes through the pipe and tee is able to save a copy of it to the file "stdout.txt"; however, we just lost control of stderr. stderr will not go through the pipe, instead it goes directly to our display.
- Gaining control of stderr and stdout.
Lets gain control again of stderr and stdout. We do this by surrounding our command with a set of parenthesis.
- Swapping stdout and stderr.
Now that we have captured stdout, we wish to capture stderr using tee as well. The pipe will only accept stdout, so we must swap stderr and stdout to do this.
Note: The switch is using the standard variable switch method -- 3 variables (buckets) are required to swap 2 variables with each other. (you have 2 variables and you need to switch the contents - you must bring in a 3rd temporary variable to hold the contents of one value so you can properly swap them).
- Capturing stderr
Now that we have swapped our stdout and stderr, lets hook up tee once again. tee will now capture stderr (tee believes that it is really stdout because stdout is the only thing that can come through the pipe).
- Gaining control of stderr and stdout, for the 2nd time.
Tee grabs stderr, but once again the channel that doesn't go through the pipe gets sent to the display and we loose it. Lets capture both our stderr and stdout, once again, by using parenthesis.
- Swapping stdout and stderr back to their normal state.
Now we have, once again, captured both stderr and stdout for our use. Currently they are reversed. Lets switch them back for proper use in our pipeline. Again, lets use the standard variable switch to swap them around.
- Gaining control of stderr and stdout, for the 3rd time.
At this point we have swapped stdout and stderr back to their normal positions; however, if we will be manipulating stdout and stderr any further, we should complete this command with either a pipe or another set of parenthesis.
Since we want to be as complete as possible in this example we will use parenthesis. Using parenthesis will gain control over both stdout and stderr. Using a pipe will only gain control over stdout.
Note: If we use a pipe or parenthesis the next process that hooks up to this command will see stderr and stdout in their proper place. If we don't add the last set of parenthesis, or go through a pipe, the order will remain messed up.
- Redirecting stdout and stderr to separate files
Now lets do something productive with stdout and stderr so that we can really prove that everything went back to their proper place. Lets tell our command to redirect stdout to "out" and stderr to "err".
Note: This last step is optional, normally you would insert your other required commands here, commands that would more than likely operate on stdout.
Please note that the results for "out" and "err" are the same as when you run the following command. This proves that we restored stdout and stderr back to their normal usable posisitions. The above command; however, gives us the capability to copy out stdout and stderr using tee and still be able to use stdout and stderr like we always have.
Capturing stderr in one file and stderr and stdout combined in another file
I had a request for stderr in 1 file and stderr and stdout combined in another file so here it is:
Please keep in mind when reading a book or web page with command line documentation
that the trailing slash signifies continuation of the current line. The reason for
this is due to the line width available on the screen and printed page. Note that
the trailing slash must be the last character on the line followed by enter. Spaces,
tabs, etc., may not apear after the trailing backslash because that does not signify
line continuation but rather some form of escape character. There are two ways that
you may enter this information at the command line:
1. Exactly as show, with the backslashes, followed by enter marks or
(((echo out >&1; echo err >&2) 3>&2 2>&1 1>&3 \
| tee stderr.txt ) 3>&2 2>&1 1>&3 ) > combined.txt 2>&1
2. Remove the trailing backslashes and do not press enter after each line, just word wrap
the lines together at the command prompt.
(((echo out >&1; echo err >&2) 3>&2 2>&1 1>&3 | tee stderr.txt ) 3>&2 2>&1 1>&3 ) > combined.txt 2>&1
Other Good Reading:
jobcontrol.html
man bash
Contents of ./myprog.sh
#!/bin/bash
echo "Standard Out" >&1
echo "Standard Error" >&2
Break out Standard Out into stdout.log, Standard Error into stderror.log, then display both on the screen:
((( ./myprog.sh | tee stdout.log ) 3>&2 2>&1 1>&3 | tee stderr.log ) 3>&2 2>&1 1>&3 )
Note, the lines may not come in the same order as without the redirection.
Note, the outside set of parenthesis (and the last set of movement just inside that last set of parenthesis) places stdout back on "1" and stderr back on "2" just in case you want to manipulate them further.
Break out Standard Out into stdout.log, Standard Error into stderror.log, combine the output (stdout and stderr), then display both on the screen:
((( ./myprog.sh | tee stdout.log ) 3>&2 2>&1 1>&3 | tee stderr.log ) 3>&2 2>&1 1>&3 ) 2>&1 | tee combined.log
Sometimes you want to be able to monitor the progress of a long running process, but you also want to save a transcript of its output so you don't have to watch it every second.tee was designed with this very purpose in mind.
% ls -l | tee foobar total 26 -rw-r--r-- 1 jeffy 28 May 9 16:12 Makefile -rwxr-xr-x 1 jeffy 24576 May 28 11:31 foo -rw-r--r-- 1 jeffy 57 May 9 16:13 foo.c % ls Makefile foo foo.c foobar % cat foobar total 26 -rw-r--r-- 1 jeffy 28 May 9 16:12 Makefile -rwxr-xr-x 1 jeffy 24576 May 28 11:31 foo -rw-r--r-- 1 jeffy 57 May 9 16:13 foo.c
Keep in mind that tee only duplicates stdout, so if you want to save error output (you probably do), you'll need to combine stderr onto stdout with "|&" if you're in csh-land, or "2>&1" if you're playing with sh.Also note that tee accepts a "-a" flag which tells it to append its output to the file instead of truncating the file first.
Why's it called tee, you ask? Named after the pipe fitting that looks like a letter "T" and sends its input to two different places.
tee is also distinguished by having one of the shorter man pages in UNIXdom.
% cd /vobs/App/Mr % clearmake |& tee Transcript
Copyright © 1996-2008 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. Submit comments This document is an industrial compilation designed and created exclusively for educational use and is placed under the copyright of the Open Content License(OPL). Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.
Standard disclaimer: The statements, views and opinions presented on this web page are those of the author and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.
Last modified: September 14, 2008