Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-)

Perl as a command line utility tool

In modern Unix programming environment Perl can be used as AWK replacement even in simple scripts. Recently Perl became a standard, installed by default  in all major Unixes including AIX, HP-UX, Linux and Solaris. It is usually available in /usr/bin.

Default availability dramatically changed role of Perl in Unix system scripting and routine text processing. For most administrators it is much easier to use Perl that any of the older alternatives. The main consideration here is the power of the language and availability of a very good, built-in debugger.  Neither bash nor AWK has built-in debugger installed by default. And that alone tips the scales in Perl favor, as each administrator has a lot of things to do to spend time guessing where he made mistake using multiple  echo/print statements. 

That's why Perl is gradually displacing older Unix utilities such as cut, sed,  wc, and, of course, AWK. Often you can replace quite complex set of pipe stages that use classic UNIX utilities with one Perl loop. Don't interpret me wrong, pipes are a blessing and extremely powerful tool that each UNIX admins should use to the max, but if you can avoid unnecessary complexity, why stick to old practices.

Perl is also amazingly Unix-friendly language that give the programmer full access to the Unix API.  For an introduction to Perl writen for system admistrators see Nikolai Bezroukov. Skeptical Introduction to Perl

The simplest way to start is to remember -e option (execute), which instructs Perl interpreter that the next argument is a Perl statement to be compiled and run.  If -e is given, Perl will not look for a script filename in the argument list.  Make sure to use semicolons where you would in a normal program. For example

perl -e 'print "Hello world of Perl command line";'

Multiple -e commands may be given to build up a multi-line script.

There are several more options that can sometimes simplify writing "in-line" Perl scripts or make them more similar to traditional AWL-scripts: 

• -n causes Perl to assume the following loop around your script, which makes it iterate over filename arguments somewhat like sed -n or awk:

  while (<>) { ... # your script goes here }

  The lines are not printed by default. You need to use -p to have lines printed. If a file named by an argument cannot be opened for some reason, Perl warns you about it, and moves on to the next file.
   

• -p causes Perl to assume the following loop around your script, which makes it iterate over filename arguments somewhat like sed:

  while (<>) { ... # your script } continue { print or die "-p destination: $!\n";}

• -a turns on autosplit mode when used with either -n or -p. An implicit split command to the @F array is done as the first thing inside the implicit while loop produced by the -n or -p.  For example

  perl -ane 'print pop(@F), "\n";'

  is equivalent to

    while (<>) { @F = split(' '); print pop(@F), "\n"; }

It is important to know that an alternate delimiter for split may be specified using option -F.
   

• -e  specified the one-line to be executed.  We already discussed it above.
   

• If a file named by an argument cannot be opened for some reason, Perl warns you about it, and moves on to the next file. Note that the lines are printed automatically. An error occurring during printing is treated as fatal. To suppress printing use the -n switch. A -p overrides a -n switch.

Please note that you can use option -d   to debug the script.

As a useful example let's look how we can combine power of Perl command line with find utility to produce a basic global string/pattern replace utility for multiple files :

$ find . -type f -exec perl -i -pe 's/something/another/g' {} \;

To make a command named lower that converts all filenames in the current directory to lower case add this to your ~/.bashrc or  ~/.kshrc

function lower { perl -e 'for (@ARGV) { rename $_, lc($_) unless -e lc($_); }' * ; }

Tom Christiansen posted a list of  on Usenet years ago, and that list is still interesting and useful for any Perl programmer. See his list plus some more modern one liners on a separate page Perl One Liners which you are encouraged to browse; the old Tom Christiansen list widely available in the file tomc.txt  in various places on the Internet.

As with everything, excessive zeal hurts.  You need to exercise sound judgment and the moment when one liner became counterproductive because of excessive complexity and should be converted into a regular script.

Emulation of Unix cut utility in Perl

If option -a ( autosplit mode) is specified each line is converted into array @F.  Here's a simple one-line script that will print out the fourth word of every line, but also skip any line beginning with a # because it's a comment line.

perl -naF 'next if /^#/; print "$F[3]\n"'

Here is example on how to print first and the second from the last columns:

perl -lane 'print "$F[0]:$F[-2]\n"'

Here is example from Tom Christiansen one liners which shows how to print each line with its characters backwards:

        perl -nle 'print scalar reverse $_' file1 file2 file3 ....

Old News ;-)

[May 21, 2006] Matz Kindahl - Collection of Perl programs

These are one liners that might be of use. Some of them are from the net and some are one that I have had to use for some simple task. If Perl 5 is required, the perl5 is used.

perl -ne '$n += $_; print $n if eof'
perl5 -ne '$n += $_; END { print "$n\n" }'

To sum numbers on a stream, where each number appears on a line by itself. That kind of output is what you get from cut(1), if you cut out a numerical field from an output. There is also a C program called sigma that does this faster.
 

perl5 -pe 's/(\w)(.*)$/\U$1\L$2/'
perl5 -pe 's/\w.+/\u\L$&/'

To capitalize the first letter on the line and convert the other letters to small case. The last one is much nicer, and also faster.
 

perl -e 'dbmopen(%H,".vacation",0666);printf("%-50s: %s\n",$K,scalar(localtime(unpack("L",$V)))while($K,$V)=each(%H)'

Well, it is a one-liner. :)
You can use it to examine who wrote you a letter while you were on vacation. It examines the file that vacation(1) produces.

 

perl5 -p000e 'tr/ \t\n\r/ /;s/(.{50,72})\s/$1\n/g;$_.="\n"x2'

This piece will read paragraphs from the standard input and reformat them in such a manner that every line is between 50 and 72 characters wide. It will only break a line at a whitespace and not in the middle of a word.

 

perl5 -pe 's#\w+#ucfirst lc reverse $&#eg'

This piece will read lines from the standard input and transform them into the Zafir language used by Zafirs troops, i.e. "Long Live Zafir!" becomes "Gnol Evil Rifaz!" (for some reason they always talk using capital letters).
Andrew Johnson and I posted slightly different versions, and we both split the string unnecessarily. This one avoids splitting the string.

 

perl -pe '$_ = " $_ "; tr/ \t/ /s; $_ = substr($_,1,-1)'

This piece will remove spaces at the beginning and end of a line and squeeze all other sequences of spaces into one single space.
This was one of the "challenges" from comp.lang.perl.misc that occurs frequently; I am just unable to resist those. :)

[Mar 22, 2006 ] NewsForge Command line Perl for sysadmins By: Hubert Chen

So the command perl -pe '' file1 file2 specifies an empty Perl script, but perl -p will still read in every line from those two files and print it to STDOUT. That line of code will perform the same function as the command cat file1 file2.

To make changes to big files I frequently use commands like this:

perl -pe 's/here is a typi/here is a typo/g' < inputfile > outputfile

In this command, 's/here is a typi/here is a typo/g' tells Perl to search for "here is a typi" and replace it with "here is a typo." Make sure to give the string match enough context to avoid accidentally substituting more times than you want. For example, if we limited the search string to "typi" we might end up generating "typong" from "typing" or "typocal" from "typical."

The g modifier tells Perl to replace the string globally. Without it, substitution occurs only once per line.

Notice that the whole script is enclosed in single quotes. The shell will expand *, $, and other shell variables that aren't in single quotes, so usually you will want to enclose a script in single quotes. Perl scripts use single and double quotes in the same way as the shell: substituting in variables when strings are enclosed in double quotes, and leaving strings in single quotes as literals. If you want to use single quotes within the script itself, Perl allows you to instead use the format q/Hello World/ instead of enclosing a string in single quotes, or qq/Hello World/ for double quotes.

Compare the results of these two commands:

perl -e "$a = qq/Hello World/; print $a" # error
perl -e '$a = qq/Hello World/; print $a' # prints "Hello World"

In the first case, the shell attempts to substitute the shell variable $a because the entire string is in double quotes. The example fails because $a is probably not set by the your shell. It will send to Perl the script " = qq/Hello World/; print " causing an invalid syntax error if you use bash or ksh, or if using csh or tcsh your shell will throw an error before sending it to Perl because you are using an undefined variable. In the second example, the shell does not expand $a but correctly sends the literal script '$a = qq/Hello World/; print $a' to Perl. This has the desired effect of setting the $a variable to "Hello World" and then printing it.

When you use perl -pe for a stream operation like perl -pe 's/here is a typi/here is a typo/g' < inputfile > outputfile, Perl allows you to edit the file in place and make a backup of the old version as well.

The -i option of Perl means that you want to edit the file in place and overwrite the original version. Use this option with caution. A safer solution is to use an argument to the -i option to store a backup copy of the original file. For example, if you used the option -i.bak on a file named foo, the new edited version of the file would be foo and the original would be saved as foo.bak.

In effect, you can shorten a command like mv file file.old; perl -pe 's/oldstring/newstring/g' file.old > file to perl -p -i.old -e 's/oldstring/newstring/g' file. Even better, you can do it as a bulk operation on a set of files, like so:

perl -p -i.old -e 's/oldstring/newstring/g' file*

Here's an easy way to change all the strings in all files recursively:

find . | xargs perl -p -i.old -e 's/oldstring/newstring/g'

Perl's -a and -F options help parse a file while you are reading it. -a turns on autosplit mode. When autosplit is enabled, after reading each line of input, Perl will automatically do a split on the line and assign the resulting array to the @F variable. Each line is split up by whitespace unless the -F parameter is used to specify a new field delimiter. These two features simplify parsing when the file is a simple record-oriented format.

Here's a simple one-line script that will print out the fourth word of every line, but also skip any line beginning with a # because it's a comment line.

perl -naF 'next if /^#/; print "$F[3]\n"'

This second one-line script will extract all usernames from /etc/passwd.

perl -na -F: -e 'print "$F[0]\n"' < /etc/passwd

You can use -F/:/ to split on a pattern instead of a string literal. Be careful, because the shell may escape characters preceded by a \ if they are not enclosed in single quotes. To split on whitespace use -F'/\s+/'.

Of course, command line Perl can be used for more general sysadmin tasks in addition to file editing. Let's say you have some HTTP log files named with a date timestamp like access_log.2005-1-1 .. access_log.2005-12-31 and you want to copy them to access_log.old.2005-1-1 .. access_log.old.2005-12-31. Copying the files by hand would be a slow and error-prone operation. You could create a script to do this, but it is a simple enough operation that you can do it with a quick line of Perl.

perl -e 'for(<access_log.*>){$a = $_; s/log/log.old/; `cp $a $_`}'

In this script the <access_log.*> uses Perl's file globbing to create a for loop of all the files. It assigns each one to the variable $_, then executes the loop. The loop first stores the old file name in the $a variable, then changes the $_ variable to substitute log and changes it to log.old. Finally it calls an external cp command to copy the filename from the old filename to the new one. An external cp might be slower than calling Perl's File::Copy module, but brevity prevails in these short code bits when you have no concerns about the script&#s performance.

In Perl, backticks call an external command and then return the output of that command. This can make your scripts much simpler. Sometimes it is the only reasonable way to accomplish a task. As an example, it is possible to write a Perl script to read the /proc file system to find out which processes are running and what they are doing, but it's easier to capture the output of a ps command and then parse that.

If you know a faster way to get your task done with another program, don't feel obligated to write a pure Perl solution. In many cases, using an external program is the only way to accomplish a task, but wrapping command line Perl around it is good way to automate your task.

The following example copies a script named command.pl to machines B, C, and D, then execute it and prints out the results. Using scp and ssh you can do this with a script like this:
perl -e 'for("B","C","D"){`scp command.pl $_:`; `ssh $_ command.pl`; }'.

This works even better if you have public key authentication set up properly.

perl.com Perl Command-Line Options

Perl has a large number of command-line options that can help to make your programs more concise and open up many new possibilities for one-off command-line scripts using Perl. In this article we'll look at some of the most useful of these.

perl.com FMTYEWTK About Mass Edits In Perl

If you only want to read in one or more files, apply a regex to the contents, and spit out the altered text as one big stream -- the best approach is probably a one-liner such as the following:

perl -p -e "s/Foo/Bar/g" <FileList>

This command calls perl with the options -p and -e "s/Foo/Bar/g" against the files listed in FileList. The first argument, -p, tells Perl to print each line it reads after applying the alteration. The second option, -e, tells Perl to evaluate the provided substitution regex rather than reading a script from a file. The Perl interpreter then evaluates this regex against every line of all (space separated) files listed on the command line and spits out one huge stream of the concatenated fixed lines.

In standard fashion, Perl allows you to concatenate options without arguments with following options for brevity and convenience. Therefore, you'll more often see the previous example written as:

perl -pe "s/Foo/Bar/g" <FileList>

In-place Editing

If you want to edit the files in place, editing each file before going on to the next, that's pretty easy, too:

perl -pi.bak -e "s/Foo/Bar/g" <FileList>

The only change from the last command is the new option -i.bak, which tells Perl to operate on files in-place, rather than concatenating them together into one big output stream. Like the -e option, -i takes one argument, an extension to add to the original file names when making backup copies; for this example I chose .bak. Warning: If you execute the command twice, you've most likely just overwritten your backups with the changed versions from the first run. You probably didn't want to do that.

Because -i takes an argument, I had to separate out the -e option, which Perl otherwise would interpret as the argument to -i, leaving us with a backup extension of .bake, unlikely to be correct unless you happen to be a pastry chef. In addition, Perl would have thought that "s/Foo/Bar/" was the filename of the script to run, and would complain when it could not find a script by that name.

Running Multiple Regexes

Of course, you may want to make more extensive changes than just one regex. To make several changes all at once, add more code to the evaluated script. Remember to separate each additional line of code with a semicolon (technically, you should place a semicolon at the end of each line of code, but the very last one in any code block is optional). For example, you could make a series of changes:

perl -pi.bak -e "s/Bill Gates/Microsoft CEO/g; s/CEO/Overlord/g" <FileList>

"Bill Gates" would then become "Microsoft Overlord" throughout the files. (Here, as in all examples, we ignore such finicky things as making sure we don't change "HERBACEOUS" to "HERBAOverlordUS"; for that kind of information, refer to a good treatise on regular expressions, such as Jeffrey Friedl's impressive book Mastering Regular Expressions, 2nd Edition. Also, I've wrapped the command to fit, but you should type it in as just one line.)

Doing Your Own Printing

You may wish to override the behavior created by -p, which prints every line read in, after any changes made by your script. In this case, change to the -n option. -p -e "s/Foo/Bar/" is roughly equivalent to -n -e "s/Foo/Bar/; print". This allows you to write interesting commands, such as removing lines beginning with hash marks (Perl comments, C-style preprocessor directives, etc.):

perl -ni.bak -e "print unless /^\s*#/;" <FileList>

[Nov 20, 2004] One-Liners in Perl by S. Lee Henry

One of the more unusual past-times of Unix geeks is determining how much logic can be crammed into a single line of code. Perl hackers do this exceedingly well. For example, here's a one-liner test for prime numbers:

	perl -le 'print "PRIME" if (1 x shift) !~ /^(11+)\1+$/' 19

Abigail@fnx.com contributed that snippet to the Perl Journal's collection of one-liners available at http://www.itknowledge.com/tpj/one-liners01.html . You substitute whatever number you want to check for the number 19 at the end.

Perl is as terse as it is powerful. Even so, few Perl programmers code as tightly as this! Most are happy producing more relaxed and readable code. Even so, one-liners are useful for a lot more than proving your worth as a tight coder. A versatile one-liner can be used to do much quick and dirty processing on the command line. Here's a simple substitute command that might come in very handy:

	perl -p -i -e 's/this/that/g' filename

This command replaces the string *this* with the string *that* in the specified file -- much the same as the scripts recently included in this column.

You can insert this command into a howto file for a quick example of how to do it, turn it into a script that prompts for the *this* and *that* strings, or just memorize it. It's not that hard, even if you're not a regular Perl user. Just remember that the arguments spell "pie" and that the substitute command looks like a substitute command in sed. Since the command replaces the original file with the changed one, it's very quick. Arguments:

    p            print
    i            in-place edit
    e            execute the following command

The Perl command:

 	perl -p -i -e 's/'

.. strips the carriage returns out of a file, turning a DOS file (which ends in both carriage returns and linefeeds) into a Unix file (which includes only linefeeds). It is basically the equivalent of:

	tr -d "\015"

Another very good use of one-liners is to test your understand about the language. The very terse nature of a one-liner might force you to look at some syntax you might not normally use.

[Nov 17, 2004] Edit HTML With a One-Line Perl Program

If you maintain a number of HTML documents on a Unix WWW server, you may sometimes want to make the same change to a number of files. Doing so by hand in a text editor can be tedious, but one time-saving option is to edit your files in place with a perl "one-liner". Best of all, you don't have to be a perl expert to do it.

Warning: Be sure to try this on a dummy copy of your files before you use it to edit the real thing! Since the editing happens in place, a mistake can be tricky to undo, even if you use the backup -i.bak option.

Sections of this page:

• Examples
• Anatomy of a perl one-liner
• More info

Examples

Change the hostname "xyz.rice.edu" to "abc.rice.edu":

perl -i.bak -p -e 's/xyz\.rice\.edu/abc.rice.edu/ig' *.html

Change localhost URLs to remote URLs:

perl -i.bak -p \
-e 's#file://localhost/localpath/#http://riceinfo.rice.edu/remotepath/#ig' \
*.html

Insert a department name at the beginning of every <TITLE>:

perl -i.bak -p \
-e 's#<title>#<title>Rice Fooology Dept.: #i' *.html

Insert a maintainer signature at the end of every file (before the closing <BODY> tag):

perl -i.bak -p \
-e 's#</body>#<p>\n<address>-- Jane Doe (jdoe\@rice.edu) 1999.12.31</address>\n</body>#i' \
*.html

Anatomy of a perl one-line substitution command

perl -i[.backup-extension] -p -e 's#pat1#pat2#ig' files

-i[.backup-extension]

Tells perl to run the command on the named files in-place, i.e., using the named files both as input and output. If a backup extension is provided, the unmodified version of each file will be saved with the extension appended.
Example: -i.bak

-p

Tells perl to assume an input loop around your one-line program and echo the output.

-e

The one-line program follows.

's#pat1#pat2#ig'

The perl "substitution" function. Matches every instance of the pattern pat1 and replaces it with pat2. The "#" used to delimit the patterns can be any character that isn't found in pat1 or pat2. The perl pattern matching used in pat1 is very powerful and somewhat complex; the main pitfall to remember is that you may need to escape special characters such as "." with a preceding backslash, e.g. "xyz\.rice\.edu". The trailing "i" flag means to ignore case when matching pat1. The trailing "g" flag means to apply the substitution multiple times on the same line (without the "g" it will only be applied to the leftmost pattern match on each line).

files

The file(s) on which the command should be run. In an HTML context, you probably want to specify a pattern in the shell to match your HTML files, taking into account any subdirectories you also want to include. Examples:

*.html                  (HTML files in current dir)
*.html blah/*.html      (HTML files in current dir and subdir "blah")
*.html */*.html         (HTML files in current dir and all subdirs one level deep)
{.,*,*/*,*/*/*}/*.html  (HTML files in current dir and all subdirs three levels deep)

For more information

• Pertinent sections of the perl man page (if unavailable on the Web, type man perl at the Unix prompt):
  • Perl regular expressions
  • The s/// substitution function
  • Perl switches
• Learning Perl (the llama book) by Randal Schwartz
• Programming Perl (the camel book) by Larry Wall and Randal Schwartz

[Nov 16, 2004] Perl One Liners

The following are a collection of perl one liners for command line use. First, quick recap of the important perl command line arguments is done before the example one liners. These examples are followed by a section that demonstrates how to convert a one liner into a full Perl script.

This page is likely not suitable for those with no Perl experience; consulting sites such as learn.perl.org may be necessary to learn about $_ and other constructs that will not be explained here.

For more background on these command line arguments, peruse perlrun.

• -e is used to specify Perl expressions to be run. More than one can be used, if needed. Other options should not follow this option.
   
• -p is to loop over and print input by default.
   
• -n is to loop over input without printing anything.
   
• -l handles newlines for you, and generally can be used by default— unless you need to do something special with the line endings themselves.

• The -ple or -nle sets are good groups to start with, depending on whether printing will be used by default or not.

• -i causes perl to operate on files in-place, and optionally also backs up the files via -i.bak or whatever.

• Using -ie '…' is a mistake, as the -i option reads the e as the backup filename suffix, followed shortly thereafter by perl failing as there is no longer any -e option to denote the following expression. Hence, I tend to use the in-place option before the loop control set.

• $ echo | perl -i -ple 42

• In general, practice constructing command lines such that -e is the last thing before an expression to avoid this sort of argument processing problem.

• -a enables autosplit of input into the @F array.

• Usually I use perl -lane … when processing input into columns, as it is easy to remember, and most often do not want to print by default. Alternatives such as cut or awk might be better to use when dealing with columns.

• -F allows one to alter the pattern input is split on with -a. Like -i it takes an argument, so should be used apart from other option sets.

• $ perl -F: -lane 'print $F[0] unless /^#/' /etc/passwd

• -0 specifies the input record separator.
   
• -M lets you load nifty modules such as File::Slurp or IO::All.

        There are more, so be sure to consult the depths of perlrun at some point.

[Nov 15, 2004] Perl one-liners for the bioinformatician

In the Perl spirit of "Programming is fun", here are some one-liners that might be actually useful. Please mail me yours, the best one-liner writer wins a Perl magnetic poetry kit. Contest closes July 31st, 2000. Please note that it is me personally running this competition, not NRCC, CBR or IMB. "Best" is subjective, and will be determined by an open vote.

Take a multiple sequence FASTA sequence file and print the non-redundant subset with the description lines for identical sequence concatenated with ;'s. Not a small one-liner, but close enough.

perl -ne 'BEGIN{$/=">";$"=";"}($d,$_)=/(.*?)\n(.+?)>?$/s;push @{$h{lc()}},$d if$_;END{for(keys%h){print">@{$h{$_}}$_"}}' filename

Split a multi-sequence FastA file into individual files named after their description lines.

perl -ne 'BEGIN{$/=">"}if(/^\s*(\S+)/){open(F,">$1")||warn"$1 write failed:$!\n";chomp;print F ">", $_}'

Take a blast output and print all of the gi's matched, one per line.

perl -pe 'next unless ($_) = /^>gi\|(\d+)/;$_.="\n"' filename

Filter all repeats of length 4 or greater from a FASTA input file. This one is thanks to Lincoln Stein and Gustavo Glusman's discussions on the bio-perl mailing list.

perl -pe 'BEGIN{$_=<>;print;undef$/}s/((.+?)\2{3,})/"N"x length$1/eg' filename

[Oct 8,  2004] Re perl oneliner for grep

By : anonymous ( Fri Oct 8 01:39:43 2004 )

Try using the -P option with grep. This enables perl regular expressions in grep e.g.
grep -P "\S+\s+\S+" file

[Sep 17, 2004] chmod all files in a directory

By : anonymous ( Fri Sep 17 20:22:02 2004 )

perl -e 'chmod 0000 $_ while <*>'

[Mar 16, 2004] Comma-delimit your output

By : anonymous ( Tue Mar 16 15:05:27 2004 )

perl -wne 'BEGIN{$" = ","} @fields = split/\s+/; print "@fields\n";'

Recommended Links

[PDF] Perl One-liners  View as HTML Perl One-liners Jeff Bay, jlb0170@yahoo.com

Abstract This article introduces some of the more common perl options found in command line programs, also known as ...

The Perl Journal One Liners

Adding a long list of numbers on the command line:

perl -e ’print eval join("+", @ARGV)’ 6 10 20 11
9 16 17 28 100 33333 14 -7

Preserving case in a substitution

To replace substring $x with an equal length substring $y, but preserving the case of $x:Z

$string =~ s/($x)/"\L$y"^"\L$1"^$1/ie;

How To Use The Perl Debugger as a Command-Line Interpreter

	perl -de 0

WLUG-Wiki - Perl One Liners

perl -pi -e 's/foo/bar/' file

Does an inplace SED on the file. GNU sed(1) v4 also supports this with -i and will probably be quicker if you only need a simple query and replacement. However, Perl's RegularExpressions are more powerful and easier on the hands than the POSIX variety offered by SED. With GNU sed(1), you can use the -r switch to get an extended RegularExpression syntax which also requires fewer backslashes than the POSIX flavour.

Removing empty lines from a file

perl -ni.bak -e'/\S/ && print' file1 file2

In Shell:

for FILE in file1 file2 ; do mv "$F"{,.bak} ; grep '[^ ]' "$F.bak" > "$F" ; done

Collapse consecutive blank lines to a single one

perl -00 -pi.bak -e1 file1 file2

Note the use of 1 as a no-op piece of Perl code. In this case, the -00 and -p switches already do all the work, so only a dummy needs to be supplied.

Binary dump of a string

perl -e 'printf "%08b\n", $_ for unpack "C*", shift' 'My String'

Replace literal "\n" and "\t" in a file with newlines and tabs

perl -pe 's!\\n!\n!g; s!\\t!\t!g' $file

Note that you can use any punctuation as the separator in an s/// command, and if you have backslashes or even need literal slashes in your pattern then doing this can increase clarity.

List all currently running processes

This is useful if you suspect that ps(1) is not reliable, whether due to a RootKit or some other cause. It prints the process ID and command line of every running process on the system (except some "special" kernel processes that lie about/don't have command lines).

perl -0777 -pe 'BEGIN { chdir "/proc"; @ARGV = sort { $a <=> $b } glob("*/cmdline") }
    $ARGV =~ m!^(\d+)/!; print "$1\t"; s/\0/ /g; $_ .= "\n";'

It runs an implicit loop over the /proc/*/cmdline files, by priming @ARGV with a list of files sorted numerically (which needs to be done explicitly using <=> -- the default sort is ASCIIbetical) and then employing the -p switch. -0777 forces files to be slurped wholesale. Per file, the digits that lead the filename are printed, followed by a tab. Since a null separates the arguments in these files, all of them are replaced by spaces to make the output printable. Finally, a newline is appended. The print call implicit in the -p switch then takes care of outputting the massaged command line.

Tom Christiansen one liners

 Matz Kindahl - Collection of Perl programs

These are one liners that might be of use. Some of them are from the net and some are one that I have had to use for some simple task. If Perl 5 is required, the perl5 is used.

perl -ne '$n += $_; print $n if eof'
perl5 -ne '$n += $_; END { print "$n\n" }'

To sum numbers on a stream, where each number appears on a line by itself. That kind of output is what you get from cut(1), if you cut out a numerical field from an output. There is also a C program called sigma that does this faster.
 

perl5 -pe 's/(\w)(.*)$/\U$1\L$2/'
perl5 -pe 's/\w.+/\u\L$&/'

To capitalize the first letter on the line and convert the other letters to small case. The last one is much nicer, and also faster.
 

perl -e 'dbmopen(%H,".vacation",0666);printf("%-50s: %s\n",$K,scalar(localtime(unpack("L",$V)))while($K,$V)=each(%H)'

Well, it is a one-liner. :)
You can use it to examine who wrote you a letter while you were on vacation. It examines the file that vacation(1) produces.
 

perl5 -p000e 'tr/ \t\n\r/ /;s/(.{50,72})\s/$1\n/g;$_.="\n"x2'

This piece will read paragraphs from the standard input and reformat them in such a manner that every line is between 50 and 72 characters wide. It will only break a line at a whitespace and not in the middle of a word.
 

perl5 -pe 's#\w+#ucfirst lc reverse $&#eg'

This piece will read lines from the standard input and transform them into the Zafir language used by Zafirs troops, i.e. "Long Live Zafir!" becomes "Gnol Evil Rifaz!" (for some reason they always talk using capital letters).
Andrew Johnson and I posted slightly different versions, and we both split the string unnecessarily. This one avoids splitting the string.

 

perl -pe '$_ = " $_ "; tr/ \t/ /s; $_ = substr($_,1,-1)'

This piece will remove spaces at the beginning and end of a line and squeeze all other sequences of spaces into one single space.
This was one of the "challenges" from comp.lang.perl.misc that occurs frequently; I am just unable to resist those. :)

Cultured Perl One-liners 101

find . -name "*.mp3" | perl -pe 's/.\/\w+-(\w+)-.*/$1/' | sort | uniq

perl -pi -e'$_ = sprintf "%04d %s", $., $_' test # inserting numbers in the file

find . -name "*.jpg" | perl -ne'chomp; $name = $_; $quote = chr(39); s/[$quote\\!]/_/ ; print "mv \"$name\" \"$_\"\n"'


Cultured Perl One-liners 102

Grep-style operations

#grep abba foo

perl -en 'print if /abba/' foo

Working with columns

Cultured Perl One-liners 102

# add first and penultimate columns
# NOTE the equivalent awk script:
# awk '{i = NF - 1; print $1 + $i}'
perl -lane 'print $F[0] + $F[-2]'

Sed-style operations

Practical uses include omitting lines matching a regular expression, printing a range of lines, inplace editing of multiple files, etc

Printing of the range of lines:

$ (echo a; echo b) | perl -nle 'print unless /b/'
a

$ (echo a; echo b) | perl -nle 'print unless $. == 1'
b

Any time the $. line number variable is being used with multiple files, the eof function may need to be used to reset the current line number counter. The following examples demonstrate this feature by reading the file input twice, and resetting the line number counter in the second case.

# print the range of lines 5 to 50
perl -ne 'print if $. >= 5; exit if $. >= 50;'

Add a line to a file

Appending data to existing files is easy. So is inserting data into arbitrary locations in a file, such as prepending a new first line to a set of files. In the following case, #!/usr/bin/perl will be added as the first line of all *.pl files in the current directory.

$ perl -i -ple 'print q{#!/usr/bin/perl} if $. == 1; close ARGV if eof' *.pl

If a recursive replace is needed, either investigate the use of the modules File::Find or IO::All, or simply have the unix shell pull in the required files as arguments to perl. While the second example is longer, it will work properly if filenames have spaces in their names, due to find -print0 and xargs -0 using the NUL character to delimit filenames instead of spaces.

$ perl -i -ple 'print q{#!/usr/bin/perl} if $. == 1; close ARGV if eof' \
`find . -type f -name "*.pl"`

$ find . -type f -name "*.pl" -print0 | \
xargs -0 perl -i -ple 'print q{#!/usr/bin/perl} if $. == 1; close ARGV if eof'

The following trick shows how to replace the second line of a file with some text, but only if that line is blank.

$ perl -ple 's/^$/some text/ if $. == 2; close ARGV if eof'

Home on the range

To skip ranges of text, use the .. operator. This operator is documented in perlop. The following examples illustrate different ways of collapsing runs of newlines. The first example eliminates all blank lines.

$ cat input
foo

bar
$ perl -ne 'print unless /^$/../^$/' input
foo
bar

The unless statement is equivalent to if not, but is different from if ! due to the associativity and precedence rules covered in perlop. A benefit of this behavior allows the reduction of runs of blank lines to a single blank line.

$ perl -ne 'print if ! /^$/../^$/' input
foo

bar

Line numbers can also be used with the range operator, for instance to remove the first four lines of a file.

$ perl -nle 'print unless 1 .. 4' input
bar

Altering record parsing

Perl uses the -0 option to allow changing the input record separator. The two main uses of this option are -00 to operate in paragraph mode, and -0777 to read all input into $_ at once. The paragraphs file contains the -0 documentation from perlrun, and is used in the following example to extract just the paragraph with the word special in it.

$ perl -00 -ne 'print if /special/' paragraphs

The special value 00 will cause Perl to slurp files in paragraph mode. The value 0777 will cause Perl to slurp files whole because there is no legal byte with that value.

Parsing the entire input file as a single line can be used to alter the newlines that otherwise require a range operator to deal with, as shown above. The following is a different way to remove runs of newlines from a file: by treating the entire file as a single line, a repeating s///g expression can be used to replace newlines as needed.

$ cat input
foo

bar
$ perl -0777 -pe 's/\n+/\n/g' input
foo
bar

Custom Quoting

Shell quoting may cause problems when writing expressions on the command line. Single quotes are usually used to delimit Perl expressions, to prevent shell interpolation of the code. To use a literal single quote inside such a single quoted string, the awkward '\'' syntax will need to be used, to end the single quoted string, include a literal quote, then restart the quoted string.

$ perl -le 'print "'\'' is a single quote"'
' is a single quote

Alternatives include using an octal escape code instead; see ascii(1) for a listing of codes.

$ perl -le 'print "\047 is a single quote"'
' is a single quote

Perl also allows different quoting operators, see the “Quote and Quote-like Operators” section under perlop for more information on these.

$ perl -le 'print q{single quoted: $$} . qq{ interpolated: $$}'
single quoted: $$ interpolated: 11506

Output to Multiple Files

To split output among multiple files, change where standard output points at based on some test. For example, the following will split a standard unix mailbox file inbox into multiple files named out.*, incrementing a number for each message in the mailbox.

$ perl -pe 'BEGIN { $n=1 } open STDOUT, ">out.$n" and $n++ if /^From /' inbox

Converting One Liners

One liners may be used as quick example code, or could be found in someone’s shell history. The following section demonstrates how to convert such one liners to full Perl scripts.

• Newline handling

• The -l command line option can easily be ported to a script by using it on the shebang line.

• #!/usr/bin/perl -w -l
  use strict;

• Loop over input (-pe or -ne)

• Printing loops can be replaced with a while block that prints by default. For a non-printing loop, remove the print statement.

• #!/usr/bin/perl -w -l
  use strict;
  
  while (<>) {
  # code from -e expressions here
  
  print;
  } continue {
  close ARGV if eof;
  }

• Special BEGIN or END blocks will need to be located outside of the while loop.

• In Place Editing

• To convert the -i in-place option, use the $^I variable, and ensure the files to be processed are in @ARGV before looping over <>.

• # trick to expand globs in input for systems with poor shells (Win32)
  local @ARGV = map glob, @ARGV;
  
  local $^I = '.orig';
  
  while (<>) {
  # code here
  
  print;
  } continue { close ARGV if eof }

In-place editing

  
# 1. in-place edit of *.c files changing all foo to bar
perl -p -i.bak -e 's/\bfoo\b/bar/g' *.c

# 2. delete first 10 lines
perl -i.old -ne 'print unless 1 .. 10' foo.txt

# 3. change all the isolated oldvar occurrences to newvar
perl -i.old -pe 's{\boldvar\b}{newvar}g' *.[chy]

# 4. increment all numbers found in these files
perl -i.tiny -pe 's/(\d+)/ 1 + $1 /ge' file1 file2 ....

# 5. delete all but lines between START and END
perl -i.old -ne 'print unless /^START$/ .. /^END$/' foo.txt

# 6. binary edit (careful!)
perl -i.bak -pe 's/Mozilla/Slopoke/g' /usr/local/bin/netscape

Last updated: November 08, 2008