This blog post will teach you how to use standard input/output and pipes to redirect input and output of commands in Linux
- You should be able to :
- Redirect I/O channels to files
- Connect commands using pipes
- Use the for loops to iterate over sets of values
Standard Input and Output
- Linux provides three I/O channels to Programs
- Standard input (STDIN) – Keyboard by Default
- Standard output (STDOUT) – terminal windows by default
- Standard error (STDERR) – terminal windows by default
Input/Output Streams
One of the most important features of Linux (and UNIX) is the ability to redirect a command’s input, output and error data. In general, this allows the input from a program to come from any source, and the output to go to any source. In addition, the output from one command can be fed directly to the input of another command.
Standard input defaults to the keyboard. Most commands use files as their source of input, but will accept standard input from other sources, via redirection.
Standard output, by default, is the screen or terminal window. Most commands send their output to standard output without explicitly being told to do so. When standard output is sent to a destination other than the screen, such as a file, one is redirecting standard output.
But there is a second output data stream. It is called the standard error. This is a secondary output stream that carries warnings, usage messages, error messages and other “out-of-band” information, in an output stream distinct from standard output. This error stream is normally sent to the screen but may be redirected elsewhere. Consider the following command and its output, which assumes that there is a file called myfile in your home directory, but no file called fakefile.
amar@amar-pc:~$ ls -l myfile fakefile
ls: cannot access 'fakefile': No such file or directory
-rw-rw-r-- 1 amar amar 0 Nov 20 00:41 myfile
Though both lines are being printed to the screen, the first line is directed to the standard error stream and the second line to the standard output. Later we will see how the standard output and standard error can be redirected to other locations.
These streams are abbreviated as STDIN (called file descriptor number 0), STDOUT (file descriptor number 1), and STDERR (file descriptor number 2). The fact that there are two output channels allows separation of error messages from normal output. For example, error messages could be saved in a file with the normal output going to the monitor.
Redirecting Output to a File
- STDOUT and STDERR can be redirected to files:
- command operator filename
- Supported operators include:
- > Redirect STDOUT to file
- 2> Redirect STDERR to file
- &> Redirect all output to file
- File contents are overwritten by default. >> appends.
Altering I/O Sources and Destinations
The standard output of commands, which ordinarily displays on the terminal, can be redirected into a file or piped into another command.
Standard error, which also ordinarily displays on the terminal, can be redirected into a file. Although it is also possible to pipe standard error into a command using some fairly complex syntax, this is generally not done.
Standard input, ordinarily coming from the keyboard, can be redirected from a file. More commonly, the standard output of one command can be piped into the standard input of another command.
Common Redirection Operators
command > file – Direct standard output of command to file:
command >> file – Append standard output of command to file
command < file – Send file as input to command
command 2> file – Redirect error messages from command to file
command 2>> file – Append error messages from command to file
Examples
- This command generates output and errors when run as non-root:
$ find /etc -name passwd
- Operators can be used to store output and errors:
$ find /etc -name passwd > find.out
$ find /etc -name passwd 2> /dev/null
$ find /etc -name passwd > find.out 2> find.err
Redirecting STDOUT and STDERR
amar@amar-pc:~$ find /etc -name passwd
/etc/passwd
find: ‘/etc/polkit-1/localauthority’: Permission denied
find: ‘/etc/cups/ssl’: Permission denied
/etc/pam.d/passwd
find: ‘/etc/ssl/private’: Permission denied
The second line of output is printed to STDERR. It is the result of find attempting to access a directory that the non-privileged user running it does not have permission to read. The third line of output is printed to STDOUT and shows an accessible file the command has found. Note what happens when the same command is run, but STDOUT is redirected to a file:
amar@amar-pc:~$ find /etc -name passwd > find.out
find: ‘/etc/polkit-1/localauthority’: Permission denied
find: ‘/etc/cups/ssl’: Permission denied
find: ‘/etc/ssl/private’: Permission denied
amar@amar-pc:~$ cat find.out
/etc/passwd
/etc/pam.d/passwd
Because STDERR and STDOUT are distinct, redirecting one does not affect the other. The following command would redirect STDERR (also known as “file descriptor 2”) instead of STDOUT:
amar@amar-pc:~$ find /etc -name passwd 2> find.err
/etc/passwd
/etc/pam.d/passwd
The trick here is to remember that within the program STDERR is identified by the number 2, hence 2>, meaning “redirect file descriptor 2”. STDOUT is identified by the number 1, but since redirecting it is the default behavior when > is used, it rarely needs to be stated explicitly.
Redirecting the STDERR of a program means that you will only see its output, which is helpful in situations like the above where you know the command will generate some errors but do not care about them enough to want to see them, But if you really do not care about the errors, why waste a file storing them? There is a special file on your system that is very useful in this sort of situation, /dev/null. /dev/null is a black hole for data. Anything that is sent to is simply ignored. Hence, to store the output of this find command, but ignore its errors, you could run:
amar@amar-pc:~$ find /etc -name passwd > find.out 2> /dev/null
Overwriting vs Appending
Try running one of the above commands repeatedly. You will notice that every time output is redirected to a file, the contents of that file are destroyed or “overwritten”. Suppose you wanted to run a command multiple times over the course of a day or a week and store its output each time? This can be done by using the >> operator (2>> for STDERR) instead of >.
Redirecting STDOUT to a Program (Piping)
- Pipes (the | character) can connect commands: –
command1 | command2- Sends STDOUT of command1 to STDIN of command2 instead of the screen.
- STDERR is not forwarded across pipes
- Used to combine the functionality of multiple tools
- command1 | command2 | command3… etc.
Understanding Pipes
Two of the basic tenets of UNIX philosophy are: make small programs that do one thing well, and expect the output of every program to become the input to another, as yet unknown, program. The use of pipes let you leverage the effects of these two design principles to create extremely powerful chains of commands to achieve your desired result. Most command-line programs that operate on files can also accept input on standard in, and output to standard out. Some commands like tr are designed to only operate within a pipe (can not operate on files directly).
Any command that writes to standard output can be used on the left-hand side of a pipe.
Any command that reads from standard input can be used on the right-hand side of a pipe.
Multiple commands can be chained together with pipes.
Example output
amar@amar-pc:man$ pwd
/usr/share/man
amar@amar-pc:man$ ls -C | tr 'a-z' 'A-Z'
CS ES FR.ISO8859-1 ID KO MAN3 MAN6 MAN9 PT RU SV ZH_CN
DA FI FR.UTF-8 IT MAN1 MAN4 MAN7 NL PT_BR SL TR ZH_TW
DE FR HU JA MAN2 MAN5 MAN8 PL RO SR UK
In the example above, lower case letters of the alphabet from the file listing are converted to upper case.
Useful Pipe Targets
- less: View input one page at a time:
$ls -l /etc | less- input can be searched with /
- input can be searched with /
- mail: Send input via email:
$echo "test email" | mail -s "test" user@example.com - lpr : Send input to a printer:
$ echo "test print" | lpr
$ echo "test print" | lpr -P printer_name
Piping to Email
The traditional mail command is rather old fashioned for a mailer. But, it is extremely useful for mailing the standard output of a command. The -s option allows you to specify a subject for the email.
Piping to a Printer
You can also pipe output to the lpr command. this will send the output to your system’s default printer. If you want to use a non-default printer you can specify one with the -P option.