Bash shell is often seen as simple launcher for other programs, when in fact it is fairly sophisticated piece of software. It has its own language, its own start-up scripts (several of them, executed depending on context) and its own job management service. In this post I explore the bash redirects, pipes and their typical use cases. I am not going to repeat basic examples that can be found in every tutorial, instead I will go over a few tricks, shortcuts and gotchas I came across while working with bash shell. I am not a bash expert, but still I hope that some of the examples may be useful.
- Need to delete a content of a file? Easy, use “>” :
dd if=/dev/random bs=1 count=16 > myfile # creates a file > myfile #removes the content of myfile
- Bash allows commands to be grouped. In this case the output streams of all commands in the list are combined:
(echo 1_error 1>&2; echo 1_output)
The line above produces two messages, “1_error” goes to the combined STDERR, while “1_output” goes to the combined STDOUT. We will use this construct below.
For more details on the command grouping see here: reference.
- To redirect the standard error to standard output use “2>&1”. There is also a shortcut “&>” for “2>&1 >”. Finally, “&|” is a shortcut for “2>&1 |”
(echo 1_error 1>&2; echo 1_output) #outputs to both STDOUT and STDERR (echo 1_error 1>&2; echo 1_output) 2> /dev/null # supresses STDERR (echo 1_error 1>&2; echo 1_output) 1> /dev/null # supresses STDOUT (echo 1_error 1>&2; echo 1_output) | tr 1 2 # only STDOUT is tr'ed (echo 1_error 1>&2; echo 1_output) 2> >(tr 1 2) # only STDERR is tr'ed (we are using 'process substitution') (echo 1_error 1>&2; echo 1_output) 2>&1 | tr 1 2 # both are tr'ed (echo 1_error 1>&2; echo 1_output) |& tr 1 2 # both are tr'ed (echo 1_error 1>&2; echo 1_output) &> >(tr 1 2) # both are tr'ed
- The order of redirects is important:
(echo 1_error 1>&2; echo 1_output) 1> /dev/null 2>&1 #outputs nothing (echo 1_error 1>&2; echo 1_output) 2>&1 1> /dev/null #outputs "1_error" to STDOUT
It is useful to think of a sequence of redirects as if it’s a program that operates on stream names and their locations. In the first example, “1> /dev/null” sets STDOUT to location “/dev/null”, then “2>&1” sets STDERR to point to the same location as STDOUT, i.e. “dev/null”. In the second example, STDERR is set to the location of STDOUT, and then STDOUT name is set to point to “/dev/null” location.
Let’s illustrate this by the following example:(echo 1_error 1>&2; echo 1_output) 3>&1 1>&2 2>&3- | tr 1 2
In this example we swap STDOUT and STDERR. First, we introduce a new file handler “3” and make it point to the current location of STDOUT, then we make STDOUT point to the current location of STDERR, and finally STDERR is made to the location of ‘3’, which is the original location of STDOUT. “-” means “close the file handle after we are done”.
Some of the examples were borrowed here: http://stackoverflow.com/questions/1507816/with-bash-how-can-i-pipe-standard-error-into-another-process - An interesting application. We can highlight error messages in the command output:
color()(set -o pipefail; bash -c "$@" 2>&1>&3|sed $'s,.*,\e[31m&\e[m,'>&2)3>&1
Now execute
color "(echo 1_error 1>&2; echo 1_output)"
to see it in action. For more details take a look here.
- Sudo and tee. Redirects will fail if there is not enough permissions to write a file. As a workaround we can use “sudo tee” trick:
touch protected.file chmod a-xrw protected.file echo test > protected.file # fails sudo echo test > protected.file #fails as well echo test | sudo tee protected.file #this works echo test | sudo tee protected.file >/dev/null #works and suppresses STDOUT as we don't need it
- Programs can detect that their output is being redirected. For example, run “ls /bin” and “ls /bin > list.txt”. In the former case the output has several columns, while in the later the output is one column. To test create outputtest.sh:
if [[ -t 1 ]] ; then echo stdout is a terminal else echo stdout is not a terminal fi
(taken from http://stackoverflow.com/questions/9340129/test-stdout-and-stderr-redirection-in-bash-script)
The script can be confused byoutputtest.sh > /dev/pts/3 #still console but not the current one
- Run-time redirection. In Linux it is possible to redirect the output of already running process. This is a bit hacky but doable. Get screenify script from here
script
Run the following script that just prints its PIDruby -e 'while (1) do print("my pid is " + $$.to_s() + " \n"); sleep(10) end'&Execute
./screenify <pid> > newoutputdestination.txt
- Pipes can have names. I haven’t been in a situation when this functionality is needed, but it still gives an idea how things work under the hood. In one terminal window run:
mkfifo /tmp/mypipe cat /tmp/mypipe
In another terminal execute:
ls /bin > /tmp/mypipe
The output of “ls” will appear in the first terminal window.
- Statuses. There are ways to obtain the status of each command in a pipe, and there is an option that ensures that a failing command status is not lost and is returned as the status of the entire pipe:
# We can get the status for all commands in a pipe false | true echo "Statuses: ${PIPESTATUS[0]} ${PIPESTATUS[1]}" # "pipefail" option forces the pipe to return the the first non-zero status (and zero otherwise) set -o pipefail false | true echo "With pipefail on: $?" # turn "pipefail" off set +o pipefail false | true echo "With pipefail off: $?" - The last command in a pipe can run in the same process as the script itself. This can be convenient if we want to use the same set of variables:
#!/bin/bash # With this option the last command in the pipe runs in the same process # so the updates to the bash variables are visible. shopt -s lastpipe n=0 yes | n=1 echo "n is $n" # If the option is unset then the updates are lost shopt -u lastpipe n=0 yes | n=1 echo "n is $n"
- Misc.
- If want to know what “>|” does, then check “noclobber” option. See here.
- It is possible to statically redirect outputs for all subsequent shell commands:
exec 2>&1
- “>&” is the same as “&>”, but the latter is preferable.