Bash shell is often seen as simple launcher for other programs, when in fact it is fairly sophisticated piece of software. It has its own language, its own start-up scripts (several of them, executed depending on context) and its own job management service. \u00a0 In this post I explore the bash redirects, pipes and their typical use cases. I am not going to repeat basic examples that can be found in every tutorial, instead I will go over a few tricks, shortcuts and gotchas I came across while working with bash shell. I am not a bash expert, but still I hope that some of the examples may be useful.<\/p>\n
dd if=\/dev\/random bs=1 count=16 > myfile # creates a file\r\n> myfile #removes the content of myfile<\/pre>\n<\/li>\n
(echo 1_error 1>&2; echo 1_output)<\/pre>\nThe line above produces two messages, “1_error” goes to the combined STDERR, while “1_output” goes to the combined STDOUT. We will use this construct below.<\/p>\n
For more details on the command grouping see here: reference<\/a>.<\/li>\n
- To redirect the standard error to standard output use “2>&1”. There is also a shortcut \u00a0 “&>” for “2>&1 \u00a0>”. \u00a0Finally, “&|” is a shortcut for “2>&1 |”\n
(echo 1_error 1>&2; echo 1_output) #outputs to both STDOUT and STDERR\r\n(echo 1_error 1>&2; echo 1_output) 2> \/dev\/null # supresses STDERR\r\n(echo 1_error 1>&2; echo 1_output) 1> \/dev\/null # supresses STDOUT\r\n(echo 1_error 1>&2; echo 1_output) | tr 1 2 # only STDOUT is tr'ed\r\n(echo 1_error 1>&2; echo 1_output) 2> >(tr 1 2) # only STDERR is tr'ed (we are using 'process substitution')\r\n(echo 1_error 1>&2; echo 1_output) 2>&1 | tr 1 2 # both are tr'ed\r\n(echo 1_error 1>&2; echo 1_output) |& tr 1 2 # both are tr'ed\r\n(echo 1_error 1>&2; echo 1_output) &> >(tr 1 2) # both are tr'ed<\/pre>\n<\/li>\n- The order of redirects is important:\n
(echo 1_error 1>&2; echo 1_output) 1> \/dev\/null 2>&1 #outputs nothing\r\n(echo 1_error 1>&2; echo 1_output) 2>&1 1> \/dev\/null #outputs "1_error" to STDOUT<\/pre>\nIt is useful to think of a sequence of redirects as if it’s a program that operates on stream names and their locations. In the first example, \u00a0“1> \/dev\/null” sets STDOUT to location “\/dev\/null”, then “2>&1” sets STDERR to point to the same location as STDOUT, i.e. “dev\/null”. In the second example, STDERR is set to the location of STDOUT, and then STDOUT name is set to point to “\/dev\/null” location.
\nLet’s illustrate this by the following example:<\/p>\n(echo 1_error 1>&2; echo 1_output) 3>&1 1>&2 2>&3- | tr 1 2<\/pre>\nIn this example we swap STDOUT and STDERR. First, we introduce a new file handler “3” and make it point to the current location of STDOUT, then we make STDOUT point to the current location of STDERR, and finally STDERR is made to the location of ‘3’, which is the original location of STDOUT. “-” means “close the file handle after we are done”.
\nSome of the examples were borrowed here: \u00a0\u00a0http:\/\/stackoverflow.com\/questions\/1507816\/with-bash-how-can-i-pipe-standard-error-into-another-process<\/a><\/li>\n- An interesting application. We can highlight error messages in the command output:\n
color()(set -o pipefail; bash -c "$@" 2>&1>&3|sed $'s,.*,\\e[31m&\\e[m,'>&2)3>&1<\/pre>\nNow execute<\/p>\n
color "(echo 1_error 1>&2; echo 1_output)"<\/pre>\nto see it in action. For more details take a look here<\/a>.<\/li>\n
- Sudo and tee. Redirects will fail if there is not enough permissions to write a file. As a workaround we can use “sudo tee” trick:\n
touch protected.file\r\nchmod a-xrw protected.file\r\necho test > protected.file # fails\r\nsudo echo test > protected.file #fails as well\r\necho test | sudo tee protected.file #this works\r\necho test | sudo tee protected.file >\/dev\/null #works and suppresses STDOUT as we don't need it<\/pre>\n<\/li>\n- Programs can detect that their output is being redirected. For example, run “ls \/bin” and “ls \/bin > list.txt”. In the former case the output has several columns, while in the later the output is one column. To test create outputtest.sh:\n
\r\nif [[ -t 1 ]] ; then\r\n echo stdout is a terminal\r\nelse\r\n echo stdout is not a terminal\r\nfi\r\n<\/pre>\n(taken from http:\/\/stackoverflow.com\/questions\/9340129\/test-stdout-and-stderr-redirection-in-bash-script)
\nThe script can be confused by<\/p>\noutputtest.sh > \/dev\/pts\/3 #still console but not the current one<\/pre>\n<\/li>\n- Run-time redirection. In Linux it is possible to redirect the output of already running process. This is a bit hacky but doable. Get screenify script from here
\nscript<\/a>
\nRun the following script that just prints its PID<\/p>\nruby -e 'while (1) do print("my pid is " + $$.to_s() + " \\n"); sleep(10) end'&<\/pre>\nExecute<\/p>\n
.\/screenify <pid> > newoutputdestination.txt<\/pre>\n<\/li>\n- Pipes can have names. I haven’t been in a situation when this functionality is needed, but it still gives an idea how things work under the hood. In one terminal window run:\n
mkfifo \/tmp\/mypipe\r\ncat \/tmp\/mypipe<\/pre>\nIn another terminal execute:<\/p>\n
ls \/bin > \/tmp\/mypipe<\/pre>\nThe output of “ls” will appear in the first terminal window.<\/li>\n
- Statuses. There are ways to obtain the status of each command in a pipe, and there is an option that ensures that a failing command status is not lost and is returned as the status of the entire pipe:\n
\r\n# We can get the status for all commands in a pipe\r\nfalse | true\r\necho "Statuses: ${PIPESTATUS[0]} ${PIPESTATUS[1]}"\r\n\r\n# "pipefail" option forces the pipe to return the the first non-zero status (and zero otherwise)\r\nset -o pipefail\r\nfalse | true\r\necho "With pipefail on: $?"\r\n\r\n# turn "pipefail" off\r\nset +o pipefail\r\nfalse | true\r\necho "With pipefail off: $?"\r\n\r\n<\/pre>\n<\/li>\n- The last command in a pipe can run in the same process as the script itself. This can be convenient if we want to use the same set of variables:\n
\r\n#!\/bin\/bash\r\n# With this option the last command in the pipe runs in the same process\r\n# so the updates to the bash variables are visible. \r\nshopt -s lastpipe\r\nn=0\r\nyes | n=1\r\necho "n is $n"\r\n# If the option is unset then the updates are lost\r\nshopt -u lastpipe\r\nn=0\r\nyes | n=1\r\necho "n is $n"\r\n<\/pre>\n<\/li>\n- Misc.\n
\n
- If want to know what “>|” does, then check “noclobber” option. See here<\/a>.<\/li>\n
- It is possible to statically redirect outputs for all subsequent shell commands:\n
exec 2>&1<\/pre>\n<\/li>\n- “>&” is the same as “&>”, but the latter is preferable.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"
Bash shell is often seen as simple launcher for other programs, when in fact it is fairly sophisticated piece of software. It has its own language, its own start-up scripts (several of them, executed depending on context) and its own job management service. \u00a0 In this post I explore the bash redirects, pipes and their\u2026 Read More »<\/a><\/span><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[10,5],"class_list":["post-8","post","type-post","status-publish","format-standard","hentry","category-linux","tag-bash","tag-i-o-redirect"],"_links":{"self":[{"href":"https:\/\/www.type.sh\/index.php\/wp-json\/wp\/v2\/posts\/8","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.type.sh\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.type.sh\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.type.sh\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.type.sh\/index.php\/wp-json\/wp\/v2\/comments?post=8"}],"version-history":[{"count":2,"href":"https:\/\/www.type.sh\/index.php\/wp-json\/wp\/v2\/posts\/8\/revisions"}],"predecessor-version":[{"id":10,"href":"https:\/\/www.type.sh\/index.php\/wp-json\/wp\/v2\/posts\/8\/revisions\/10"}],"wp:attachment":[{"href":"https:\/\/www.type.sh\/index.php\/wp-json\/wp\/v2\/media?parent=8"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.type.sh\/index.php\/wp-json\/wp\/v2\/categories?post=8"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.type.sh\/index.php\/wp-json\/wp\/v2\/tags?post=8"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}