The bash, zsh and ksh have a nice feature to detect a non-zero status in a pipe: from the bash(1) manual page: The return status of a pipeline is the exit status of the last command, unless the pipefail option is enabled. If pipefail is enabled, the pipeline's return status is the value of the last (rightmost) command to exit with a non-zero status, or zero if all commands exit successfully. $ bash -c 'set -o pipefail; false | true; echo $?' 1 $ ksh93 -c 'set -o pipefail; false | true; echo $?' 1 $ zsh -c 'set -o pipefail; false | true; echo $?' 1 However, this feature is missing in /bin/sh: $ sh -c 'set -o pipefail; false | true; echo $?' set: Illegal option -o pipefail $ sh -c 'false | true; echo $?' 0 There are workaround for shell which don’t check the pipe status, as described in https://unix.stackexchange.com/questions/14270/get-exit-status-of-process-thats-piped-to-another (((((exec 3>&- 4>&-; someprog); echo $? >&3) | filter >&4) 3>&1) | (read xs; exit $xs)) 4>&1 but this looks bizarre and you have to do it for every command in the pipe line. I really wish that we have the 'set -o pipefail' option setting in FreeBSD /bin/sh too.
I don't think this is the first request for this sort of feature in sh(1) :-). I certainly would like it, too, and I think Bryan Drewery has expressed interest as well. I don't see another pipefail enhancement request bug searching bugzilla, though I thought there was one.
The implementation of set -o pipefail would be fairly easy, but I don't like that a SIGPIPE exit of a non-final process will also be treated as an error.
(In reply to Jilles Tjoelker from comment #2) Plenty of people like that behavior, though. It seems like if we implement pipefail we should follow existing expectations. If you want a looser option, maybe you could implement that as well under a different name?
(In reply to Jilles Tjoelker from comment #2) You mean a pipe like this: ( set -o pipefail; man tcsh | head | wc -l ); echo $? 10 141 You can use a sub shell if you need the pipefail option for a special action only.
The pipefail option was proposed for POSIX at http://austingroupbugs.net/view.php?id=789 but this discussion seems to be stalled. Apart from the SIGPIPE issue, it was pointed out that shell options interact poorly with functions since they are dynamically scoped (for example, set -o pipefail; cmd1 | cmd2 | cmd3; r=$?; set +o pipefail not only affects this pipe's exit status, but also everything done by cmd1, cmd2 and cmd3 if they are functions). I will try to get this moving again. However, the implementation of set -o pipefail is simple and doing without it or a similar feature in scripts is complicated.
Hi Jilles, thanks for the pointer to the POSIX discussion. We should be compatible with the /bin/bash. BASH is the industry standard for shell scripts, unfortunately. For me the only reason to use the bash is the pipefail option. I wish I could replace the first line #!/bin/bash with #!/bin/sh and everything works as expected. Regarding the implementation: the POSIX issue mention in the Description section that “busybox sh” has the feature implemented. busybox is using the ash, re-ported from NetBSD and debianized. I don’t think we can use the code from ash due the GPL licence, but we could use some of the busybox regression tests (./ash_test/ash-misc/pipefail.*) to make sure that our implementation runs fine.
There is a reusable piece of code in a StackOverflow answer https://unix.stackexchange.com/questions/76162/how-do-i-capture-the-return-status-and-use-tee-at-the-same-time-in-korn-shell/76171#76171 This supports simple commands not containing words "|". By defining a function containing the command, any command can be used. It is not the most beautiful option but it will work and isolate the file descriptor manipulation. It will not help if you want to set -eo pipefail to make shell "do what I mean" but I think shell will not do proper error handling automatically with any option setting anyway. I already committed a small refactoring in support of -o pipefail to sh which seems good anyway but I'm not entirely sure yet whether to add this feature.
Created attachment 198515 [details] possible patch I would be very interested in having this feature. I just hacked up a quick patch that seems to work. However, I have never looked at the bin/sh code before so I probably missed something. I could upload this to phabricator if it makes sense. $ ./bin/sh -c "set -o pipefail; yes | head -n1 | wc -l"; echo $? 1 141 $ bash -c "set -o pipefail; yes | head -n1 | wc -l"; echo $? 1 141 $ ./bin/sh -c "set -o pipefail; false | true"; echo $? 1 $ bash -c "set -o pipefail; false | true"; echo $? 1
Created attachment 198594 [details] my set -o pipefail patch A while ago, I created a patch for set -o pipefail but did not publish it anywhere. The Austin group process for set -o pipefail recently started moving again, and it seems now likely it will be accepted.
That patch looks great, is there any chance we could get it committed? I am currently using various hacks to use bash with -o pipefail as my build shell but having it in bin/sh would make it much easier. I am using -o pipefail to test my changes to build FreeBSD on Linux/macOS to get sensible errors instead of empty .c/.h files being generated from something like this: foo.c: some_command_that_doesnt_exist | sed .... > ${.TARGET}
I recently ran into another build system issue where it would have been nice to have pipefail. Would it be possible to commit this patch since it seems compatible with other shells and is extremely useful?
A commit references this bug: Author: jilles Date: Sun Feb 24 21:05:16 UTC 2019 New revision: 344502 URL: https://svnweb.freebsd.org/changeset/base/344502 Log: sh: Add set -o pipefail The pipefail option allows checking the exit status of all commands in a pipeline more easily, at a limited cost of complexity in sh itself. It works similarly to the option in bash, ksh93 and mksh. Like ksh93 and unlike bash and mksh, the state of the option is saved when a pipeline is started. Therefore, even in the case of commands like A | B & a later change of the option does not change the exit status, the same way (A | B) & works. Since SIGPIPE is not handled specially, more work in the script is required for a proper exit status for pipelines containing commands such as head that may terminate successfully without reading all input. This can be something like ( cmd1 r=$? if [ "$r" -gt 128 ] && [ "$(kill -l "$r")" = PIPE ]; then exit 0 else exit "$r" fi ) | head PR: 224270 Relnotes: yes Changes: head/bin/sh/jobs.c head/bin/sh/options.h head/bin/sh/sh.1 head/bin/sh/tests/execution/Makefile head/bin/sh/tests/execution/pipefail1.0 head/bin/sh/tests/execution/pipefail2.42 head/bin/sh/tests/execution/pipefail3.42 head/bin/sh/tests/execution/pipefail4.42 head/bin/sh/tests/execution/pipefail5.42 head/bin/sh/tests/execution/pipefail6.42 head/bin/sh/tests/execution/pipefail7.0
jilles@ could we have this MFC to 12-stable as well?
(In reply to Dave Cottlehuber from comment #13) This purely additive feature seems suitable for MFC, yes.
A commit references this bug: Author: jilles Date: Tue Mar 26 20:47:31 UTC 2019 New revision: 345556 URL: https://svnweb.freebsd.org/changeset/base/345556 Log: MFC r327475: sh: Move various structs from jobs.h to jobs.c These implementation details of jobs.c need not be exposed. PR: 224270 Changes: _U stable/11/ stable/11/bin/sh/jobs.c stable/11/bin/sh/jobs.h
A commit references this bug: Author: jilles Date: Tue Mar 26 21:30:26 UTC 2019 New revision: 345559 URL: https://svnweb.freebsd.org/changeset/base/345559 Log: MFC r328818: sh: Refactor job status printing, preparing for -o pipefail and similar No functional change is intended. PR: 224270 Changes: _U stable/11/ stable/11/bin/sh/jobs.c
A commit references this bug: Author: jilles Date: Tue Mar 26 22:34:10 UTC 2019 New revision: 345561 URL: https://svnweb.freebsd.org/changeset/base/345561 Log: MFC r344502: sh: Add set -o pipefail The pipefail option allows checking the exit status of all commands in a pipeline more easily, at a limited cost of complexity in sh itself. It works similarly to the option in bash, ksh93 and mksh. Like ksh93 and unlike bash and mksh, the state of the option is saved when a pipeline is started. Therefore, even in the case of commands like A | B & a later change of the option does not change the exit status, the same way (A | B) & works. Since SIGPIPE is not handled specially, more work in the script is required for a proper exit status for pipelines containing commands such as head that may terminate successfully without reading all input. This can be something like ( cmd1 r=$? if [ "$r" -gt 128 ] && [ "$(kill -l "$r")" = PIPE ]; then exit 0 else exit "$r" fi ) | head PR: 224270 Relnotes: yes Changes: _U stable/11/ stable/11/bin/sh/jobs.c stable/11/bin/sh/options.h stable/11/bin/sh/sh.1 stable/11/bin/sh/tests/execution/Makefile stable/11/bin/sh/tests/execution/pipefail1.0 stable/11/bin/sh/tests/execution/pipefail2.42 stable/11/bin/sh/tests/execution/pipefail3.42 stable/11/bin/sh/tests/execution/pipefail4.42 stable/11/bin/sh/tests/execution/pipefail5.42 stable/11/bin/sh/tests/execution/pipefail6.42 stable/11/bin/sh/tests/execution/pipefail7.0