Bug 261200 - Unable to pipe the output of jobs in sh
Summary: Unable to pipe the output of jobs in sh
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 13.0-STABLE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-bugs (Nobody)
Depends on:
Reported: 2022-01-14 14:33 UTC by Ron Wills
Modified: 2022-01-24 14:26 UTC (History)
3 users (show)

See Also:

Basic example of the jobs builtin being piped to other commands (438 bytes, application/x-shellscript)
2022-01-14 14:33 UTC, Ron Wills
no flags Details
Jobs piping testcase (662 bytes, application/x-shellscript)
2022-01-15 16:23 UTC, Ron Wills
no flags Details
Update man page about piping with job control commands (1.08 KB, patch)
2022-01-19 01:09 UTC, Ron Wills
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Ron Wills 2022-01-14 14:33:16 UTC
Created attachment 231005 [details]
Basic example of the jobs builtin being piped to other commands

The output of the builtin command jobs in /bin/sh cannot be piped to other commands. With other shells like bash the output can be piped to wc or cat and works as expected. With sh it appears all the output is lost.

Simply doing 'jobs | cat' in sh never outputs anything.
Comment 1 Graham Perrin 2022-01-15 08:23:52 UTC
(In reply to Ron Wills from comment #0)

If not a bug, might this be a question for Stack Exchange? 

Here, without using sh: 

% sleep 10 &
[1] 12372
% sleep 10 &
[2] 12374
% sleep 10 &
[3] 12376
% jobs
[1]  + Running                       sleep 10
[2]  - Running                       sleep 10
[3]    Running                       sleep 10
% jobs | wc -l
% jobs | cat
[1]    Done                          sleep 10
% jobs
[2]  + Done                          sleep 10
[3]  + Done                          sleep 10
% jobs
% jobs | cat
% echo $SHELL
% uname -KU
1400047 1400047
Comment 2 Ron Wills 2022-01-15 16:22:07 UTC
(In reply to Graham Perrin from comment #1)

It's hard to tell if this is a feature or a bug. It's completely undocumented from what I can tell.

As I do more testing, I can pipe the output of other builtin commands like echo or pwd but not the command jobs. The shell /bin/sh isn't the only shell that this doesn't work for.

$ bash jobs-test1.sh
*** Jobs ***
[1]   Running                       sleep 10 &
[2]-  Running                       sleep 10 &
[3]+  Running                       sleep 10 &
*** Job Count ***
*** Jobs Piped ***
[1]   Running                       sleep 10 &
[2]-  Running                       sleep 10 &
[3]+  Running                       sleep 10 &

Works as expected.

$ sh jobs-test1.sh
$ tcsh jobs-test1.sh
$ zsh jobs-test1.sh
*** Jobs ***
[1]   Running
[2] - Running
[3] + Running
*** Job Count ***
*** Jobs Piped ***

None of these pipe the output of builtin command jobs.

It just seems to be an odd feature to have this one command, to my knowledge, not be able to pipe its output. I'm updating the testcase script to make the output clearer.
Comment 3 Ron Wills 2022-01-15 16:23:26 UTC
Created attachment 231025 [details]
Jobs piping testcase

Makes the test output a bit clearer.
Comment 4 Fernando Apesteguía freebsd_committer 2022-01-15 18:02:16 UTC
To me, the behavior is erratic to say the least:

$sleep 5000&
$sleep 3000&
[1] - Running                 sleep 5000
[2] + Running                 sleep 3000
$jobs | wc -l
$jobs %1
[1] - Running                 sleep 5000
$jobs %1 | wc -l
jobs: No such job: %1

^Triage: Adding jilles@ to CC since he has fixed some output issues for builtins before.
Comment 5 Ron Wills 2022-01-18 18:15:15 UTC
While tracing a simple script like:

sleep 1000&
jobs | cat

I find that the first execution of jobs is executed by the shell process itself and the second piped jobs is forked then executed in a child process. In both cases the jobtab, found in jobs.c, has four entries in it but in the child process all the jobs are marked unused.

In the function forkshell(), found in jobs.c, there's a section of code for the new child process that clears the jobtab.

for (i = njobs, p = jobtab ; -- i >= 0 ; p++)
    if (p->used)

And this is the reason why jobs shows no entries when piped.

I'm not sure why this is done. I'm assuming this is possibly to prevent the child process from doing some kind of job clean up later... I'll keep digging into this as time permits ;)
Comment 6 Jilles Tjoelker freebsd_committer 2022-01-18 23:33:34 UTC
The reason that jobs cannot be piped is that an element of a pipeline (with more than one element) is run in a subshell environment, and the subshell environment has its own jobs. For example,

sh -c ':& { :& jobs; }|wc -l'

writes 1.

There are, however, some exceptions to this.

One such exception is that if a command substitution contains a single jobs command, this jobs command returns information about the parent shell environment. This exception is documented in the man page under "Command Substitution". The command is otherwise still executed in a subshell environment, so, for example, variable assignments in expansions do not persist. Technically, this is implemented by executing certain command substitutions in the same process; among other things, resetting the jobs table is skipped and anything that would alter it does not follow this code path.

This exception is commonly available (like the one for `trap` which has an Austin Group interpretation: https://www.austingroupbugs.net/view.php?id=53 ), but is not always implemented the way FreeBSD sh implements it. Some other shells instead implement it by making `jobs` (or `trap`) return the information from just before the subshell environment was entered if no change had been made yet, but in the case of `jobs` this is a bit unfortunate: either it creates an observable difference between (non-special) builtins and external programs, or it requires trickery to ensure a foreground job can be run without disturbing the jobs table for display by `jobs`.

In bash (5.1.16(0)-release), `:& jobs | wc -l` and `:& J=jobs; $J | wc -l` work, but `:& { jobs; } | wc -l` does not, so bash appears to use a similar analysis like FreeBSD sh uses for command substitutions.
Comment 7 Ron Wills 2022-01-19 00:25:16 UTC
Okay now this is making sense. Simple examples:

$ jobs

Outputs from the current shell.

$ jobs | cat

Jobs is now in a subshell, created by the pipe, with a new jobs table so no output.

$ echo $(jobs) | cat
$ echo `jobs` | cat

Allows you to pipe the jobs output in a work around the whole subshell thing.
Comment 8 Ron Wills 2022-01-19 01:09:11 UTC
Created attachment 231147 [details]
Update man page about piping with job control commands

Here's a proposed update to the man page to add a reference to the Command Substitution subsection for the builtin commands jobid, jobs and trap. 

A line is appended to each command description as follows:

Piping the output of "command" will not work as expected. See the Command Substitution subsection for details.

Maybe this might make this little quirk clearer for the future.
Comment 9 Fernando Apesteguía freebsd_committer 2022-01-24 14:26:37 UTC
Thanks for the explanation Jills!

Would you consider adding the modification to the man page? I am not an src committer or a member of manpages.