Bug 176713 - [patch] nc(1) closes network socket too soon
Summary: [patch] nc(1) closes network socket too soon
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 9.1-RELEASE
Hardware: Any Any
: Normal Affects Only Me
Assignee: freebsd-bugs mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-03-07 02:50 UTC by Ronald F. Guilmette
Modified: 2017-12-31 22:27 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Ronald F. Guilmette 2013-03-07 02:50:00 UTC
As described in this thread:

  https://bugs.launchpad.net/ubuntu/+source/netcat-openbsd/+bug/544935

numerous people have, like me, encountered a serious problem with nc(1),
specifically that it prematurely closes the connection to the remote network
server being accessed as soon as it senses EOF on its stdin channel.

As can been see by perusing the above web page, this problem can cause,
and has caused much gnashing of teeth and tearing of hair in many quarters.

It is certainly possible that some existing scripts and/or applications
may perhaps depend on the current/existing behavior, and thus, it may
perhaps not be considered wise to change the existing behavior at this
(late?) point in time.  However there are quite certainly many situations
in which the different behavior (of gathering and outputting all output
from the remote server before closing down) can be unambiguously useful,
if not to say utterly irreplacable.  For these reasons I propse the addition
of a new "-q" option (no arguments) for nc(1) which, when used would change
the program's behavior, causing it to wait for EOF from the remote network
server before closing down and exiting.

I provide implementation patches for exactly such a change below.

How-To-Repeat: 
echo 193.0.6.139 | nc whois.ripe.net 43

((note how the output produced by the above command pipeline gets chopped
off prematurely))

Fix:

Simple/trivial context diff patches (implementing -q) attached below.


*** netcat.c	2012-12-03 10:55:02.000000000 -0800
--- netcat.c	2013-03-06 18:10:10.000000000 -0800
***************
*** 82,87 ****
--- 82,88 ----
  int	FreeBSD_Oflag;				/* Do not use TCP options */
  char   *Pflag;					/* Proxy username */
  char   *pflag;					/* Localport flag */
+ int	qflag;					/* quit on remote EOF flag */
  int	rflag;					/* Random ports flag */
  char   *sflag;					/* Source Address */
  int	tflag;					/* Telnet Emulation */
***************
*** 153,159 ****
  	sv = NULL;
  
  	while ((ch = getopt_long(argc, argv,
! 	    "46DdEe:hI:i:jklnoO:P:p:rSs:tT:UuV:vw:X:x:z",
  	    longopts, NULL)) != -1) {
  		switch (ch) {
  		case '4':
--- 154,160 ----
  	sv = NULL;
  
  	while ((ch = getopt_long(argc, argv,
! 	    "46DdEe:hI:i:jklnoO:P:p:qrSs:tT:UuV:vw:X:x:z",
  	    longopts, NULL)) != -1) {
  		switch (ch) {
  		case '4':
***************
*** 224,229 ****
--- 225,233 ----
  		case 'p':
  			pflag = optarg;
  			break;
+ 		case 'q':
+ 			qflag = 1;
+ 			break;
  		case 'r':
  			rflag = 1;
  			break;
***************
*** 823,829 ****
  			if ((n = read(wfd, buf, plen)) < 0)
  				return;
  			else if (n == 0) {
! 				shutdown(nfd, SHUT_WR);
  				pfd[1].fd = -1;
  				pfd[1].events = 0;
  			} else {
--- 827,834 ----
  			if ((n = read(wfd, buf, plen)) < 0)
  				return;
  			else if (n == 0) {
! 				if (!qflag)
! 					shutdown(nfd, SHUT_WR);
  				pfd[1].fd = -1;
  				pfd[1].events = 0;
  			} else {
Comment 1 Mark Linimon 2013-07-20 18:08:18 UTC
----- Forwarded message from Fernando Apesteguía <fernando.apesteguia@gmail.com> -----

Date: Sat, 20 Jul 2013 17:58:52 +0200
From: Fernando Apesteguía <fernando.apesteguia@gmail.com>
To: "Ronald F. Guilmette" <rfg@tristatelogic.com>
Cc: FreeBSD Hackers <freebsd-hackers@freebsd.org>
Subject: Re: bin/176713: [patch] nc(1) closes network socket too soon

On Sat, Jul 20, 2013 at 4:31 PM, Ronald F. Guilmette
<rfg@tristatelogic.com>wrote:

> Could someone please take a look at this bug report (bin/176713) and
> also at the simple patch that I provided to fix the problem?
>
> This is quite a serious problem, and my PR has been pending with no
> action since Wed, 6 Mar 2013.
>
>
> Regards,
> rfg
>
>
> P.S.  Please note that in reality, I do not believe that it is necessary
> to add a new -q option in order to preserve backwards compatability,
> however
> I proposed one because in my experience there is always going to be some
> purist who will claim that *any* change in semantics, however small or
> helpful, in *any* software ``may'' break something.  In reality, the
> problem in this can most easily be solved simply by removing the one
> statement:
>
>         shutdown(nfd, SHUT_WR);
>
> entirely from the source, rather than making it conditional on -q, as I
> suggested in my PR, bin/176713.
>

It seems to work for me:

Old version:

$ echo 193.0.6.139 | nc whois.ripe.net 43
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

with the patched version:

echo 193.0.6.139 | ./nc -q whois.ripe.net 43
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

% Note: this output has been filtered.
%       To receive output for a database update, use the "-B" flag.

% Information related to '193.0.0.0 - 193.0.7.255'

% Abuse contact for '193.0.0.0 - 193.0.7.255' is 'abuse@ripe.net'

inetnum:        193.0.0.0 - 193.0.7.255
netname:        RIPE-NCC
descr:          RIPE Network Coordination Centre
org:            ORG-RIEN1-RIPE
descr:          Amsterdam, Netherlands
remarks:        Used for RIPE NCC infrastructure.
country:        NL
admin-c:        JDR-RIPE
admin-c:        BRD-RIPE
tech-c:         OPS4-RIPE
status:         ASSIGNED PI
source:         RIPE # Filtered
mnt-by:         RIPE-NCC-MNT
mnt-lower:      RIPE-NCC-MNT

organisation:   ORG-RIEN1-RIPE
org-name:       Reseaux IP Europeens Network Coordination Centre (RIPE NCC)
org-type:       RIR
descr:          RIPE NCC Operations
address:        Reseaux IP Europeens Network Coordination Centre (RIPE NCC)
                P.O. Box 10096
                1016 EB Amsterdam
                Netherlands
phone:          +31205354444
fax-no:         +31205354445
admin-c:        AP110-RIPE
admin-c:        MENN1-RIPE
abuse-c:        ops4-ripe
mnt-ref:        RIPE-NCC-MNT
mnt-ref:        RIPE-NCC-HM-MNT
mnt-by:         RIPE-NCC-HM-MNT
source:         RIPE # Filtered

role:           RIPE NCC Operations
address:        Singel 258
address:        1016 AB Amsterdam
address:        The Netherlands
phone:          +31 20 535 4444
fax-no:         +31 20 535 4445
abuse-mailbox:  abuse@ripe.net
admin-c:        JDR-RIPE
admin-c:        BRD-RIPE
tech-c:         GL7321-RIPE
tech-c:         JA47
tech-c:         MENN1-RIPE
tech-c:         EMIL-RIPE
tech-c:         RCO-RIPE
tech-c:         APZ-RIPE
tech-c:         CNAG-RIPE
tech-c:         RPM-RIPE
tech-c:         SO2011-RIPE
tech-c:         RH5357-RIPE
tech-c:         DCW-RIPE
tech-c:         FMUL-RIPE
nic-hdl:        OPS4-RIPE
mnt-by:         RIPE-NCC-MNT
source:         RIPE # Filtered

person:         Brian Riddle
address:        IT Manager
address:        RIPE NCC - Operations
address:        Singel 258
address:        1016AB Amsterdam
address:        The Netherlands
phone:          +31 20 535 4444
fax-no:         +31 20 535 4445
nic-hdl:        BRD-RIPE
source:         RIPE # Filtered
mnt-by:         RIPE-NCC-MNT

person:         Jochem de Ruig
address:        RIPE NCC
address:        Singel 258
address:        1016AB Amsterdam
address:        The Netherlands
phone:          +31 20 535 4444
fax-no:         +31 20 535 4445
nic-hdl:        JDR-RIPE
source:         RIPE # Filtered
mnt-by:         RIPE-NCC-MNT

% Information related to '193.0.0.0/21AS3333'

route:          193.0.0.0/21
descr:          RIPE-NCC
origin:         AS3333
mnt-by:         RIPE-NCC-MNT
source:         RIPE # Filtered

% This query was served by the RIPE Database Query Service version 1.66.3
(WHOIS2)

Tested on:

FreeBSD hammer 9.1-RELEASE-p3 FreeBSD 9.1-RELEASE-p3 #0: Mon Apr 29
18:27:25 UTC 2013
root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC
amd64


>
> P.P.S.  I have just realized that yet one more critical enhancement for
> nc is called for, and I will be filing a separate and additional PR for
> that soon.  I hope that someone will be able to take a look at that as
> soon as it is filed.
>

Just out of curiosity. What is it about?

----- End forwarded message -----
Comment 2 Mark Linimon 2013-07-24 01:59:41 UTC
----- Forwarded message from "Ronald F. Guilmette" <rfg@tristatelogic.com> -----

Date: Tue, 23 Jul 2013 16:48:33 -0700
From: "Ronald F. Guilmette" <rfg@tristatelogic.com>
To: freebsd-hackers@freebsd.org
Subject: Re: bin/176713: [patch] nc(1) closes network socket too soon


In message <CAJ-Vmonk-8v9ej0w4qycNNBkieOeE9dL3BTVp6VqipxKH2jSUA@mail.gmail.com>
Adrian Chadd <adrian@freebsd.org> wrote:

>Right, and your patch just stops the shutdown(), right?

The shutdown that occurs when EOF is encountered on stdin, yes.

>Rather than
>teaching nc to correctly check BOTH socket states before deciding to
>close things.

In effect, nc *is* currently "checking both sockets" and that is exactly
the problem.  It terminates (prematurely in some cases) whenever it sees
an EOF _either_ from the remote host _or_ from stdin.

My patch casuses nc to wait for EOF from the remote server before exiting,
EVEN IF prior to the time it sees that EOF from the remote server it sees
an EOF (first) on stdin.

This code change demonstratably makes the functionality of nc better and
more pragmatically useful in typical use cases.

You appear to be proposing something else, but I'm sorry to say that
I cannot decypher what, exactly you are attempting to propose.

I have proposed specific code changes.  If you have some different ones
that you would like to propose, then I feel sure that everyone on the
hackers list, including myself, would be interested to take a look at
what you have in mind, and also what problem you are solving.

>I'd personally rather see nc taught to check to see whether it can
>possibly make ANY more progress before deciding to shut things down.

I believe that that is exactly what the patch that I proposed does.
I'm not sure why you feel otherwise.

Look, there are only two scenarios... either (a) EOF arrives from stdin
first or else (b) EOF arrives from the remote server first.

My patch causes nc to continue gathering data from the remote server
(and copying it all to stdout) in case (a).

In case (b) there is no point in nc continuing to run (and/or continuing
to read from stdin) if the remote server has shut down the connection.
In this case, the data that nc might yet gather from its stdin channel
has noplace to go!  So whenever nc has sensed an EOF from the remote
server it can (and should) immediately shut down... and that is exactly
what it is _already_ programmed to do.

So, what problem do you want to solve that is not solved by the patch
that I already proposed?

Also, with respect, if you think there really is some other problem,
then proposing actual concrete patches to solve that other problem
would perhaps allow folks, including myself, to better understand what
it is that you are driving at.


Regards,
rfg
_______________________________________________
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"


----- End forwarded message -----
Comment 3 Eitan Adler freebsd_committer freebsd_triage 2017-12-31 07:59:19 UTC
For bugs matching the following criteria:

Status: In Progress Changed: (is less than) 2014-06-01

Reset to default assignee and clear in-progress tags.

Mail being skipped