Bug 225344 - fetch(1) awaits EOF (or ftp credentials) even when they are provided in env vars
Summary: fetch(1) awaits EOF (or ftp credentials) even when they are provided in env vars
Status: Closed Not Accepted
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Only Me
Assignee: Dag-Erling Smørgrav
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-01-20 19:38 UTC by Matthias Apitz
Modified: 2018-05-14 11:03 UTC (History)
3 users (show)

See Also:


Attachments
Improve netrcfd handling and add debugging aids for FTP authentication (2.43 KB, patch)
2018-05-13 13:37 UTC, Dag-Erling Smørgrav
no flags Details | Diff
Improve netrcfd handling and add debugging aids for FTP authentication (3.09 KB, patch)
2018-05-13 13:46 UTC, Dag-Erling Smørgrav
no flags Details | Diff
output from: kdump -tcnstu (40.34 KB, text/plain)
2018-05-14 07:45 UTC, Matthias Apitz
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Matthias Apitz 2018-01-20 19:38:05 UTC
fetch(1) hangs (for example on ports 'make install') awaiting the input of ftp credentials via STDIN:

The man page of fetch(3), i.e. the used library for this, explains the
usage of some env vars:

FTP_LOGIN=anonymous export FTP_LOGIN
FTP_PASSWORD=guru@sisis.de export FTP_PASSWORD

and if you set this and run:

$ fetch ftp://ftp.muc.de

you will see, that the values of the env vars are used, but only *after*
reading and getting EOF on stdin, i.e. if you run:

$ fetch ftp://ftp.muc.de < /dev/null

it works right away.

How this is supposed to work in automated scripts, like 'make install'?
Comment 1 Conrad Meyer freebsd_committer 2018-01-23 20:49:56 UTC
query_auth (the hang you describe) is predicated on v_tty, which is only true if stderr is a tty (interactive console).  In scripts, it won't be.
Comment 2 Conrad Meyer freebsd_committer 2018-01-23 20:52:40 UTC
Part of the problem is that the environment variables aren't detected until the library, while the interactive authentication is done from the binary.  So there is a layering violation needed to determine to use the environment variables without prompting.

I'd suggest moving the interactive authentication code into the library instead, and predicating it on the necessary environment variables not being set.
Comment 3 Dag-Erling Smørgrav freebsd_committer 2018-05-12 16:38:32 UTC
Matthias: please provide a real example, because I am unable to reproduce the issue.  Also, note that a) ftp.muc.de neither expects nor accepts authentication and b) `FOO=bar export FOO` neither sets nor exports FOO.

Conrad: the FTP code doesn't use fetchAuthMethod, it uses getlogin(3) (which is a bug in itself), but only *after* checking FTP_PASSWORD.
Comment 4 Conrad Meyer freebsd_committer 2018-05-12 18:59:23 UTC
Ah, I stand corrected.  Thanks. :-)
Comment 5 Matthias Apitz 2018-05-12 19:14:08 UTC
b)
[guru@c720-r314251 ~]$ FOO=bar export FOO
[guru@c720-r314251 ~]$ env | grep FOO
FOO=bar

a)
for real world example try:

fetch ftp://sunsite.informatik.rwth-aachen.de/pub/mirror/eclipse.org/tools/cdt/releases/9.0/sr1/features/org.eclipse.cdt.msw_9.0.1.201607151550.jar
Comment 6 Dag-Erling Smørgrav freebsd_committer 2018-05-12 22:09:50 UTC
a) That server does not require authentication, and what you describe is impossible since the only interactive element in the FTP code is the getlogin() call which happens *only* if FTP_PASSWORD is unset.

b) Which shell are you using?

c) Are you actually running that fetch command from the command line, or within a script?

d) If the former, have you tried 'fetch -vv ftp://...'?

e) If the latter, have you tried running your script with sh -x to verify that it is indeed fetch which is waiting for input?
Comment 7 Matthias Apitz 2018-05-12 22:22:37 UTC
$ cat f
env | grep FTP_PASSWORD
fetch  -vv ftp://sunsite.informatik.rwth-aachen.de/pub/mirror/eclipse.org/tools/cdt/releases/9.0/sr1/features/org.eclipse.cdt.msw_9.0.1.201607151550.jar
$ sh f
scheme:   [ftp]
user:     []
password: []
host:     [sunsite.informatik.rwth-aachen.de]
port:     [0]
document: [/pub/mirror/eclipse.org/tools/cdt/releases/9.0/sr1/features/org.eclipse.cdt.msw_9.0.1.201607151550.jar]
---> sunsite.informatik.rwth-aachen.de:21
resolving server address: sunsite.informatik.rwth-aachen.de:21
<<< 220 Welcome to SunSITE CEUR
(hangs)

The above fetch is issued by some port's make engine, but this does not matter.
Comment 8 Dag-Erling Smørgrav freebsd_committer 2018-05-13 13:37:55 UTC
Created attachment 193353 [details]
Improve netrcfd handling and add debugging aids for FTP authentication

So 1) FTP_PASSWORD was not set, 2) the only way it could be trying to read from stdin at this point is if url->netrcfd is 0 in fetch_netrc_auth(), which shouldn't happen, because it is always initialized to -2, and 3) I am still unable to reproduce on either 11.1 or the latest head.

Are you by any chance behind a corporate firewall with a transparent FTP proxy?  Something like a Cisco PIX or ASA?

Could you add `ktrace -di` in front of the `fetch` command line and attach the full output from `kdump -tcnstu`?

Could you also try the attached patch?  It should print a few additional lines of debugging output, in addition to handling netrcfd more robustly.
Comment 9 Dag-Erling Smørgrav freebsd_committer 2018-05-13 13:46:33 UTC
Created attachment 193354 [details]
Improve netrcfd handling and add debugging aids for FTP authentication

Improved patch with slightly more debugging output.
Comment 10 Matthias Apitz 2018-05-14 07:43:47 UTC
I'm not behind any proxy/router; just connected to Internet via data mobile and my own firewall (ipf) is switched off.



$ cat ./225344-fetch.sh
#!/bin/sh

env | grep FTP_PASSWORD
ktrace -di fetch  -vv ftp://sunsite.informatik.rwth-aachen.de/pub/mirror/eclipse.org/tools/cdt/releases/9.0/sr1/features/org.eclipse.cdt.msw_9.0.1.201607151550.jar

$ ./225344-fetch.sh
scheme:   [ftp]
user:     []
password: []
host:     [sunsite.informatik.rwth-aachen.de]
port:     [0]
document: [/pub/mirror/eclipse.org/tools/cdt/releases/9.0/sr1/features/org.eclipse.cdt.msw_9.0.1.201607151550.jar]
---> sunsite.informatik.rwth-aachen.de:21
resolving server address: sunsite.informatik.rwth-aachen.de:21
<<< 220 Welcome to SunSITE CEUR

(it hangs here until I do use Ctrl-C)

^C^C

If I run the same with STDIN redirected from /dev/null to make
the read(2) seeing EOF it works fine:

$ ./225344-fetch.sh < /dev/null
scheme:   [ftp]
user:     []
password: []
host:     [sunsite.informatik.rwth-aachen.de]
port:     [0]
document: [/pub/mirror/eclipse.org/tools/cdt/releases/9.0/sr1/features/org.eclipse.cdt.msw_9.0.1.201607151550.jar]
---> sunsite.informatik.rwth-aachen.de:21
resolving server address: sunsite.informatik.rwth-aachen.de:21
<<< 220 Welcome to SunSITE CEUR
>>> USER anonymous
<<< 331 Please specify the password.
>>> PASS guru@c720-r314251
<<< 230 Login successful.
>>> PWD
<<< 257 "/"
>>> CWD pub/mirror/eclipse.org/tools/cdt/releases/9.0/sr1/features
<<< 250 Directory successfully changed.
>>> MODE S
<<< 200 Mode set to S.
>>> TYPE I
<<< 200 Switching to Binary mode.
>>> SIZE org.eclipse.cdt.msw_9.0.1.201607151550.jar
<<< 213 18070
size: [18070]
>>> MDTM org.eclipse.cdt.msw_9.0.1.201607151550.jar
<<< 213 20160715182025
last modified: [2016-07-15 18:20:25]
>>> MODE S
<<< 200 Mode set to S.
>>> TYPE I
<<< 200 Switching to Binary mode.
setting passive mode
>>> PASV
<<< 227 Entering Passive Mode (137,226,34,227,232,8)
opening data connection
initiating transfer
>>> RETR org.eclipse.cdt.msw_9.0.1.201607151550.jar
<<< 150 Opening BINARY mode data connection for org.eclipse.cdt.msw_9.0.1.201607151550.jar (18070 bytes).
local size / mtime: 18070 / 1468606825
remote size / mtime: 18070 / 1468606825
org.eclipse.cdt.msw_9.0.1.201607151550.jar    100% of   17 kB   35 kBps 00m01s
Waiting for final status
<<< 226 File send OK.


The attached kdump.txt is from the run without redirecting STDIN.
Comment 11 Matthias Apitz 2018-05-14 07:45:28 UTC
Created attachment 193380 [details]
output from: kdump -tcnstu
Comment 12 Dag-Erling Smørgrav freebsd_committer 2018-05-14 09:42:00 UTC
One quick correction: I said earlier that getlogin(3) is interactive...  of course it isn't.  I was thinking of getpass(3).  All getlogin(3) does is return your user name.  So there is no interactive element whatsoever in the ftp code path.

The only explanation I can think of for this behavior is, as mentioned earlier, that netrcfd is 0.  But that would not explain the fstat(2), fcntl(2) etc., which suggest something similar to getpasss().  Neither does it explain the three successful read(2) calls, returning a total of 40 bytes.

Is the NETRC environment variable set?  Does $HOME/.netrc (or ~/.netrc if HOME is not set) exist?

And can you please try the patch like I asked you?
Comment 13 Matthias Apitz 2018-05-14 10:29:32 UTC
I do not have the file ~/.netrc

The patch failes partly:

root@c720-r314251:/usr/src # patch < ~guru/fetch-netrcfd.diff
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|Index: lib/libfetch/common.c
|===================================================================
|--- lib/libfetch/common.c	(revision 333574)
|+++ lib/libfetch/common.c	(working copy)
--------------------------
Patching file lib/libfetch/common.c using Plan A...
Hunk #1 succeeded at 1342 (offset -19 lines).
Hunk #2 succeeded at 1366 (offset -19 lines).
Hunk #3 succeeded at 1382 (offset -19 lines).
Hunk #4 succeeded at 1435 (offset -19 lines).
Hmm...  The next patch looks like a unified diff to me...
The text leading up to this was:
--------------------------
|Index: lib/libfetch/fetch.c
|===================================================================
|--- lib/libfetch/fetch.c	(revision 333574)
|+++ lib/libfetch/fetch.c	(working copy)
--------------------------
Patching file lib/libfetch/fetch.c using Plan A...
Hunk #1 succeeded at 270 (offset -2 lines).
Hunk #2 succeeded at 285 (offset -2 lines).
Hunk #3 failed at 350.
1 out of 3 hunks failed--saving rejects to lib/libfetch/fetch.c.rej
Hmm...  The next patch looks like a unified diff to me...
The text leading up to this was:
--------------------------
|Index: lib/libfetch/ftp.c
|===================================================================
|--- lib/libfetch/ftp.c	(revision 333574)
|+++ lib/libfetch/ftp.c	(working copy)
--------------------------
Patching file lib/libfetch/ftp.c using Plan A...
Hunk #1 succeeded at 909 (offset -2 lines).
Hunk #2 succeeded at 929 (offset -2 lines).
done
Comment 14 Matthias Apitz 2018-05-14 10:34:28 UTC
the rejected is:

# cat lib/libfetch/fetch.c.rej
@@ -350,7 +350,7 @@
 		fetch_syserr();
 		return (NULL);
 	}
-	u->netrcfd = -2;
+	u->netrcfd = -1;
 
 	/* scheme name */
 	if ((p = strstr(URL, ":/"))) {

# grep 'u->netrcfd' lib/libfetch/fetch.c.orig 
	u->netrcfd = -2;

should I compile it anyway?
Comment 15 Dag-Erling Smørgrav freebsd_committer 2018-05-14 10:45:02 UTC
# cd lib/libfetch
# for f in common fetch ftp ; do mv $f.c.orig $f.c ; done
# svn diff >/tmp/reporting-bugs-in-code-that-you-have-modified-yourself-is-a-waste-of-everybody_s-time.diff
# svn revert -R .
# make && make install

That should fix your problem.
Comment 16 Matthias Apitz 2018-05-14 10:53:55 UTC
[root@c720-r314251 /usr/src/lib/libfetch]# for f in common fetch ftp ; do mv $f.c.orig $f.c ; done
[root@c720-r314251 /usr/src/lib/libfetch]# svn diff
[root@c720-r314251 /usr/src/lib/libfetch]# 

Blaming people for things they have not done is not helpful and unpolitely.
Comment 17 Matthias Apitz 2018-05-14 10:58:11 UTC
the source is and was as from SVN and the problem is unsolved
Comment 18 Dag-Erling Smørgrav freebsd_committer 2018-05-14 11:03:23 UTC
I take that back.  You don't have local modifications, but you're running a 15-month-old CURRENT.  Either way, I feel justified in rejecting the PR.