fetch(1) hangs (for example on ports 'make install') awaiting the input of ftp credentials via STDIN: The man page of fetch(3), i.e. the used library for this, explains the usage of some env vars: FTP_LOGIN=anonymous export FTP_LOGIN FTP_PASSWORD=guru@sisis.de export FTP_PASSWORD and if you set this and run: $ fetch ftp://ftp.muc.de you will see, that the values of the env vars are used, but only *after* reading and getting EOF on stdin, i.e. if you run: $ fetch ftp://ftp.muc.de < /dev/null it works right away. How this is supposed to work in automated scripts, like 'make install'?
query_auth (the hang you describe) is predicated on v_tty, which is only true if stderr is a tty (interactive console). In scripts, it won't be.
Part of the problem is that the environment variables aren't detected until the library, while the interactive authentication is done from the binary. So there is a layering violation needed to determine to use the environment variables without prompting. I'd suggest moving the interactive authentication code into the library instead, and predicating it on the necessary environment variables not being set.
Matthias: please provide a real example, because I am unable to reproduce the issue. Also, note that a) ftp.muc.de neither expects nor accepts authentication and b) `FOO=bar export FOO` neither sets nor exports FOO. Conrad: the FTP code doesn't use fetchAuthMethod, it uses getlogin(3) (which is a bug in itself), but only *after* checking FTP_PASSWORD.
Ah, I stand corrected. Thanks. :-)
b) [guru@c720-r314251 ~]$ FOO=bar export FOO [guru@c720-r314251 ~]$ env | grep FOO FOO=bar a) for real world example try: fetch ftp://sunsite.informatik.rwth-aachen.de/pub/mirror/eclipse.org/tools/cdt/releases/9.0/sr1/features/org.eclipse.cdt.msw_9.0.1.201607151550.jar
a) That server does not require authentication, and what you describe is impossible since the only interactive element in the FTP code is the getlogin() call which happens *only* if FTP_PASSWORD is unset. b) Which shell are you using? c) Are you actually running that fetch command from the command line, or within a script? d) If the former, have you tried 'fetch -vv ftp://...'? e) If the latter, have you tried running your script with sh -x to verify that it is indeed fetch which is waiting for input?
$ cat f env | grep FTP_PASSWORD fetch -vv ftp://sunsite.informatik.rwth-aachen.de/pub/mirror/eclipse.org/tools/cdt/releases/9.0/sr1/features/org.eclipse.cdt.msw_9.0.1.201607151550.jar $ sh f scheme: [ftp] user: [] password: [] host: [sunsite.informatik.rwth-aachen.de] port: [0] document: [/pub/mirror/eclipse.org/tools/cdt/releases/9.0/sr1/features/org.eclipse.cdt.msw_9.0.1.201607151550.jar] ---> sunsite.informatik.rwth-aachen.de:21 resolving server address: sunsite.informatik.rwth-aachen.de:21 <<< 220 Welcome to SunSITE CEUR (hangs) The above fetch is issued by some port's make engine, but this does not matter.
Created attachment 193353 [details] Improve netrcfd handling and add debugging aids for FTP authentication So 1) FTP_PASSWORD was not set, 2) the only way it could be trying to read from stdin at this point is if url->netrcfd is 0 in fetch_netrc_auth(), which shouldn't happen, because it is always initialized to -2, and 3) I am still unable to reproduce on either 11.1 or the latest head. Are you by any chance behind a corporate firewall with a transparent FTP proxy? Something like a Cisco PIX or ASA? Could you add `ktrace -di` in front of the `fetch` command line and attach the full output from `kdump -tcnstu`? Could you also try the attached patch? It should print a few additional lines of debugging output, in addition to handling netrcfd more robustly.
Created attachment 193354 [details] Improve netrcfd handling and add debugging aids for FTP authentication Improved patch with slightly more debugging output.
I'm not behind any proxy/router; just connected to Internet via data mobile and my own firewall (ipf) is switched off. $ cat ./225344-fetch.sh #!/bin/sh env | grep FTP_PASSWORD ktrace -di fetch -vv ftp://sunsite.informatik.rwth-aachen.de/pub/mirror/eclipse.org/tools/cdt/releases/9.0/sr1/features/org.eclipse.cdt.msw_9.0.1.201607151550.jar $ ./225344-fetch.sh scheme: [ftp] user: [] password: [] host: [sunsite.informatik.rwth-aachen.de] port: [0] document: [/pub/mirror/eclipse.org/tools/cdt/releases/9.0/sr1/features/org.eclipse.cdt.msw_9.0.1.201607151550.jar] ---> sunsite.informatik.rwth-aachen.de:21 resolving server address: sunsite.informatik.rwth-aachen.de:21 <<< 220 Welcome to SunSITE CEUR (it hangs here until I do use Ctrl-C) ^C^C If I run the same with STDIN redirected from /dev/null to make the read(2) seeing EOF it works fine: $ ./225344-fetch.sh < /dev/null scheme: [ftp] user: [] password: [] host: [sunsite.informatik.rwth-aachen.de] port: [0] document: [/pub/mirror/eclipse.org/tools/cdt/releases/9.0/sr1/features/org.eclipse.cdt.msw_9.0.1.201607151550.jar] ---> sunsite.informatik.rwth-aachen.de:21 resolving server address: sunsite.informatik.rwth-aachen.de:21 <<< 220 Welcome to SunSITE CEUR >>> USER anonymous <<< 331 Please specify the password. >>> PASS guru@c720-r314251 <<< 230 Login successful. >>> PWD <<< 257 "/" >>> CWD pub/mirror/eclipse.org/tools/cdt/releases/9.0/sr1/features <<< 250 Directory successfully changed. >>> MODE S <<< 200 Mode set to S. >>> TYPE I <<< 200 Switching to Binary mode. >>> SIZE org.eclipse.cdt.msw_9.0.1.201607151550.jar <<< 213 18070 size: [18070] >>> MDTM org.eclipse.cdt.msw_9.0.1.201607151550.jar <<< 213 20160715182025 last modified: [2016-07-15 18:20:25] >>> MODE S <<< 200 Mode set to S. >>> TYPE I <<< 200 Switching to Binary mode. setting passive mode >>> PASV <<< 227 Entering Passive Mode (137,226,34,227,232,8) opening data connection initiating transfer >>> RETR org.eclipse.cdt.msw_9.0.1.201607151550.jar <<< 150 Opening BINARY mode data connection for org.eclipse.cdt.msw_9.0.1.201607151550.jar (18070 bytes). local size / mtime: 18070 / 1468606825 remote size / mtime: 18070 / 1468606825 org.eclipse.cdt.msw_9.0.1.201607151550.jar 100% of 17 kB 35 kBps 00m01s Waiting for final status <<< 226 File send OK. The attached kdump.txt is from the run without redirecting STDIN.
Created attachment 193380 [details] output from: kdump -tcnstu
One quick correction: I said earlier that getlogin(3) is interactive... of course it isn't. I was thinking of getpass(3). All getlogin(3) does is return your user name. So there is no interactive element whatsoever in the ftp code path. The only explanation I can think of for this behavior is, as mentioned earlier, that netrcfd is 0. But that would not explain the fstat(2), fcntl(2) etc., which suggest something similar to getpasss(). Neither does it explain the three successful read(2) calls, returning a total of 40 bytes. Is the NETRC environment variable set? Does $HOME/.netrc (or ~/.netrc if HOME is not set) exist? And can you please try the patch like I asked you?
I do not have the file ~/.netrc The patch failes partly: root@c720-r314251:/usr/src # patch < ~guru/fetch-netrcfd.diff Hmm... Looks like a unified diff to me... The text leading up to this was: -------------------------- |Index: lib/libfetch/common.c |=================================================================== |--- lib/libfetch/common.c (revision 333574) |+++ lib/libfetch/common.c (working copy) -------------------------- Patching file lib/libfetch/common.c using Plan A... Hunk #1 succeeded at 1342 (offset -19 lines). Hunk #2 succeeded at 1366 (offset -19 lines). Hunk #3 succeeded at 1382 (offset -19 lines). Hunk #4 succeeded at 1435 (offset -19 lines). Hmm... The next patch looks like a unified diff to me... The text leading up to this was: -------------------------- |Index: lib/libfetch/fetch.c |=================================================================== |--- lib/libfetch/fetch.c (revision 333574) |+++ lib/libfetch/fetch.c (working copy) -------------------------- Patching file lib/libfetch/fetch.c using Plan A... Hunk #1 succeeded at 270 (offset -2 lines). Hunk #2 succeeded at 285 (offset -2 lines). Hunk #3 failed at 350. 1 out of 3 hunks failed--saving rejects to lib/libfetch/fetch.c.rej Hmm... The next patch looks like a unified diff to me... The text leading up to this was: -------------------------- |Index: lib/libfetch/ftp.c |=================================================================== |--- lib/libfetch/ftp.c (revision 333574) |+++ lib/libfetch/ftp.c (working copy) -------------------------- Patching file lib/libfetch/ftp.c using Plan A... Hunk #1 succeeded at 909 (offset -2 lines). Hunk #2 succeeded at 929 (offset -2 lines). done
the rejected is: # cat lib/libfetch/fetch.c.rej @@ -350,7 +350,7 @@ fetch_syserr(); return (NULL); } - u->netrcfd = -2; + u->netrcfd = -1; /* scheme name */ if ((p = strstr(URL, ":/"))) { # grep 'u->netrcfd' lib/libfetch/fetch.c.orig u->netrcfd = -2; should I compile it anyway?
# cd lib/libfetch # for f in common fetch ftp ; do mv $f.c.orig $f.c ; done # svn diff >/tmp/reporting-bugs-in-code-that-you-have-modified-yourself-is-a-waste-of-everybody_s-time.diff # svn revert -R . # make && make install That should fix your problem.
[root@c720-r314251 /usr/src/lib/libfetch]# for f in common fetch ftp ; do mv $f.c.orig $f.c ; done [root@c720-r314251 /usr/src/lib/libfetch]# svn diff [root@c720-r314251 /usr/src/lib/libfetch]# Blaming people for things they have not done is not helpful and unpolitely.
the source is and was as from SVN and the problem is unsolved
I take that back. You don't have local modifications, but you're running a 15-month-old CURRENT. Either way, I feel justified in rejecting the PR.