Sending and receiving systems are running the same version of FreeBSD (updated via freebsd-update) and the same versions of all software (using the quarterly pkg repo -- except for bbcp which was manually pulled from latest pkg repo and installed via "pkg add"). # uname -a FreeBSD alphadrive.sd73.bc.ca 9.3-RELEASE-p5 FreeBSD 9.3-RELEASE-p5 #0: Mon Nov 3 22:38:58 UTC 2014 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 The normal command I run is as follows (pipe a "zfs send" over a bbcp connection and then pipe that into a "zfs recv"): bbcp -N io "zfs send -I poolname/backups/folder@2015-01-21 poolname/backups/folder@2015-01-22" username@remotehost:"zfs recv -d poolname" The only output is "Segmentation fault (core dumped)." with the resulting bbcp.core file created. dmesg (and /var/log/messages) just shows: pid 91510 (bbcp), uid 0: exited on signal 11 (core dumped) Sending and receiving system had bbcp-20150113 installed. Downgraded the receiving system to bbcp-20120520. Same result. Downgraded the sending system to bbcp-20120520 as well, and everything is transferring correctly. I haven't tested the older version on the sending side with the newer version on the receiving side. Have to wait for the current transfer to finish before I can change the software on the receiving system. The core file is 8 MB in size. It can be downloaded from here for the next couple of days: http://www.sd73.bc.ca/downloads/bbcp.core If there's anything else you need, let me know. I see there's a newer version available in bugzilla (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=196814). I don't have the ports tree installed on these systems. If the patch in that PR is committed before next week's pkg build, I can test it next week. :)
Maintainer CC'd
(In reply to fjwcash from comment #0) I believe this issue has been resolved already by bugzilla pr 196814: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=196814 This is a new version that I submitted on 2015-01-17 02:23 UTC Could someone with a commit bit please address this? Thank you for all your time, and consideration. --Chris
Sending server: bbcp-20120520 Receiving side: bbcp-20150113 # bbcp -N io "zfs send -I poolname/backups/folder@2015-01-21 poolname/backups/folder@2015-01-22" username@remotehost:"zfs recv -d poolname" bbcp: Invalid argument unable to connect to omegadrive.sd73.bc.ca port 5031 bbcp: Unable to allocate more than 0 of 4 data streams. bbcp: Accept timed out on port 5031 bbcp: Unable to allocate more than 0 of 4 data streams. Nothing is transferred. No segmentation fault, though. And nothing in the logs on either server (dmesg or /var/log/messages). Sending server: bbcp-20150113 Receiving side: bbcp-20150113 Results in segmentation fault. Sending server: bbcp-20150113 Receiving side: bbcp-20120520 Results in segmentation fault. Sending server: bbcp-20120520 Receiving side: bbcp-20120520 Results in successful transfer. I believe that is all permutations of versions.
(In reply to fjwcash from comment #3) > Sending server: bbcp-20120520 > Receiving side: bbcp-20150113 > > # bbcp -N io "zfs send -I poolname/backups/folder@2015-01-21 > poolname/backups/folder@2015-01-22" username@remotehost:"zfs recv -d > poolname" > bbcp: Invalid argument unable to connect to omegadrive.sd73.bc.ca port 5031 > bbcp: Unable to allocate more than 0 of 4 data streams. > bbcp: Accept timed out on port 5031 > bbcp: Unable to allocate more than 0 of 4 data streams. > > > Nothing is transferred. No segmentation fault, though. And nothing in the > logs on either server (dmesg or /var/log/messages). > > > Sending server: bbcp-20150113 > Receiving side: bbcp-20150113 > > Results in segmentation fault. > > > Sending server: bbcp-20150113 > Receiving side: bbcp-20120520 > > Results in segmentation fault. > > > Sending server: bbcp-20120520 > Receiving side: bbcp-20120520 > > Results in successful transfer. > > > I believe that is all permutations of versions. Version 20150116 was committed 2015-02-24 11:27:33 UTC. I think a package should be available by now. --Chris
Same error messages and results (core dumped; segmentation fault). /var/log/messages shows the following error: Mar 3 08:53:38 betadrive kernel: pid 21163 (bbcp), uid 0: exited on signal 11 Eventually, the following messages show on the console (usually 30 seconds after the core dump): bbcp: Accept timed out on port 5031 bbcp: Unable to allocate more than 0 of 4 data streams. Tested the following configurations, all with the same results: Send: 20150116 Recv: 20150116 Send: 20150116 Recv: 20120520 The following configuration gives difference error messages and doesn't core dump: Send: 20120520 Recv: 20150116 bbcp: Invalid argument unable to connect to offsite.backups.sd73.bc.ca port 5031 bbcp: Unable to allocate more than 0 of 4 data streams. bbcp: Accept timed out on port 5031 bbcp: Unable to allocate more than 0 of 4 data streams. The only configuration that works is: Send: 20120520 Recv: 20120520 Command used for testing: bbcp -N io "zfs send -I poolname/backups/folder@2015-01-21 poolname/backups/folder@2015-01-22" username@remotehost:"zfs recv -d poolname" I'll make a new core file available here shortly: http://www.sd73.bc.ca/downloads/bbcp-20150116.core
Problem is probably with ipv6 bcpp code, try -4 argument The reason of core dump is: n = getaddrinfo(iP, 0, hostHints, &rP); It fails with n==3 (EAI_BADFLAGS) and then MyHost is NULL which results in segmentation fault in strdup(). The reason of getaddrinfo failure is that hostHints.ai_flags contains AI_V4MAPPED | AI_CANONNAME AI_V4MAPPED flag is rejected if net.inet6.v6only==1 But fixing this was not sufficient, remote bcpp started listening on TCP6 socket, but local bcpp couldn't connect.
(In reply to lampa from comment #6) > Problem is probably with ipv6 bcpp code, try -4 argument > > The reason of core dump is: > > n = getaddrinfo(iP, 0, hostHints, &rP); > > It fails with n==3 (EAI_BADFLAGS) and then MyHost is NULL which results > in segmentation fault in strdup(). > > The reason of getaddrinfo failure is that hostHints.ai_flags contains > AI_V4MAPPED | AI_CANONNAME > AI_V4MAPPED flag is rejected if net.inet6.v6only==1 > > But fixing this was not sufficient, remote bcpp started listening on TCP6 > socket, but local bcpp couldn't connect. Thanks for the hints, lampa! I'll take those pointers, and see if I can cobble (in) some (better && FreeBSD) IPv6 support. If that appears unfeasible. I'll see if I can convert it all to strictly IPv4. Thanks again! --Chris
(In reply to lampa from comment #6) > Problem is probably with ipv6 bcpp code, try -4 argument > ... > > It fails with n==3 (EAI_BADFLAGS) and then MyHost is NULL which results > in segmentation fault in strdup(). > n==3, as I read it, indicates it's in bracketed format. In much the same way Apache expects it in httpd.conf eg; NameVirtualHost [nnnn:an:anan:annn:nnn:nnaa:aaan:aana]:80 the eval and designator is in bbcp_NetAddrInfo.C @134 // Format address // if (IP.Addr.sa_family == AF_INET6) {if (bLen < (INET6_ADDRSTRLEN+2)) return QFill(bAddr, bLen); if (fmtOpts & old6Map4 && IN6_IS_ADDR_V4MAPPED(&IP.v6.sin6_addr)) {if (ipRaw) {strcpy(bAddr, "::"); n = 2;} else {strcpy(bAddr, "[::"); n = 3; addBrak=1;} ___________________________^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ if (!inet_ntop(AF_INET, &IP.v6.sin6_addr.s6_addr32[3], bAddr+n, bLen-n)) return QFill(bAddr, bLen); } else { if (!ipRaw) {*bAddr = '['; n = 1; addBrak = 1;} else n = 0; if (!inet_ntop(AF_INET6,&(IP.v6.sin6_addr),bAddr+n,bLen-n)) return QFill(bAddr, bLen); } } At least as *I* read it. For the record. There were zero *BSD defines when I stepped into this, and to some extent I simply OR'd FreeBSD against MacOS, where it applied. I may have to dig deep into FreeBSD specific NET foo -- specifically INET6. So as to add additional/specific DEFINES. Or simply disable IPv6 entirely, if FreeBSD is detected. It's still early in my investigation. But just thought I might mention it FWIW. ... > > But fixing this was not sufficient, remote bcpp started listening on TCP6 > socket, but local bcpp couldn't connect. --Chris
Hi, Any update on this?
(In reply to Martin Wilke from comment #9) > Hi, > > Any update on this? Still waiting for feedback. Bracketing the IPv6 address should fix it. As should using the IPv4 switch ( -4 ). Thanks! --Chris
I just installed bbcp (built from a current ports tree on 10-STABLE svn updated as of today) and ran into a variety of odd segfaults and error messages. I tried building the older bbcp-20120520 version of the port and it worked fine. From this, I would assume that there is an issue with one of the FreeBSD-specific patches, as if the upstream was bad, other people besides FreeBSD users would be unhappy. I must admit that I was in a hurry, so I didn't try the -4 switch on the latest version (I only found this bug report after I'd built the older version and it was working). I subscribed this bug, so if more information is needed, let me know.
Sorry, didn't realise you were waiting for feedback; thought you were still working on the IPv6 stuff. Installed bbcp-20150116 onto FreeBSD 9.3p5. Updated my bbcp script to use -4 and get: bbcp: Value not specified for '-4'. Usage: bbcp [Options] [Inspec] Outspec Options: [-a [dir]] [-A] [-b [+]bf] [-B bsz] [-c [lvl]] [-C cfn] [-D] [-d path] [-e] [-E csa] [-f] [-F] [-g] [-h] [-i idfn] [-I slfn] [-k] [-K] [-L opts[@logurl]] [-l logf] [-m mode] [-n] [-N nio] [-o] [-O] [-p] [-P sec] [-r] [-R [args]] [-q qos] [-s snum] [-S srcxeq] [-T trgxeq] [-t sec] [-v] [-V] [-u loc] [-U wsz] [-w [=]wsz] [-x rate] [-y] [-z] [-Z pnf[:pnl]] [-4 [loc]] [-~] [-@ {copy|follow|ignore}] [-$] [-#] [--] I/Ospec: [user@][host:]file Seems -4 needs an argument, but not sure what [loc] means. Adding an IP address changes the error message to: bbcp: Invalid ipv4 options - 10.10.10.105 Moving the -4 to different places in the command-line changes the error message, but never actually connects to send anything: bbcp: i '. Not sure how to use the -4 option. And this package doesn't include a man page (although I'm not sure if the previous 2012xxxx version did either). Command-line used is: bbcp -N io "zfs send -I ${localpool}/${localfs}/${site}@${lastsnap} \ ${localpool}/${localfs}/${site}@${yesterday}" \ ${user}@${remotehost}:"${zfs} recv ${recvopts} ${rempool}"
Previous test was done with bbcp-20120520 installed on the remote server (receiving side). After upgrading it to bbcp-20150116 as well, no change. Same error messages when trying to use -4. After some trial and error, I managed to make "-4 c" work (no idea why, or what it does). But, then it spits out this error message: bbcp: Invalid argument unable to connect to port remoteserver:5031 bbcp: Unable to allocate more than 0 of 4 data streams. bbcp: Accept timed out on port 5031 bbcp: Unable to allocate more than 0 of 4 data streams. And the bbcp connection fails. Same error message when the receiving server is using bbcp-20120520. And, just to make sure all the bases are covered, using bbcp-20120520 on the sending side and bbcp-20150116 on the receiving side gives the same error message as listed above: bbcp: Invalid argument unable to connect to omegadrive.sd73.bc.ca port 5031 bbcp: Unable to allocate more than 0 of 4 data streams. bbcp: Accept timed out on port 5031 bbcp: Unable to allocate more than 0 of 4 data streams. bbcp-20120520 on both the sending and receiving side works great. So there's still something, somewhere in the updated codebase that doesn't like us. :(
Aha! Reading all of the comments in depth is helpful. :) I didn't notice the mention of the net.inet6.ip6.v6only sysctl. The following makes things work: On the receiving server: sysctl net.inet6.ip6.v6only=0 install bbcp-20150116.txz On the sending server: sysctl net.inet6.ip6.v6only=0 install bbcp-20150116.txz add "-4 c" to the start of the bbcp command With the above, I am able to use the latest version of the bbcp package to send data between servers. Command-line used: bbcp -4 c -N io "zfs send -I ${localpool}/${localfs}/${site}@${lastsnap} \ ${localpool}/${localfs}/${site}@${yesterday}" \ ${user}@${remotehost}:"${zfs} recv ${recvopts} ${rempool}" Would it be possible to get a brief blurb about what the -4 option parameters are?
Taken from the bbcp website (when in doubt, google it out): --ipv4 [who] | -4 [who] uses the IPv4 TCP stack for command processing and source-target connections. This option is incompatible with IPv6 addresses. The optional who argument restricts IPv4 mode to the command if c is specified, the source if s is specified or the target if t is specified. The default is cst (i.e. everywhere). If who is omitted and –-ipv4 is the last option on the command line it must be followed by the -- option (double dash). So, it looks like the "-h" output should be updated to use: [-4 [cst]] Or similar, with an explanation of what c, s, and t stand for? On another note, "-4 s" and "-4 t" end with segmentation faults. The only option that works is "-4 c". If needed, I can post core dumps for those two segfaults.
(In reply to fjwcash from comment #15) > Taken from the bbcp website (when in doubt, google it out): > > --ipv4 [who] | -4 [who] > uses the IPv4 TCP stack for command processing and source-target > connections. This option is incompatible with IPv6 addresses. The optional > who argument restricts IPv4 mode to the command if c is specified, the > source if s is specified or the target if t is specified. The default is cst > (i.e. everywhere). If who is omitted and –-ipv4 is the last option on the > command line it must be followed by the -- option (double dash). > > So, it looks like the "-h" output should be updated to use: > > [-4 [cst]] > > Or similar, with an explanation of what c, s, and t stand for? > > > > On another note, "-4 s" and "-4 t" end with segmentation faults. The only > option that works is "-4 c". > > If needed, I can post core dumps for those two segfaults. A *huge* thanks for all your testing, fjwcash@! I had hoped to do some myself today. But the updating of my dev box last night, went a stray. :-( I'll see if I can get some done tomorrow. But in any event, it looks like we've come a long way, baby! :-) Thanks again, fjwcash! --Chris
Is this still relevant? 9.3 is gone. Version is 20150116.
(In reply to w.schwarzenfeld from comment #17) Why not try it and see? But, since you asked, I did... Clean install on 11-STABLE r326615 via building from ports and installing: (0:9) pool12:/usr/ports/sysutils/bbcp% bbcp -P 2 /dev/zero localhost:/dev/null Password for terry@pool12.glaver.org: bbcp: Connection refused unable to connect to localhost port 5031 bbcp: Unable to allocate more than 0 of 4 data streams. ^C (1:10) pool12:/usr/ports/sysutils/bbcp% bbcp -# 14.09.02.00.0 Exact same system, using bbcp 20120520 binary copied from 10.x: (0:5) pool12:/usr/ports/sysutils/bbcp% bbcp -P 2 /dev/zero localhost:/dev/null Password for terry@pool12.glaver.org: bbcp: Creating /dev/null/zero bbcp: 180112 20:08:48 0% done; 2.1 GB/s bbcp: 180112 20:08:50 0% done; 2.1 GB/s ^C (1:6) pool12:/usr/ports/sysutils/bbcp% bbcp -# bbcp: Invalid option, '-#'. Usage: bbcp [Options] [Inspec] Outspec Options: [-a [dir]] [-b [+]bf] [-B bsz] [-c [lvl]] [-C cfn] [-D] [-d path] [-e] [-E csa] [-f] [-F] [-h] [-i idfn] [-I slfn] [-k] [-K] [-L opts[@logurl]] [-l logf] [-m mode] [-n] [-N nio] [-o] [-O] [-p] [-P sec] [-r] [-R [args]] [-q qos] [-s snum] [-S srcxeq] [-T trgxeq] [-t sec] [-v] [-V] [-u loc] [-U wsz] [-w [=]wsz] [-x rate] [-y] [-z] [- -] I/Ospec: [user@][host:]file "Of course", the older bbcp doesn't support version identification* via -#, but a "strings /usr/local/bin/bbcp | grep ersion" reports: Version: 12.01.30.01.0 I also built bbcp 20120520 on this r326615 system using the current ports framework and the system clang 5.0.0 (the one I copied over was built under an older ports framework with the older system clang 3.4.1) and it worked properly. * Not that this is particularly useful, as the reported "versions" don't correspond with the port / distname "versions".
I walk through the buglist and search for old and obsolete PR's. Some thing I try and some not, and I only ask (it will be too much to try all, and some tries will not possible). Thanks for answer.
(In reply to w.schwarzenfeld from comment #19) Ok, sorry for the tone of my prior reply. It has just been 3 years in this PR alone, with cycles of: Q: Is it still broken? A: Yes! <long pause> On the other hand, perhaps no-one else is using this port and those of us who do have found various work-arounds. bsdstats (which itself just started working again after years in the same limbo) reports no installs of sysutils/bbc - it will be interesting to see if in a few months after a few report cycles, if anyone (at least anyone using bsdstats) is using bbcp (and what versions).
(In reply to w.schwarzenfeld from comment #17) Hey, this is my bug. Totally forgot about it. :) I'll have to double-check the versions we use when back at work on Monday. I vaguely remember something about upgrading the package after upgrading to FreeBSD 10.4 and having to force an install of an older saved package. Cheers, Freddie
(In reply to fjwcash from comment #21) Any updates on this, Freddie? I'm now in situation that I really need working bbcp from the ports, so I offer my helping hand. Vinca
(In reply to Vinca Kol from comment #22) I don't know if this will be useful for you, but you can fetch an older version of the port from https://svnweb.freebsd.org/ports/head/sysutils/bbcp/?pathrev=351085 and that one will work.
(In reply to Terry Kennedy from comment #23) Yep, I know and I'm using that. But I'm on embedded system, so it's not straightforward to edit the /etc/services. And the old version also doesn't work on 11.1, because of missing library libstdc++.so.6 (and i can't easily install it on the embedded system).
Sending server: # freebsd-version -ku 10.3-RELEASE-p20 10.3-RELEASE-p22 # pkg info -x bbcp bbcp-20150116 Command used: bbcp -4 c -N io "${zfs} send -I ${localpool}/${localfs}/${site}@${lastsnap} ${localpool}/${localfs}/${site}@${yesterday}" rsbackup@${remserver}:"${zfs} recv ${recvopts} ${rempool}" Receiving server: # freebsd-version -ku 10.3-RELEASE-p20 10.3-RELEASE-p22 # pkg info -x bbcp bbcp-20150116 Everything works beautifully. And, reading back through this bug report, Comment 14 shows things working. :) At least for IPv4 connections. I have no IPv6 systems to test that side of things with. Reading through Comment 14, it was the "-4 c" that made things work without segmentation faults for us. As far as our setup goes, this bug has been worked around / mitigated / avoided (solved?) by the use of "-4 c". As far as I am concerned, this is no longer an issue.
First off a HUGE thanks go out to fjwcash@ for his relentless, and tireless work with this. I can't thank you enough! That said, and for all others whom claim it doesn't work; The answer was given above. It goes as follows: The following makes things work: On the receiving server: sysctl net.inet6.ip6.v6only=0 install bbcp-20150116.txz On the sending server: sysctl net.inet6.ip6.v6only=0 install bbcp-20150116.txz add "-4 c" to the start of the bbcp command With the above, I am able to use the latest version of the bbcp package to send data between servers. Command-line used: bbcp -4 c -N io "zfs send -I ${localpool}/${localfs}/${site}@${lastsnap} \ ${localpool}/${localfs}/${site}@${yesterday}" \ ${user}@${remotehost}:"${zfs} recv ${recvopts} ${rempool}" Taken from the bbcp website (when in doubt, google it out): --ipv4 [who] | -4 [who] uses the IPv4 TCP stack for command processing and source-target connections. This option is incompatible with IPv6 addresses. The optional who argument restricts IPv4 mode to the command if c is specified, the source if s is specified or the target if t is specified. The default is cst (i.e. everywhere). If who is omitted and –-ipv4 is the last option on the command line it must be followed by the -- option (double dash). So, it looks like the "-h" output should be updated to use: [-4 [cst]] Or similar, with an explanation of what c, s, and t stand for? On another note, "-4 s" and "-4 t" end with segmentation faults. The only option that works is "-4 c". So, until, or unless I'm able to ivercome the problems with the -s, and -t switches; don't use them. Thanks again, fjwcash@ ! --Chris
P.S. This pr(1) can be closed with the reason: not a bug --Chris
Please change the state of this PR back to open, because: 1) Behavior was introduced some time ago which causes this port to segfault unless specific command-line options are given. 2) Getting things working using the suggested workaround involves changing a sysctl which affects other things besides this port. 3) Having the port apparently default to using IPv6 (and then not working using IPv6) is counterintuitive to the way it used to work and the way [at least some] users expect it to work. The fact is, it used to work "out of the box". It doesn't any more.
Reopen. See Comment28.
(In reply to Terry Kennedy from comment #28) > Please change the state of this PR back to open, because: > > 1) Behavior was introduced some time ago which causes this port to segfault > unless specific command-line options are given. > > 2) Getting things working using the suggested workaround involves changing a > sysctl which affects other things besides this port. > > 3) Having the port apparently default to using IPv6 (and then not working > using IPv6) is counterintuitive to the way it used to work and the way [at > least some] users expect it to work. > > The fact is, it used to work "out of the box". It doesn't any more. I was unable to get it to dump core. But then again, I also (as I suggested in this thread) bracketed my IPv6 address -- [ipv6:addr:ess1::...] which worked fine for me having IPv6. I introduced this newer version, because several people felt that FreeBSD should have the newer version that was available. Indicating that I wasn't doing a good job as it's Maintainer. The newer version was NOT supported by FreeBSD, and I was forced to add that support. If it had been up to me, I'd have left it at the last version (which you, and others indicated "just worked"). I'll try to find time to do additional work. But I [currently] have no open time slots available, and as a result can't give you a time frame. If the directions laid out in this pr(1) don't work for you, you would probably be best served by rolling back to the previous version. Or if that won't work for you; trying to hack up a patch to fix the problem(s) you indicate you're experiencing (if you're pressed for time, and can't wait for me). HTH --Chris
OK for those still experiencing trouble, or just want to test this port out. I have added some changes to this port that should improve things as compared to what's currently in the ports tree. Until those changes have been committed. Please see: bug #229115 HTH and please report your findings! --Chris
(In reply to Chris Hutchinson from comment #31) I've been working with a much newer BBCP version (2017.12.00.00.0) with encouraging results - please refer to PR 229115 for the details.
Port removed. Close here "overcome by events". sysutils/bbcp||2019-08-03|Has expired: the port is broken for more than a year