Bug 192283

Summary: irc/epic4 fails to fetch (tries to look in distinfo for tarball, not distfiles) after r363371
Product: Ports & Packages Reporter: Jeremy Chadwick <jdc>
Component: Individual Port(s)Assignee: Josh Paetzel <jpaetzel>
Status: Closed FIXED    
Severity: Affects Only Me CC: adamw
Priority: ---    
Version: Latest   
Hardware: amd64   
OS: Any   
Attachments:
Description Flags
Fix USES ordering none

Description Jeremy Chadwick 2014-07-31 03:30:09 UTC
I'm not sure what is going on here.  This is very uncomfortable -- look very closely at the error message (how it's attempting to look for the tarball within the actual port distinfo file (?!?!), rather than /usr/ports/distfiles (a directory):

root@icarus:/usr/ports/irc/epic4 # make clean
===>  Cleaning for epic4-2.10.5

root@icarus:/usr/ports/irc/epic4 # make fetch
===>  License BSD3CLAUSE accepted by the user
===>  Found saved configuration for epic4-2.10.5
===>   epic4-2.10.5 depends on file: /usr/local/sbin/pkg - found
=> epic4-2.10.5.tar.gz is not in /usr/ports/irc/epic4/distinfo.
=> Either /usr/ports/irc/epic4/distinfo is out of date, or
=> epic4-2.10.5.tar.gz is spelled incorrectly.
*** [do-fetch] Error code 1

Stop in /usr/ports/irc/epic4.

I haven't figured out exactly what's wrong here (nothing in the Makefile stands out) and this is the only port which does this.  But it's 100% reproducible on two separate RELENG_9 boxes with completely different hardawre, both with ports SVN r363529 (latest as of this PR).

I should note at some point in the semi-recent past this port did work correctly, so my gut feeling is that someone broke something within the Mk/ framework.
Comment 1 John Marino freebsd_committer 2014-07-31 08:45:41 UTC
Over to maintainer
Comment 2 Josh Paetzel freebsd_committer 2014-08-01 04:18:17 UTC
Can you paste the contents on /usr/ports/irc/epic4/distinfo?  I am unable to reproduce this error.  Port builds fine on my test system using the svn checkout of the ports tree.
Comment 3 Jeremy Chadwick 2014-08-01 04:35:59 UTC
From both systems, including an MD5 of the distinfo file.  These systems are completely unrelated and both pull from svn directly (not portsnap, etc.):

root@icarus:/usr/ports/irc/epic4 # cat distinfo
SHA256 (epic4-2.10.5.tar.bz2) = fca7b1788bde52687fd01c33c5e74d0e16fcc656a76b387de18504eda764ff80
SIZE (epic4-2.10.5.tar.bz2) = 636364
SHA256 (epic4-help-20050315.tar.bz2) = a7b7026ecfc00ab7040ef5e435d72fd347d48fab128722d4e213dba194cd5bbe
SIZE (epic4-help-20050315.tar.bz2) = 238390
root@icarus:/usr/ports/irc/epic4 # md5 distinfo
MD5 (distinfo) = e8e8d4349e725aef2a048df3b21541d1
root@icarus:/usr/ports/irc/epic4 # ls -l
total 56
-rw-r--r--  1 root  wheel   2256 Jul 29 22:12 Makefile
-rw-r--r--  1 root  wheel    282 Jul 29 22:12 distinfo
-rw-r--r--  1 root  wheel    438 Jul 29 22:12 pkg-descr
-rw-r--r--  1 root  wheel    256 Jul 29 22:12 pkg-message
-rw-r--r--  1 root  wheel  40871 Jul 29 22:12 pkg-plist

root@mambo:/usr/ports/irc/epic4 # cat distinfo
SHA256 (epic4-2.10.5.tar.bz2) = fca7b1788bde52687fd01c33c5e74d0e16fcc656a76b387de18504eda764ff80
SIZE (epic4-2.10.5.tar.bz2) = 636364
SHA256 (epic4-help-20050315.tar.bz2) = a7b7026ecfc00ab7040ef5e435d72fd347d48fab128722d4e213dba194cd5bbe
SIZE (epic4-help-20050315.tar.bz2) = 238390
root@mambo:/usr/ports/irc/epic4 # md5 distinfo
MD5 (distinfo) = e8e8d4349e725aef2a048df3b21541d1
root@mambo:/usr/ports/irc/epic4 # ls -l
total 56
-rw-r--r--  1 root  wheel   2256 Jul 29 17:34 Makefile
-rw-r--r--  1 root  wheel    282 Apr  2 00:33 distinfo
-rw-r--r--  1 root  wheel    438 Mar  3 17:07 pkg-descr
-rw-r--r--  1 root  wheel    256 Mar  3 17:07 pkg-message
-rw-r--r--  1 root  wheel  40871 Mar  3 17:07 pkg-plist
Comment 4 Jeremy Chadwick 2014-08-01 04:44:47 UTC
I just noticed: it looks to me like the problem is that the distinfo contains references to .tar.bz2 files, while something else (likely some part of the Mk/ framework, thus Makefile) contains references to .tar.gz files.  Note the error very carefully:

=> epic4-2.10.5.tar.gz is not in /usr/ports/irc/epic4/distinfo.

Compared to this in distinfo:

SHA256 (epic4-2.10.5.tar.bz2) = fca7b1788bde52687fd01c33c5e74d0e16fcc656a76b387de18504eda764ff80

You want my gut feeling?  r363371 is responsible somehow for this.
Comment 5 Josh Paetzel freebsd_committer 2014-08-01 05:35:36 UTC
[jpaetzel@roadrash] /usr/ports% svn info
Path: .
Working Copy Root Path: /usr/ports
URL: svn+ssh://svn/ports/head
Relative URL: ^/head
Repository Root: svn+ssh://svn/ports
Repository UUID: 35697150-7ecd-e111-bb59-0022644237b5
Revision: 363686

[jpaetzel@roadrash] /usr/ports/irc/epic4# make distclean
===>  Cleaning for epic4-2.10.5
===>  Deleting distfiles for epic4-2.10.5
[jpaetzel@roadrash] /usr/ports/irc/epic4# make fetch
===>  License BSD3CLAUSE accepted by the user
===>  Found saved configuration for epic4-2.10.5
===>   epic4-2.10.5 depends on file: /usr/local/sbin/pkg - found
=> epic4-2.10.5.tar.bz2 doesn't seem to exist in /usr/ports/distfiles/.
=> Attempting to fetch http://ftp.epicsol.org/pub/epic/EPIC4-PRODUCTION/epic4-2.10.5.tar.bz2
epic4-2.10.5.tar.bz2                          100% of  621 kB  456 kBps 00m01s
=> epic4-help-20050315.tar.bz2 doesn't seem to exist in /usr/ports/distfiles/.
=> Attempting to fetch http://ftp.epicsol.org/pub/epic/EPIC4-PRODUCTION/epic4-help-20050315.tar.bz2
epic4-help-20050315.tar.bz2                   100% of  232 kB  327 kBps 00m00s
===> Fetching all distfiles required by epic4-2.10.5 for building

I simply cannot reproduce this.
Comment 6 Jeremy Chadwick 2014-08-01 16:57:08 UTC
Is "roadrash" RELENG_10 or RELENG_9?

The fact I have two completely different boxes (one is bare metal, the other is a VPS hosted in southern California and operating under kqemu) behaving the exact same way -- and when the port used to build just fine -- seems to indicate it's more of a software thing than something wrong with my setups.

The tar.gz vs. tar.bz2 thing is very suspicious.  As such, I'm going to try rolling back ports to the commit prior to r363371.
Comment 7 Jeremy Chadwick 2014-08-01 17:09:57 UTC
I could have used svn update -r, but I had already tried other svn commands by that point, so this was easiest.  I checked out r363370, which is the commit immediately prior to r363371 (the one I suspect is the cause).

root@icarus:~ # rm -fr /usr/ports
root@icarus:~ # mkdir /usr/ports && chmod 755 /usr/ports && chown root:wheel /usr/ports
root@icarus:~ # svn checkout -r 363370 svn://svn.freebsd.org/ports/head /usr/ports
{...snip...}
 U   /usr/ports
Checked out revision 363370.

root@icarus:~ # cd /usr/ports
root@icarus:/usr/ports # svn info
Path: .
Working Copy Root Path: /usr/ports
URL: svn://svn.freebsd.org/ports/head
Relative URL: ^/head
Repository Root: svn://svn.freebsd.org/ports
Repository UUID: 35697150-7ecd-e111-bb59-0022644237b5
Revision: 363370
Node Kind: directory
Schedule: normal
Last Changed Author: amdmi3
Last Changed Rev: 363370
Last Changed Date: 2014-07-29 11:24:05 -0700 (Tue, 29 Jul 2014)

root@icarus:/usr/ports/irc/epic4 # make distclean
===>  Cleaning for epic4-2.10.5
===>  Deleting distfiles for epic4-2.10.5
root@icarus:/usr/ports/irc/epic4 # make fetch
===>  License BSD3CLAUSE accepted by the user
===>  Found saved configuration for epic4-2.10.5
===>   epic4-2.10.5 depends on file: /usr/local/sbin/pkg - found
=> epic4-2.10.5.tar.bz2 doesn't seem to exist in /usr/ports/distfiles/.
=> Attempting to fetch http://ftp.epicsol.org/pub/epic/EPIC4-PRODUCTION/epic4-2.10.5.tar.bz2
epic4-2.10.5.tar.bz2                          100% of  621 kB  783 kBps 00m01s
=> epic4-help-20050315.tar.bz2 doesn't seem to exist in /usr/ports/distfiles/.
=> Attempting to fetch http://ftp.epicsol.org/pub/epic/EPIC4-PRODUCTION/epic4-help-20050315.tar.bz2
epic4-help-20050315.tar.bz2                   100% of  232 kB  468 kBps 00m01s
===> Fetching all distfiles required by epic4-2.10.5 for building

Okay, so we know r363370 works.  Now let's update to r363371 and retry that:

root@icarus:/usr/ports/irc/epic4 # cd ../..
root@icarus:/usr/ports # svn update -r 363371
Updating '.':
{...snip...}
Updated to revision 363371.

root@icarus:/usr/ports # svn info
Path: .
Working Copy Root Path: /usr/ports
URL: svn://svn.freebsd.org/ports/head
Relative URL: ^/head
Repository Root: svn://svn.freebsd.org/ports
Repository UUID: 35697150-7ecd-e111-bb59-0022644237b5
Revision: 363371
Node Kind: directory
Schedule: normal
Last Changed Author: adamw
Last Changed Rev: 363371
Last Changed Date: 2014-07-29 11:41:15 -0700 (Tue, 29 Jul 2014)

root@icarus:/usr/ports # cd irc/epic4
root@icarus:/usr/ports/irc/epic4 # make distclean
===>  Cleaning for epic4-2.10.5
===>  Deleting distfiles for epic4-2.10.5
root@icarus:/usr/ports/irc/epic4 # make fetch
===>  License BSD3CLAUSE accepted by the user
===>  Found saved configuration for epic4-2.10.5
===>   epic4-2.10.5 depends on file: /usr/local/sbin/pkg - found
=> epic4-2.10.5.tar.gz is not in /usr/ports/irc/epic4/distinfo.
=> Either /usr/ports/irc/epic4/distinfo is out of date, or
=> epic4-2.10.5.tar.gz is spelled incorrectly.
*** [do-fetch] Error code 1

Stop in /usr/ports/irc/epic4.
root@icarus:/usr/ports/irc/epic4 #

Yup, whatever adamw committed broke this.  Rephrased: the irc/epic4 port maintainer is not responsible for the direct breakage, adamw is.

Can someone please CC him on this ticket?
Comment 8 Jeremy Chadwick 2014-08-01 17:13:21 UTC
Ah, I just figured out how to add people, heh.  I'm used to a significantly older Bugzilla from my past job... :-)
Comment 9 Josh Paetzel freebsd_committer 2014-08-01 17:15:12 UTC
I still don't understand why I can't reproduce this, why the builders aren't going off about this, etc etc etc.

roadrash is RELENG_10, but how would that affect anything related to fetching ports?
Comment 10 Jeremy Chadwick 2014-08-01 17:22:51 UTC
You would be surprised how it would affect things.  It would be the 2nd occasion I've personally seen where someone working on the Mk framework did not test on an ""older"" FreeBSD before committing, such as assuming certain utilities existed when on ""older"" they did not.  The QAT/tinderbox thing may also be being run on RELENG_10, so that might explain lack of automated test breakgage.  Or it may be because the only ports r363371 touched were ""unstaged"" ports, thus only ~426 ports are potentially impacted.

http://svnweb.freebsd.org/ports?limit_changes=0&view=revision&revision=363371

To me, it looks like something is messing with DISTINFO_FILE or possibly MASTERDIR.  Debugging Mk/ framework is tedious.  All I know is that reverting to r363370 or earlier fixes the problem.
Comment 11 Jeremy Chadwick 2014-08-01 17:44:07 UTC
Okay, after a bit of time debugging Mk framework code, I've figured out the problem and why it's not affecting you.  Here's the issue:

r363371 removed USE_BZIP2=yes contextually, meaning it turned that into a USES line that includes tar:bzip2, ex:

However, what is not made clear to other committers or ports maintainers is that USES+=tar:bzip2 actually modifies EXTRACT_SUFX.

Now look at the port Makefile (with line numbers):

     12 DISTFILES=      ${DISTNAME}${EXTRACT_SUFX} \
     13                 ${PORTNAME}-help-20050315.tar.bz2
    ...
     52 .if ${PORT_OPTIONS:MPERL}
     53 USES+=          perl5 tar:bzip2

Starting to see the problem?  The problem is that the USES+=tar:bzip2 part becomes conditionally dependent upon the PERL port option -- which I have disabled/unchecked!  I'm certain you have PERL enabled, thus you never saw the bug.

The solution is quite simple and I have confirmed works: remove the "tar:bzip2" part from that USES line, and instead further up (say before the OPTIONS part), add USES+=tar:bzip2.  Voila, it works.

Review the actual change -- look at what got removed and where and what got added and where (line-number-wise):

http://svnweb.freebsd.org/ports/head/irc/epic4/Makefile?limit_changes=0&r1=363371&r2=363370&pathrev=363371

What this means is that r363371 may have actually broken things very very badly -- the author appears to have blindly assumed that if there was an existing USES line, simply append tar:bzip2 to that line, regardless of where it is in the Makefile (and remove USE_BZIP2).  That is wrong / incorrect logic.  All 425 other ports need to be reviewed -- this is potentially major breakage.

I'll provide a patch for epic4 in a moment.
Comment 12 Jeremy Chadwick 2014-08-01 17:49:05 UTC
Created attachment 145217 [details]
Fix USES ordering
Comment 13 Jeremy Chadwick 2014-08-01 17:49:28 UTC
Patch available here:

http://jdc.koitsu.org/freebsd/192283/Makefile.diff
Comment 14 commit-hook freebsd_committer 2014-08-01 20:09:58 UTC
A commit references this bug:

Author: jpaetzel
Date: Fri Aug  1 20:09:43 UTC 2014
New revision: 363762
URL: http://svnweb.freebsd.org/changeset/ports/363762

Log:
  Fix regression introduced in r363371

  If you had the perl dependancy switched off this port was unfetchable.
  PR:	192283
  Submitted by:	jdc@koitsu.org

Changes:
  head/irc/epic4/Makefile
Comment 15 Josh Paetzel freebsd_committer 2014-08-01 20:11:03 UTC
Fixed in r363762

Thanks for chasing this down