Bug 235275 - Multiply bundled GH_TUPLE repositories result in broken links
Summary: Multiply bundled GH_TUPLE repositories result in broken links
Status: Closed Works As Intended
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Ports Framework (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Port Management Team
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-01-28 08:34 UTC by Yuri Victorovich
Modified: 2019-11-18 00:33 UTC (History)
2 users (show)

See Also:


Attachments
Makefile (753 bytes, text/plain)
2019-01-28 08:34 UTC, Yuri Victorovich
no flags Details
patch (593 bytes, patch)
2019-07-31 22:34 UTC, Yuri Victorovich
no flags Details | Diff
testcase.shar (77.84 KB, text/plain)
2019-07-31 23:02 UTC, Yuri Victorovich
no flags Details
Makefile (1.76 KB, text/plain)
2019-09-19 07:31 UTC, Mathieu Arnold
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Yuri Victorovich freebsd_committer 2019-01-28 08:34:12 UTC
Created attachment 201474 [details]
Makefile

The port with the attached Makefile creates broken links for multiply defined GH_TUPLE elements.

How to reproduce:
1. Save the attached Makefile as /usr/ports/devel/om/Makefile
2. cd /usr/ports/devel/om
3. Run: make makesum extract
4. Observe how all 'common' links except for one are broken:
> $ ls -l ./work/OpenModelica-1.14.0-dev/OMEdit/common
> lrwxr-xr-x  1 yuri  users  41 Jan 28 00:28 ./work/OpenModelica-1.14.0-dev/OMEdit/common -> OpenModelica-1.14.0-dev/OMCompiler/common
> $ ls -l ./work/OpenModelica-1.14.0-dev/OMEdit/common/
> ls: ./work/OpenModelica-1.14.0-dev/OMEdit/common/: No such file or directory
> $ ls -l OpenModelica-1.14.0-dev/OMCompiler/common/
> ls: OpenModelica-1.14.0-dev/OMCompiler/common/: No such file or directory

Only the first instance of 'common' is usable: ./work/OpenModelica-1.14.0-dev/OMCompiler/common
All subsequent links that are supposed to link to it are broken.
Comment 1 Yuri Victorovich freebsd_committer 2019-07-31 22:34:38 UTC
Created attachment 206184 [details]
patch

The attached patch solves the problem.

It should create links with the full path to the original directory.
Comment 2 Yuri Victorovich freebsd_committer 2019-07-31 23:02:01 UTC
Created attachment 206185 [details]
testcase.shar

Attaching the sample port that is fixed by this patch.
Comment 3 Mathieu Arnold freebsd_committer 2019-09-17 12:07:48 UTC
Damn, it took me a whole hour trying to see what you were doing and why it was wrong and broken.

You cannot have the *exact same* account/project/commit more than once when using USE_GITHUB and/or USE_GITLAB. it breaks, like you demonstrated here.

Remove the two extra OpenModelica-common:c9c9085 tuples and handle the symlinks yourself in post-extract (or maybe post-patch as post-extract may not be late enough).

Your patch that "fixes" the problem is wrong, and only hides the bug in your port.
Comment 4 Mathieu Arnold freebsd_committer 2019-09-17 12:23:24 UTC
I admit, your ports were allowed to be broken because of how I implemented the GH_SUBDIR thing.

review D21686 fixes the implementation and now your ports fail early.
Comment 5 Yuri Victorovich freebsd_committer 2019-09-19 03:29:46 UTC
(In reply to Mathieu Arnold from comment #4)

They aren't broken. It's about net/zeek, not OpenModelica. net/zeek has a deep bundling structure and some projects are bundled multiple times at multiple different locations. There is nothing wrong with this, and this should be allowed. The attached patch fixes the invalid links that framework creates. It makes links valid, and the build of net/zeek succeeds.
Comment 6 Mathieu Arnold freebsd_committer 2019-09-19 07:17:21 UTC
Like I said, a account:project:hash tuple can only happen once in G[HL]_TUPLES, it is like adding the same file to DISTFILES more than once, it only bloats the distinfo file.
If a port needs to have the same distfile accessible at multiple places, you can either extract it manually, or create symlinks yourself in post-extract.
Comment 7 Mathieu Arnold freebsd_committer 2019-09-19 07:21:30 UTC
To expand a bit, the framework is not there to automatically handle every whim of every port, it allows you to do most things automatically, and when you run into something that is does not do, it allows you to do so in the port's Makefile.
Comment 8 Mathieu Arnold freebsd_committer 2019-09-19 07:31:05 UTC
Created attachment 207614 [details]
Makefile

The corrected Makefile.
Comment 9 Dmitri Goutnik freebsd_committer 2019-11-18 00:33:50 UTC
(In reply to Mathieu Arnold from comment #6)
This situation with duplicates often happens with Go ports [1], [2] due to the way Go dependencies work. E.g.

fsnotify:fsnotify:v1.4.7:fsnotify_fsnotify/vendor/github.com/fsnotify/fsnotify \
fsnotify:fsnotify:v1.4.7:fsnotify_fsnotify_1/vendor/gopkg.in/fsnotify.v1 \
fsnotify:fsnotify:v1.4.7:fsnotify_fsnotify_2/vendor/gopkg.in/fsnotify/fsnotify.v1 \

are 3 different packages as far as Go is concerned, but we get duplicates because we have to map them to Github mirrors.

Looking at bsd.sites.mk code [3], it's not entirely clear why it does ${MV} first and then ${LN}:

post-extract-gh-${_group}:
	@${RMDIR} ${WRKSRC}/${GH_SUBDIR_${_group}} 2>/dev/null || :
	@${MKDIR} ${WRKSRC}/${GH_SUBDIR_${_group}:H} 2>/dev/null || :
	@${MV} ${WRKSRC_${_group}} ${WRKSRC}/${GH_SUBDIR_${_group}}
	@${LN} -s ${WRKSRC:T}/${GH_SUBDIR_${_group}} ${WRKSRC_${_group}}

After the move, there's nothing to link to anymore and two remaining fsnotify tuples from the example above get broken GH_SUBDIR symlink.

Would changing ${MV}/${LN} to just symlinking tuple's WRKSRC to GH_SUBDIR break anything? E.g.

post-extract-gh-${_group}:
	@${RMDIR} ${WRKSRC}/${GH_SUBDIR_${_group}} 2>/dev/null || :
	@${MKDIR} ${WRKSRC}/${GH_SUBDIR_${_group}:H} 2>/dev/null || :
	@${LN} -s ${WRKSRC_${_group}} ${WRKSRC}/${GH_SUBDIR_${_group}}

It does seem to work in my (limited) testing and fixes broken symlink problem.

[1] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241637
[2] https://bugs.freebsd.org/bugzilla/attachment.cgi?id=209217&action=diff
[3] https://svnweb.freebsd.org/ports/head/Mk/bsd.sites.mk?annotate=511849#l496