Bug 209814
| Summary: | games/diaspora: Split off data files into separate port | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Ports & Packages | Reporter: | lightside <lightside> | ||||||||||||||||||||||||||||||
| Component: | Individual Port(s) | Assignee: | Dmitry Marakasov <amdmi3> | ||||||||||||||||||||||||||||||
| Status: | Closed Not Accepted | ||||||||||||||||||||||||||||||||
| Severity: | Affects Some People | CC: | mat | ||||||||||||||||||||||||||||||
| Priority: | --- | Keywords: | feature, patch | ||||||||||||||||||||||||||||||
| Version: | Latest | ||||||||||||||||||||||||||||||||
| Hardware: | Any | ||||||||||||||||||||||||||||||||
| OS: | Any | ||||||||||||||||||||||||||||||||
| Attachments: |
|
||||||||||||||||||||||||||||||||
|
Description
lightside
2016-05-28 17:01:15 UTC
Created attachment 170761 [details]
The games/diaspora-data port in shar format
Created attachment 170769 [details]
The poudriere testport log (FreeBSD 10.2 amd64): games/diaspora-data
Added games/diaspora-data poudriere testport log.
The games/diaspora port was tested manually.
Created attachment 170793 [details]
The poudriere testport log (FreeBSD 10.2 amd64): games/diaspora
Added poudriere testport log for games/diaspora port, just for completion.
Overall, if games/diaspora-data port is built, which took about 45 minutes for poudriere build and package generation, the build of games/diaspora port took about 10 minutes.
Without splitting of data files (as it is now), it may take about 50 minutes (which depends on used hardware).
Also, since games/diaspora-data is a architecture neutral port, this may reduce space requirements for various architectures or operating system versions (even if capacity may seem unlimited). Looks like, I needed to propose this, when some distfile was unavailable (on bug 209738). Now, there are available packages, e.g. on PortsMon: http://portsmon.freebsd.org/portoverview.py?category=games&portname=diaspora and each package is about 1.8 Gb. (In reply to comment #4) > and each package is about 1.8 Gb about 1573 Gb, if to be precise. (In reply to lightside from comment #5) > about 1573 Gb, if to be precise. wrong, 1.54 Gb. At least, now I know, that games/diaspora port builds on FreeBSD ARM platform. But PortsMon shows "runaway_process" error type, because of interrupted packaging stage (timed out build after 3600 seconds): http://beefy8.nyi.freebsd.org/data/head-armv6-default/p416340_s301291/logs/errors/diaspora-1.1.1.log 2 hours and 37 minutes is a "new record" for this port. If seriously, I recommend to return MANUAL_PACKAGE_BUILD=yes, in case if splitting of data files hasn't helped (but it should, if packaging of games/diaspora-data will be done on amd64 or i386 build server, while arm build server uses amd64/i386 result, because it's a architecture neutral port). This will save many resources on build servers. CC: mat Why split it ? Using MANUAL_PACKAGE_BUILD is *only* for ports that can't be built in poudriere (and they should be fixed, because a package must build in poudriere) Hello Mathieu Arnold. Thanks for your attention. (In reply to comment #8) > Why split it ? I explained this in comment #0, comment #3 and comment #4. If breafly, because data files can be splitted and become architecture neutral package (for games/diaspora-data port). Then games/diaspora port will build source files only (and package some options related files). This allows to reduce build time in case of PORTREVISION change(s) (e.g. USE_GL+=gl needed, because of new stage-qa) for different architecures, as well as save capacity. You can see exact numbers in a provided poudriere testport logs (e.g. 7 Mb for games/diaspora and 3 Gb for games/diaspora-data uncompressed). (In reply to comment #9) > Using MANUAL_PACKAGE_BUILD is *only* for ports that can't be built in > poudriere (and they should be fixed, because a package must build in > poudriere) The issue with ARM build server is related to limited resources, which doesn't allow to create package in time. The poudriere build log shows it: http://beefy8.nyi.freebsd.org/data/head-armv6-default/p416340_s301291/logs/errors/diaspora-1.1.1.log The PortsMon shows fine builds for other architectures: http://portsmon.freebsd.org/portoverview.py?category=games&portname=diaspora (In reply to lightside from comment #10) > Hello Mathieu Arnold. Thanks for your attention. > > (In reply to comment #8) > > Why split it ? > I explained this in comment #0, comment #3 and comment #4. > If breafly, because data files can be splitted and become architecture > neutral package (for games/diaspora-data port). Then games/diaspora port > will build source files only (and package some options related files). This > allows to reduce build time in case of PORTREVISION change(s) (e.g. > USE_GL+=gl needed, because of new stage-qa) for different architecures, as > well as save capacity. You can see exact numbers in a provided poudriere > testport logs (e.g. 7 Mb for games/diaspora and 3 Gb for games/diaspora-data > uncompressed). Ok, I don't really see the problem with the sizes you talk about, but why not. > (In reply to comment #9) > > Using MANUAL_PACKAGE_BUILD is *only* for ports that can't be built in > > poudriere (and they should be fixed, because a package must build in > > poudriere) > The issue with ARM build server is related to limited resources, which > doesn't allow to create package in time. The poudriere build log shows it: > http://beefy8.nyi.freebsd.org/data/head-armv6-default/p416340_s301291/logs/ > errors/diaspora-1.1.1.log > The PortsMon shows fine builds for other architectures: > http://portsmon.freebsd.org/portoverview.py?category=games&portname=diaspora Mmmm, maybe the 3600 seconds timeout needs to be bumped, or something else. (In reply to comment #11) > Mmmm, maybe the 3600 seconds timeout needs to be bumped, or something else. There is a PKG_CREATE_VERBOSE variable available in ports, with following description: "If set, pass the -v option to pkg create which ensures periodic output during packaging and will help prevent timeouts by build monitors". I can add PKG_CREATE_VERBOSE=yes to the games/diaspora-data port (shar archive currently), if this may solve timed out build(s) on ARM build server. What do you think? Created attachment 171303 [details]
The games/diaspora-data port in shar format
Added PKG_CREATE_VERBOSE=yes. Just in case.
I tested it on other (e.g. games/pioneer) port and it works with following output:
-8<--
% make package
===> Building package for pioneer-0.0.20160610
file sizes/checksums [2238]: 100%
packing files [2238]: 100%
packing directories [0]: 100%
-->8-
where percentage changes periodically.
(In reply to comment #13) > Added PKG_CREATE_VERBOSE=yes. Just in case. The same may be needed for following ports: games/crafty-tablebase-pawn: http://portsmon.freebsd.org/portoverview.py?category=games&portname=crafty-tablebase-pawn http://beefy8.nyi.freebsd.org/data/head-armv6-default/p416340_s301291/logs/errors/crafty-tablebase-pawn-20070910.log http://beefy3.nyi.freebsd.org/data/head-i386-default/p416574_s301708/logs/errors/crafty-tablebase-pawn-20070910.log http://beefy7.nyi.freebsd.org/data/head-mips-default/p416340_s301291/logs/errors/crafty-tablebase-pawn-20070910.log games/flightgear-data: http://portsmon.freebsd.org/portoverview.py?category=games&portname=flightgear-data http://beefy8.nyi.freebsd.org/data/head-armv6-default/p416340_s301291/logs/errors/flightgear-data-2016.2.1.log http://beefy7.nyi.freebsd.org/data/head-mips-default/p416340_s301291/logs/errors/flightgear-data-2016.2.1.log http://beefy8.nyi.freebsd.org/data/head-mips64-default/p416340_s301291/logs/errors/flightgear-data-2016.2.1.log games/ufoai-data: http://portsmon.freebsd.org/portoverview.py?category=games&portname=ufoai-data http://beefy8.nyi.freebsd.org/data/head-armv6-default/p416340_s301291/logs/errors/ufoai-data-2.5.log http://beefy7.nyi.freebsd.org/data/head-mips-default/p416340_s301291/logs/errors/ufoai-data-2.5.log http://beefy8.nyi.freebsd.org/data/head-mips64-default/p416340_s301291/logs/errors/ufoai-data-2.5.log But this shows, that data package will build across different architectures anyway, even if port is architecture neutral. So, it may save capacity for future builds only, in case previous result is available on the same build server. The full list of ports in games category, related to current "runaway_process" error type: games/crafty-tablebase-no-pawn games/crafty-tablebase-pawn games/diaspora games/flightgear-data games/ufoai-data games/urbanterror-data I got it with following query.sh script: -8<-- #!/bin/sh category=$1 portname=$2 url="http://portsmon.freebsd.org/portoverview.py?category=${category}&portname=${portname}" wget -qO - "$url" | grep -o "runaway_process" 1>/dev/null && echo "$category/$portname" -->8- and command: % find /usr/ports/games -type d -maxdepth 1 | sort | xargs basename | xargs -L1 ./query.sh games Created attachment 172977 [details]
The games/diaspora-data port in shar format
Use POST_PLIST+=build-plist-empty, instead of add-plist-post: build-plist-empty.
Created attachment 173011 [details]
Proposed patch (since 415836 revision)
Changed approach, with following overall changes:
- Bump PORTREVISION
- Add runtime dependency on games/diaspora-data port.
- Add gl to USE_GL
- Use EXTRACT_* variables
- Use new options helpers for WXLAUNCHER option
- Split data files
- Add PKG_CREATE_VERBOSE=yes to prevent timeouts by build monitors
Created attachment 173012 [details]
The games/diaspora-data port in shar format
Created attachment 173013 [details]
Proposed patch (since 415836 revision)
Removed "-f" option for ${RM} command, because it's already defined.
Created attachment 174740 [details] Proposed patch (since 415836 revision) - Update variables for Creative Commons license after ports r421995 changes Conditionals in diaspora are too complicated. I suggest to create separate data port not linked to diaspora. (In reply to comment #21) > Conditionals in diaspora are too complicated. I suggest to create separate > data port not linked to diaspora. Agreed, it may be complicated for others, except maintainer. I merged some logic in one main port to simplify maintainership, in my opinion. The separate games/diaspora-data port (like in attachment 172977 [details]) will require to duplicate MASTER_SITES, DISTFILES, LICENSE* variables and post-extract stage. (In reply to comment #21) > Conditionals in diaspora are too complicated. I suggest to create separate > data port not linked to diaspora. Possible to create so called Makefile.inc file with shared variables, like maintainer of devel/eric6 port did (for example). Ok, I will try to propose different variant after some time. Created attachment 174761 [details]
Proposed patch (since 415836 revision)
Created attachment 174762 [details] Proposed patch (since 415836 revision) (In reply to comment #23) > Possible to create so called Makefile.inc file with shared variables I think, the Makefile.common file is more appropriate (like in graphics/libGL port), instead of Makefile.inc. Created attachment 174763 [details] The games/diaspora-data port in shar format (In reply to comment #23) > Ok, I will try to propose different variant after some time. Done. To Dmitry Marakasov: Thanks for your attention. Sorry about first reaction in comment #22. I reconsidered the used approach (like what some maintainers did in other ports). Created attachment 174766 [details]
Proposed patch (since 415836 revision)
- Move docs installation to do-install-DOCS-on target
This PR was closed, because proposed changes were obsoleted. There is alternative PR in bug 215539, which contains some improvements from this bug report. |
