Bug 152877 - [bsd.port.mk] Patch to add locking to ports make operations
Summary: [bsd.port.mk] Patch to add locking to ports make operations
Status: In Progress
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Ports Framework (show other bugs)
Version: Latest
Hardware: Any Any
: Normal Affects Only Me
Assignee: Port Management Team
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-12-07 01:20 UTC by Ivan Brawley
Modified: 2014-06-17 00:53 UTC (History)
1 user (show)

See Also:


Attachments
file.diff (4.99 KB, patch)
2010-12-07 01:20 UTC, Ivan Brawley
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Ivan Brawley 2010-12-07 01:20:11 UTC
With the use of machines with multiple cores, being able to use "make -j" for ports becomes more and more desirable.

Work has been done to allow for individual ports to be built with "make -j", however making multiple ports at once can cause problems. Problems arise when multiple ports try to build/install the same dependency at the same time.

So I've worked on a patch to Mk/bsd.port.mk that will use the lockf(1) command to lock the port's Makefile before doing any of the seven major targets (fetch, extract, patch, configure, build, install and package).

This method will cause the 2nd make (or 3rd, etc) to block it's make while another is running. It does cause the two to swap between being blocked and doing the work, but at least it's safe in that only one of them is doing the work.

It adds "WITH_PORT_LOCKING" and "WITHOUT_PORT_LOCKING" variables to control whether to do locking and "LOCKF_CMD" that does the actual locking. I've chosen the port's Makefile to control the locking of the port as every port has one and it allows for multiple logins/xterms to do makes with locking.

Fix: The attached patch to Mk/bsd.port.mk contains the magic for port locking.

Include it if its decided it's good enough.

Patch attached with submission follows:
Comment 1 Mark Linimon freebsd_committer freebsd_triage 2010-12-07 01:24:33 UTC
Responsible Changed
From-To: freebsd-ports-bugs->portmgr

portmgr territory.
Comment 2 Ivan Brawley 2010-12-08 05:53:31 UTC
Sorry, but the diff I uploaded is missing a character...

The last hunk is the offending one. I don't know how it slipped through, 
as I made sure it was working before making the patch file.

The line

if [ -d "$$dir" -a X"${WITH_PORT_LOCKING}" != "X" -a 
X"${WITHOUT_PORT_LOCKING" = "X" ]; then \

should be

if [ -d "$$dir" -a X"${WITH_PORT_LOCKING}" != "X" -a 
X"${WITHOUT_PORT_LOCKING}" = "X" ]; then \

ie, there was a missing '}' character at the end of WITHOUT_PORT_LOCKING.

ivan.
--
Ivan Brawley
Comment 3 Mark Linimon freebsd_committer freebsd_triage 2014-06-02 03:03:55 UTC
Infrastructure PR.
Comment 4 Baptiste Daroussin freebsd_committer 2014-06-16 06:48:16 UTC
This locking mechanism is not good enough, first it locks the Makefile which may be readonly, while it should lock the target, imho it will need a plain lock file in the WRKDIR directory.
Comment 5 Ivan Brawley 2014-06-16 07:10:22 UTC
Two points...

(1) The lockf command appears to still work if the Makefile is read-only. I've just tried two tests. One was using the lockf command as a regular user that can't write to the Makefile and by having my /usr/ports as a ZFS filesystem and setting the readonly flag to on. Both of these test cases, the lockf command was able to be paused until the other completed.

(2) This patch was made 3.5 years ago, and a lot has changed in the way that ports operates. I would be surprised if the patch can still be applied without pain, so I'll need to revisit the whole patch anyway if it would be worth while.
Comment 6 Baptiste Daroussin freebsd_committer 2014-06-16 08:08:30 UTC
I do like the general idea of this patch, I'm sorry it took so long for one to actually have a look at it, if you refresh your patch I'll try to make sure this time it does not take that long :)
Comment 7 Ivan Brawley 2014-06-17 00:53:47 UTC
OK, I'll take another look at the way ports are now built and come up with a new patch. May take a little while.

I'm open to suggestions to how this should work (inc. using a lock file in WRKDIR as that may help out on other edge cases). Any other suggestions?