Bug 155970 - [exp-run] [patch] lang/python: speed up upgrade-site-packages
[exp-run] [patch] lang/python: speed up upgrade-site-packages
Status: Closed FIXED
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s)
Latest
Any Any
: Normal Affects Only Me
Assigned To: Sofian Brabez
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2011-03-27 14:20 UTC by kimelto
Modified: 2012-09-27 15:00 UTC (History)
1 user (show)

See Also:


Attachments
python-2.7,2.patch (1.70 KB, patch)
2011-03-27 14:20 UTC, kimelto
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description kimelto 2011-03-27 14:20:10 UTC
Instead of asking each file in lib/python which package it comes from, ask each package if it has a file in lib/python.
This speed things up significantly on my system.

Port maintainer (python@FreeBSD.org) is cc'd.

Generated with FreeBSD Port Tools 0.99
Comment 1 Edwin Groothuis freebsd_committer 2011-03-27 14:20:22 UTC
Responsible Changed
From-To: freebsd-ports-bugs->freebsd-python

Over to maintainer (via the GNATS Auto Assign Tool)
Comment 2 Sofian Brabez freebsd_committer 2012-08-27 09:38:36 UTC
Responsible Changed
From-To: freebsd-python->sbz

I'll take it.
Comment 3 Sofian Brabez freebsd_committer 2012-09-04 12:16:51 UTC
Responsible Changed
From-To: sbz->portmgr

assign to portmgr for a exp-run 

http://www.freebsd.org/cgi/query-pr.cgi?pr=155970 

Date: Tue, 4 Sep 2012 13:13:05 +0200
Comment 4 Beat Gaetzi freebsd_committer 2012-09-04 20:22:21 UTC
Hi,

PKG_WHICH_CMD in case of USE_PORTMASTER hardcodes /usr/sbin/pkg_info. This
may breaks systems using pkgng. Could you please consider using PKG_INFO?

> grep PKGNG /etc/make.conf
#WITH_PKGNG=yes
> make -V PKG_INFO -C /usr/ports/
/usr/sbin/pkg_info

> grep PKGNG /etc/make.conf
WITH_PKGNG=yes
> make -V PKG_INFO -C /usr/ports/
/sbin/pkg info -g

Sorry I don't have a pkgng system which needs a python upgrade to test the
PKG_INFO change :)

Thanks,
Beat
Comment 5 Beat Gaetzi freebsd_committer 2012-09-04 20:25:20 UTC
On Sep 4, 2012, at 9:22 PM, Beat G=E4tzi wrote:
> Hi,
>=20
> PKG_WHICH_CMD in case of USE_PORTMASTER hardcodes /usr/sbin/pkg_info. =
This
> may breaks systems using pkgng. Could you please consider using =
PKG_INFO?

Argh, please ignore my last mail. Those lines were removed=85

Sorry,
Beat=
Comment 6 Beat Gaetzi freebsd_committer 2012-09-04 20:30:15 UTC
Responsible Changed
From-To: portmgr->beat

Take for exp-run.
Comment 7 Beat Gaetzi freebsd_committer 2012-09-08 09:04:30 UTC
Responsible Changed
From-To: beat->sbz

Exp-runs finished: 
http://pointyhat.freebsd.org/errorlogs/amd64-9-exp-latest 
http://pointyhat.freebsd.org/errorlogs/amd64-8-exp-latest 

The first exp-run on 9-exp had some unrelated ghostscript failures so 
I've stared a second one on 8-exp. The result looks ok but please review 
them too. Please note that an exp-run doesn't run the 
upgrade-site-packages target so this functionality wasn't tested.
Comment 8 dfilter freebsd_committer 2012-09-27 14:53:08 UTC
Author: sbz
Date: Thu Sep 27 13:52:53 2012
New Revision: 304962
URL: http://svn.freebsd.org/changeset/ports/304962

Log:
  - Speed up upgrade-site-packages [1]
  - Use make modifiers [2]
  
  PR:		ports/155970
  Submitted by:	Julien Laffaye <kimelto at gmail.com> [1], sbz (myself) [2]
  Exp-run by:	beat

Modified:
  head/lang/python/Makefile

Modified: head/lang/python/Makefile
==============================================================================
--- head/lang/python/Makefile	Thu Sep 27 13:44:01 2012	(r304961)
+++ head/lang/python/Makefile	Thu Sep 27 13:52:53 2012	(r304962)
@@ -25,15 +25,9 @@ do-install:
 
 # Major upgrade support
 PORTUPGRADE_CMD=	${LOCALBASE}/sbin/portupgrade
-PKG_WHICH_CMD=	${LOCALBASE}/sbin/pkg_which
-XARGS_CMD=	${XARGS} -0
 
 .if defined(USE_PORTMASTER)
 PORTUPGRADE_CMD=	${LOCALBASE}/sbin/portmaster
-PORTUPGRADE_ARGS?=	""
-PKG_WHICH_CMD=	/usr/sbin/pkg_info -qW
-JOBS=	${SYSCTL} -n kern.smp.cpus
-XARGS_CMD=	${XARGS} -L 1 -0 -P $$(${JOBS})
 .endif
 
 upgrade-site-packages:
@@ -42,22 +36,21 @@ upgrade-site-packages:
 		${FALSE}; \
 	 fi
 
-	@for ver in `echo ${_PYTHON_ALLBRANCHES}|${SED} 's/${PYTHON_VER}//'`; do \
-		if [ -d ${PREFIX}/lib/python$$ver ]; then \
-			UPD=`${FIND} ${PREFIX}/lib/python$$ver \
-					-type f -print0 | \
-				${XARGS_CMD} ${PKG_WHICH_CMD} | \
-				${GREP} -Fv '?' | \
-				${EGREP} -v '^python2?[0-5]?-2' | \
-				${SORT} -u`; \
-			if [ "$$UPD" ]; then \
-				if [ -n ${PORTUPGRADE_ARGS} ]; then \
-					${PORTUPGRADE_CMD} ${PORTUPGRADE_ARGS} -f $$UPD; \
-				else \
-					${PORTUPGRADE_CMD} -f $$UPD; \
-				fi; \
-			fi; \
+	@GREP_ARGS="-q"; \
+	for ver in `echo ${_PYTHON_ALLBRANCHES:C/${PYTHON_VER}//}`; do \
+		GREP_ARGS="$${GREP_ARGS} -e lib/python$$ver"; \
+	done; \
+	UPD=""; \
+	for pkg in `${PKG_INFO} -Ea`; do \
+		${PKG_INFO} -L $${pkg} | ${GREP} $${GREP_ARGS} && \
+			UPD="$${UPD} $${pkg}"; \
+	done; \
+	if [ "$${UPD}" ]; then \
+		if [ -n ${PORTUPGRADE_ARGS} ]; then \
+			${PORTUPGRADE_CMD} ${PORTUPGRADE_ARGS} -f $${UPD}; \
+		else \
+			${PORTUPGRADE_CMD} -f $${UPD}; \
 		fi; \
-	 done \
+	fi; \
 
 .include <bsd.port.mk>
_______________________________________________
svn-ports-all@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/svn-ports-all
To unsubscribe, send any mail to "svn-ports-all-unsubscribe@freebsd.org"
Comment 9 Sofian Brabez freebsd_committer 2012-09-27 14:54:32 UTC
State Changed
From-To: open->closed

Committed, with minor changes. Thanks!