Bug 151695 - [nanobsd] [patch] Enhance tools/nanobsd/fill_pkg.sh
Summary: [nanobsd] [patch] Enhance tools/nanobsd/fill_pkg.sh
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: misc (show other bugs)
Version: 8.1-STABLE
Hardware: Any Any
: Normal Affects Only Me
Assignee: Warner Losh
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-10-24 19:30 UTC by Lev A. Serebryakov
Modified: 2021-07-16 18:30 UTC (History)
0 users

See Also:


Attachments
Newer version, for pkgng (4.65 KB, text/plain)
2014-08-15 11:23 UTC, Lev A. Serebryakov
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Lev A. Serebryakov freebsd_committer freebsd_triage 2010-10-24 19:30:11 UTC
  NanoBSD has helper script "fill_pkg.sh" which links all packages and ther dependencies from "package dump" (like /usr/ports/packages/All) to specified directory.
  But "fill_pkg.sh" has some limitations:
  (1) It needs ports tree, which should have exactly same versions as "package dump".
  (2) It requires full paths to needed ports, including "/usr/ports" part.
  (3) It has assumptions about Nano Package Dir (it assumes, that it specified rtelative to current directory).
  (4) It does not have any diagnostics (almost).

  This PR enhances "fill_pkg.sh" script in several ways:

  (1) Nano package dir could be absolute path.
  (2) Script understands four ways to specify "root" ports/packages:
     (a) Absolute directory with port (old one)
     (b) Relative directory with port, relative to ${PORTSDIR} or /usr/ports
     (c) Absolute path to file with package (with .tbz suffix)
     (d) Name of package in dump dir, with or without .tbz suffix

     These ways can be mixed in one call. Dependencies for
     packages are obtained with 'pkg_info -r' call, and are searched for
     in same directory as "parent" package. Dependencies for ports are
     obtained in old way from port's Makefile.
  (3) Three levels of diagnostic (and -v option, could be repeated) are added.
  (4) All path variables are enclosed in quotes, to make script work with paths,
      containing spaces.

   As changes are massive, I attach new version of script iteslf, not
   patch/diff output.

Fix: 

#!/bin/sh
#
# Copyright (c) 2010 Lev Serebryakov.
# Copyright (c) 2009 Poul-Henning Kamp.
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright
#    notice, this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.
#
# $FreeBSD: src/tools/tools/nanobsd/fill_pkg.sh,v 1.1.2.1 2009/08/03 08:13:06 kensmith Exp $
#
# Usage:
# 	$0 PACKAGE_DUMP NANO_PACKAGE_DIR /usr/ports/foo/bar [package.tbz]...
#
# Will symlink the packages listed, including their runtime dependencies,
# from the PACKAGE_DUMP to the NANO_PACKAGE_DIR.
#

: ${PORTSDIR:=/usr/ports}

usage () {
	echo "Usage: $0 [-v] package-dump-dir nano-package-dir port-dir-or-pkg ..." 1>&2
	exit 2
}

msg () {
	local l=$1
	shift
	[ "$l" -le "$VERBOSE" ] && echo $*
}

ports_recurse() (
	local outputfile=$1
	shift
	local dumpdir=$1
	shift
	local type
	local fullpath
	for p
	do
		if [ -d "$p" -a -f "$p/Makefile" ] ; then
			msg 3 "$p: full path to port"
			PKGNAME=`cd "$p" && make -V PKGNAME`
			type=port
			fullpath=$p
		elif [ -d "${PORTSDIR}/$p" -a -f "${PORTSDIR}/$p/Makefile" ] ; then
			msg 3 "$p: path to port relative to ${PORTSDIR}}"
			PKGNAME=`cd "${PORTSDIR}/$p" && make -V PKGNAME`
			type=port
			fullpath=${PORTSDIR}/$p
		elif [ "${p%.tbz}" != "$p" -a -f "$p" ] && pkg_info "$p" > /dev/null 2>&1 ; then
			msg 3 "$p: full package file name"
			PKGNAME=`basename "$p" | sed 's/\.tbz$//I'`
			type=pkg
			fullpath=$p
		elif [ "${p%.tbz}" != "$p" -a -f "$dumpdir/$p" ] && pkg_info "$dumpdir/$p" > /dev/null 2>&1 ; then
			msg 3 "$p: package file name relative to $dumpdir"
			PKGNAME=`basename "$p" | sed 's/\.tbz$//I'`
			type=pkg
			fullpath=$dumpdir/$p
		elif [ -f "$dumpdir/$p.tbz" ] && pkg_info "$dumpdir/$p.tbz" > /dev/null 2>&1 ; then
			msg 3 "$p: package name relative to $dumpdir"
			PKGNAME=`basename "$p"`
			type=pkg
			fullpath=$dumpdir/$p.tbz
		else
			echo "Missing port or package $p" 1>&2
			exit 2
		fi
		if grep -q "^$PKGNAME\$" "$outputfile" ; then
			msg 3 "$PKGNAME was added already"
			true
		elif [ "$type" = "port" ] ; then
			(
			cd "$fullpath"
			rd=`make -V RUN_DEPENDS`	
			ld=`make -V LIB_DEPENDS`	
			
			for dep in $rd $ld
			do
				arg=`echo $dep | sed 's/^[^:]*:\([^:]*\).*$/\1/'`
				msg 2 "Check $arg as requirement for $PKGNAME"
				ports_recurse "$outputfile" "$dumpdir" "$arg"
			done
			)
			msg 1 "Add $PKGNAME"
			echo "$PKGNAME" >> "$outputfile"
		else
			dir=`dirname "$p"` # Get directory from SPECIFIED path, not from full path
			if [ "$dir" = "." ] ; then
			  dir=""
			else
			  dir=${dir}/
			fi
			deps=`pkg_info -r "$fullpath" | grep "Dependency:" | cut -d " " -f 2-`
			for dep in $deps
			do
				arg=`echo $dep | sed -e "s|^|$dir|" -e 's/$/.tbz/'`
				msg 2 "Check $arg as requirement for $PKGNAME"
				ports_recurse "$outputfile" "$dumpdir" "$arg"
			done
			msg 1 "Add $PKGNAME"
			echo "$PKGNAME" >> "$outputfile"
		fi
	done
)

VERBOSE=0

args=`getopt v $*`
if [ $? -ne 0 ] ; then
	usage
	exit 2
fi

set -- $args
for i
do
	case "$i" 
	in
	-v)
		VERBOSE=$(($VERBOSE + 1))
		shift
		;;
	--)
		shift
		break
	esac
done

NANO_PKG_DUMP=$1
shift;
if [ ! -d "$NANO_PKG_DUMP" ] ; then
	echo "$NANO_PKG_DUMP is not a directory" 1>&2
	usage
fi

NANO_PACKAGE_DIR=$1
shift;

if [ -e "$NANO_PKG_DIR" -a ! -d "$NANO_PKG_DIR" ] ; then
	echo "$NANO_PKG_DIR is not a directory" 1>&2
	usage
fi

case "$NANO_PKG_DIR"  in
	/*)	true ;;
	*)	NANO_PKG_DIR=`pwd`/$NANO_PKG_DIR ;;
esac

rm -rf "$NANO_PACKAGE_DIR"
mkdir -p "$NANO_PACKAGE_DIR"

PL=$NANO_PACKAGE_DIR/_list
true > "$PL"

for p
do
	ports_recurse "$PL" "$NANO_PKG_DUMP" "$p"
done

for i in `cat "$PL"`
do
	if [ -f "$NANO_PKG_DUMP/$i.tbz" ] ; then
		ln -s "$NANO_PKG_DUMP/$i.tbz" "$NANO_PACKAGE_DIR"
	else
		echo "Package $i misssing in $NANO_PKG_DUMP" 1>&2
		exit 1
	fi
done

rm -f "$PL"
exit 0
Comment 1 Lev A. Serebryakov freebsd_committer freebsd_triage 2010-10-24 19:46:01 UTC
Responsible Changed
From-To: freebsd-bugs->freebsd-embedded


Over to maintainer(s)
Comment 2 Warner Losh freebsd_committer freebsd_triage 2011-05-13 20:55:48 UTC
Responsible Changed
From-To: freebsd-embedded->imp

I think this looks good.  I'll evaluate in more depth and commit or 
iterate with author.
Comment 3 Lev A. Serebryakov freebsd_committer freebsd_triage 2014-08-15 11:23:21 UTC
Created attachment 145813 [details]
Newer version, for pkgng

It is version updated to pkgng.
Comment 4 Eitan Adler freebsd_committer freebsd_triage 2018-05-28 19:47:14 UTC
batch change:

For bugs that match the following
-  Status Is In progress 
AND
- Untouched since 2018-01-01.
AND
- Affects Base System OR Documentation

DO:

Reset to open status.


Note:
I did a quick pass but if you are getting this email it might be worthwhile to double check to see if this bug ought to be closed.
Comment 5 Warner Losh freebsd_committer freebsd_triage 2021-07-07 21:39:58 UTC
https://reviews.freebsd.org/D31101
Comment 6 commit-hook freebsd_committer freebsd_triage 2021-07-11 15:07:18 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=36cfb5d50f8e8856695780a6792fb7e81816e9ee

commit 36cfb5d50f8e8856695780a6792fb7e81816e9ee
Author:     Lev A. Serebryakov <lev@FreeBSD.org>
AuthorDate: 2021-07-11 15:04:39 +0000
Commit:     Warner Losh <imp@FreeBSD.org>
CommitDate: 2021-07-11 15:05:16 +0000

    nanobsd: enhance fill_pkg.sh

    NanoBSD has helper script "fill_pkg.sh" which links all packages and
    ther dependencies from "package dump" (like /usr/ports/packages/All) to
    specified director. fill_pkg.sh has some limitations:

    1) It needs ports tree, which should have exactly same versions as
       "package dump".
    2) It requires full paths to needed ports, including "/usr/ports" part.
    3) It has assumptions about Nano Package Dir (it assumes, that it
       specified rtelative to current directory).
    4) It does not have any diagnostics (almost).

    This PR enhances "fill_pkg.sh" script in several ways:

    1) Nano package dir could be absolute path.
    2) Script understands four ways to specify "root" ports/packages:
       (a) Absolute directory with port (old one)
       (b) Relative directory with port, relative to ${PORTSDIR} or /usr/ports
       (c) Absolute path to file with package (with .tbz suffix)
       (d) Name of package in dump dir, with or without .tbz suffix

       These ways can be mixed in one call. Dependencies for
       packages are obtained with 'pkg_info -r' call, and are searched for
       in same directory as "parent" package. Dependencies for ports are
       obtained in old way from port's Makefile.
    3) Three levels of diagnostic (and -v option, could be repeated) are added.
    4) All path variables are enclosed in quotes, to make script work with paths,
       containing spaces.

    Note: imp merged in the changes to fill_pkg.sh since this has been a PR.

    PR:                     151695
    Reviewed by:            imp@
    MFC After:              3 days
    Differential Revision:  https://reviews.freebsd.org/D31101

 tools/tools/nanobsd/fill_pkg.sh | 154 +++++++++++++++++++++++++++++-----------
 1 file changed, 113 insertions(+), 41 deletions(-)
Comment 7 commit-hook freebsd_committer freebsd_triage 2021-07-16 18:30:59 UTC
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=1405d761ba36f6adc59986999befddaf1bedee3f

commit 1405d761ba36f6adc59986999befddaf1bedee3f
Author:     Lev A. Serebryakov <lev@FreeBSD.org>
AuthorDate: 2021-07-11 15:04:39 +0000
Commit:     Warner Losh <imp@FreeBSD.org>
CommitDate: 2021-07-16 18:28:44 +0000

    nanobsd: enhance fill_pkg.sh

    NanoBSD has helper script "fill_pkg.sh" which links all packages and
    ther dependencies from "package dump" (like /usr/ports/packages/All) to
    specified director. fill_pkg.sh has some limitations:

    1) It needs ports tree, which should have exactly same versions as
       "package dump".
    2) It requires full paths to needed ports, including "/usr/ports" part.
    3) It has assumptions about Nano Package Dir (it assumes, that it
       specified rtelative to current directory).
    4) It does not have any diagnostics (almost).

    This PR enhances "fill_pkg.sh" script in several ways:

    1) Nano package dir could be absolute path.
    2) Script understands four ways to specify "root" ports/packages:
       (a) Absolute directory with port (old one)
       (b) Relative directory with port, relative to ${PORTSDIR} or /usr/ports
       (c) Absolute path to file with package (with .tbz suffix)
       (d) Name of package in dump dir, with or without .tbz suffix

       These ways can be mixed in one call. Dependencies for
       packages are obtained with 'pkg_info -r' call, and are searched for
       in same directory as "parent" package. Dependencies for ports are
       obtained in old way from port's Makefile.
    3) Three levels of diagnostic (and -v option, could be repeated) are added.
    4) All path variables are enclosed in quotes, to make script work with paths,
       containing spaces.

    Note: imp merged in the changes to fill_pkg.sh since this has been a PR.

    PR:                     151695
    Reviewed by:            imp@
    MFC After:              3 days
    Differential Revision:  https://reviews.freebsd.org/D31101

    (cherry picked from commit 36cfb5d50f8e8856695780a6792fb7e81816e9ee)

 tools/tools/nanobsd/fill_pkg.sh | 154 +++++++++++++++++++++++++++++-----------
 1 file changed, 113 insertions(+), 41 deletions(-)