Bug 224871

Summary: `make fetchindex' may leave a corrupt INDEX file
Product: Ports & Packages Reporter: Wolfram Schneider <wosch>
Component: Ports FrameworkAssignee: Port Management Team <portmgr>
Status: Closed FIXED    
Severity: Affects Only Me CC: dch, portmgr, ports-bugs, rene
Priority: ---    
Version: Latest   
Hardware: Any   
OS: Any   
URL: https://reviews.freebsd.org/D14088
Bug Depends on: 224893    
Bug Blocks:    
Description Flags
/usr/ports/Makefile patch none

Description Wolfram Schneider freebsd_committer 2018-01-03 17:15:09 UTC
In /usr/ports/Makefile is a target fetchindex to fetch the latest INDEX file for the given OS.

$ cd /usr/ports; make fetchindex

In case of an error the target will fail with an error status, but leaves the ./INDEX file in an undefined corrupt state. An error can occurs if the disk is full, or the kernel kills the bunzip2 command (e.g. out of swap).


$ make fetchindex
bunzip2: I/O or other error, bailing out.  Possible reason follows.
bunzip2: No space left on device
	Input file = (stdin), output file = (stdout)
*** Error code 1

make: stopped in /home/projects/freebsd-ports

$ du -hs INDEX-12
 32M	INDEX-12

The expected size is 36MB.

The problem is in the line
    bunzip2 < ${INDEXDIR}/${INDEXFILE}.bz2 > ${INDEXDIR}/${INDEXFILE}
Comment 1 Wolfram Schneider freebsd_committer 2018-01-03 17:19:12 UTC
Another issue is: the fetch command runs in mirror mode (-am), but the INDEX.bz2 file will be deleted after extracted. That does not make sense.

make -n fetchindex
/usr/bin/env  fetch -am -o /home/projects/freebsd-ports/INDEX-12.bz2 http://www.FreeBSD.org/ports/INDEX-12.bz2
bunzip2 < /home/projects/freebsd-ports/INDEX-12.bz2 > /home/projects/freebsd-ports/INDEX-12 &&  chmod a+r /home/projects/freebsd-ports/INDEX-12 && /bin/rm -f /home/projects/freebsd-ports/INDEX-12.bz2

To run fetch(1) in mirror mode, we need to keep the bzip2'd INDEX file.
Comment 2 Wolfram Schneider freebsd_committer 2018-01-03 17:23:37 UTC
Created attachment 189374 [details]
/usr/ports/Makefile patch
Comment 3 Rene Ladan freebsd_committer 2019-04-08 19:23:53 UTC
Reassign to portmgr
Comment 4 commit-hook freebsd_committer 2021-11-08 15:00:33 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=cd8518854f3e96fb86dd7ed7b8f1c0258e839157

commit cd8518854f3e96fb86dd7ed7b8f1c0258e839157
Author:     Baptiste Daroussin <bapt@FreeBSD.org>
AuthorDate: 2021-11-08 14:58:02 +0000
Commit:     Baptiste Daroussin <bapt@FreeBSD.org>
CommitDate: 2021-11-08 14:59:55 +0000

    fetchindex: do not leave a corrupted INDEX

    When extracting the fetched index, do it in a temporary file, and only
    when extraction succeed, move it to the final file, it makes it more
    atomic and less likely to live a corrupted file behind

    PR:     224871

 Makefile | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)