Bug 224871 - `make fetchindex' may leave a corrupt INDEX file
Summary: `make fetchindex' may leave a corrupt INDEX file
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Ports Framework (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Port Management Team
URL: https://reviews.freebsd.org/D14088
Keywords:
Depends on: 224893
Blocks:
  Show dependency treegraph
 
Reported: 2018-01-03 17:15 UTC by Wolfram Schneider
Modified: 2021-11-22 13:07 UTC (History)
4 users (show)

See Also:


Attachments
/usr/ports/Makefile patch (613 bytes, patch)
2018-01-03 17:23 UTC, Wolfram Schneider
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Wolfram Schneider freebsd_committer freebsd_triage 2018-01-03 17:15:09 UTC
In /usr/ports/Makefile is a target fetchindex to fetch the latest INDEX file for the given OS.

$ cd /usr/ports; make fetchindex

In case of an error the target will fail with an error status, but leaves the ./INDEX file in an undefined corrupt state. An error can occurs if the disk is full, or the kernel kills the bunzip2 command (e.g. out of swap).

Example:

$ make fetchindex
bunzip2: I/O or other error, bailing out.  Possible reason follows.
bunzip2: No space left on device
	Input file = (stdin), output file = (stdout)
*** Error code 1

Stop.
make: stopped in /home/projects/freebsd-ports

$ du -hs INDEX-12
 32M	INDEX-12

The expected size is 36MB.

The problem is in the line
    bunzip2 < ${INDEXDIR}/${INDEXFILE}.bz2 > ${INDEXDIR}/${INDEXFILE}
Comment 1 Wolfram Schneider freebsd_committer freebsd_triage 2018-01-03 17:19:12 UTC
Another issue is: the fetch command runs in mirror mode (-am), but the INDEX.bz2 file will be deleted after extracted. That does not make sense.

make -n fetchindex
/usr/bin/env  fetch -am -o /home/projects/freebsd-ports/INDEX-12.bz2 http://www.FreeBSD.org/ports/INDEX-12.bz2
bunzip2 < /home/projects/freebsd-ports/INDEX-12.bz2 > /home/projects/freebsd-ports/INDEX-12 &&  chmod a+r /home/projects/freebsd-ports/INDEX-12 && /bin/rm -f /home/projects/freebsd-ports/INDEX-12.bz2

To run fetch(1) in mirror mode, we need to keep the bzip2'd INDEX file.
Comment 2 Wolfram Schneider freebsd_committer freebsd_triage 2018-01-03 17:23:37 UTC
Created attachment 189374 [details]
/usr/ports/Makefile patch
Comment 3 Rene Ladan freebsd_committer freebsd_triage 2019-04-08 19:23:53 UTC
Reassign to portmgr
Comment 4 commit-hook freebsd_committer freebsd_triage 2021-11-08 15:00:33 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=cd8518854f3e96fb86dd7ed7b8f1c0258e839157

commit cd8518854f3e96fb86dd7ed7b8f1c0258e839157
Author:     Baptiste Daroussin <bapt@FreeBSD.org>
AuthorDate: 2021-11-08 14:58:02 +0000
Commit:     Baptiste Daroussin <bapt@FreeBSD.org>
CommitDate: 2021-11-08 14:59:55 +0000

    fetchindex: do not leave a corrupted INDEX

    When extracting the fetched index, do it in a temporary file, and only
    when extraction succeed, move it to the final file, it makes it more
    atomic and less likely to live a corrupted file behind

    PR:     224871

 Makefile | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)