Bug 253946 - Cirrus-CI runs encountering disk full failures
Summary: Cirrus-CI runs encountering disk full failures
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: misc (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-03-01 21:33 UTC by Ed Maste
Modified: 2021-03-02 03:10 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Ed Maste freebsd_committer 2021-03-01 21:33:10 UTC
For some time Cirrus-CI runs of the FreeBSD base system src tree (and forks) have been failing due to a full disk. These runs use standard FreeBSD GCP images.

https://cirrus-ci.com/github/freebsd/freebsd-src/

Failure snippet:

--- all_subdir_qlnx ---
ERROR: ctfconvert: qlnxr_cm.o.ctf: Cannot finalize temp file: I/O error: No space left on device
--- all_subdir_vmm ---
fatal error: error in backend: IO failure on output stream: No space left on device
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.	Program arguments: /usr/local/llvm11/bin/clang -target x86_64-unknown-freebsd14.0 --sysroot=/usr/obj/tmp/cirrus-ci-build/amd64.amd64/tmp -B/var/empty -O2 -pipe -fno-common -DVMM_KEEP_STATS -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -DKLD_TIED -nostdinc -I/tmp/cirrus-ci-build/sys/amd64/vmm -I/tmp/cirrus-ci-build/sys/amd64/vmm/io -I/tmp/cirrus-ci-build/sys/amd64/vmm/intel -I/tmp/cirrus-ci-build/sys/amd64/vmm/amd -DHAVE_KERNEL_OPTION_HEADERS -include /usr/obj/tmp/cirrus-ci-build/amd64.amd64/sys/GENERIC/opt_global.h -I. -I/tmp/cirrus-ci-build/sys -I/tmp/cirrus-ci-build/sys/contrib/ck/include -fno-common -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fdebug-prefix-map=./machine=/tmp/cirrus-ci-build/sys/amd64/include -fdebug-prefix-map=./x86=/tmp/cirrus-ci-build/sys/x86/include -I/usr/obj/tmp/cirrus-ci-build/amd64.amd64/sys/GENERIC -MD -MF.depend.vmm.o -MTvmm.o -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -Wno-error-unused-function -Wno-error-pointer-sign -Wno-error-shift-negative-value -Wno-address-of-packed-member -Wno-format-zero-length -mno-aes -mno-avx -std=iso9899:1999 -c /tmp/cirrus-ci-build/sys/amd64/vmm/vmm.c -o vmm.o 

I'm not sure of the exact bounds but main at 5f1b1f184b7f12330cf4a027e3db7c6700c67640 was green for me and 1bf86687c2eb744497b6d9fd02b80e9821b1f4d6 was red. There does not appear to be anything particularly interesting in this range, so I suspect that a change in the GCP image or the installed ports is responsible.
Comment 1 Ed Maste freebsd_committer 2021-03-01 21:34:44 UTC
Passing case:

New packages to be INSTALLED:
	adwaita-icon-theme: 3.38.0
	argyllcms: 1.9.2_5
	at-spi2-atk: 2.34.2
	at-spi2-core: 2.36.0
	atk: 2.36.0
	avahi-app: 0.8
	bash: 5.1.4_1
	bash-completion: 2.11,2
	cairo: 1.16.0_1,3
	colord: 1.3.5_1
	cups: 2.3.3op2
	cyrus-sasl: 2.1.27_1
	dbus: 1.12.20_3
	dbus-glib: 0.110
	dejavu: 2.37_1
	dtc: 1.6.0
	encodings: 1.0.5,1
	font-bh-ttf: 1.0.3_4
	font-misc-ethiopic: 1.0.4
	font-misc-meltho: 1.0.3_4
	fontconfig: 2.13.93,1
	freetype2: 2.10.4
	fribidi: 1.0.10
	gdbm: 1.19
	gdk-pixbuf2: 2.40.0
	glib: 2.66.7,1
	gnome_subr: 1.0
	gnutls: 3.6.15
	graphite2: 1.3.14
	gtk-update-icon-cache: 3.24.24
	gtk3: 3.24.24
	harfbuzz: 2.7.4
	hicolor-icon-theme: 0.17
	jbigkit: 2.1_1
	jpeg-turbo: 2.0.6
	lcms2: 2.12
	libICE: 1.0.10,1
	libSM: 1.2.3,1
	libX11: 1.6.12,1
	libXScrnSaver: 1.2.3_2
	libXau: 1.0.9
	libXcomposite: 0.4.5,1
	libXcursor: 1.2.0
	libXdamage: 1.1.5
	libXdmcp: 1.1.3
	libXext: 1.3.4,1
	libXfixes: 5.0.3_2
	libXft: 2.3.3
	libXi: 1.7.10,1
	libXinerama: 1.1.4_2,1
	libXrandr: 1.5.2
	libXrender: 0.9.10_2
	libXtst: 1.2.3_2
	libXxf86vm: 1.1.4_3
	libdaemon: 0.14_1
	libdrm: 2.4.104,1
	libedit: 3.1.20191231,1
	libepoll-shim: 0.0.20210213
	libepoxy: 1.5.4
	libevent: 2.1.12
	libffi321: 3.2.1_2
	libfontenc: 1.1.4
	libgsf: 1.14.47_1
	libidn2: 2.3.0_1
	liblz4: 1.9.3,1
	libpaper: 1.1.24.4
	libpcap: 1.10.0
	libpciaccess: 0.16
	libpthread-stubs: 0.4
	librsvg2-rust: 2.50.3_1
	libtasn1: 4.16.0_1
	libunistring: 0.9.10_1
	libunwind: 20201110
	libxcb: 1.14_1
	libxkbcommon: 1.0.3
	libxml2: 2.9.10_3
	libxshmfence: 1.3
	libyaml: 0.2.5
	llvm11: 11.0.1
	lua52: 5.2.4
	mesa-libs: 20.2.3
	mkfontscale: 1.2.1
	nettle: 3.6
	nspr: 4.29
	p11-kit: 0.23.22
	pango: 1.42.4_5
	pciids: 20210131
	pcre: 8.44
	pcre2: 10.36
	perl5: 5.32.1_1
	pixman: 0.40.0_1
	png: 1.6.37_1
	polkit: 0.118
	qemu42: 4.2.1
	sdl2: 2.0.12_3
	shared-mime-info: 2.0
	spidermonkey78: 78.7.0_1
	tiff: 4.2.0
	tpm-emulator: 0.7.4_2
	trousers: 0.3.14_3
	uefi-edk2-qemu-x86_64: g20191122
	vde2: 2.3.2_5
	vte3: 0.62.1_1
	wayland: 1.19.0
	wayland-protocols: 1.20
	xkeyboard-config: 2.31
	xorg-fonts-truetype: 7.7_1
	xorgproto: 2020.1
	zstd: 1.4.8
Installed packages to be UPGRADED:
	boost-libs: 1.72.0_2 -> 1.72.0_3
	icu: 67.1,1 -> 68.2,1
	libffi: 3.2.1_3 -> 3.3_1

Failing case:

New packages to be INSTALLED:
	adwaita-icon-theme: 3.38.0
	argyllcms: 1.9.2_5
	at-spi2-atk: 2.34.2
	at-spi2-core: 2.36.0
	atk: 2.36.0
	avahi-app: 0.8
	bash: 5.1.4_1
	bash-completion: 2.11,2
	cairo: 1.16.0_1,3
	colord: 1.3.5_1
	cups: 2.3.3op2
	cyrus-sasl: 2.1.27_1
	dbus: 1.12.20_3
	dbus-glib: 0.110
	dejavu: 2.37_1
	dtc: 1.6.0
	encodings: 1.0.5,1
	font-bh-ttf: 1.0.3_4
	font-misc-ethiopic: 1.0.4
	font-misc-meltho: 1.0.3_4
	fontconfig: 2.13.93,1
	freetype2: 2.10.4
	fribidi: 1.0.10
	gdbm: 1.19
	gdk-pixbuf2: 2.40.0
	glib: 2.66.7,1
	gnome_subr: 1.0
	gnutls: 3.6.15
	graphite2: 1.3.14
	gtk-update-icon-cache: 3.24.24
	gtk3: 3.24.24
	harfbuzz: 2.7.4
	hicolor-icon-theme: 0.17
	jbigkit: 2.1_1
	jpeg-turbo: 2.0.6
	lcms2: 2.12
	libICE: 1.0.10,1
	libSM: 1.2.3,1
	libX11: 1.6.12,1
	libXScrnSaver: 1.2.3_2
	libXau: 1.0.9
	libXcomposite: 0.4.5,1
	libXcursor: 1.2.0
	libXdamage: 1.1.5
	libXdmcp: 1.1.3
	libXext: 1.3.4,1
	libXfixes: 5.0.3_2
	libXft: 2.3.3
	libXi: 1.7.10,1
	libXinerama: 1.1.4_2,1
	libXrandr: 1.5.2
	libXrender: 0.9.10_2
	libXtst: 1.2.3_2
	libXxf86vm: 1.1.4_3
	libdaemon: 0.14_1
	libdrm: 2.4.104,1
	libedit: 3.1.20191231,1
	libepoll-shim: 0.0.20210213
	libepoxy: 1.5.4
	libevent: 2.1.12
	libffi321: 3.2.1_2
	libfontenc: 1.1.4
	libgsf: 1.14.47_1
	libidn2: 2.3.0_1
	liblz4: 1.9.3,1
	libpaper: 1.1.24.4
	libpcap: 1.10.0
	libpciaccess: 0.16
	libpthread-stubs: 0.4
	librsvg2-rust: 2.50.3_1
	libtasn1: 4.16.0_1
	libunistring: 0.9.10_1
	libunwind: 20201110
	libxcb: 1.14_1
	libxkbcommon: 1.0.3
	libxml2: 2.9.10_3
	libxshmfence: 1.3
	libyaml: 0.2.5
	llvm11: 11.0.1
	lua52: 5.2.4
	mesa-libs: 20.2.3
	mkfontscale: 1.2.1
	nettle: 3.6
	nspr: 4.29
	p11-kit: 0.23.22
	pango: 1.42.4_5
	pciids: 20210131
	pcre: 8.44
	pcre2: 10.36
	perl5: 5.32.1_1
	pixman: 0.40.0_1
	png: 1.6.37_1
	polkit: 0.118
	qemu42: 4.2.1
	sdl2: 2.0.12_3
	shared-mime-info: 2.0
	spidermonkey78: 78.7.0_1
	tiff: 4.2.0
	tpm-emulator: 0.7.4_2
	trousers: 0.3.14_3
	uefi-edk2-qemu-x86_64: g20191122
	vde2: 2.3.2_5
	vte3: 0.62.1_1
	wayland: 1.19.0
	wayland-protocols: 1.20
	xkeyboard-config: 2.31
	xorg-fonts-truetype: 7.7_1
	xorgproto: 2020.1
	zstd: 1.4.8
Installed packages to be UPGRADED:
	boost-libs: 1.72.0_2 -> 1.72.0_3
	icu: 67.1,1 -> 68.2,1
	libffi: 3.2.1_3 -> 3.3_1
Comment 2 Ed Maste freebsd_committer 2021-03-01 21:36:01 UTC
Packages and versions are unchanged
Comment 3 Ed Maste freebsd_committer 2021-03-02 03:10:14 UTC
I've temporarily stopped installing qemu and running the smoke test:

---
commit a7d593dd1da27833b5384349700bc3c7bcae6aad
Author: Ed Maste <emaste@FreeBSD.org>
Date:   Mon Mar 1 16:39:25 2021 -0500

    Cirrus-CI: temporarily avoid qemu smoke test boot
    
    Cirrus-CI has been red for some time because we're running out of disk
    space on the ephemeral GCP VMs.  For now remove the package + qemu boot,
    and just check for build regressions.
    
    This change to be reverted once we have identified and addressed the
    underlying issue.
    
    Sponsored by:   The FreeBSD Foundation
---

It looks like we were probably very close to the limit before and some uninteresting change pushed it over. After the commit above it seems there is 9.5GB free on the VM after installing packages (excl qemu and its dependencies), and /usr/obj is 8.5GB.