Bug 231810 - [build] release always fails with "mkimg: partition 2: No space left on device"
Summary: [build] release always fails with "mkimg: partition 2: No space left on device"
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: misc (show other bugs)
Version: 11.2-RELEASE
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-bugs mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-09-30 00:33 UTC by leeb
Modified: 2018-10-03 15:32 UTC (History)
1 user (show)

See Also:
koobs: mfc-stable11?


Attachments
svn diff for release/amd64/make-memstick.sh (602 bytes, patch)
2018-09-30 00:33 UTC, leeb
no flags Details | Diff
Screenshot (14.02 KB, image/png)
2018-10-02 20:54 UTC, leeb
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description leeb 2018-09-30 00:33:37 UTC
Created attachment 197609 [details]
svn diff for release/amd64/make-memstick.sh

11.2-RELEASE-p3 r338761

make release always produces this error for me:
Populating `mini-memstick.img.part'
Image `mini-memstick.img.part' complete
mkimg: partition 2: No space left on device
*** Error code 74

Stop.
make[1]: stopped in /usr/src/release
*** Error code 1

Stop.
make: stopped in /usr/src/release

The cause appears to be the nested use of mkimg, though why it can build memstick.img without problems I don't know.

Tested both images on physical hardware, begun installer, went to shell, unmame looked good.
Comment 1 Matthias Apitz 2018-09-30 05:41:49 UTC
I have the same problem in a recent CURRENT; here are the details:


# TMPDIR=/usr/tmp export TMPDIR
# ./make-memstick.sh /usr/local/r338641/root.r338641 /usr/local/r338641/memstick.im
+ makefs -B little -M 50331648b -m 50331648b -o 'label=FreeBSD_Install' -o 'version=2' /usr/local/r338641/memstick.im.part /usr/local/r338641/root.r338641
Calculated size of `/usr/local/r338641/memstick.im.part': 25769803776 bytes, 24530 inodes
Extent size set to 32768
/usr/local/r338641/memstick.im.part: 24576.0MB (50331648 sectors) block size 32768, fragment size 4096
        using 28 cylinder groups of 901.44MB, 28846 blks, 1024 inodes.
super-block backups (for fsck -b #) at:
      192,  1846336,  3692480,  5538624,  7384768,  9230912, 11077056,
 12923200, 14769344, 16615488, 18461632, 20307776, 22153920, 24000064,
 25846208, 27692352, 29538496, 31384640, 33230784, 35076928, 36923072,
 38769216, 40615360, 42461504, 44307648, 46153792, 47999936, 49846080,
Populating `/usr/local/r338641/memstick.im.part'
Image `/usr/local/r338641/memstick.im.part' complete
+ rm /usr/local/r338641/root.r338641/etc/fstab
+ rm /usr/local/r338641/root.r338641/etc/rc.conf.local
+ mkimg -C 28G -s mbr -b /usr/local/r338641/root.r338641/boot/mbr -p 'efi:=/usr/local/r338641/root.r338641/boot/boot1.efifat' -p 'freebsd:-mkimg -C 28G -s bsd -b /usr/local/r338641/root.r338641/boot/boot -p freebsd-ufs:=/usr/local/r338641/memstick.im.part' -a 2 -o /usr/local/r338641/memstick.im
...


While the cascade of mkimg(1) is running a *big* temp file is created,
which at the end eats up all memory of the disk:

# ls -lh /usr/local/r338641 /usr/tmp
/usr/local/r338641:
total 25172008
-rwxr-xr-x   1 root  wheel   1.3K Sep 29 16:41 make-memstick.sh
-rw-r--r--   1 root  wheel     0B Sep 29 16:50 memstick.im
-rw-r--r--   1 root  wheel    24G Sep 29 16:50 memstick.im.part
drwxr-xr-x  18 root  wheel   512B Sep 24 07:11 root.r338641

/usr/tmp:
total 11307168
-rw-------  1 root  wheel   456G Sep 29 17:01 mkimg-LmntlL    <<<******** 456G !!!
-rw-------  1 root  wheel     0B Sep 29 16:50 mkimg-yfU8Lr

/: write failed, filesystem is full

/: write failed, filesystem is full

But the 'memstick.im' is created fine at the end:

# ls -lh /usr/local/r338641 /usr/tmp
/usr/local/r338641:
total 2591752
-rwxr-xr-x   1 root  wheel   1.3K Sep 29 16:41 make-memstick.sh
-rw-r--r--   1 root  wheel    24G Sep 29 17:22 memstick.im
drwxr-xr-x  18 root  wheel   512B Sep 24 07:11 root.r338641

/usr/tmp:
total 0

And the UBS stick produced from 'memstick.im' with dd(1) boots fine and the root
file system has around 20 GB free space.

Why it is mkimg(1) creating such a big temp. file?

I glanced through the sources of src/usr.bin/mkimg and it seems that the big file
is some kind of swap space used by mkimg.
Comment 2 Matthias Apitz 2018-09-30 17:15:12 UTC
(In reply to leeb from comment #0)

I think, in your case it will help to set an env var to some file system with more space than one normally has below /tmp, for example:

# mkdir /usr/tmp
# TMPDIR=/usr/tmp export TMPDIR

AT least, this helped me for a tiny root file system to make an image off. 

But if you specify in the command mkfs(8) that your file system should have plenty space for additional thinks, it will create there in TMPDIR some swap file with giga o even terra bytes.

There is something broken with mkimg(1).

Some time ago, in March 2017 on r314251, this was all working fine and w/o any TMPDIR.
Comment 3 leeb 2018-10-01 01:21:33 UTC
My /tmp is 12G tmpfs.  Using the patch I supplied, it doesn't really go over 100M, just checking with frequent df -h in another console during the mkimg process.

This happens with a GENERIC kernel on a stock installation.  Nothing added.
cd /usr/src
make -j16 buildworld
make -j16 buildkernel
cd release
make release

I don't know why your mkfs is 24G, mine is more like 350M.

sh /usr/src/release/amd64/make-memstick.sh bootonly mini-memstick.img
Calculated size of `mini-memstick.img.part': 361332736 bytes, 11416 inodes
Extent size set to 32768
mini-memstick.img.part: 344.6MB (705728 sectors) block size 32768, fragment size 4096
        using 1 cylinder groups of 344.59MB, 11027 blks, 12672 inodes.
super-block backups (for fsck -b #) at:
 192,
Populating `mini-memstick.img.part'
Image `mini-memstick.img.part' complete
Comment 4 Matthias Apitz 2018-10-01 06:21:05 UTC
(In reply to leeb from comment #3)

My file system produced by mkfs(8) is around 24 GByte by intention because I added '-M 50331648b -m 50331648b' in the script to the parms of mkfs(8). The idea is to produce a file system in which I can install packages (a complete KDE4) for testing before I do move to the new CURRENT and my own packages produced with poudriere.

I applied your patch to my script make-memstick.sh and it now produced fine the memestick.img without any memory problems. It seems that the problems is caused by chaining the mkimg(1) calls, which your patch exactly addresses.

I will copy over the memstick.img to an USB key and test further.
Comment 5 leeb 2018-10-01 20:43:28 UTC
I pulled the disc1.iso from freebsd.org last night, installed it on a Xen VM (16GB RAM, 32GB Disk, no swap) and ran the make buldworld/buildkernel/release which went fine.  I binary updated it to -p4, same result was OK.

My build system (16GB RAM, 32GB Disk, no swap, tmpfs) was built from source and is currently -p3, no ports installed, no src.conf, no make.conf.  I'll src update it and see if -p4 works correctly or still exhibits the problem, which will narrow it to -p3 (unlikely) or my environment.

The Xen hosts are different.
Comment 6 leeb 2018-10-02 03:15:37 UTC
tmpfs was the cause of my problem.  After removing it the release worked correctly.

It was mounted on /tmp as rw,mode=01777,size=12g

Consider this closed from my perspective, but I'll leave it for Matthias.
Comment 7 Matthias Apitz 2018-10-02 05:03:13 UTC
I removed tmpfs from rc.conf, but the memory issue is still there.
When this proc chain is running:

mkimg -s mbr -b /usr/local/r338641/root.r338641/boot/mbr -p 'efi:=/usr/local/r338641/root.r338641/boot/boot1.efifat' -p 'freebsd:-mkimg -s bsd -b /usr/local/r338641/root.r338641/boot/boot -p freebsd-ufs:=/home/guru/zdata/r338641/memstick.im.part' -a 2 -o /home/guru/zdata/r338641/memstick.im

the temp file is growing and growing:

# ls -lh /usr/tmp
total 7683968
-rw-------  1 root  wheel     0B Oct  2 06:52 mkimg-8lenP0
-rw-------  1 root  wheel   390G Oct  2 07:00 mkimg-EfzEvr

...

# ls -lh /usr/tmp
total 9515808
-rw-------  1 root  wheel     0B Oct  2 06:52 mkimg-8lenP0
-rw-------  1 root  wheel   408G Oct  2 07:01 mkimg-EfzEvr

Please re-open the PR again for CURRENT.
Comment 8 leeb 2018-10-02 06:12:47 UTC
Per comment #7
Comment 9 Matthias Apitz 2018-10-02 12:25:39 UTC
(In reply to leeb from comment #6)

Lee, could you please do a test for double check: modify the script

/usr/src/release/amd64/make-memstick.sh

and insert size values in the line for mkfs like this; change the line:

makefs -B little ...

to:

makefs -B little -M 50331648b -m 50331648b ...

Be prepared for the target memstick.img needs ~25 GByte
Comment 10 leeb 2018-10-02 20:33:27 UTC
That created a 219G file on /tmp which is on a 10G partition
Comment 11 leeb 2018-10-02 20:54:09 UTC
Created attachment 197721 [details]
Screenshot
Comment 12 Matthias Apitz 2018-10-03 08:26:32 UTC
(In reply to leeb from comment #10)

How is this possible, a 219G file in a 10G partition? What type of file is this?
Comment 13 leeb 2018-10-03 15:32:15 UTC
Sorry, I had increased the size of the partition that /usr/src and /usr/obj were on, but forgot /tmp was on my root partition.  They are all UFS.  The src and obj are nullfs mounts.

I suppose a sparse file would act differently if it were piped vs. stored?  That's the only difference that patch makes.  The only reason I did that was to better isolate my debug output, but it didn't help in that respect.