This is a upstream report of https://lists.gnu.org/archive/html/bug-coreutils/2024-03/msg00045.html. Broken down we have the following problem: Consider this simple C program: > #include <sys/types.h> > #include <sys/stat.h> > #include <unistd.h> > #include <stdio.h> > #include <string.h> > > int main(int argc, char *argv[]) { > char *fs = argv[1]; > struct stat sb; > if (!stat(fs, &sb)) { > int major = major(sb.st_dev); > int minor = minor(sb.st_dev); > printf("%s:%ld, major: %d, minor: %d\n", fs, sb.st_dev, major, minor); > } > return 0; > } Compile it once on FreeBSD and Linux and run both on FreeBSD with the following UFS volumes mounted (not reproducible with ZFS or NFS mounts): > $ df -t ufs > Filesystem 1K-blocks Used Avail Capacity Mounted on > /dev/da0p2 2031132 629168 1239476 34% / > /dev/da0p4 4053308 245928 3483116 7% /tmp > /dev/da0p5 4053308 864504 2864540 23% /var > /dev/da0p6 8106716 798660 6659520 11% /var/tmp > /dev/da0p7 12180252 7877268 3328564 70% /usr > /dev/gvinum/local 24360604 16555848 5855908 74% /usr/local > /dev/gvinum/ports 48741436 23930092 20912032 53% /usr/ports > /dev/gvinum/obj 20307196 15515764 3166860 83% /usr/obj > /dev/gvinum/pgsql 32487548 5266684 24621864 18% /usr/local/pgsql > /dev/gvinum/nexus2 32487548 13744292 16144256 46% /var/nexus2 > /dev/gvinum/svn 8106716 5183236 2274944 69% /var/svn > /dev/gvinum/osipovmi 30450780 21861900 6152820 78% /var/osipovmi > /dev/gvinum/poudriere 64995324 37001360 22794340 62% /var/poudriere > /dev/gvinum/compat 4053308 984876 2744168 26% /compat > /dev/gvinum/bastille 81249212 59453292 15295984 80% /usr/local/bastille FreeBSD executable: > $ df -t ufs | cut -f 6 -w | sed 1d | xargs -I% ./stat-freebsd % > /:108, major: 0, minor: 108 > /tmp:110, major: 0, minor: 110 > /var:111, major: 0, minor: 111 > /var/tmp:112, major: 0, minor: 112 > /usr:113, major: 0, minor: 113 > /usr/local:161, major: 0, minor: 161 > /usr/ports:142, major: 0, minor: 142 > /usr/obj:141, major: 0, minor: 141 > /usr/local/pgsql:140, major: 0, minor: 140 > /var/nexus2:162, major: 0, minor: 162 > /var/svn:163, major: 0, minor: 163 > /var/osipovmi:164, major: 0, minor: 164 > /var/poudriere:166, major: 0, minor: 166 > /compat:165, major: 0, minor: 165 > /usr/local/bastille:139, major: 0, minor: 139 now the Linux executable: > $ file stat-linux > stat-linux: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=4f0f500b4e20d2a43770db4afa7e94c834d8ccca, not stripped > $ df -t ufs | cut -f 6 -w | sed 1d | xargs -I% ./stat-linux % > /:108, major: 0, minor: 108 > /tmp:165, major: 0, minor: 165 > /var:165, major: 0, minor: 165 > /var/tmp:112, major: 0, minor: 112 > /usr:165, major: 0, minor: 165 > /usr/local:161, major: 0, minor: 161 > /usr/ports:142, major: 0, minor: 142 > /usr/obj:141, major: 0, minor: 141 > /usr/local/pgsql:140, major: 0, minor: 140 > /var/nexus2:162, major: 0, minor: 162 > /var/svn:163, major: 0, minor: 163 > /var/osipovmi:164, major: 0, minor: 164 > /var/poudriere:166, major: 0, minor: 166 > /compat:165, major: 0, minor: 165 > /usr/local/bastille:139, major: 0, minor: 139 So if you use df(1) from /compat/linux those incorrectly FS are omitted: > $ /compat/linux/usr/bin/linux-df -a -B 512 -P -t ufs -T > Dateisystem Typ 512-Blöcke Benutzt Verfügbar Kapazität Eingehängt auf > /dev/da0p2 ufs 4062264 1258336 2478952 34% / > /dev/da0p4 ufs 8106616 1969752 5488336 27% /tmp > /dev/da0p5 - - - - - /var > /dev/da0p6 ufs 16213432 1597320 13319040 11% /var/tmp > /dev/da0p7 - - - - - /usr > /dev/gvinum/local ufs 48721208 33111696 11711816 74% /usr/local > /dev/gvinum/ports ufs 97482872 47860184 41824064 54% /usr/ports > /dev/gvinum/obj ufs 40614392 31031528 6333720 84% /usr/obj > /dev/gvinum/pgsql ufs 64975096 10533368 49243728 18% /usr/local/pgsql > /dev/gvinum/nexus2 ufs 64975096 27488584 32288512 46% /var/nexus2 > /dev/gvinum/svn ufs 16213432 10366472 4549888 70% /var/svn > /dev/gvinum/osipovmi ufs 60901560 43723800 12305640 79% /var/osipovmi > /dev/gvinum/poudriere ufs 129990648 74002720 45588680 62% /var/poudriere > /dev/gvinum/compat - - - - - /compat > /dev/gvinum/bastille ufs 162498424 118906584 30591968 80% /usr/local/bastille This can be reproduced with 13-STABLE: FreeBSD deblndw011x.ad001.siemens.net 13.3-STABLE FreeBSD 13.3-STABLE 77814c959 GENERIC amd64 and on 15-STABLE: FreeBSD deblndw013x5v.ad001.siemens.net 15.0-CURRENT FreeBSD 15.0-CURRENT #0 main-n268454-9097284b98be: Thu Feb 22 03:00:34 UTC 2024 root@releng3.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 I assume the bug is somewhere here: * https://github.com/freebsd/freebsd-src/blob/e29be07861173f87b2dd46db1d0c7cbcf72d4ad0/sys/compat/linux/linux_stats.c#L707 * https://github.com/freebsd/freebsd-src/blob/main/sys/compat/linux/linux.h#L55
You can also use stat(1) from FreeBSD userland or GNU userland to reproduce the problem with appropriate format string.
A simple 15-CURRENT vm: > root@deblndw013x5v:/usr # df -h > Filesystem Size Used Avail Capacity Mounted on > /dev/vtbd0p2 1.9G 1.5G 290M 84% / > devfs 1.0K 0B 1.0K 0% /dev > /dev/vtbd0p1 260M 1.3M 259M 1% /boot/efi > /dev/vtbd0p4 3.9G 33M 3.5G 1% /tmp > /dev/vtbd0p5 3.9G 171M 3.4G 5% /var > /dev/vtbd0p6 7.7G 64M 7.0G 1% /var/tmp > linprocfs 8.0K 0B 8.0K 0% /compat/linux/proc > linsysfs 8.0K 0B 8.0K 0% /compat/linux/sys > devfs 1.0K 0B 1.0K 0% /compat/linux/dev > fdescfs 1.0K 0B 1.0K 0% /compat/linux/dev/fd > tmpfs 11G 4.0K 11G 0% /compat/linux/dev/shm Linux executable: root@deblndw013x5v:/tmp # df -t ufs | cut -f 6 -w | sed 1d | xargs -I% ./stat-linux % /:52, major: 0, minor: 52 /tmp:54, major: 0, minor: 54 /var:52, major: 0, minor: 52 /var/tmp:56, major: 0, minor: 56 FreeBSD executable: > root@deblndw013x5v:/tmp # df -t ufs | cut -f 6 -w | sed 1d | xargs -I% ./stat-freebsd % > /:52, major: 0, minor: 52 > /tmp:54, major: 0, minor: 54 > /var:55, major: 0, minor: 55 > /var/tmp:56, major: 0, minor: 56
I am not sure what happens there. Most likely, you would need to add some printf()s in your linux64.ko to get more diagnostic there. First, look at the long comment at the start of sys/compat/linux/linux.h describing arrangements for Linux and FreeBSD dev_t <-> (major, minor) layouts. I do not see anything special in the value of e.g. 55 vs. 52 for the /var dev_t in the last message. I even wrote a simple check to ensure that I did not missed anything: #include <sys/param.h> #include <sys/conf.h> #include <stdbool.h> #include <stdio.h> #include "linux.h" int main(void) { dev_t d = 55; dev_t ld = linux_encode_dev(major(d), minor(d)); printf("%#lx %#lx %#x %#x\n", d, ld, linux_decode_major(ld), linux_decode_minor(ld)); } (you need to provide -I <path to src>/sys/compat/linux when compiling it. So perhaps check what is the dev_t for /var before linux_new_encode_dev() in linux_kern_statat() on your machine.
No change for d = 55: > $ ./test_stat > 0x37 0x37 0 0x37 I will set up a separate VM and add printf() statements. Something is fishly here. Will take me a few days.
Is the Linux binary finding /compat/linux/var instead of /var?
(In reply to John F. Carr from comment #5) John, this totally makes sense -- in fact that is the case: > $ stat -f %d / > 108 > $ stat -f %d /tmp > 110 > $ stat -f %d /var > 111 > $ stat -f %d /usr > 113 > $ stat -f %d /compat/linux/ > 165 > $ /compat/linux/bin/stat --format %d / > 108 > $ /compat/linux/bin/stat --format %d /tmp > 165 > $ /compat/linux/bin/stat --format %d /usr > 165 > $ /compat/linux/bin/stat --format %d /compat/linux > 165 > $ /compat/linux/bin/stat --format %d /usr/local > 161 > $ stat -f %d /usr/local > 161 man 4 linux says: > The path translation mechanism makes Linux processes look up file paths > under emul_path (defaulting to /compat/linux) before /. For example, > when Linux process attempts to open /etc/passwd, it will really access > /compat/linux/etc/passwd, unless the latter does not exist. This is used > to make sure Linux processes load Linux shared libraries instead of their > similarly-named FreeBSD counterparts, and also to provide alternative > versions of certain other files and virtual file systems. with: > $ ll /compat/linux/ > total 25 > lrwxr-xr-x 1 root wheel 7 2018-04-11 06:59 bin@ -> usr/bin > dr-xr-xr-x 34 root wheel 512 2024-02-20 19:19 dev/ > drwxr-xr-x 25 root wheel 2560 2024-02-20 18:50 etc/ > lrwxr-xr-x 1 root wheel 7 2018-04-11 06:59 lib@ -> usr/lib > lrwxr-xr-x 1 root wheel 9 2018-04-11 06:59 lib64@ -> usr/lib64 > drwxr-xr-x 4 root wheel 512 2022-01-17 09:44 opt/ > dr-xr-xr-x 1 root wheel 0 2024-03-20 14:07 proc/ > lrwxr-xr-x 1 root wheel 8 2024-02-16 20:28 run@ -> /var/run > lrwxr-xr-x 1 root wheel 8 2018-04-11 06:59 sbin@ -> usr/sbin > drwxr-xr-x 2 root wheel 512 2024-01-30 15:39 srv/ > dr-xr-xr-x 1 root wheel 0 2024-03-20 14:07 sys/ > drwxrwxrwt 7 root wheel 512 2023-05-25 15:52 tmp/ > drwxr-xr-x 8 root wheel 512 2024-02-20 18:50 usr/ > drwxr-xr-x 16 root wheel 512 2024-02-20 18:50 var/ Thanks John, I didn't think about this. Although my report looks like bogus then, but how to go from here? Do docs need improvement or something else or say "works as designed"?
IMHO, documentation can be improved around this, but this bug should be closed as "not a bug".
(In reply to Gleb Smirnoff from comment #7) Very likely, where would you imrove docs? Handbook? Manpages?
On Thu Mar 28 11:42:26 2024 UTC, michaelo@FreeBSD.org wrote: > Very likely, where would you imrove docs? Handbook? Manpages? Reading linux(4) carefully you can actually see this already documented. You may find a different file (and directory is a file) accessing the same path by a Linux ABI process compared to native ABI. This leads to all possible consequences of a different file, including this particular one - the st_dev. As you see some people see that in the manual page, like John, and some not, like you. This means the manual page potentially can be improved. I don't know exactly how. You can think of what exact info you'd want to add to make it more clear and then coordinate with a doc committer or just a native speaker to improve you manual page improvement and then commit it.
(In reply to Gleb Smirnoff from comment #9) I agree, I'll try to provide a hunk which is more exact.