This is a upstream report of https://lists.gnu.org/archive/html/bug-coreutils/2024-03/msg00045.html. Broken down we have the following problem: Consider this simple C program: > #include <sys/types.h> > #include <sys/stat.h> > #include <unistd.h> > #include <stdio.h> > #include <string.h> > > int main(int argc, char *argv[]) { > char *fs = argv[1]; > struct stat sb; > if (!stat(fs, &sb)) { > int major = major(sb.st_dev); > int minor = minor(sb.st_dev); > printf("%s:%ld, major: %d, minor: %d\n", fs, sb.st_dev, major, minor); > } > return 0; > } Compile it once on FreeBSD and Linux and run both on FreeBSD with the following UFS volumes mounted (not reproducible with ZFS or NFS mounts): > $ df -t ufs > Filesystem 1K-blocks Used Avail Capacity Mounted on > /dev/da0p2 2031132 629168 1239476 34% / > /dev/da0p4 4053308 245928 3483116 7% /tmp > /dev/da0p5 4053308 864504 2864540 23% /var > /dev/da0p6 8106716 798660 6659520 11% /var/tmp > /dev/da0p7 12180252 7877268 3328564 70% /usr > /dev/gvinum/local 24360604 16555848 5855908 74% /usr/local > /dev/gvinum/ports 48741436 23930092 20912032 53% /usr/ports > /dev/gvinum/obj 20307196 15515764 3166860 83% /usr/obj > /dev/gvinum/pgsql 32487548 5266684 24621864 18% /usr/local/pgsql > /dev/gvinum/nexus2 32487548 13744292 16144256 46% /var/nexus2 > /dev/gvinum/svn 8106716 5183236 2274944 69% /var/svn > /dev/gvinum/osipovmi 30450780 21861900 6152820 78% /var/osipovmi > /dev/gvinum/poudriere 64995324 37001360 22794340 62% /var/poudriere > /dev/gvinum/compat 4053308 984876 2744168 26% /compat > /dev/gvinum/bastille 81249212 59453292 15295984 80% /usr/local/bastille FreeBSD executable: > $ df -t ufs | cut -f 6 -w | sed 1d | xargs -I% ./stat-freebsd % > /:108, major: 0, minor: 108 > /tmp:110, major: 0, minor: 110 > /var:111, major: 0, minor: 111 > /var/tmp:112, major: 0, minor: 112 > /usr:113, major: 0, minor: 113 > /usr/local:161, major: 0, minor: 161 > /usr/ports:142, major: 0, minor: 142 > /usr/obj:141, major: 0, minor: 141 > /usr/local/pgsql:140, major: 0, minor: 140 > /var/nexus2:162, major: 0, minor: 162 > /var/svn:163, major: 0, minor: 163 > /var/osipovmi:164, major: 0, minor: 164 > /var/poudriere:166, major: 0, minor: 166 > /compat:165, major: 0, minor: 165 > /usr/local/bastille:139, major: 0, minor: 139 now the Linux executable: > $ file stat-linux > stat-linux: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=4f0f500b4e20d2a43770db4afa7e94c834d8ccca, not stripped > $ df -t ufs | cut -f 6 -w | sed 1d | xargs -I% ./stat-linux % > /:108, major: 0, minor: 108 > /tmp:165, major: 0, minor: 165 > /var:165, major: 0, minor: 165 > /var/tmp:112, major: 0, minor: 112 > /usr:165, major: 0, minor: 165 > /usr/local:161, major: 0, minor: 161 > /usr/ports:142, major: 0, minor: 142 > /usr/obj:141, major: 0, minor: 141 > /usr/local/pgsql:140, major: 0, minor: 140 > /var/nexus2:162, major: 0, minor: 162 > /var/svn:163, major: 0, minor: 163 > /var/osipovmi:164, major: 0, minor: 164 > /var/poudriere:166, major: 0, minor: 166 > /compat:165, major: 0, minor: 165 > /usr/local/bastille:139, major: 0, minor: 139 So if you use df(1) from /compat/linux those incorrectly FS are omitted: > $ /compat/linux/usr/bin/linux-df -a -B 512 -P -t ufs -T > Dateisystem Typ 512-Blöcke Benutzt Verfügbar Kapazität Eingehängt auf > /dev/da0p2 ufs 4062264 1258336 2478952 34% / > /dev/da0p4 ufs 8106616 1969752 5488336 27% /tmp > /dev/da0p5 - - - - - /var > /dev/da0p6 ufs 16213432 1597320 13319040 11% /var/tmp > /dev/da0p7 - - - - - /usr > /dev/gvinum/local ufs 48721208 33111696 11711816 74% /usr/local > /dev/gvinum/ports ufs 97482872 47860184 41824064 54% /usr/ports > /dev/gvinum/obj ufs 40614392 31031528 6333720 84% /usr/obj > /dev/gvinum/pgsql ufs 64975096 10533368 49243728 18% /usr/local/pgsql > /dev/gvinum/nexus2 ufs 64975096 27488584 32288512 46% /var/nexus2 > /dev/gvinum/svn ufs 16213432 10366472 4549888 70% /var/svn > /dev/gvinum/osipovmi ufs 60901560 43723800 12305640 79% /var/osipovmi > /dev/gvinum/poudriere ufs 129990648 74002720 45588680 62% /var/poudriere > /dev/gvinum/compat - - - - - /compat > /dev/gvinum/bastille ufs 162498424 118906584 30591968 80% /usr/local/bastille This can be reproduced with 13-STABLE: FreeBSD deblndw011x.ad001.siemens.net 13.3-STABLE FreeBSD 13.3-STABLE 77814c959 GENERIC amd64 and on 15-STABLE: FreeBSD deblndw013x5v.ad001.siemens.net 15.0-CURRENT FreeBSD 15.0-CURRENT #0 main-n268454-9097284b98be: Thu Feb 22 03:00:34 UTC 2024 root@releng3.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 I assume the bug is somewhere here: * https://github.com/freebsd/freebsd-src/blob/e29be07861173f87b2dd46db1d0c7cbcf72d4ad0/sys/compat/linux/linux_stats.c#L707 * https://github.com/freebsd/freebsd-src/blob/main/sys/compat/linux/linux.h#L55
You can also use stat(1) from FreeBSD userland or GNU userland to reproduce the problem with appropriate format string.
A simple 15-CURRENT vm: > root@deblndw013x5v:/usr # df -h > Filesystem Size Used Avail Capacity Mounted on > /dev/vtbd0p2 1.9G 1.5G 290M 84% / > devfs 1.0K 0B 1.0K 0% /dev > /dev/vtbd0p1 260M 1.3M 259M 1% /boot/efi > /dev/vtbd0p4 3.9G 33M 3.5G 1% /tmp > /dev/vtbd0p5 3.9G 171M 3.4G 5% /var > /dev/vtbd0p6 7.7G 64M 7.0G 1% /var/tmp > linprocfs 8.0K 0B 8.0K 0% /compat/linux/proc > linsysfs 8.0K 0B 8.0K 0% /compat/linux/sys > devfs 1.0K 0B 1.0K 0% /compat/linux/dev > fdescfs 1.0K 0B 1.0K 0% /compat/linux/dev/fd > tmpfs 11G 4.0K 11G 0% /compat/linux/dev/shm Linux executable: root@deblndw013x5v:/tmp # df -t ufs | cut -f 6 -w | sed 1d | xargs -I% ./stat-linux % /:52, major: 0, minor: 52 /tmp:54, major: 0, minor: 54 /var:52, major: 0, minor: 52 /var/tmp:56, major: 0, minor: 56 FreeBSD executable: > root@deblndw013x5v:/tmp # df -t ufs | cut -f 6 -w | sed 1d | xargs -I% ./stat-freebsd % > /:52, major: 0, minor: 52 > /tmp:54, major: 0, minor: 54 > /var:55, major: 0, minor: 55 > /var/tmp:56, major: 0, minor: 56
I am not sure what happens there. Most likely, you would need to add some printf()s in your linux64.ko to get more diagnostic there. First, look at the long comment at the start of sys/compat/linux/linux.h describing arrangements for Linux and FreeBSD dev_t <-> (major, minor) layouts. I do not see anything special in the value of e.g. 55 vs. 52 for the /var dev_t in the last message. I even wrote a simple check to ensure that I did not missed anything: #include <sys/param.h> #include <sys/conf.h> #include <stdbool.h> #include <stdio.h> #include "linux.h" int main(void) { dev_t d = 55; dev_t ld = linux_encode_dev(major(d), minor(d)); printf("%#lx %#lx %#x %#x\n", d, ld, linux_decode_major(ld), linux_decode_minor(ld)); } (you need to provide -I <path to src>/sys/compat/linux when compiling it. So perhaps check what is the dev_t for /var before linux_new_encode_dev() in linux_kern_statat() on your machine.
No change for d = 55: > $ ./test_stat > 0x37 0x37 0 0x37 I will set up a separate VM and add printf() statements. Something is fishly here. Will take me a few days.
Is the Linux binary finding /compat/linux/var instead of /var?
(In reply to John F. Carr from comment #5) John, this totally makes sense -- in fact that is the case: > $ stat -f %d / > 108 > $ stat -f %d /tmp > 110 > $ stat -f %d /var > 111 > $ stat -f %d /usr > 113 > $ stat -f %d /compat/linux/ > 165 > $ /compat/linux/bin/stat --format %d / > 108 > $ /compat/linux/bin/stat --format %d /tmp > 165 > $ /compat/linux/bin/stat --format %d /usr > 165 > $ /compat/linux/bin/stat --format %d /compat/linux > 165 > $ /compat/linux/bin/stat --format %d /usr/local > 161 > $ stat -f %d /usr/local > 161 man 4 linux says: > The path translation mechanism makes Linux processes look up file paths > under emul_path (defaulting to /compat/linux) before /. For example, > when Linux process attempts to open /etc/passwd, it will really access > /compat/linux/etc/passwd, unless the latter does not exist. This is used > to make sure Linux processes load Linux shared libraries instead of their > similarly-named FreeBSD counterparts, and also to provide alternative > versions of certain other files and virtual file systems. with: > $ ll /compat/linux/ > total 25 > lrwxr-xr-x 1 root wheel 7 2018-04-11 06:59 bin@ -> usr/bin > dr-xr-xr-x 34 root wheel 512 2024-02-20 19:19 dev/ > drwxr-xr-x 25 root wheel 2560 2024-02-20 18:50 etc/ > lrwxr-xr-x 1 root wheel 7 2018-04-11 06:59 lib@ -> usr/lib > lrwxr-xr-x 1 root wheel 9 2018-04-11 06:59 lib64@ -> usr/lib64 > drwxr-xr-x 4 root wheel 512 2022-01-17 09:44 opt/ > dr-xr-xr-x 1 root wheel 0 2024-03-20 14:07 proc/ > lrwxr-xr-x 1 root wheel 8 2024-02-16 20:28 run@ -> /var/run > lrwxr-xr-x 1 root wheel 8 2018-04-11 06:59 sbin@ -> usr/sbin > drwxr-xr-x 2 root wheel 512 2024-01-30 15:39 srv/ > dr-xr-xr-x 1 root wheel 0 2024-03-20 14:07 sys/ > drwxrwxrwt 7 root wheel 512 2023-05-25 15:52 tmp/ > drwxr-xr-x 8 root wheel 512 2024-02-20 18:50 usr/ > drwxr-xr-x 16 root wheel 512 2024-02-20 18:50 var/ Thanks John, I didn't think about this. Although my report looks like bogus then, but how to go from here? Do docs need improvement or something else or say "works as designed"?
IMHO, documentation can be improved around this, but this bug should be closed as "not a bug".
(In reply to Gleb Smirnoff from comment #7) Very likely, where would you imrove docs? Handbook? Manpages?
On Thu Mar 28 11:42:26 2024 UTC, michaelo@FreeBSD.org wrote: > Very likely, where would you imrove docs? Handbook? Manpages? Reading linux(4) carefully you can actually see this already documented. You may find a different file (and directory is a file) accessing the same path by a Linux ABI process compared to native ABI. This leads to all possible consequences of a different file, including this particular one - the st_dev. As you see some people see that in the manual page, like John, and some not, like you. This means the manual page potentially can be improved. I don't know exactly how. You can think of what exact info you'd want to add to make it more clear and then coordinate with a doc committer or just a native speaker to improve you manual page improvement and then commit it.
(In reply to Gleb Smirnoff from comment #9) I agree, I'll try to provide a hunk which is more exact.
I would like to close this PR if nobody objects. In effect the information is there at the very beginning of the manual page: DESCRIPTION The linux kernel module provides limited Linux ABI (application binary interface) compatibility, making it possible to run many unmodified Linux applications without the need for virtualization or emulation. Some of the facilities provided are: • Linux to native system call translation • Linux-specific system calls • Special signal handling for Linux processes • Path translation mechanism • Linux-specific virtual file systems The path translation mechanism makes Linux processes look up file paths under emul_path (defaulting to /compat/linux) before /. For example... I don't know how we can actually make it more explicit.
(In reply to Fernando Apesteguía from comment #11) I agree all of the information is in the man page, but do you think this would be a slight clarification: diff --git a/share/man/man4/linux.4 b/share/man/man4/linux.4 index 212dd2526f3f..b404c9e1c04d 100644 --- a/share/man/man4/linux.4 +++ b/share/man/man4/linux.4 @@ -63,9 +63,11 @@ before .Pa / . For example, when Linux process attempts to open .Pa /etc/passwd , -it will really access +it will first access .Pa /compat/linux/etc/passwd , -unless the latter does not exist. +falling back to +.Pa /etc/passwd +if the former does not exist. This is used to make sure Linux processes load Linux shared libraries instead of their similarly-named FreeBSD counterparts, and also to provide alternative versions of certain other files and virtual
(In reply to Ed Maste from comment #12) The change removes an inconsistency between the description (accessing /compat/linux **before** /) and the example that can be interpreted as accessing linux files **instead of** the FreeBSD counterparts. I think the improvement is small, but it is still and improvement :-) Ed, do you want to take care of it?
(In reply to Fernando Apesteguía from comment #13) I have the change staged locally and will push it
(In reply to Ed Maste from comment #12) I find "the former" confusing because the sentence is talking about three things (the original name, the compat name and what it does if the compat name doesn't exist) and "the former" is referring to the second thing. I think it would be clearer to explicitly say "if /compat/linux/etc/passwd does not exist." Also "when Linux process" is bad grammar. It should be "when a Linux process".
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=f66e71fa78e16164339f7fd4791306fb30165581 commit f66e71fa78e16164339f7fd4791306fb30165581 Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2024-08-19 13:43:37 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2024-08-19 14:14:28 +0000 linux.4: clarify path translation Try to be a little more explicit about the path translation mechanism accessing /compat/linux/<path> then falling back to /<path>. PR: 277804 Reviewed by: fernape Sponsored by: The FreeBSD Foundation share/man/man4/linux.4 | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=d1daec3d358eb5aaa38fa7c95fbfa330c46a69a1 commit d1daec3d358eb5aaa38fa7c95fbfa330c46a69a1 Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2024-08-19 14:26:26 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2024-08-19 14:29:19 +0000 linux.4: improve the path translation clarificiation As suggested by martin@lispworks.com, refer to the compat path explicitly, and correct an existing grammaro. PR: 277804 Fixes: f66e71fa78e1 ("linux.4: clarify path translation") Sponsored by: The FreeBSD Foundation share/man/man4/linux.4 | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
These manpage changes are really helpful, thank you guys. Can we MFC them? They are harmless, yet useful in stable branches as well.
(In reply to Michael Osipov from comment #18) Yes I will MFC the updates after several days.
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=226c733737383478f3c1d3f8c119207136c05de0 commit 226c733737383478f3c1d3f8c119207136c05de0 Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2024-08-19 13:43:37 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2024-08-26 13:22:03 +0000 linux.4: clarify path translation Try to be a little more explicit about the path translation mechanism accessing /compat/linux/<path> then falling back to /<path>. As suggested by martin@lispworks.com, refer to the compat path explicitly, and correct an existing grammaro. PR: 277804 Reviewed by: fernape Sponsored by: The FreeBSD Foundation (cherry picked from commit f66e71fa78e16164339f7fd4791306fb30165581) (cherry picked from commit d1daec3d358eb5aaa38fa7c95fbfa330c46a69a1) share/man/man4/linux.4 | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=6a847b6d1ee8ccf7e109d13bece2ad634672e6ff commit 6a847b6d1ee8ccf7e109d13bece2ad634672e6ff Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2024-08-19 13:43:37 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2024-08-26 13:22:58 +0000 linux.4: clarify path translation Try to be a little more explicit about the path translation mechanism accessing /compat/linux/<path> then falling back to /<path>. As suggested by martin@lispworks.com, refer to the compat path explicitly, and correct an existing grammaro. PR: 277804 Reviewed by: fernape Sponsored by: The FreeBSD Foundation (cherry picked from commit f66e71fa78e16164339f7fd4791306fb30165581) (cherry picked from commit d1daec3d358eb5aaa38fa7c95fbfa330c46a69a1) (cherry picked from commit 226c733737383478f3c1d3f8c119207136c05de0) share/man/man4/linux.4 | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
Committed and MFCed.