Bug 244962

Summary: emulators/virtualbox-ose-additions: mount -t vboxvfs ⋯: Operation not supported by device
Product: Ports & Packages Reporter: mikhail.rokhin
Component: Individual Port(s)Assignee: Graham Perrin <grahamperrin>
Status: Closed Overcome By Events    
Severity: Affects Many People CC: grahamperrin, groenveld, jkim, jlduran, mjg, mmokhi, vbox
Priority: ---    
Version: Latest   
Hardware: Any   
OS: Any   
URL: https://github.com/freebsd/freebsd-ports/commit/b8cb39c59f206c07bc52c9196bff389cf8ceecf7#diff-09a804455f0e1f154f95eb352b6298e1df1bad38679897eb14c8209dd01db303R37
See Also: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=249377
Attachments:
Description Flags
stack trace screen snapshot none

Description mikhail.rokhin 2020-03-21 21:30:00 UTC
When use `mount_vboxvfs ...` it shows:
link_elf_obj: symbol bcmp undefined
linker_load_file: /boot/modules/vboxvfs.ko - unsupported file type

FreeBSD 13-Current
Additions: 5.2.34_2 (installed via pkg, not compiled)
Comment 1 Yuri Pankov 2020-03-22 19:53:34 UTC
Do versions in output of `sysctl kern.osreldate` and `pkg query %Av virtualbox-ose-additions` match?  Try rebuilding from ports if they don't.
Comment 2 mikhail.rokhin 2020-03-23 13:36:17 UTC
(In reply to Yuri Pankov from comment #1)
Identical: 1300084
Comment 3 groenveld 2020-03-27 13:38:00 UTC
The kernel does not offer bcmp:
$ nm /boot/kernel/kernel|grep -i bcmp
ffffffff80cb4610 T timingsafe_bcmp

The port links vboxvfs.ko with a library that references bcmp
$ cat /usr/ports/emulators/virtualbox-ose-additions/work/VirtualBox-5.2.34/out/freebsd.amd64/release/obj/vboxvfs/vboxvfs.dep
define vboxvfs_CMDS_PREV_

@kmk_builtin_rm -f -- /usr/ports/emulators/virtualbox-ose-additions/work/VirtualBox-5.2.34/out/freebsd.amd64/release/obj/vboxvfs/vboxvfs.dep /usr/ports/emulators/virtualbox-ose-additions/work/VirtualBox-5.2.34/out/freebsd.amd64/release/obj/vboxvfs/vboxvfs.ko  /usr/ports/emulators/virtualbox-ose-additions/work/VirtualBox-5.2.34/out/freebsd.amd64/release/obj/vboxvfs/vboxvfs.map
ld               -d --warn-common   -m elf_x86_64_fbsd   -r -o  /usr/ports/emulators/virtualbox-ose-additions/work/VirtualBox-5.2.34/out/freebsd.amd64/release/obj/vboxvfs/vboxvfs.ko /usr/ports/emulators/virtualbox-ose-additions/work/VirtualBox-5.2.34/out/freebsd.amd64/release/obj/vboxvfs/vboxvfs_vfsops.o /usr/ports/emulators/virtualbox-ose-additions/work/VirtualBox-5.2.34/out/freebsd.amd64/release/obj/vboxvfs/vboxvfs_vnops.o /usr/ports/emulators/virtualbox-ose-additions/work/VirtualBox-5.2.34/out/freebsd.amd64/release/obj/vboxvfs/vboxvfs_prov.o    /usr/ports/emulators/virtualbox-ose-additions/work/VirtualBox-5.2.34/out/freebsd.amd64/release/lib/additions/VBoxGuestR0Lib.a   /usr/ports/emulators/virtualbox-ose-additions/work/VirtualBox-5.2.34/out/freebsd.amd64/release/lib/additions/RuntimeGuestR0.a
awk -f /usr/src/sys/conf/kmod_syms.awk  /usr/ports/emulators/virtualbox-ose-additions/work/VirtualBox-5.2.34/out/freebsd.amd64/release/obj/vboxvfs/vboxvfs.ko /dev/null | xargs -J% objcopy % /usr/ports/emulators/virtualbox-ose-additions/work/VirtualBox-5.2.34/out/freebsd.amd64/release/obj/vboxvfs/vboxvfs.ko
endef
$ nm /usr/ports/emulators/virtualbox-ose-additions/work/VirtualBox-5.2.34/out/freebsd.amd64/release/lib/additions/RuntimeGuestR0.a | grep -B13 bcmp
strformatrt.o:
                 U VBoxGuest_RTStrFormat
                 U VBoxGuest_RTStrFormatNumber
0000000000000000 r _ZL13g_szHexDigits
0000000000002060 t _ZL15rtstrFormatIPv6PFmPvPKcmES_PK10RTUINT128U
0000000000000000 r _ZZ13rtstrFormatRtE12s_wszBigNull
0000000000000280 r _ZZ13rtstrFormatRtE8s_aTypes
0000000000000870 r _ZZ13rtstrFormatRtE8s_aTypes_0
0000000000000850 r _ZZ13rtstrFormatRtE8s_szNull
0000000000000857 r _ZZ13rtstrFormatRtE8s_szTrue
000000000000085d r _ZZ13rtstrFormatRtE9s_szFalse
                 U __start_set_pcpu
                 U __stop_set_pcpu
                 U bcmp

Someone with an understanding of FreeBSD kernel modules will have to explain where things went sideways.
John
groenveld@acm.org
Comment 4 Mateusz Guzik freebsd_committer freebsd_triage 2020-03-27 21:32:33 UTC
There is no bcmp symbol on amd64 for almost 2 years now. In sys/systm.h there is:

int     bcmp(const void *b1, const void *b2, size_t len);
[..]
#define bcmp(b1, b2, len) __builtin_memcmp((b1), (b2), (len))

The first line should probably get removed, but it should not be affecting vbox.

vbox likes to *disable* several warnings. Thus what most likely happens is that some place using bcmp *fails* to include the systm.h header, due to warns disabled nobody is even told and the compiler ends up generating a call to bcmp (instead of memcmp, or no call in the first place).

Thus someone(tm) should make sure the header is included in files which end up doing bcmp. Also will not hurt to make sure the build environment is clean and up to date -- if thep ort pulls in *OLD* files it will only see bcmp declared, but no macro telling it to use __builtin_memcmp and then it will stick to generating the call.
Comment 5 mikhail.rokhin 2020-04-05 16:59:14 UTC
Link to VB bugtracker: https://www.virtualbox.org/ticket/19420
Comment 6 Jung-uk Kim freebsd_committer freebsd_triage 2020-04-06 17:48:42 UTC
(In reply to mikhail.rokhin from comment #5)
Note the driver is ours, not theirs.

https://svnweb.freebsd.org/changeset/ports/446145

Please close the upstream PR.
Comment 7 Jung-uk Kim freebsd_committer freebsd_triage 2020-04-06 17:53:24 UTC
Mahdi, can you please take a look?
Comment 8 groenveld 2020-04-06 20:15:47 UTC
The invocation of bcmp(3) appears to be hidden in strformatrt.cpp in this switch:
//                                /*
//                                 * Regular hex dump with dittoing.
//                                 */
//                                case 'D':

John
groenveld@acm.org
Comment 9 Mateusz Guzik freebsd_committer freebsd_triage 2020-04-06 21:37:01 UTC
Quick grep reveals stuff like:
src/VBox/Additions/x11/x11include/XFree86-4.3/X11/Xfuncs.h:
#else /* else vanilla BSD */
#define memmove(dst,src,len) bcopy((char *)(src),(char *)(dst),(int)(len))
#define memcpy(dst,src,len) bcopy((char *)(src),(char *)(dst),(int)(len))
#define memcmp(b1,b2,len) bcmp((char *)(b1),(char *)(b2),(int)(len))

All defines of the sort should be eliminated from all files -- memmove, memcpy and memcmp are provided as first class citizens both in the kernel and libc. For kernel code, previously mentioned sys/systm.h header needs to be included.
Comment 10 groenveld 2020-04-07 21:12:41 UTC
Any thoughts on why when I compile strformatrt.cpp with -O2 I get the bcmp symbol, but with -O0 I don't?

Would that be a clang10 misfeature?

John
groenveld@acm.org

$ c++ -c -O0 -g -pipe -pedantic -Wshadow -Wall -Wextra -Wno-missing-field-initializers -Wno-unused -Wno-trigraphs -fdiagnostics-show-option -Wno-unused-parameter -Wno-language-extension-token -Wno-gnu-folding-constant -Wno-gnu-anonymous-struct -Wno-string-plus-int -Wno-nested-anon-types -Wno-variadic-macros -Wno-long-long -Wno-return-type-c-linkage -Wno-overloaded-virtual -Wno-variadic-macros -O0 -fno-omit-frame-pointer -fno-strict-aliasing -m64 -I/usr/ports/emulators/virtualbox-ose-additions/work/VirtualBox-5.2.34/src/VBox/Runtime/include -I/usr/ports/emulators/virtualbox-ose-additions/work/VirtualBox-5.2.34/src/libs/liblzf-3.4 -I/usr/local/include -I/usr/local/include/libxml2 -I/usr/ports/emulators/virtualbox-ose-additions/work/VirtualBox-5.2.34/include -I/usr/ports/emulators/virtualbox-ose-additions/work/VirtualBox-5.2.34/out/freebsd.amd64/release -DVBOX -DVBOX_OSE -DVBOX_WITH_64_BITS_GUESTS -DVBOX_WITH_REM -DVBOX_WITH_RAW_MODE -DRT_OS_FREEBSD -D__FREEBSD__ -DRT_ARCH_AMD64 -D__AMD64__ -DVBOX_WITH_DEBUGGER -DVBOX_WITH_HARDENING -DRTPATH_APP_PRIVATE=\"/usr/local/lib/virtualbox\" -DRTPATH_APP_PRIVATE_ARCH=\"/usr/local/lib/virtualbox\" -DRTPATH_SHARED_LIBS=\"/usr/local/lib/virtualbox\" -DRTPATH_APP_DOCS=\"/usr/local/lib/virtualbox\" -D_REENTRANT -DIN_RING3 -DLOG_DISABLED -DIN_BLD_PROG -DIN_RT_R3 -DIN_ADV_BLD_PROG -DIN_RT_R3 -DLDR_WITH_NATIVE -DLDR_WITH_ELF32 -DLDR_WITH_LX -DLDR_WITH_MACHO -DLDR_WITH_PE -DRT_WITH_VBOX -DRT_NO_GIP -DRT_WITHOUT_NOCRT_WRAPPERS -DNOFILEID -DRT_WITH_ICONV_CACHE -DIPRT_WITHOUT_LDR_VERIFY -DRT_NO_GIP -Wp,-MD,/usr/ports/emulators/virtualbox-ose-additions/work/VirtualBox-5.2.34/out/freebsd.amd64/release/obj/RuntimeBldProg/common/string/strformatrt.o.dep -Wp,-MT,/usr/ports/emulators/virtualbox-ose-additions/work/VirtualBox-5.2.34/out/freebsd.amd64/release/obj/RuntimeBldProg/common/string/strformatrt.o -Wp,-MP -o /usr/ports/emulators/virtualbox-ose-additions/work/VirtualBox-5.2.34/out/freebsd.amd64/release/obj/RuntimeBldProg/common/string/strformatrt.o /usr/ports/emulators/virtualbox-ose-additions/work/VirtualBox-5.2.34/src/VBox/Runtime/common/string/strformatrt.cpp

$ nm ./work/VirtualBox-5.2.34/out/freebsd.amd64/release/obj/RuntimeBldProg/common/string/strformatrt.o | tail -8
0000000000000add r _ZZ13rtstrFormatRtE9s_szFalse
                 U memchr
                 U memcmp
                 U memset
                 U rtStrFormatKernelAddress
0000000000000000 T rtstrFormatRt
                 U strlen
                 U strncmp
Comment 11 Mateusz Guzik freebsd_committer freebsd_triage 2020-04-07 21:22:52 UTC
This is probably an optimization which is only there because of something wrong with headers.

There is one fundamental discrepancy between bcmp and memcmp: the former only indicates there is a difference while the latter computes what it is on the first differing byte. Thus, if bcmp is provided and buffers differ, it has less work to do than memcmp and is a better pick if available.

However, FreeBSD headers explicitly define bcmp as builtin_memcmp, therefore the above should not be happening if they are properly included and the defines are not overwritten. Once more this is just some build mess.
Comment 12 groenveld 2020-04-08 12:03:59 UTC
strformatrt.cpp compiles with gcc9 without the bcmp(3) reference but the package fails later due to a clang vs gcc flag conflict that I'll try to track down later.

I would like to try to build the package with clang and -O0.
The second invocation of -O2 is set in Config.kmk which I can patch.
But the first one is a mystery.
I tried setting the port Makefile with CXXFLAGS and CFLAGS but that didn't take.

John
groenveld@acm.org
Comment 13 groenveld 2020-04-10 14:22:10 UTC
Created attachment 213246 [details]
stack trace screen snapshot
Comment 14 groenveld 2020-04-10 20:53:53 UTC
Comment on attachment 213246 [details]
stack trace screen snapshot



VBOXVFS[1]: sfprov_mount: path: [browser]
sfprov_mount(browser): error=0 rc=0
panic: getnewvnode: not registered vector op 0xffffffff82a78418

cpuid = 0
time = 1586551831
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00250ee5c0
vpanic() at vpanic+0x182/frame 0xfffffe00250ee610
panic() at panic+0x43/frame 0xfffffe00250ee670
getnewvnode() at getnewvnode+0x254/frame 0xfffffe00250ee6b0
vboxfs_alloc_vp() at vboxfs_alloc_vp+0x470/frame 0xfffffe00250ee790
vboxfs_root() at vboxfs_root+0x32/frame 0xfffffe00250ee7c0
vfs_domount() at vfs_domount+0x85c/frame 0xfffffe00250ee9f0
vfs_donmount() at vfs_donmount+0x871/frame 0xfffffe00250eea90
sys_nmount() at sys_nmount+0x69/frame 0xfffffe00250eead0
amd64_syscall() at amd64_syscall+0x140/frame 0xfffffe00250eebf0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00250eebf0
--- syscall (378, FreeBSD ELF64, sys_nmount), rip = 0x8002f9fea, rsp = 0x7fffffffd3e8, rbp = 0x7fffffffd950 ---
KDB: enter: panic
[ thread pid 862 tid 100464 ]
Stopped at      kdb_enter+0x37: movq    $0,0x10927d6(%rip)
db>
Comment 15 Mateusz Guzik freebsd_committer freebsd_triage 2020-04-10 21:12:18 UTC
This one should be easy: it needs VFS_VOP_VECTOR_REGISTER to be added for the argument passed to getnewvnode, grep will show you plenty of examples in the tree.
Comment 16 groenveld 2020-04-10 22:46:16 UTC
Thank you!

vboxvfs_vnops.c
+ #if __FreeBSD_version > 1300000
+ VFS_VOP_VECTOR_REGISTER(vboxfs_vnodeops);
+ #endif

John
groenveld@acm.org
Comment 17 groenveld 2020-05-09 22:15:36 UTC
John-Mark Gurney's patch worked for me with FreeBSD 13.0-CURRENT #0 r360723:
<URL:https://lists.freebsd.org/pipermail/freebsd-virtualization/2020-May/008448.html>

John
groenveld@acm.org
Comment 18 groenveld 2020-07-19 17:54:22 UTC
(In reply to groenveld from comment #17)
John-Mark Gurney's patch almost applies to the new virtualbox-ose-5.2.44.
Update patch-src_VBox_Additions_freebsd_vboxvfs_Makefile.kmk

--- src/VBox/Additions/freebsd/vboxvfs/Makefile.kmk.orig        2020-05-13 19:37
:06 UTC
+++ src/VBox/Additions/freebsd/vboxvfs/Makefile.kmk
@@ -33,32 +33,37 @@ vboxvfs_INCS          = \
        $(vboxvfs_0_OUTDIR)
 vboxvfs_SOURCES       = \
        vboxvfs_vfsops.c \
-       vboxvfs_vnops.c
+       vboxvfs_vnops.c \
+       bcmp.c \
+       vboxvfs_prov.c


John
groenveld@acm.org
Comment 19 groenveld 2021-01-25 20:02:09 UTC
appears to be fixed in ports rev 561713
John
groenveld@acm.org
Comment 20 Graham Perrin freebsd_committer freebsd_triage 2021-03-14 00:50:28 UTC
mikhail, please, can you confirm that this is no longer an issue for you? 

If it's still an issue: 5.2.44_3 or 6.1.18?
Comment 21 mikhail.rokhin 2021-03-14 01:13:31 UTC
(In reply to Graham Perrin from comment #20)

No, it's the same, except there is no message about bcmp: 
link_elf_obj: symbol bcmp undefined

Just:
KLD vboxvfs.ko: depends on kernel - not available or version mismatch
linker_load_file: /boot/modules/vboxvfs.ko - unsupported file type

And earlier during boot:
KLD vboxguest.ko: depends on kernel - not available or version mismatch
linker_load_file: /boot/modules/vboxguest.ko - unsupported file type

virtualbox-ose-additions-6.1.18 (latest from `pkg ins ...`)

Kernel & world revision is latest ( 369447 ; FreeBSD 13.0-CURRENT).
Comment 22 Mateusz Guzik freebsd_committer freebsd_triage 2021-06-06 08:14:30 UTC
Is this sorted out now?
Comment 23 mikhail.rokhin 2021-06-06 15:49:09 UTC
(In reply to Mateusz Guzik from comment #22)
No, nothing changed.

Additionally:

Starting vboxservice.
VBoxService: error: VbglR3Init failed with rc=VERR_FILE_NOT_FOUND
/etc/rc: WARNING: failed to start vboxservice
Comment 24 Graham Perrin freebsd_committer freebsd_triage 2021-06-06 17:29:25 UTC
(In reply to mikhail.rokhin from comment #23)

> VBoxService: error: VbglR3Init failed with rc=VERR_FILE_NOT_FOUND

UFS in the guest?
Comment 25 mikhail.rokhin 2021-06-08 15:09:58 UTC
(In reply to Graham Perrin from comment #24)

No, ZFS and it succeeded 'til some recent changes.
Comment 26 Graham Perrin freebsd_committer freebsd_triage 2021-06-08 23:51:17 UTC
Thanks. 

(In reply to mikhail.rokhin from comment #23)

> … VbglR3Init failed with rc=VERR_FILE_NOT_FOUND

As far as I can tell, probably not related to this bug 244962. 

If you raise the issue elsewhere, please rope me in. Thanks.
Comment 27 Graham Perrin freebsd_committer freebsd_triage 2023-08-18 21:02:43 UTC
> mount -t vboxsf 

Typo?


(In reply to mikhail.rokhin from comment #0)

> When use `mount_vboxvfs ...` …

% pkg provides mount_vboxvfs
% 

What am I missing?
Comment 28 groenveld 2023-08-18 23:06:14 UTC
(In reply to Graham Perrin from comment #27)

Working for me:

# uname -srvKU
FreeBSD 14.0-ALPHA1 FreeBSD 14.0-ALPHA1 amd64 1400094 #0 main-n264678-136fc495615f: Fri Aug 11 08:18:04 UTC 2023     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC 1400094 1400094

# pkg info virtualbox-ose-additions
virtualbox-ose-additions-6.1.46
Name           : virtualbox-ose-additions
Version        : 6.1.46
Installed on   : Fri Aug 18 04:48:59 2023 EDT
Origin         : emulators/virtualbox-ose-additions
Architecture   : FreeBSD:14:amd64
Prefix         : /usr/local
Categories     : kld emulators
Licenses       : GPLv2
Maintainer     : vbox@FreeBSD.org
WWW            : https://www.virtualbox.org/
Comment        : VirtualBox additions for FreeBSD guests
Options        :
        DBUS           : on
        DEBUG          : off
        INVARIANTS     : on
        X11            : on
Shared Libs required:
        libXt.so.6
        libXmu.so.6
        libXext.so.6
        libX11.so.6
Annotations    :
        FreeBSD_version: 1400094
        cpe            : cpe:2.3:a:oracle:vm_virtualbox:6.1.46:::::freebsd14:x64
Flat size      : 7.41MiB
Description    :
VirtualBox Guest Additions provide closer integration between host and
guest, and improve the performance of guest systems.

This FreeBSD provided package can be installed in FreeBSD guest machines.

Additions are not intended for use on Host machines.

WWW: https://www.virtualbox.org/

# mount -t vboxvfs fbsd14 /mnt
VBOXVFS[1]: sfprov_mount: Enter
VBOXVFS[1]: sfprov_mount: path: [fbsd14]
sfprov_mount(fbsd14): error=0 rc=0

# kldstat
kldstat
Id Refs Address                Size Name
 1   22 0xffffffff80200000  1d4efb8 kernel
 2    1 0xffffffff81f4f000   6f2698 zfs.ko
 3    1 0xffffffff82643000     7938 cryptodev.ko
 4    1 0xffffffff83110000     3210 intpm.ko
 5    1 0xffffffff83114000     2178 smbus.ko
 6    2 0xffffffff83117000    35af0 vboxguest.ko
 7    1 0xffffffff8314d000    214c1 vboxvfs.ko


John
groenveld@acm.org
Comment 29 Graham Perrin freebsd_committer freebsd_triage 2023-08-19 04:45:42 UTC
(In reply to groenveld from comment #19)

> appears to be fixed in ports rev 561713

<https://cgit.freebsd.org/ports/commit/?id=b8cb39c59f206c07bc52c9196bff389cf8ceecf7> (2021-01-16)

Line 37 highlighted in GitHub. Whilst I don't perceive b8cb39c59f20 alone as a fix for e.g. 'Operation not supported by device', it certainly fixed something that arose later than opening comment 0. 


(In reply to groenveld from comment #28)

> # mount -t vboxvfs fbsd14 /mnt

Thanks. Summary line updated. 

It seems that we no longer have 'Operation not supported by device', so let's close. 

Any more recent symptom that's specific to FreeBSD, e.g. not explicable upstream in a virtualbox.org area, can be in a separate report (with reference to 244962, if necessary).

Thanks