Created attachment 166235 [details] [workaround] nvidia linux_ioctl global handler (for both linux and linux64) (In reply to ohartman from bug 201340 comment #17) > on CURRENT amd64 I would at least expect linux64.ko nvidia-driver requires kernel to provide linux_ioctl_{,un}register_handler to allow linux userland access /dev/nvidia*. According to /sys/modules/linux/Makefile the symbol is only exported for 32bit version. Obviously, linux64 needs to export 64bit symbol. And its not possible to export by building into kernel: option COMPAT_LINUX doesn't exist on amd64. How to make 64bit OpenGL apps work with nvidia & linuxulator: 1. Switch to 64bit linux_base or populate manually (I've used CentOS 7) 2. Apply *attached* patch and rebuild kernel or just linux* modules 3. Disable linux_ioctl handler in nvidia-driver (i.e. rebuild with LINUX=off) 4. Install 64bit linux driver under /compat/linux 5. Remove sbin/nvidia-modprobe binary to enable userland fallback (as used by FreeBSD driver) CUDA 64bit may still not work e.g., $ export PATH=/compat/linux/bin:/usr/local/cuda/bin:$PATH $ nvcc -o foo -m32 -lcuda a.c $ nvcc -o bar -lcuda a.c $ ./foo $ ./bar cuInit failed: 999 $ cat a.c #include <cuda.h> #include <stdio.h> int main() { CUresult r = cuInit(0); if (r != CUDA_SUCCESS) { printf("cuInit failed: %d\n", r); return 1; } return 0; }
Move to canonical assignee.
nvidia-driver install ioctl_handler, please, look to nvidia_linux_init/nvidia_linux_exit pair that called from nvidia_modevent(). as far as I can see, nvidia driver does not support 64 bit Linuxulator, so nvidia_linux_init() not called for linux64 module. also nvidia driver does not install 64 bit linux libs. this should be fixed
(In reply to Dmitry Chagin from comment #2) The patch here is the result of failed attempt to register 64bit |struct linux_ioctl_handler| from within nvidia-driver. All in-tree kernel modules register only 32bit ioctls, the rest is done by linux64 (drm2, v4l2, console, etc). How do you expect ports drivers to support linux64 without examples? $ nm -A /boot/kernel/*.ko | fgrep ioctl_register_handler /boot/kernel/amr_linux.ko: U linux_ioctl_register_handler /boot/kernel/ipmi_linux.ko: U linux_ioctl_register_handler /boot/kernel/linux.ko: 0000000000009480 t linux_ioctl_register_handler /boot/kernel/linux64.ko: 00000000000093e0 t linux_ioctl_register_handler /boot/kernel/mfi_linux.ko: U linux_ioctl_register_handler /boot/kernel/mrsas_linux.ko: U linux_ioctl_register_handler nvidia_linux.c doesn't have much MD code but the following has no effect even on the hashsum of its object file. --- src/nvidia/nvidia_linux.c~ +++ src/nvidia/nvidia_linux.c @@ -18,13 +18,8 @@ #define LINUX_IOCTL_NVIDIA_MIN 0x4600 #define LINUX_IOCTL_NVIDIA_MAX 0x46ff -#if defined(NVCPU_X86) #include "machine/../linux/linux.h" #include "machine/../linux/linux_proto.h" -#elif defined(NVCPU_X86_64) -#include "machine/../linux32/linux.h" -#include "machine/../linux32/linux32_proto.h" -#endif #include <compat/linux/linux_ioctl.h>
Andy, can you confirm this bug needs to be fixed before nvidia-driver can support linux64 (64bit linuxulator) available since FreeBSD 10.3-RELEASE ?
Sorry, I admit I haven't looked at 64-bit Linuxulator support. Certainly, 64-bit Linux user-space NVIDIA driver components will make ioctls on /dev/nvidia* device files, and we expect those ioctls to be routed to the NVIDIA FreeBSD kernel drivers. But, I don't know off hand what that plumbing looks like. I can try to take a look...
Any update on this problem? New CUDA (>7.0) only provides 64-bit libraries, so supporting 64 bit linuxulator is required for using recent CUDA programs.
(In reply to Henry Hu from comment #6) > New CUDA (>7.0) only provides 64-bit libraries CUDA toolkit 7.* and 8.x were 64-bit only themselves but could target 32-bit. 9.0 is when 32-bit application support disappeared. https://docs.nvidia.com/cuda/archive/9.0/cuda-toolkit-release-notes/index.html#deprecated-features > supporting 64 bit linuxulator is required for using recent CUDA programs NVIDIA-FreeBSD-x86_64-396.54.tar.gz still provides libcuda.so.1 for 32-bit linuxulator.
@Jan: Thanks for pointing me to this issue. Any news from someone else on this PR? Are said patches expected to still work? Is there any ETA for getting this into the port/kernel? I know quite a few people who are affected by this and the linux-base ports are now 64bit by default so this has become more important from my POV. Thanks!
A commit references this bug: Author: tijl Date: Tue Nov 6 13:51:09 UTC 2018 New revision: 340181 URL: https://svnweb.freebsd.org/changeset/base/340181 Log: On amd64 both Linux compat modules, linux.ko and linux64.ko, provide linux_ioctl_(un)register_handler that allows other driver modules to register ioctl handlers. The ioctl syscall implementation in each Linux compat module iterates over the list of handlers and forwards the call to the appropriate driver. Because the registration functions have the same name in each module it is not possible for a driver to support both 32 and 64 bit linux compatibility. Move the list of ioctl handlers to linux_common.ko so it is shared by both Linux modules and all drivers receive both 32 and 64 bit ioctl calls with one registration. These ioctl handlers normally forward the call to the FreeBSD ioctl handler which can handle both 32 and 64 bit. Keep the special COMPAT_LINUX32 ioctl handlers in linux.ko in a separate list for now and let the ioctl syscall iterate over that list first. Later, COMPAT_LINUX32 support can be added to the 64 bit ioctl handlers via a runtime check for ILP32 like is done for COMPAT_FREEBSD32 and then this separate list would disappear again. That is a much bigger effort however and this commit is meant to be MFCable. This enables linux64 support in x11/nvidia-driver*. PR: 206711 Reviewed by: kib MFC after: 3 days Changes: head/sys/amd64/linux32/linux32_sysvec.c head/sys/compat/linux/linux_common.c head/sys/compat/linux/linux_ioctl.c head/sys/compat/linux/linux_ioctl.h
A commit references this bug: Author: tijl Date: Thu Nov 8 19:56:30 UTC 2018 New revision: 340258 URL: https://svnweb.freebsd.org/changeset/base/340258 Log: MFC r340181, r340185: On amd64 both Linux compat modules, linux.ko and linux64.ko, provide linux_ioctl_(un)register_handler that allows other driver modules to register ioctl handlers. The ioctl syscall implementation in each Linux compat module iterates over the list of handlers and forwards the call to the appropriate driver. Because the registration functions have the same name in each module it is not possible for a driver to support both 32 and 64 bit linux compatibility. Move the list of ioctl handlers to linux_common.ko so it is shared by both Linux modules and all drivers receive both 32 and 64 bit ioctl calls with one registration. These ioctl handlers normally forward the call to the FreeBSD ioctl handler which can handle both 32 and 64 bit. Keep the special COMPAT_LINUX32 ioctl handlers in linux.ko in a separate list for now and let the ioctl syscall iterate over that list first. Later, COMPAT_LINUX32 support can be added to the 64 bit ioctl handlers via a runtime check for ILP32 like is done for COMPAT_FREEBSD32 and then this separate list would disappear again. That is a much bigger effort however and this commit is meant to be MFCable. This enables linux64 support in x11/nvidia-driver*. PR: 206711 Reviewed by: kib Approved by: re (gjb) Changes: _U stable/12/ stable/12/sys/amd64/linux32/linux32_sysvec.c stable/12/sys/compat/linux/linux_common.c stable/12/sys/compat/linux/linux_ioctl.c stable/12/sys/compat/linux/linux_ioctl.h
A commit references this bug: Author: tijl Date: Thu Nov 8 19:56:46 UTC 2018 New revision: 340259 URL: https://svnweb.freebsd.org/changeset/base/340259 Log: MFC r340181, r340185: On amd64 both Linux compat modules, linux.ko and linux64.ko, provide linux_ioctl_(un)register_handler that allows other driver modules to register ioctl handlers. The ioctl syscall implementation in each Linux compat module iterates over the list of handlers and forwards the call to the appropriate driver. Because the registration functions have the same name in each module it is not possible for a driver to support both 32 and 64 bit linux compatibility. Move the list of ioctl handlers to linux_common.ko so it is shared by both Linux modules and all drivers receive both 32 and 64 bit ioctl calls with one registration. These ioctl handlers normally forward the call to the FreeBSD ioctl handler which can handle both 32 and 64 bit. Keep the special COMPAT_LINUX32 ioctl handlers in linux.ko in a separate list for now and let the ioctl syscall iterate over that list first. Later, COMPAT_LINUX32 support can be added to the 64 bit ioctl handlers via a runtime check for ILP32 like is done for COMPAT_FREEBSD32 and then this separate list would disappear again. That is a much bigger effort however and this commit is meant to be MFCable. This enables linux64 support in x11/nvidia-driver*. PR: 206711 Reviewed by: kib Changes: _U stable/11/ stable/11/sys/amd64/linux32/linux32_sysvec.c stable/11/sys/compat/linux/linux_common.c stable/11/sys/compat/linux/linux_ioctl.c stable/11/sys/compat/linux/linux_ioctl.h