Bug 206711 - Export linux_ioctl_{,un}register_handler from linux64 for x11/nvidia-driver
Summary: Export linux_ioctl_{,un}register_handler from linux64 for x11/nvidia-driver
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Only Me
Assignee: Tijl Coosemans
URL:
Keywords:
Depends on:
Blocks: 217901
  Show dependency treegraph
 
Reported: 2016-01-28 13:54 UTC by Jan Beich
Modified: 2018-11-08 19:58 UTC (History)
10 users (show)

See Also:
tijl: maintainer-feedback+
tijl: mfc-stable11+
tijl: mfc-stable12+


Attachments
[workaround] nvidia linux_ioctl global handler (for both linux and linux64) (1.52 KB, patch)
2016-01-28 13:54 UTC, Jan Beich
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Jan Beich freebsd_committer 2016-01-28 13:54:28 UTC
Created attachment 166235 [details]
[workaround] nvidia linux_ioctl global handler (for both linux and linux64)

(In reply to ohartman from bug 201340 comment #17)
> on CURRENT amd64 I would at least expect linux64.ko

nvidia-driver requires kernel to provide linux_ioctl_{,un}register_handler to allow linux userland access /dev/nvidia*. According to /sys/modules/linux/Makefile the symbol is only exported for 32bit version. Obviously, linux64 needs to export 64bit symbol. And its not possible to export by building into kernel: option COMPAT_LINUX doesn't exist on amd64.

How to make 64bit OpenGL apps work with nvidia & linuxulator:
1. Switch to 64bit linux_base or populate manually (I've used CentOS 7)
2. Apply *attached* patch and rebuild kernel or just linux* modules
3. Disable linux_ioctl handler in nvidia-driver (i.e. rebuild with LINUX=off)
4. Install 64bit linux driver under /compat/linux
5. Remove sbin/nvidia-modprobe binary to enable userland fallback (as used by FreeBSD driver)

CUDA 64bit may still not work e.g.,

  $ export PATH=/compat/linux/bin:/usr/local/cuda/bin:$PATH
  $ nvcc -o foo -m32 -lcuda a.c
  $ nvcc -o bar -lcuda a.c

  $ ./foo
  $ ./bar
  cuInit failed: 999

  $ cat a.c
  #include <cuda.h>
  #include <stdio.h>

  int main()
  {
      CUresult r = cuInit(0);
      if (r != CUDA_SUCCESS) {
          printf("cuInit failed: %d\n", r);
          return 1;
      }
      return 0;
  }
Comment 1 Jan Beich freebsd_committer 2016-01-28 14:45:51 UTC
Move to canonical assignee.
Comment 2 Dmitry Chagin freebsd_committer 2016-02-22 19:14:57 UTC
nvidia-driver install ioctl_handler, please, look to nvidia_linux_init/nvidia_linux_exit pair that called from nvidia_modevent().

as far as I can see, nvidia driver does not support 64 bit Linuxulator, so nvidia_linux_init() not called for linux64 module. also nvidia driver does not install 64 bit linux libs.
this should be fixed
Comment 3 Jan Beich freebsd_committer 2016-02-23 14:28:11 UTC
(In reply to Dmitry Chagin from comment #2)
The patch here is the result of failed attempt to register 64bit |struct linux_ioctl_handler| from within nvidia-driver. All in-tree kernel modules register only 32bit ioctls, the rest is done by linux64 (drm2, v4l2, console, etc). How do you expect ports drivers to support linux64 without examples?

  $ nm -A /boot/kernel/*.ko | fgrep ioctl_register_handler
  /boot/kernel/amr_linux.ko:                  U linux_ioctl_register_handler
  /boot/kernel/ipmi_linux.ko:                 U linux_ioctl_register_handler
  /boot/kernel/linux.ko:     0000000000009480 t linux_ioctl_register_handler
  /boot/kernel/linux64.ko:   00000000000093e0 t linux_ioctl_register_handler
  /boot/kernel/mfi_linux.ko:                  U linux_ioctl_register_handler
  /boot/kernel/mrsas_linux.ko:                U linux_ioctl_register_handler

nvidia_linux.c doesn't have much MD code but the following has no effect even on the hashsum of its object file.

  --- src/nvidia/nvidia_linux.c~
  +++ src/nvidia/nvidia_linux.c
  @@ -18,13 +18,8 @@
   #define LINUX_IOCTL_NVIDIA_MIN 0x4600
   #define LINUX_IOCTL_NVIDIA_MAX 0x46ff

  -#if defined(NVCPU_X86)
   #include "machine/../linux/linux.h"
   #include "machine/../linux/linux_proto.h"
  -#elif defined(NVCPU_X86_64)
  -#include "machine/../linux32/linux.h"
  -#include "machine/../linux32/linux32_proto.h"
  -#endif

   #include <compat/linux/linux_ioctl.h>
Comment 4 Jan Beich freebsd_committer 2016-04-19 08:17:01 UTC
Andy, can you confirm this bug needs to be fixed before nvidia-driver can support linux64 (64bit linuxulator) available since FreeBSD 10.3-RELEASE ?
Comment 5 Andy Ritger 2016-04-19 15:44:30 UTC
Sorry, I admit I haven't looked at 64-bit Linuxulator support.

Certainly, 64-bit Linux user-space NVIDIA driver components will make ioctls on /dev/nvidia* device files, and we expect those ioctls to be routed to the NVIDIA FreeBSD kernel drivers.  But, I don't know off hand what that plumbing looks like.

I can try to take a look...
Comment 6 Henry Hu 2017-12-15 05:33:42 UTC
Any update on this problem? New CUDA (>7.0) only provides 64-bit libraries, so supporting 64 bit linuxulator is required for using recent CUDA programs.
Comment 7 Jan Beich freebsd_committer 2018-08-30 21:50:08 UTC
(In reply to Henry Hu from comment #6)
> New CUDA (>7.0) only provides 64-bit libraries

CUDA toolkit 7.* and 8.x were 64-bit only themselves but could target 32-bit. 9.0 is when 32-bit application support disappeared.

https://docs.nvidia.com/cuda/archive/9.0/cuda-toolkit-release-notes/index.html#deprecated-features

> supporting 64 bit linuxulator is required for using recent CUDA programs

NVIDIA-FreeBSD-x86_64-396.54.tar.gz still provides libcuda.so.1 for 32-bit linuxulator.
Comment 8 Hannes Hauswedell 2018-10-15 16:16:23 UTC
@Jan: Thanks for pointing me to this issue.

Any news from someone else on this PR? Are said patches expected to still work? Is there any ETA for getting this into the port/kernel? I know quite a few people who are affected by this and the linux-base ports are now 64bit by default so this has become more important from my POV.

Thanks!
Comment 9 commit-hook freebsd_committer 2018-11-06 13:52:08 UTC
A commit references this bug:

Author: tijl
Date: Tue Nov  6 13:51:09 UTC 2018
New revision: 340181
URL: https://svnweb.freebsd.org/changeset/base/340181

Log:
  On amd64 both Linux compat modules, linux.ko and linux64.ko, provide
  linux_ioctl_(un)register_handler that allows other driver modules to
  register ioctl handlers.  The ioctl syscall implementation in each Linux
  compat module iterates over the list of handlers and forwards the call to
  the appropriate driver.  Because the registration functions have the same
  name in each module it is not possible for a driver to support both 32 and
  64 bit linux compatibility.

  Move the list of ioctl handlers to linux_common.ko so it is shared by
  both Linux modules and all drivers receive both 32 and 64 bit ioctl calls
  with one registration.  These ioctl handlers normally forward the call
  to the FreeBSD ioctl handler which can handle both 32 and 64 bit.

  Keep the special COMPAT_LINUX32 ioctl handlers in linux.ko in a separate
  list for now and let the ioctl syscall iterate over that list first.
  Later, COMPAT_LINUX32 support can be added to the 64 bit ioctl handlers
  via a runtime check for ILP32 like is done for COMPAT_FREEBSD32 and then
  this separate list would disappear again.  That is a much bigger effort
  however and this commit is meant to be MFCable.

  This enables linux64 support in x11/nvidia-driver*.

  PR:		206711
  Reviewed by:	kib
  MFC after:	3 days

Changes:
  head/sys/amd64/linux32/linux32_sysvec.c
  head/sys/compat/linux/linux_common.c
  head/sys/compat/linux/linux_ioctl.c
  head/sys/compat/linux/linux_ioctl.h
Comment 10 commit-hook freebsd_committer 2018-11-08 19:57:20 UTC
A commit references this bug:

Author: tijl
Date: Thu Nov  8 19:56:30 UTC 2018
New revision: 340258
URL: https://svnweb.freebsd.org/changeset/base/340258

Log:
  MFC r340181, r340185:

  On amd64 both Linux compat modules, linux.ko and linux64.ko, provide
  linux_ioctl_(un)register_handler that allows other driver modules to
  register ioctl handlers.  The ioctl syscall implementation in each Linux
  compat module iterates over the list of handlers and forwards the call to
  the appropriate driver.  Because the registration functions have the same
  name in each module it is not possible for a driver to support both 32 and
  64 bit linux compatibility.

  Move the list of ioctl handlers to linux_common.ko so it is shared by
  both Linux modules and all drivers receive both 32 and 64 bit ioctl calls
  with one registration.  These ioctl handlers normally forward the call
  to the FreeBSD ioctl handler which can handle both 32 and 64 bit.

  Keep the special COMPAT_LINUX32 ioctl handlers in linux.ko in a separate
  list for now and let the ioctl syscall iterate over that list first.
  Later, COMPAT_LINUX32 support can be added to the 64 bit ioctl handlers
  via a runtime check for ILP32 like is done for COMPAT_FREEBSD32 and then
  this separate list would disappear again.  That is a much bigger effort
  however and this commit is meant to be MFCable.

  This enables linux64 support in x11/nvidia-driver*.

  PR:		206711
  Reviewed by:	kib
  Approved by:	re (gjb)

Changes:
_U  stable/12/
  stable/12/sys/amd64/linux32/linux32_sysvec.c
  stable/12/sys/compat/linux/linux_common.c
  stable/12/sys/compat/linux/linux_ioctl.c
  stable/12/sys/compat/linux/linux_ioctl.h
Comment 11 commit-hook freebsd_committer 2018-11-08 19:57:25 UTC
A commit references this bug:

Author: tijl
Date: Thu Nov  8 19:56:46 UTC 2018
New revision: 340259
URL: https://svnweb.freebsd.org/changeset/base/340259

Log:
  MFC r340181, r340185:

  On amd64 both Linux compat modules, linux.ko and linux64.ko, provide
  linux_ioctl_(un)register_handler that allows other driver modules to
  register ioctl handlers.  The ioctl syscall implementation in each Linux
  compat module iterates over the list of handlers and forwards the call to
  the appropriate driver.  Because the registration functions have the same
  name in each module it is not possible for a driver to support both 32 and
  64 bit linux compatibility.

  Move the list of ioctl handlers to linux_common.ko so it is shared by
  both Linux modules and all drivers receive both 32 and 64 bit ioctl calls
  with one registration.  These ioctl handlers normally forward the call
  to the FreeBSD ioctl handler which can handle both 32 and 64 bit.

  Keep the special COMPAT_LINUX32 ioctl handlers in linux.ko in a separate
  list for now and let the ioctl syscall iterate over that list first.
  Later, COMPAT_LINUX32 support can be added to the 64 bit ioctl handlers
  via a runtime check for ILP32 like is done for COMPAT_FREEBSD32 and then
  this separate list would disappear again.  That is a much bigger effort
  however and this commit is meant to be MFCable.

  This enables linux64 support in x11/nvidia-driver*.

  PR:		206711
  Reviewed by:	kib

Changes:
_U  stable/11/
  stable/11/sys/amd64/linux32/linux32_sysvec.c
  stable/11/sys/compat/linux/linux_common.c
  stable/11/sys/compat/linux/linux_ioctl.c
  stable/11/sys/compat/linux/linux_ioctl.h