Bug 217901 - x11/nvidia-driver: Linux compatibility support doesn't install linux64 libs
Summary: x11/nvidia-driver: Linux compatibility support doesn't install linux64 libs
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Some People
Assignee: Alexey Dokuchaev
URL:
Keywords:
: 219698 (view as bug list)
Depends on: 206711
Blocks: 224358 228287
  Show dependency treegraph
 
Reported: 2017-03-18 22:02 UTC by Callum
Modified: 2018-12-14 16:27 UTC (History)
6 users (show)

See Also:
bugzilla: maintainer-feedback? (danfe)


Attachments
patch (33.63 KB, patch)
2018-11-09 20:59 UTC, Tijl Coosemans
no flags Details | Diff
patch3 (43.50 KB, patch)
2018-11-13 15:00 UTC, Tijl Coosemans
no flags Details | Diff
ktrace (50.49 KB, application/octet-stream)
2018-11-13 21:31 UTC, Hannes Hauswedell
no flags Details
native ktrace (49.23 KB, application/octet-stream)
2018-11-13 23:00 UTC, Hannes Hauswedell
no flags Details
redev_test (1.05 KB, text/plain)
2018-11-14 01:33 UTC, Alex S
no flags Details
linux.sendmsg.patch (3.27 KB, patch)
2018-11-19 10:32 UTC, Tijl Coosemans
no flags Details | Diff
patch4 (45.35 KB, patch)
2018-11-19 10:44 UTC, Tijl Coosemans
no flags Details | Diff
Linux compat ktrace of glxinfo (47.28 KB, application/octet-stream)
2018-11-19 20:52 UTC, Hannes Hauswedell
no flags Details
native ktrace of glxinfo (46.27 KB, application/octet-stream)
2018-11-19 20:53 UTC, Hannes Hauswedell
no flags Details
linux.sendmsg.2.patch (1.94 KB, patch)
2018-11-20 14:29 UTC, Tijl Coosemans
no flags Details | Diff
patch5 (56.20 KB, patch)
2018-11-22 14:27 UTC, Tijl Coosemans
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Callum 2017-03-18 22:02:04 UTC
When installing x11/nvidia-driver and selecting the LINUX configuration option only 32-bit libraries are installed in /compat/linux/lib/.

linux-c7 and FreeBSD 11 support 64-bit Linux and thus 64-bit libraries should ideally be installed, where supported, in /compat/linux/lib64/ for compatibility with 64-bit applications.

I would propose checking for the existence of /compat/linux/lib64 as the trigger for installing the lib64 libraries.

Thanks.
Comment 1 Tijl Coosemans freebsd_committer 2017-06-01 09:15:48 UTC
*** Bug 219698 has been marked as a duplicate of this bug. ***
Comment 2 Hannes Hauswedell 2017-10-07 14:28:21 UTC
Yes, this is really important! However, I think it is not only an issue of missing libraries, but an actual driver issue. I have tried putting the right libraries in place, but it doesn't work.

However, more and more linux binaries are now 64bit (only) so this is a real problem, rendering the linux compatibility useless for very many apps, including, but not limited to all modern games.
Comment 3 Tijl Coosemans freebsd_committer 2018-11-09 20:59:49 UTC
Created attachment 199099 [details]
patch

This patch changes the x11/nvidia-driver ports to fetch and install the Linux driver package under /compat/linux.  This includes both 32 bit and 64 bit libraries, unlike the Linux libraries included in the FreeBSD driver package which are only 32 bit.

The 64 bit libraries require at least r340259 of FreeBSD stable/11, stable/12 or head.

I've been able to test x11/nvidia-driver-304 successfully, but I cannot test the other versions.  The Linux libraries in x11/nvidia-driver are libglvnd based and I'm not sure that will work, so it would be nice if somebody could test that before this patch is committed.
Comment 4 Alex S 2018-11-10 21:28:03 UTC
(In reply to Tijl Coosemans from comment #3)

> The Linux libraries in x11/nvidia-driver are libglvnd based

The Linux driver provides both libglvnd compatible and legacy GL libraries. See https://devtalk.nvidia.com/default/topic/915640/multiple-glx-client-libraries-in-the-nvidia-linux-driver-installer-package/ and http://us.download.nvidia.com/XFree86/Linux-x86_64/390.87/README/installedcomponents.html.
Comment 5 Hannes Hauswedell 2018-11-11 16:18:13 UTC
Thank you for working on this, it is really appreciated!

Do these changes make the driver compatible with linux_base-c7, too? I think the main issue are changed paths (/usr is a symlink now) which broke stuff for me on install/uninstall some times...
Comment 6 Tijl Coosemans freebsd_committer 2018-11-11 20:39:38 UTC
(In reply to Hannes Hauswedell from comment #5)
I haven't tried linux_base-c7, but /usr isn't a symlink there.  /(s)bin and /lib(64) are, but that's also the case with linux_base-c6, so that shouldn't be a problem.
Comment 7 Hannes Hauswedell 2018-11-12 23:20:42 UTC
I have updated to 13-BETA4, removed all -c7 ports, installed linux_base-c6, applied the patch here (which applied cleanly), installed x11/nvidia-driver (nvidia-driver-390.87).

But running /compat/linux/usr/bin/glxinfo leads to immediate dump of the kernel :'(

I have made the dump available here:
https://hauswedell.net/crash/core.tar.zst

I have a GTX 750 Ti, I think I can't try any of the older drivers...

Thanks for your help!
Comment 8 Tijl Coosemans freebsd_committer 2018-11-13 15:00:41 UTC
Created attachment 199205 [details]
patch3

The linux ioctl handler in the nvidia driver didn't wrap the FreeBSD ioctl handler properly.  This patch should fix that.
Comment 9 Hannes Hauswedell 2018-11-13 18:22:39 UTC
(In reply to Tijl Coosemans from comment #8)

Thanks, this fixes the coredump! But now I run into #228287 , i.e. I get 

% /compat/linux/usr/bin/glxinfo
name of display: unix:0.0
X Error of failed request:  BadValue (integer parameter out of range for operation)
  Major opcode of failed request:  154 (GLX)
  Minor opcode of failed request:  3 (X_GLXCreateContext)
  Value in failed request:  0x0
  Serial number of failed request:  25
  Current serial number in output stream:  26

I tried setting IndirectRendering as described in #228287, but then OpenGL-Apps won't start either because they can't allocate resources. I assume IndirectRendering deactivates proper 3D anyway?
Comment 10 Tijl Coosemans freebsd_committer 2018-11-13 20:03:51 UTC
(In reply to Hannes Hauswedell from comment #9)
What is the output of "/compat/linux/bin/sh ldd /bin/glxinfo"?
Comment 11 Hannes Hauswedell 2018-11-13 20:51:32 UTC
(In reply to Tijl Coosemans from comment #10)

% /compat/linux/bin/sh ldd /bin/glxinfo
        linux_vdso.so.1 =>  (0x00007fffffffe000)
        libGLU.so.1 => /lib64/libGLU.so.1 (0x0000000801400000)
        libGL.so.1 => /lib64/libGL.so.1 (0x0000000801800000)
        libm.so.6 => /lib64/libm.so.6 (0x0000000801c00000)
        libX11.so.6 => /lib64/libX11.so.6 (0x0000000802000000)
        libXext.so.6 => /lib64/libXext.so.6 (0x0000000802400000)
        libc.so.6 => /lib64/libc.so.6 (0x0000000802800000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x0000000802c00000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000000803000000)
        libdl.so.2 => /lib64/libdl.so.2 (0x0000000803400000)
        libGLX.so.0 => /lib64/libGLX.so.0 (0x0000000803800000)
        libGLdispatch.so.0 => /lib64/libGLdispatch.so.0 (0x0000000803c00000)
        libxcb.so.1 => /lib64/libxcb.so.1 (0x0000000804000000)
        /lib64/ld-linux-x86-64.so.2 (0x0000000001021000)
        libXau.so.6 => /lib64/libXau.so.6 (0x0000000804400000)
Comment 12 Tijl Coosemans freebsd_committer 2018-11-13 21:05:20 UTC
(In reply to Hannes Hauswedell from comment #11)
Please run the following commands and attach ktrace.txt to this bug.

ktrace -i -f /where/you/want/ktrace.out /compat/linux/bin/glxinfo
kdump -H -f /where/you/want/ktrace.out >  /where/you/want/ktrace.txt
Comment 13 Hannes Hauswedell 2018-11-13 21:31:14 UTC
Created attachment 199213 [details]
ktrace

The file is attached, thanks for debugging this!
Comment 14 Tijl Coosemans freebsd_committer 2018-11-13 22:30:12 UTC
(In reply to Hannes Hauswedell from comment #13)
Looks like a problem with the nvidia-kms module.  Can you install graphics/mesa-demos and create a ktrace of the FreeBSD glxinfo?  Can you also give the non-kms module a try?
Comment 15 Hannes Hauswedell 2018-11-13 23:00:44 UTC
Created attachment 199218 [details]
native ktrace

This is the ktrace from the native glxinfo.

I tried booting with the nvidia-module instead of nvidia-modeset, but then I never make it into X. There are no warnings or errors in Xorg.0.log or dmesg, it just hangs when switching resolution. (It's possible to stop/restart X via ssh, but also with startx I get no suspicious output, just a black screen in console-resolution with white cursor).
I remember having had this a long time ago already which is why I switched to the kms-module in the first place, I think.
Comment 16 Alex S 2018-11-14 01:33:40 UTC
Created attachment 199220 [details]
redev_test

This test triggers a slightly different codepath in libGL where it doesn't try to recreate /nvidia* device files. This seems to be enough to get glxgears working for me with 415.13 driver. Try it with:

% /compat/linux/bin/cc -m64 -std=c99 -shared -fPIC -lm -pthread -I/compat/linux/usr/include -L/compat/linux/lib64 -o test.so stat_rdev_override.c
% env LD_PRELOAD=$PWD/test.so /compat/linux/bin/glxgears

Does that help?
Comment 17 Hannes Hauswedell 2018-11-14 09:01:54 UTC
(In reply to Alex S from comment #16)
What's the recommended way to get a compiler into the linux base? There seem to be no linux-gcc packages on FreeBSD and inside the compat there isn't even rpm to install original CentOS packages from a mirror...

Also you mentionened an nvidia driver of version 415.13. Where can I get that and does it work with the patches from this PR?

Thanks for your help with this!
Comment 18 Tijl Coosemans freebsd_committer 2018-11-14 09:23:32 UTC
(In reply to Hannes Hauswedell from comment #17)
There's a compiler in devel/linux-c6-devtools.
Comment 19 Hannes Hauswedell 2018-11-14 09:39:18 UTC
Ah, ok. Why don't they just call it GCC like every one else :|

Anyway, preloading the above object while using nvidia-driver-390.87 leads to another coredump:
https://hauswedell.net/crash/core4.tar.zst
Comment 20 Alex S 2018-11-14 20:04:56 UTC
> Also you mentioned an nvidia driver of version 415.13. 

Well, a knee-jerk patchjob on top of existing port takes about 10 min or so. However, 410.57+ actually requires libglvnd to work (a compiled copy is provided with the driver) and it really should be packaged separately. Somebody should look into making libglvnd work with Mesa as well.

> Where can I get that and does it work with the patches from this PR?

I'm actually using 11.2-RELEASE and a slightly modified patch from PR 206711 (too lazy to bother with compiling whole kernel), but generally the latest driver should work with these patches as well, there is no reason not to.

> Anyway, preloading the above object while using nvidia-driver-390.87 leads to another coredump

linux_sendmsg_common? Looks like a separate bug to me.
Comment 21 Hannes Hauswedell 2018-11-17 23:30:24 UTC
Hm, what are the next steps? I had the feeling we were making progress!

Anything I can do to help move this forward? Should I open a different PR with the coredump? Will the patches here be merged?
Comment 22 Tijl Coosemans freebsd_committer 2018-11-18 21:37:26 UTC
(In reply to Hannes Hauswedell from comment #21)
The patch for the second panic is currently under review.
Comment 23 Tijl Coosemans freebsd_committer 2018-11-19 10:32:50 UTC
Created attachment 199335 [details]
linux.sendmsg.patch

This patch should fix the second panic.  You can test it as follows:

cd /usr/src
svn patch /path/to/linux.sendmsg.patch
cd sys/modules/linux
make
make install
cd ../linux64
make
make install
mv /boot/modules/linux.ko /boot/kernel/
mv /boot/modules/linux64.ko /boot/kernel/
kldxref /boot/modules
kldxref /boot/kernel
shutdown -r now
Comment 24 Tijl Coosemans freebsd_committer 2018-11-19 10:44:43 UTC
Created attachment 199336 [details]
patch4

This version of the patch for x11/nvidia-driver adds an extra ktrace point to the nvidia-modeset driver.  Can you create another ktrace of both the Linux and FreeBSD glxinfo?
Comment 25 commit-hook freebsd_committer 2018-11-19 15:32:33 UTC
A commit references this bug:

Author: tijl
Date: Mon Nov 19 15:31:55 UTC 2018
New revision: 340631
URL: https://svnweb.freebsd.org/changeset/base/340631

Log:
  Do proper copyin of control message data in the Linux sendmsg syscall.

  Instead of calling m_append with a user address, allocate an mbuf cluster
  and copy data into it using copyin.  For the SCM_CREDS case, instead of
  zeroing a stack variable and appending that to the mbuf, zero part of the
  mbuf cluster directly.  One mbuf cluster is also the size limit used by
  the FreeBSD sendmsg syscall (uipc_syscalls.c:sockargs()).

  PR:		217901
  Reviewed by:	kib
  MFC after:	3 days

Changes:
  head/sys/compat/linux/linux_socket.c
Comment 26 Hannes Hauswedell 2018-11-19 20:51:54 UTC
I have applied the kernel patch and the new nvidia patch. Linux glxinfo still produces:
name of display: unix:0.0
X Error of failed request:  BadValue (integer parameter out of range for operation)
  Major opcode of failed request:  154 (GLX)
  Minor opcode of failed request:  3 (X_GLXCreateContext)
  Value in failed request:  0x0
  Serial number of failed request:  25
  Current serial number in output stream:  26


I will add updated ktraces.
Comment 27 Hannes Hauswedell 2018-11-19 20:52:59 UTC
Created attachment 199359 [details]
Linux compat ktrace of glxinfo
Comment 28 Hannes Hauswedell 2018-11-19 20:53:30 UTC
Created attachment 199360 [details]
native ktrace of glxinfo
Comment 29 Alex S 2018-11-19 21:05:28 UTC
(In reply to Hannes Hauswedell from comment #27)

>  1550 100663 glxinfo  NAMI  "/compat/linux/dev/nvidiactl"
>  1550 100663 glxinfo  NAMI  "/dev/nvidiactl"
>  1550 100663 glxinfo  RET   linux_unlink -1 errno -13 Permission denied

You forgot to test this with LD_PRELOAD trick.
Comment 30 Hannes Hauswedell 2018-11-20 09:34:36 UTC
(In reply to Alex S from comment #29)

Yes, but doing that still results in a coredump for me:
https://hauswedell.net/crash/core6.tar.zst

BTW: I had problems with updating the module as described above, after just copying the module the nvidia driver claimed that some things were of different versions. So I updated to 12-stable, applied the patch, rebuilt and installled entire world and kernel. I assume that's supposed to work right, too?
Comment 31 commit-hook freebsd_committer 2018-11-20 14:19:12 UTC
A commit references this bug:

Author: tijl
Date: Tue Nov 20 14:18:57 UTC 2018
New revision: 340674
URL: https://svnweb.freebsd.org/changeset/base/340674

Log:
  Fix another user address dereference in linux_sendmsg syscall.

  This was hidden behind the LINUX_CMSG_NXTHDR macro which dereferences its
  second argument.  Stop using the macro as well as LINUX_CMSG_FIRSTHDR.  Use
  the size field of the kernel copy of the control message header to obtain
  the next control message.

  PR:		217901
  MFC after:	2 days
  X-MFC-With:	r340631

Changes:
  head/sys/compat/linux/linux_socket.c
Comment 32 Tijl Coosemans freebsd_committer 2018-11-20 14:29:45 UTC
Created attachment 199382 [details]
linux.sendmsg.2.patch

Patch for the second linux_sendmsg panic.  To be applied on top of the first patch.
Comment 33 Tijl Coosemans freebsd_committer 2018-11-21 08:31:01 UTC
Does the LD_PRELOAD method still panic?
Comment 34 Hannes Hauswedell 2018-11-21 22:04:51 UTC
I wasn't at the affected computer the last days, but I have now verified that everything works with the second kernel patch applied!

After building another test.so with -m32 I can successfully start linux OpenGL in either 32bit or 64bit by prefixing the respective shared object file. Even "modern" programs like linux-unigine-valley (whose port I maintain) work.

Very nice, thanks for debugging this :)
Now we only need to make linux-base-c7 default and I can remove the dual-boot.

Should I mark this resolved now, or do we wait till everything has landed in ports and Kernel?
Comment 35 Tijl Coosemans freebsd_committer 2018-11-21 22:41:32 UTC
(In reply to Hannes Hauswedell from comment #34)
Thanks for testing.  I'll update the patch so you don't need to preload anything.  We'll close the bug when everything is committed.
Comment 36 Hannes Hauswedell 2018-11-21 23:26:40 UTC
Thanks! I am busy testing applications right now, is there a place I should report failing applications that I know work on Linux? It's currently hit and miss, roughly half work flawlessly, some crash on start, some hang...
Comment 37 commit-hook freebsd_committer 2018-11-22 09:42:47 UTC
A commit references this bug:

Author: tijl
Date: Thu Nov 22 09:41:46 UTC 2018
New revision: 340755
URL: https://svnweb.freebsd.org/changeset/base/340755

Log:
  MFC r340631:

  Do proper copyin of control message data in the Linux sendmsg syscall.

  Instead of calling m_append with a user address, allocate an mbuf cluster
  and copy data into it using copyin.  For the SCM_CREDS case, instead of
  zeroing a stack variable and appending that to the mbuf, zero part of the
  mbuf cluster directly.  One mbuf cluster is also the size limit used by
  the FreeBSD sendmsg syscall (uipc_syscalls.c:sockargs()).

  PR:		217901
  Reviewed by:	kib

Changes:
_U  stable/12/
  stable/12/sys/compat/linux/linux_socket.c
Comment 38 commit-hook freebsd_committer 2018-11-22 09:42:51 UTC
A commit references this bug:

Author: tijl
Date: Thu Nov 22 09:41:54 UTC 2018
New revision: 340756
URL: https://svnweb.freebsd.org/changeset/base/340756

Log:
  MFC r340631:

  Do proper copyin of control message data in the Linux sendmsg syscall.

  Instead of calling m_append with a user address, allocate an mbuf cluster
  and copy data into it using copyin.  For the SCM_CREDS case, instead of
  zeroing a stack variable and appending that to the mbuf, zero part of the
  mbuf cluster directly.  One mbuf cluster is also the size limit used by
  the FreeBSD sendmsg syscall (uipc_syscalls.c:sockargs()).

  PR:		217901
  Reviewed by:	kib

Changes:
_U  stable/11/
  stable/11/sys/compat/linux/linux_socket.c
Comment 39 commit-hook freebsd_committer 2018-11-22 09:47:57 UTC
A commit references this bug:

Author: tijl
Date: Thu Nov 22 09:47:43 UTC 2018
New revision: 340757
URL: https://svnweb.freebsd.org/changeset/base/340757

Log:
  MFC r340674:

  Fix another user address dereference in linux_sendmsg syscall.

  This was hidden behind the LINUX_CMSG_NXTHDR macro which dereferences its
  second argument.  Stop using the macro as well as LINUX_CMSG_FIRSTHDR.  Use
  the size field of the kernel copy of the control message header to obtain
  the next control message.

  PR:		217901

Changes:
_U  stable/12/
  stable/12/sys/compat/linux/linux_socket.c
Comment 40 commit-hook freebsd_committer 2018-11-22 09:49:01 UTC
A commit references this bug:

Author: tijl
Date: Thu Nov 22 09:47:52 UTC 2018
New revision: 340758
URL: https://svnweb.freebsd.org/changeset/base/340758

Log:
  MFC r340674:

  Fix another user address dereference in linux_sendmsg syscall.

  This was hidden behind the LINUX_CMSG_NXTHDR macro which dereferences its
  second argument.  Stop using the macro as well as LINUX_CMSG_FIRSTHDR.  Use
  the size field of the kernel copy of the control message header to obtain
  the next control message.

  PR:		217901

Changes:
_U  stable/11/
  stable/11/sys/compat/linux/linux_socket.c
Comment 41 commit-hook freebsd_committer 2018-11-22 13:10:36 UTC
A commit references this bug:

Author: tijl
Date: Thu Nov 22 13:09:43 UTC 2018
New revision: 340761
URL: https://svnweb.freebsd.org/changeset/base/340761

Log:
  MFS r340755:

  Do proper copyin of control message data in the Linux sendmsg syscall.

  Instead of calling m_append with a user address, allocate an mbuf cluster
  and copy data into it using copyin.  For the SCM_CREDS case, instead of
  zeroing a stack variable and appending that to the mbuf, zero part of the
  mbuf cluster directly.  One mbuf cluster is also the size limit used by
  the FreeBSD sendmsg syscall (uipc_syscalls.c:sockargs()).

  PR:		217901
  Reviewed by:	kib
  Approved by:	re (marius)

Changes:
_U  releng/12.0/
  releng/12.0/sys/compat/linux/linux_socket.c
Comment 42 commit-hook freebsd_committer 2018-11-22 13:12:42 UTC
A commit references this bug:

Author: tijl
Date: Thu Nov 22 13:12:17 UTC 2018
New revision: 340762
URL: https://svnweb.freebsd.org/changeset/base/340762

Log:
  MFS r340757:

  Fix another user address dereference in linux_sendmsg syscall.

  This was hidden behind the LINUX_CMSG_NXTHDR macro which dereferences its
  second argument.  Stop using the macro as well as LINUX_CMSG_FIRSTHDR.  Use
  the size field of the kernel copy of the control message header to obtain
  the next control message.

  PR:		217901
  Approved by:	re (marius)

Changes:
_U  releng/12.0/
  releng/12.0/sys/compat/linux/linux_socket.c
Comment 43 Tijl Coosemans freebsd_committer 2018-11-22 14:27:23 UTC
Created attachment 199451 [details]
patch5

This version of the patch for x11/nvidia-driver adds calls to linux_device_register_handler so the kernel reports the right device major and minor numbers to linux programs.  You should no longer need to preload anything.
Comment 44 Tijl Coosemans freebsd_committer 2018-11-22 14:47:43 UTC
(In reply to Hannes Hauswedell from comment #36)
> I am busy testing applications right now, is there a place I should report
> failing applications that I know work on Linux? It's currently hit and miss,
> roughly half work flawlessly, some crash on start, some hang...

Well, let's get the nvidia-driver working first.  Then you can file bugs for applications that fail.  Some general tips:

Make sure you have tmpfs mounted on /compat/linux/dev/shm, linprocfs on /compat/linux/proc, and linsysfs on /compat/linux/sys.
Run Linux shell scripts using /compat/linux/bin/sh, not /bin/sh.
You can check for missing libraries with "/compat/linux/bin/sh ldd /path/to/program_or_library".
Applications that need to be installed are best installed under /compat/linux (e.g. /compat/linux/opt).  You can then put a little startup script in /usr/local/bin that looks like this:

#! /bin/sh
exec /compat/linux/path/to/program "$@"

Or this if program is a shell script:

#! /bin/sh
exec /compat/linux/bin/sh /path/to/program "$@"
Comment 45 Tijl Coosemans freebsd_committer 2018-11-28 14:01:06 UTC
Have you been able to test patch5?
Comment 46 Hannes Hauswedell 2018-11-28 17:54:19 UTC
(In reply to Tijl Coosemans from comment #45)

Yes, the patch makes glxinfo, glxgears and at least Pillars of Eternity work on my system without preloading anything. I had recently also updated to linux_base-c7 which broke all unigine-engine based games/benchmarks, but I suspect that's unrelated.

Here's my current status:

Works with patch, both -c6 and -c7:
 Pillars Of Eternity (64Bit) [no sound, though]
 games/linux-doom3-demo (32Bit)

Works with patch, but only linux_base-c6, not linux_base-c7:
 games/linux-unigine-valley (64Bit)
 OilRush (64Bit)
 [both segfault on -c7]

Works with neither, application crash (segfault or sigabrt):
 Dungeons 2 (64Bit GOG)
 Torment: Tides of Numenera (64Bit GOG)

Works with neither, application hang:
 Dungeons 3 (64Bit GOG)
 games/linux-quake4-demo (32Bit) [not tested on -c6]

So all-in-all, I strongly support merging this patch, but I would be really grateful if we could work on debugging some things further as currently most linux-apps don't work, yet.

Thanks for all the time invested into this!
Comment 47 Tijl Coosemans freebsd_committer 2018-11-29 14:07:14 UTC
Alexey,

I think patch5 (generated using --show-copies-as-adds) is ready to be committed (minus PORTREVISION bumps).  It does not include nvidia-uvm like earlier patches because that's out of scope for now.  Below is a detailed description.  Can you look into it, suggest changes, approve it?

M       Makefile
M       distinfo
D       files/extra-patch-src-Makefile
        > moved to files/extra-patch-src-Makefile.in
A  +    files/extra-patch-src-Makefile.in
        > moved from files/extra-patch-src-Makefile
D       files/extra-patch-src-nv-freebsd.h
        > moved to files/extra-patch-src-nv-freebsd.h.in
A  +    files/extra-patch-src-nv-freebsd.h.in
        > moved from files/extra-patch-src-nv-freebsd.h
D       files/extra-patch-src-nv-misc.h
        > moved to files/extra-patch-src-nv-misc.h.in
A  +    files/extra-patch-src-nv-misc.h.in
        > moved from files/extra-patch-src-nv-misc.h
A       files/extra-patch-src-nvidia_ctl.c.in
A       files/extra-patch-src-nvidia_dev.c.in
A       files/extra-patch-src-nvidia_linux.c.in
A       files/extra-patch-src-nvidia_pci.c.in
M       files/extra-patch-src_nvidia-modeset_nvidia-modeset-freebsd.c
D       files/extra-patch-src_nvidia_Makefile
D       files/extra-patch-src_nvidia_nv-freebsd.h
D       files/extra-patch-src_nvidia_nv-misc.h
M       pkg-plist

Log:
Install both 32 bit and 64 bit Linux libraries from the official Linux driver distribution instead of the ones bundled with the FreeBSD driver which are incomplete and 32 bit only.  This makes amd64 Linux OpenGL programs work on FreeBSD 12.0 and recent 11-STABLE.  This does not enable CUDA.

- Initialise DISTFILES so a file can be added by the LINUX option.
- Set EXTRACT_ONLY because the file added by the LINUX option is a self-extracting archive handled in post-extract.
- The patches for <355.06 and >=355.06 are identical except for the path.  Keep only one set (listed in SUB_PATCHES) because more patches have been added.  Modify their path in pre-patch.  The patches are renamed from extra-patch-* to extra-patch-*.in.
- Set OPTIONS_SUB and use option helpers for the LINUX option.
- Define LINUX_ARCH_SUFX like ARCH_SUFX because LINUX_ARCH can be i386 on amd64.
- Set LINUX_LIBDIR to the main lib directory: (/compat/linux)/usr/lib on i386 and (/compat/linux)/usr/lib64 on amd64.  (The compat32 directory is (/compat/linux)/usr/lib.)
- Define LINUX_LIBS, LINUX_LINKS, LINUX32_LIBS, LINUX32_LINKS, LINUX_PROGS, LINUX_MANS, and LINUX_DATA for all driver versions from 304.137 to 390.87.  LINUX_LIBS and LINUX32_LIBS contain main libraries (i386/amd64) and compat32 libraries respectively.  These are installed with INSTALL_LIB in post-install.  LINUX_LINKS and LINUX32_LINKS contain symbolic link sources and their target.  LINUX_PROGS are binaries installed with INSTALL_PROGRAM.  LINUX_MANS are manuals installed with INSTALL_MAN.  LINUX_DATA are data files and their destination, installed with INSTALL_DATA.  The .if blocks for the different versions are sorted alphabetically on the first file.
- Define PLIST_RE for all versions.  These are sed commands applied to TMPPLIST in post-install.
- Include bsd.port.pre.mk instead of bsd.port.options.mk to obtain LINUX_ARCH.
- Adjust post-patch to disable installation of Linux libraries bundled with the FreeBSD driver.
- Add post-patch-LINUX-on to setup nvidia_icd.json like nvidia-installer does on Linux.
- Adjust existing post-install TMPPLIST edits to apply only to non-Linux files.
- Add post-install-LINUX-on to install the Linux files.
- Add files/extra-patch-src-nvidia_ctl.c.in and extra-patch-src-nvidia_dev.c.in.  These patches make Linux stat(2) return the right major and minor number for /dev/nvidiactl and /dev/nvidia[0-9]+.  Recent versions of the Linux libraries verify these. [1]  Define the struct passed to linux_device_register_handler on the stack in nvidia_*_attach instead of as a global variable in case multiple graphics cards are attached in parallel.
- Add files/extra-patch-src-nvidia_linux.c.in.  It rewrites the Linux ioctl handler.  The current implementation does not copyin arguments leading to kernel panics on FreeBSD 12 on machines that support Supervisor Mode Access Prevention (SMAP).
- Add files/extra-patch-src-nvidia_pci.c.in.  It adds a dependency on the linux_common kernel module for linux_device_register_handler.
- Modify files/extra-patch-src_nvidia-modeset_nvidia-modeset-freebsd.c to add major/minor numbers (global struct here, because it is constant), rewrite ioctl, and add a dependency on linux_common for /dev/nvidia-modeset.

PR: 217901
Analysed by: Alex S <iwtcex@gmail.com> [1]
Tested by: Hannes Hauswedell <h2+fbsdports@fsfe.org> (390.87)
Tested by: tijl (304.137)
Comment 48 Alex S 2018-11-29 15:10:26 UTC
> These patches make Linux stat(2) return the right major and minor number for /dev/nvidiactl and /dev/nvidia[0-9]+.  Recent versions of the Linux libraries verify these.

A minor nitpick: all versions of the driver, including legacy, have this check. Disabling the check through "ModifyDeviceFiles: 0" setting (should be placed in /proc/driver/nvidia/params) or faking successful remove/chmod/chown/xmknod sequence seems to work as well, so the new thing there is some kind of regression on some fallback path. It's not a deliberate feature.
Comment 49 Tijl Coosemans freebsd_committer 2018-11-29 21:58:15 UTC
(In reply to Alex S from comment #48)
Hmm, with version 304 I don't see calls to stat(2) or unlink(2).
Comment 50 Alex S 2018-11-29 23:40:46 UTC
(In reply to Tijl Coosemans from comment #49)

Looks like this particular check was introduced in addition to nvidia-modprobe utility in 319.12. Compare https://download.nvidia.com/XFree86/Linux-x86_64/304.137/README/faq.html#devicenodes against https://download.nvidia.com/XFree86/Linux-x86_64/319.12/README/faq.html#devicenodes.
Comment 51 commit-hook freebsd_committer 2018-12-14 16:25:06 UTC
A commit references this bug:

Author: tijl
Date: Fri Dec 14 16:24:05 UTC 2018
New revision: 487446
URL: https://svnweb.freebsd.org/changeset/ports/487446

Log:
  Install both 32 bit and 64 bit Linux libraries from the official Linux
  driver distribution instead of the ones bundled with the FreeBSD driver
  which are incomplete and 32 bit only.  This makes amd64 Linux OpenGL
  programs work on FreeBSD 12.0 and recent 11-STABLE.  This does not enable
  CUDA.

  - Initialise DISTFILES so a file can be added by the LINUX option.
  - Set EXTRACT_ONLY because the file added by the LINUX option is a
    self-extracting archive handled in post-extract.
  - The patches for <355.06 and >=355.06 are identical except for the path.
    Keep only one set (listed in SUB_PATCHES) because more patches have been
    added.  Modify their path in pre-patch.  The patches have been renamed
    from extra-patch-* to extra-patch-*.in.
  - Set OPTIONS_SUB and use option helpers for the LINUX option.
  - Define LINUX_ARCH_SUFX like ARCH_SUFX because LINUX_ARCH can be i386 on
    amd64.
  - Set LINUX_LIBDIR to the main lib directory: (/compat/linux)/usr/lib on
    i386 and (/compat/linux)/usr/lib64 on amd64.  (The compat32 directory is
    (/compat/linux)/usr/lib.)
  - Define LINUX_LIBS, LINUX_LINKS, LINUX32_LIBS, LINUX32_LINKS, LINUX_PROGS,
    LINUX_MANS, and LINUX_DATA for all driver versions from 304.137 to 390.87.
    LINUX_LIBS and LINUX32_LIBS contain main libraries (i386/amd64) and
    compat32 libraries respectively.  These are installed with INSTALL_LIB in
    post-install.  LINUX_LINKS and LINUX32_LINKS contain symbolic link sources
    and their target.  LINUX_PROGS are binaries installed with
    INSTALL_PROGRAM.  LINUX_MANS are manuals installed with INSTALL_MAN.
    LINUX_DATA are data files and their destination, installed with
    INSTALL_DATA.  The .if blocks for the different versions are sorted
    alphabetically on the first file.
  - Define PLIST_RE for all versions.  These are sed commands applied to
    TMPPLIST in post-install.
  - Include bsd.port.pre.mk instead of bsd.port.options.mk to obtain
    LINUX_ARCH.
  - Adjust post-patch to disable installation of Linux libraries bundled with
    the FreeBSD driver.
  - Add post-patch-LINUX-on to setup nvidia_icd.json like nvidia-installer
    does on Linux.
  - Adjust existing post-install TMPPLIST edits to apply only to non-Linux
    files.
  - Add post-install-LINUX-on to install the Linux files.
  - Add files/extra-patch-src-nvidia_ctl.c.in and
    extra-patch-src-nvidia_dev.c.in.  These patches make Linux stat(2) return
    the right major and minor number for /dev/nvidiactl and /dev/nvidia[0-9]+.
    Recent versions of the Linux libraries verify these. [1]  Define the
    struct passed to linux_device_register_handler on the stack in
    nvidia_*_attach instead of as a global variable in case multiple graphics
    cards are attached in parallel.
  - Add files/extra-patch-src-nvidia_linux.c.in.  It rewrites the Linux ioctl
    handler.  The current implementation does not copyin arguments leading to
    kernel panics on FreeBSD 12 on machines that support Supervisor Mode
    Access Prevention (SMAP).
  - Add files/extra-patch-src-nvidia_pci.c.in.  It adds a dependency on the
    linux_common kernel module for linux_device_register_handler.
  - Modify files/extra-patch-src_nvidia-modeset_nvidia-modeset-freebsd.c to
    add major/minor numbers (global struct here, because it is constant),
    rewrite ioctl, and add a dependency on linux_common for
    /dev/nvidia-modeset.

  PR:		217901
  Analysed by:	Alex S <iwtcex@gmail.com> [1]
  Tested by:	Hannes Hauswedell <h2+fbsdports@fsfe.org> (390.87)
  Tested by:	tijl (304.137)
  Approved by:	maintainer timeout (15 days)

Changes:
  head/x11/nvidia-driver/Makefile
  head/x11/nvidia-driver/distinfo
  head/x11/nvidia-driver/files/extra-patch-src-Makefile
  head/x11/nvidia-driver/files/extra-patch-src-Makefile.in
  head/x11/nvidia-driver/files/extra-patch-src-nv-freebsd.h
  head/x11/nvidia-driver/files/extra-patch-src-nv-freebsd.h.in
  head/x11/nvidia-driver/files/extra-patch-src-nv-misc.h
  head/x11/nvidia-driver/files/extra-patch-src-nv-misc.h.in
  head/x11/nvidia-driver/files/extra-patch-src-nvidia_ctl.c.in
  head/x11/nvidia-driver/files/extra-patch-src-nvidia_dev.c.in
  head/x11/nvidia-driver/files/extra-patch-src-nvidia_linux.c.in
  head/x11/nvidia-driver/files/extra-patch-src-nvidia_pci.c.in
  head/x11/nvidia-driver/files/extra-patch-src_nvidia-modeset_nvidia-modeset-freebsd.c
  head/x11/nvidia-driver/files/extra-patch-src_nvidia_Makefile
  head/x11/nvidia-driver/files/extra-patch-src_nvidia_nv-freebsd.h
  head/x11/nvidia-driver/files/extra-patch-src_nvidia_nv-misc.h
  head/x11/nvidia-driver/pkg-plist
  head/x11/nvidia-driver-304/Makefile
  head/x11/nvidia-driver-340/Makefile