Bug 228287 - x11/nvidia-driver linux compatibility not working anymore
Summary: x11/nvidia-driver linux compatibility not working anymore
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: amd64 Any
: --- Affects Only Me
Assignee: Alexey Dokuchaev
: 231812 (view as bug list)
Depends on: 217901
  Show dependency treegraph
Reported: 2018-05-16 08:25 UTC by Stefan Rumetshofer
Modified: 2019-01-18 03:04 UTC (History)
5 users (show)

See Also:
bugzilla: maintainer-feedback? (danfe)

output of the command linux-doom3 (2 bytes, text/plain)
2018-05-16 08:25 UTC, Stefan Rumetshofer
no flags Details
output of the command linux-doom3 (3.77 KB, text/plain)
2018-05-16 08:29 UTC, Stefan Rumetshofer
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Stefan Rumetshofer 2018-05-16 08:25:54 UTC
Created attachment 193445 [details]
output of the command linux-doom3

I tried to run linux-doom3 but get a Segmentation fault:

Fatal X Error:
  Major opcode of failed request: 153
  Minor opcode of failed request: 3
  Serial number of failed request: 42

See attachement linux-doom3

NVIDIA driver is : nvidia-driver-390.48  =   up-to-date with remote
Hardware is a GeForce GTX 560
nvidia driver an the linux modules are loaded:

% kldstat: 
Id Refs Address            Size     Name
 1   76 0xffffffff80200000 1f6e480  kernel
 2    1 0xffffffff82170000 316728   zfs.ko
 3    2 0xffffffff82487000 cb78     opensolaris.ko
 4    1 0xffffffff82b11000 5936     fdescfs.ko
 5    1 0xffffffff82b17000 a8c4     linprocfs.ko
 6    4 0xffffffff82b22000 7b27     linux_common.ko
 7    1 0xffffffff82b2a000 5aaa     linsysfs.ko
 8    2 0xffffffff82b30000 c40d3f   nvidia.ko
 9    2 0xffffffff83771000 4286e    linux.ko
10    1 0xffffffff837b4000 f408c    nvidia-modeset.ko
11    1 0xffffffff838a9000 85b9     cuse.ko
12    1 0xffffffff838b2000 e580     fuse.ko
13    3 0xffffffff838c1000 54416    vboxdrv.ko
14    1 0xffffffff83916000 7ca0     vkbd.ko
15    1 0xffffffff8391e000 245c     ulpt.ko
16    1 0xffffffff83921000 2986     uhid.ko
17    2 0xffffffff83924000 2cbf     vboxnetflt.ko
18    2 0xffffffff83927000 c57d     netgraph.ko
19    1 0xffffffff83934000 43da     ng_ether.ko
20    1 0xffffffff83939000 4016     vboxnetadp.ko
22    1 0xffffffff8397b000 bb55     tmpfs.ko
23    1 0xffffffff83987000 7fd      cd9660_iconv.ko
24    2 0xffffffff83988000 44ca     libiconv.ko
25    1 0xffffffff8398d000 6679     nullfs.ko
26    1 0xffffffff83994000 7e9      udf_iconv.ko
27    1 0xffffffff83995000 914e     udf.ko
28    1 0xffffffff8393e000 3c947    linux64.ko

running /compat/linux/usr/bin/glxgears gives me a similar error:

libGL error: No matching fbConfigs or visuals found
libGL error: failed to load driver: swrast
X Error of failed request:  BadValue (integer parameter out of range for operation)
  Major opcode of failed request:  153 (GLX)
  Minor opcode of failed request:  3 (X_GLXCreateContext)
  Value in failed request:  0x0
  Serial number of failed request:  35
  Current serial number in output stream:  37

I have seen that the following files in /compat/linux/usr/lib are empty:
Comment 1 Stefan Rumetshofer 2018-05-16 08:29:45 UTC
Created attachment 193446 [details]
output of the command linux-doom3

Sorry, first attachement was empty.
Comment 2 Stefan Rumetshofer 2018-05-18 09:25:29 UTC
(In reply to Stefan Rumetshofer from comment #0)

I did some further testing with different FreeBSD versions and different nvidia-driver version. Testet with linux-doom3.

FreeBSD 11.1-RELEASE nvidia-driver-390.48 -> not working
FreeBSD 11.1-RELEASE nvidia-driver-340-340.106 -> not working

FreeBSD 10.4-RELEASE nvidia-driver-390.48 -> not working
FreeBSD 10.4-RELEASE nvidia-driver-340-340.106 -> working

FreeBSD 12.0-CURRENT nvidia-driver-390.48 -> not working
FreeBSD 12.0-CURRENT nvidia-driver-340-340.106 -> working (only nvidia.ko can be loaded)
Comment 3 Sean Farley freebsd_committer 2018-07-22 00:04:49 UTC
I also am unable to run Linux 32-bit games such as Neverwinter Nights (games/linux-nwnclient) with nvidia-driver-390.67.  Those three empty Linux libraries are suspicious.

Does anyone have a good contact with Nvidia?  As the forum (https://devtalk.nvidia.com/default/board/97/freebsd/) seems unresponsive, possibly the E-mail address freebsd-gfx-bugs@nvidia.com would work.  Has anyone tried it?  I will no one else has already done it.
Comment 4 Patrick Mackinlay 2018-07-31 22:45:02 UTC
Did you hear back from nvidia? This issue affects me as well
Comment 5 Sean Farley freebsd_committer 2018-08-04 21:29:31 UTC
I have just sent a request for help to the E-mail address I mentioned.  There are actually two issues.  One is that there are three empty libraries.  To what issues that causes, I can only see that a tool such as eglinfo32 (from the Arch Linux repo) cannot run:

./eglinfo32: error while loading shared libraries: /lib/libEGL.so.1: file too short

As for the issue of running 32-bit Linux binaries such as games, I have found the trigger:  indirect GLX.  Apparently, indirect GLX is needed for the games to run locally.  At some point in the past, Xorg server disabled it by default.  With it enabled, I am able to run Linux games.

However, from I have read, iglx is not needed for direct application access, so I am puzzled why the Nvidia libGL library needs it.  I am hoping that it is a simple fix on their part or ours to stop requiring it.

If you wish to try with xdm, change /usr/local/etc/X11/xdm/Xservers to include +iglx:
:0 local /usr/local/bin/X :0 +iglx

For those of us with /etc/X11/xorg.conf files, supposedly add to the Screen section:
Option "AllowIndirectGLXProtocol" "True"
Comment 6 Patrick Mackinlay 2018-08-06 00:04:41 UTC
I found that it makes no difference if you add  Option "AllowIndirectGLXProtocol" "True" to xorg.conf, that does not enable indirect GLX.

Enabling indirect glx (start X with +iglx) does fix some issues. I am now able to run /compat/linux/usr/bin/glxgears. However, the performance is atrocious, it reports 60fps on the command line, but visually the app doesn't work. If renders at 1 fps and if I maximize the window the gears dont change size for over 3 seconds. Also the linux doom 3 demo doesn't run (doesn't crash but its so slow it never renders the menu even).

I tried both the 390.77 and 304.137 drivers. Same results for both. Other than the ridiculously slow graphics all else works and cpu usage is close to 0 when running glxgears.
Comment 7 Sean Farley freebsd_committer 2018-08-06 02:49:51 UTC
I think I read somewhere that settings in xorg.conf were not working for some people to enable indirect GLX.

I tested Linux glxgears (32-bit) using nvidia-driver-390.77, xorg-server-1.18.4_9,1 and linux_base-c7-7.4.1708_6 on FreeBSD 11.2-STABLE r336115 with better success.

# ./glxgears32
234420 frames in 5.0 seconds = 46881.246 FPS
235089 frames in 5.0 seconds = 47017.338 FPS

# file glxgears32
glxgears32: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=b3c02e6e2ba1944afa8b311c18e8b86c730fbc0d, stripped

# ./glxinfo32
name of display: :0.0
display: :0  screen: 0
direct rendering: No (If you want to find out why, try setting LIBGL_DEBUG=verbose)
server glx vendor string: NVIDIA Corporation
server glx version string: 1.4
OpenGL vendor string: NVIDIA Corporation
OpenGL renderer string: GeForce GTX 960/PCIe/SSE2
OpenGL version string: 3.0.0 NVIDIA 390.77

Some thoughts:
1. Do you see information like that or something else?  Perhaps some warnings?  If I run glxgears from linux-c7-glx-utils-8.2.0_4, then I see warnings, but it still runs just as fast.
libGL error: No matching fbConfigs or visuals found
libGL error: failed to load driver: swrast

2. Anything suspicious in /var/log/Xorg.0.log about GLX?  Do you see Nvidia loaded within it?

3. What permissions do you have on the nvidia* devices in /dev?
Comment 8 Patrick Mackinlay 2018-08-09 22:06:38 UTC
My X is definitely setup correctly, to turn on indirect rendering, as you mentioned, you need to start X with +iglx. I use startx so I can do this in

exec X +iglx

or in xorg.conf:

Section "ServerFlags"
    Option      "AllowIndirectGLX" "True"
    Option      "IndirectGLX" "True"

note that you need the IndirectGLX option, not the AllowIndirectGLX. The /var/log/Xorg.0.log log acknowledges the IndirectGLX option with
[   187.137] (**) Option "IndirectGLX" "on"
There is no mention of AllowIndirectGLX in the log (but it is mentioned on various source son the internet), my guess is that doesn't do anything anymore (why turn it on and then disallow it?).

Once I have X running with indirect rendering then the two linux glxinfo commands work, and report the glx vendor as NVIDIA. They also report direct rendering as No. I thing my glxgears is just slow because my NVIDIA card is old (GeForce GT-520), probably indirect rendering is slow for such a card.

When running glxgears from linux-c7-glx-utils-8.2.0_4 I get the same error as you do.
Comment 9 Sean Farley freebsd_committer 2018-08-12 03:14:32 UTC
(In reply to Patrick Mackinlay from comment #8)
I doubt that your card is that much slower than mine.  One thing I notice is that you are getting 60fps.  That sound like you have "Sync to VBlank" enabled.  I am not sure if there is a xorg.conf option for it, but it would be seen in nvidia-settings under:  X Screen 0 -> OpenGL Settings -> Sync to VBlank.  It would sync it with the display rate of your monitor (i.e., 60Hz).  There are several places within the nvidia-driver README file for vblank.

Also, the glxgears file at /compat/linux/usr/bin/glxgears is 64-bit on my machine.  I do not know if that will necessarily take advantage of the Nvidia drivers or not.

Possibly, the inability to change indirect GLX within xorg.conf is a bug with our version of the X server.  We have 1.18 while Linux is on 1.20.  It may be fixed there (or not).
Comment 10 Sean Farley freebsd_committer 2018-10-04 20:41:25 UTC
I posted about this issue on Nvidia forum.  I hope this catches a developer's eye there.

Comment 11 Tijl Coosemans freebsd_committer 2018-11-09 21:11:01 UTC
*** Bug 231812 has been marked as a duplicate of this bug. ***
Comment 12 Tijl Coosemans freebsd_committer 2018-11-09 21:13:24 UTC
Can you try the patch in bug 217901?  It reworks the installation of the Linux libraries.
Comment 13 Tijl Coosemans freebsd_committer 2018-12-14 16:29:59 UTC
Should be fixed by ports r487446.
Comment 14 Sean Farley freebsd_committer 2018-12-29 17:05:15 UTC
I confirm that the 32-bit Linux Neverwinter Nights works without the Indirect GLX flag.  Also, both the 32-bit and 64-bit Linux glxgears binaries work.

The only oddity that I experience is when stopping a running glxgears, regardless if FreeBSD or Linux binary.  Sometimes when I hit Ctrl-C in the xterm to stop it, the entire X session freezes for a few seconds.  If I have a running top in a separate window, I can see that the X server is at 100%.

Thank you for all your hard work!
Comment 15 Sean Farley freebsd_committer 2018-12-29 17:19:44 UTC
Another oddity is with xdriinfo.  The Linux binary produces:
    Screen 0: not direct rendering capable.
    Screen 1: not direct rendering capable.

The FreeBSD binary produces:
    libGL is too old.
Comment 16 Tijl Coosemans freebsd_committer 2018-12-29 19:51:32 UTC
(In reply to Sean Farley from comment #15)
xdriinfo is probably only for mesa dri.  If you want to know if direct rendering is available you can run glxinfo.
Comment 17 Alex S 2019-01-03 22:38:30 UTC
(In reply to Sean Farley from comment #14)

> stopping a running glxgears, regardless if FreeBSD or Linux binary.
> Sometimes when I hit Ctrl-C in the xterm to stop it,
> the entire X session freezes for a few seconds.

I get somewhat similar freezes when I'm close to exhausting video memory (courtesy of Firefox Quantum, of course).