After I upgraded from 346.96 to 367.44 (with replacement of nvidia.ko -> nvidia-modeset.ko), Xorg began to misbehave. In kde4 after a while mouse clicks stop working, though the cursor itself moves. When I unplug the USB mouse and plug it back it works for a while and then stops working again. This happened on 10.3 after the nvidia-driver upgrade. I upgraded OS to 11.0 with full package reinstall, and the problem persisted. Then I downgraded nvidia-driver back to 346.96 and the problem is gone. So nvidia-driver is clearly the cause. This problem makes the system unusable.
I don't understand why nvidia driver would affect the mouse, but that is what I am observing.
Sorry, I probably got confused. It wasn't nvidia-driver that caused the problem.