Memory mapping of framebuffer managed by vesa syscons driver fails with EINVAL in some video modes, while working as expected in other ones. This is caused by erroneous range check in kernel mode code. Fix: In file /usr/src/sys/dev/fb/vesa.c in function vesa_mmap() comment out the following lines: /* va_window_size == va_buffer_size/vi_planes */ /* XXX: is this correct? */ if (offset > adp->va_window_size - PAGE_SIZE) return (-1); Then recompile kernel, install it and reboot. It's a temporary workaround just to localize the cause of the problem. Proper range check needs to be inserted in that place. Probably: if( offset >= adp->va_info.vi_buffer_size ) return (-1); How-To-Repeat: 1) Install FreeBSD 8.2 Release on VmWare (6.5.3/Win64 in my environment, but I don't think it does really matter) 2) Enable VESA framebuffer support in kernel (add the following lines to kernel config file in /usr/src/sys/i386/conf): options SC_PIXEL_MODE options VESA 3) Rebuild kernel, install new kernel, reboot 4) Dump list of available video modes with vidcontrol(1) vidcontrol -i mode | more Make sure 1024x768x32 and 800x600x32 are there. Write down mode numbers. Under VmWare they are 321 and 320. 5) Make sure both modes work: vidcontrol MODE_321 vidcontrol MODE_320 6) Prepare small program to test video mode mapping: #include <sys/mman.h> #include <stdio.h> int main( int argc, char* argv[] ) { void* p; if( argc >= 3 ) { p = mmap( 0, atoi(argv[1])*atoi(argv[2])*4, PROT_READ | PROT_WRITE, MAP_SHARED, 1 /*stdout*/, 0 ); if( !p ) perror("mmap"); else printf("mmapped OK\n"); } return 0; } # gcc test_mmap.c -o test_mmap 7) Test memory mapping in 1024x768: $ vidcontrol MODE_321 $ test_mmap 1024 768 mmapped OK 8) Test memory mapping in 800x600: $ vidcontrol MODE_320 $ test_mmap 800 600 mmap: invalid argument
For bugs matching the following criteria: Status: In Progress Changed: (is less than) 2014-06-01 Reset to default assignee and clear in-progress tags. Mail being skipped
A commit references this bug: Author: bde Date: Sun Mar 24 16:47:44 UTC 2019 New revision: 345474 URL: https://svnweb.freebsd.org/changeset/base/345474 Log: Fix libvgl to not always fail to initialize due to its invalid mmap() args (neither MAP_PRIVATE nor MAP_SHARED). It was broken in r271635 and/or r271724 by stricter checking. The compatibility code in r271724 doesn't work for my old binaries (actually new binaries with old libraries). PR: needed to test the fix for PR 162373 Changes: head/lib/libvgl/main.c
This affects more than just 1 person. The suggested fix isn't quite right. The simplest fix is to just take the "- PAGE_SIZE" out. The page maps are set up to cover the whole buffer, in fact to cover the largest buffer size in any VESA mode, and va_window_size is the right field to use. There's a similar check, with a similar query, in the VGA driver. That's never been questioned probably because the VGA memory windows are usually a whole multiple of the page size. Applications programs have been working around this for years by rounding va_window_size down to the next lower whole multiple of the page size. But this is at the expense of their being unable to memory map a number of pixel rows at the bottom of the screen.