Created attachment 210977 [details] fix vt_fb_blank(). I'm developing a frame buffer driver for hardware using 3 bytes per pixel but the hardware requires the stride to be a multiple of 256 bytes. Because the stride is not a multiple of 3 bytes, the way vt_fb_blank() is coded, it writes past the end of each stride and, on the last line, writes past the end of the frame buffer. This is caught by a KASSERT in vt_fb_mem_wr1(). I think the loops in vt_fb_blank() could just stop at the end of the line (fb_width) instead of clearing memory all the way to the end of a stride. The other way would be to limit the loops with fb_stride - 1, fb_stride - 2, fb_stride - 3 for the cases of 2,3,4 bytes per pixel.
Created attachment 210991 [details] fix vt_fb_blank().
Comment on attachment 210991 [details] fix vt_fb_blank(). My previous patch was wrong. fb_width is the width in pixels, not bytes. This was my other suggested fix.
A commit references this bug: Author: emaste Date: Sat Apr 4 00:31:31 UTC 2020 New revision: 359626 URL: https://svnweb.freebsd.org/changeset/base/359626 Log: vt: avoid overrun when stride is not a multiple of bytes per pixel The reporter is developing a frame buffer driver for hardware using 3 bytes per pixel, but a stride that's a multiple of 256. Previously this resulted in writing beyond the end of each stride. On the last row this attempted to write past the end of the frame buffer, triggering the assertion in vt_fb_mem_wr1(). PR: 243533 MFC after: 2 weeks Submitted by: Thomas Skibo Changes: head/sys/dev/vt/hw/fb/vt_fb.c
A commit references this bug: Author: emaste Date: Sat Apr 25 15:17:43 UTC 2020 New revision: 360308 URL: https://svnweb.freebsd.org/changeset/base/360308 Log: MFC r359626: vt: avoid overrun when stride is not a multiple of bytes per pixel PR: 243533 Submitted by: Thomas Skibo Changes: _U stable/12/ stable/12/sys/dev/vt/hw/fb/vt_fb.c
A commit references this bug: Author: emaste Date: Sat Apr 25 15:27:45 UTC 2020 New revision: 360309 URL: https://svnweb.freebsd.org/changeset/base/360309 Log: MFC r359626: vt: avoid overrun when stride is not a multiple of bytes per pixel The reporter is developing a frame buffer driver for hardware using 3 bytes per pixel, but a stride that's a multiple of 256. Previously this resulted in writing beyond the end of each stride. On the last row this attempted to write past the end of the frame buffer, triggering the assertion in vt_fb_mem_wr1(). PR: 243533 Submitted by: Thomas Skibo Changes: _U stable/11/ stable/11/sys/dev/vt/hw/fb/vt_fb.c