Bug 243533 - vt_fb.c can overwrite frame buffer bounds if stride length is not a multiple of bytes-per-pixel
Summary: vt_fb.c can overwrite frame buffer bounds if stride length is not a multiple ...
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Only Me
Assignee: Ed Maste
URL:
Keywords: patch, vt
Depends on:
Blocks:
 
Reported: 2020-01-23 00:44 UTC by Thomas Skibo
Modified: 2020-04-25 15:51 UTC (History)
1 user (show)

See Also:


Attachments
fix vt_fb_blank(). (1.14 KB, text/plain)
2020-01-23 00:44 UTC, Thomas Skibo
no flags Details
fix vt_fb_blank(). (955 bytes, text/plain)
2020-01-23 16:57 UTC, Thomas Skibo
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Thomas Skibo 2020-01-23 00:44:15 UTC
Created attachment 210977 [details]
fix vt_fb_blank().

I'm developing a frame buffer driver for hardware using 3 bytes per pixel but the hardware requires the stride to be a multiple of 256 bytes.  Because the stride is not a multiple of 3 bytes, the way vt_fb_blank() is coded, it writes past the end of each stride and, on the last line, writes past the end of the frame buffer.  This is caught by a KASSERT in vt_fb_mem_wr1().

I think the loops in vt_fb_blank() could just stop at the end of the line (fb_width) instead of clearing memory all the way to the end of a stride.  The other way would be to limit the loops with fb_stride - 1, fb_stride - 2, fb_stride - 3 for the cases of 2,3,4 bytes per pixel.
Comment 1 Thomas Skibo 2020-01-23 16:57:47 UTC
Created attachment 210991 [details]
fix vt_fb_blank().
Comment 2 Thomas Skibo 2020-01-23 16:59:08 UTC
Comment on attachment 210991 [details]
fix vt_fb_blank().

My previous patch was wrong.  fb_width is the width in pixels, not bytes.  This was my other suggested fix.
Comment 3 commit-hook freebsd_committer 2020-04-04 00:33:16 UTC
A commit references this bug:

Author: emaste
Date: Sat Apr  4 00:31:31 UTC 2020
New revision: 359626
URL: https://svnweb.freebsd.org/changeset/base/359626

Log:
  vt: avoid overrun when stride is not a multiple of bytes per pixel

  The reporter is developing a frame buffer driver for hardware using
  3 bytes per pixel, but a stride that's a multiple of 256.  Previously
  this resulted in writing beyond the end of each stride.  On the last
  row this attempted to write past the end of the frame buffer, triggering
  the assertion in vt_fb_mem_wr1().

  PR:		243533
  MFC after:	2 weeks
  Submitted by:	Thomas Skibo

Changes:
  head/sys/dev/vt/hw/fb/vt_fb.c
Comment 4 commit-hook freebsd_committer 2020-04-25 15:18:17 UTC
A commit references this bug:

Author: emaste
Date: Sat Apr 25 15:17:43 UTC 2020
New revision: 360308
URL: https://svnweb.freebsd.org/changeset/base/360308

Log:
  MFC r359626: vt: avoid overrun when stride is not a multiple of bytes per pixel

  PR:		243533
  Submitted by:	Thomas Skibo

Changes:
_U  stable/12/
  stable/12/sys/dev/vt/hw/fb/vt_fb.c
Comment 5 commit-hook freebsd_committer 2020-04-25 15:28:19 UTC
A commit references this bug:

Author: emaste
Date: Sat Apr 25 15:27:45 UTC 2020
New revision: 360309
URL: https://svnweb.freebsd.org/changeset/base/360309

Log:
  MFC r359626: vt: avoid overrun when stride is not a multiple of bytes per pixel

  The reporter is developing a frame buffer driver for hardware using
  3 bytes per pixel, but a stride that's a multiple of 256.  Previously
  this resulted in writing beyond the end of each stride.  On the last
  row this attempted to write past the end of the frame buffer, triggering
  the assertion in vt_fb_mem_wr1().

  PR:		243533
  Submitted by:	Thomas Skibo

Changes:
_U  stable/11/
  stable/11/sys/dev/vt/hw/fb/vt_fb.c