Currently, the devel/sdl12 port depends on X11 libraries unconditionally.
SDL can work without X11 though, using alternative video drivers such as svgalib or vgl. Setting WITHOUT_X11=yes in /etc/make.conf has no effect on devel/sdl12, it still depends on X11 libraries.
Fix: Proposed patch, tested and working for me:
How-To-Repeat: Set WITHOUT_X11=yes in /etc/make.conf
Install SDL and note X11 libraries being installed as a dependency.
Over to maintainer