Bug 18074

This machine is a print server only.
Running the periodic/weekly script at 3:30 am on Saturday mornings
leaves the machine in a bad state.  Usually it panics (page fault)
and locks up on syncing disks or succeeds in rebooting, sometimes the
process dies and the machine soldiers on uselessly until it gets
rebooted.

Here's what on the console when it locks:

Fatal trap 12: page fault while in kernel mode
fault virtual address = 0x18c
fault code = supervisor read, page not present
instruction pointer = 0x8:0xc0208186
stack pointer = 0x10:0xc0228nnn
frame pointer = 0xc10:0xc0228nnn
code segment = base rx0, limit 0xfffff, type 0x16
             = DPL 0 , pres 1 def32 1, gran 1
processor eflags = interrupt enable, resume, IOPL=0
current process = Idle
interrupt mask = bio
trap number = 12
panic: page fault
syncing disks...


It panics whilst running the makewhatis perl script called from
the /etc/periodic/weekly cron job.
The call sequence is
  /etc/periodic/weekly/320.whatis
  /usr/libexec/makewhatis.local
  /usr/bin/makewhatis
  /usr/bin/perl

To investigate further, I decided to recompile the kernel
with "options DDB".
However, the problem then disappears!

So now I have two kernels with the only difference between
them being the presence of the kernel debugger; the one without
DDB panics, the one with DDB does not.

The kernel config is essentially just GENERIC with all unwanted device
drivers removed and two extra ppc devices inserted.


I have some basic questions:

  Is this a bad bug in the VM software?

  (I've seen similar errors on 3.2-RELEASE on system shutdown, but
  they haven't been critical to us.)

  Is it related to the other similar kern PRs I've found which
  have status "o":
    10397, 10542, 11330, 11196, 15644, 16890, 4186, 5975, 9961, 11266, 17680, 17961
  I understand that the VM system has been rewritten for FreeBSD 4.0
  which perhaps explains why the above PRs are still at status "o".
  
  Is it advisable to go to 4.0-RELEASE?


My faith in the reliability of FreeBSD has been undermined since 2.2.8
and I would like to know whether there is anything I can do or any way
I can help in order to restore it.  I am not a professional system
programmer or administrator, rather a fairly advanced user.

I have workarounds, but they are not fixes to what appears to be a
critical bug which could easily reoccur given the right circumstances.

I append the dmesg and kernel config files:

Copyright (c) 1992-1999 FreeBSD Inc.
Copyright (c) 1982, 1986, 1989, 1991, 1993
        The Regents of the University of California. All rights reserved.
FreeBSD 3.4-RELEASE #1: Mon Apr 17 19:04:02 BST 2000
    root@maths02.bangor.ac.uk:/usr/src/sys/compile/MATHS02
Timecounter "i8254"  frequency 1193182 Hz
CPU: Cyrix 486DX4 (486-class CPU)
  Origin = "CyrixInstead"  DIR=0x361f  Stepping=3  Revision=6
real memory  = 16777216 (16384K bytes)
config> en ed0
config> po ed0 0x300
config> ir ed0 5
config> iom ed0 0xca000
config> f ed0 0
config> q
avail memory = 13881344 (13556K bytes)
Preloaded elf kernel "kernel" at 0xc02aa000.
Preloaded userconfig_script "/boot/kernel.conf" at 0xc02aa09c.
Probing for devices on PCI bus 0:
chip0: <SiS 85c496> rev 0x31 on pci0.5.0
Probing for PnP devices:
Probing for devices on the ISA bus:
sc0 on isa
sc0: MDA/Hercules <16 virtual consoles, flags=0x0>
ed0 at 0x300-0x31f irq 5 maddr 0xca000 msize 8192 on isa
ed0: address 00:00:c0:03:b3:25, type WD8003EP (8 bit) 
atkbdc0 at 0x60-0x6f on motherboard
atkbd0 irq 1 on isa
psm0 not found
sio0 at 0x3f8-0x3ff irq 4 flags 0x10 on isa
sio0: type 16550A
sio1 at 0x2f8-0x2ff irq 3 on isa
sio1: type 16550A
ppc0 at 0x3bc irq 7 flags 0x40 on isa
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppb0: IEEE1284 device found /NIBBLE
Probing for PnP devices on ppbus0:
ppbus0: <Hewlett-Packard HP LaserJet 6L/0101.01> PRINTER HP ENHANCED PCL5,PJL
lpt0: <generic printer> on ppbus 0
lpt0: Interrupt-driven port
ppi0: <generic parallel i/o> on ppbus 0
plip0: <PLIP network interface> on ppbus 0
ppc1 at 0x378 flags 0x40 on isa
ppc1: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppb1: IEEE1284 device found /NIBBLE/ECP
Probing for PnP devices on ppbus1:
ppbus1: <HEWLETT-PACKARD DESKJET 895C> MLC,PCL,PML
lpt1: <generic printer> on ppbus 1
ppi1: <generic parallel i/o> on ppbus 1
ppc2 at 0x278 flags 0x40 on isa
ppc2: Generic chipset (NIBBLE-only) in COMPATIBLE mode
lpt2: <generic printer> on ppbus 2
ppi2: <generic parallel i/o> on ppbus 2
fdc0 at 0x3f0-0x3f7 irq 6 drq 2 on isa
fdc0: FIFO enabled, 8 bytes threshold
fd0: 1.44MB 3.5in
wdc0 at 0x1f0-0x1f7 irq 14 on isa
wdc0: unit 0 (wd0): <WDC AC2635F>
wd0: 610MB (1249920 sectors), 1240 cyls, 16 heads, 63 S/T, 512 B/S
wdc1 not found at 0x170
vga0 at 0x3b0-0x3bb maddr 0xb0000 msize 32768 on isa
npx0 on motherboard
npx0: INT 16 interface
changing root device to wd0s1a


#gwr FreeBSD-3.4 maths02 printer server
#
# GENERIC -- Generic machine with WD/AHx/NCR/BTx family disks
#
# For more information on this file, please read the handbook section on
# Kernel Configuration Files:
#
#    http://www.freebsd.org/handbook/kernelconfig-config.html
#
# The handbook is also available locally in /usr/share/doc/handbook
# if you've installed the doc distribution, otherwise always see the
# FreeBSD World Wide Web server (http://www.FreeBSD.ORG/) for the
# latest information.
#
# An exhaustive list of options and more detailed explanations of the
# device lines is also present in the ./LINT configuration file. If you are
# in doubt as to the purpose or necessity of a line, check first in LINT.
#
# $FreeBSD: src/sys/i386/conf/GENERIC,v 1.143.2.24 1999/12/05 01:56:42 luoqi Exp
 $

machine         "i386"
#gwr cpu                "I386_CPU"
cpu             "I486_CPU"
cpu             "I586_CPU"
cpu             "I686_CPU"
#gwr ident              GENERIC
ident           "MATHS02"
maxusers        32

options         DDB                     # Enable the kernel debugger.

#makeoptions    DEBUG="-g"              #Build kernel with debug symbols.

options         MATH_EMULATE            #Support for x87 emulation
options         INET                    #InterNETworking
options         FFS                     #Berkeley Fast Filesystem
options         FFS_ROOT                #FFS usable as root device [keep this!]
options         MFS                     #Memory Filesystem
options         MFS_ROOT                #MFS usable as root device, "MFS" req'ed
options         NFS                     #Network Filesystem
options         NFS_ROOT                #NFS usable as root device, "NFS" req'ed
options         MSDOSFS                 #MSDOS Filesystem
options         "CD9660"                #ISO 9660 Filesystem
options         "CD9660_ROOT"           #CD-ROM usable as root. "CD9660" req'ed
options         PROCFS                  #Process filesystem
options         "COMPAT_43"             #Compatible with BSD 4.3 [KEEP THIS!]
options         SCSI_DELAY=15000        #Be pessimistic about Joe SCSI device
options         UCONSOLE                #Allow users to grab the console
options         FAILSAFE                #Be conservative
options         USERCONFIG              #boot -c editor
options         VISUAL_USERCONFIG       #visual boot -c editor
options         KTRACE                  #ktrace(1) syscall trace support
options         SYSVSHM                 #SYSV-style shared memory
options         SYSVMSG                 #SYSV-style message queues
options         SYSVSEM                 #SYSV-style semaphores

config          kernel  root on wd0

# To make an SMP kernel, the next two are needed
#options        SMP                     # Symmetric MultiProcessor Kernel
#options        APIC_IO                 # Symmetric (APIC) I/O
# Optionally these may need tweaked, (defaults shown):
#options        NCPU=2                  # number of CPUs
#options        NBUS=4                  # number of busses
#options        NAPIC=1                 # number of IO APICs
#options        NINTR=24                # number of INTs

controller      isa0
controller      pnp0                    # PnP support for ISA
controller      eisa0
controller      pci0

# Floppy drives
controller      fdc0    at isa? port "IO_FD1" bio irq 6 drq 2
disk            fd0     at fdc0 drive 0
disk            fd1     at fdc0 drive 1

# IDE controller and disks
options         "CMD640"        # work around CMD640 chip deficiency
controller      wdc0    at isa? port "IO_WD1" bio irq 14
disk            wd0     at wdc0 drive 0
disk            wd1     at wdc0 drive 1

controller      wdc1    at isa? port "IO_WD2" bio irq 15
disk            wd2     at wdc1 drive 0
disk            wd3     at wdc1 drive 1

# ATAPI devices
options         ATAPI           #Enable ATAPI support for IDE bus
options         ATAPI_STATIC    #Don't do it as an LKM
device          acd0            #IDE CD-ROM
device          wfd0            #IDE Floppy (e.g. LS-120)

# SCSI Controllers
# A single entry for any of these controllers (ncr, ahb, ahc) is
# sufficient for any number of installed devices.
#gwr controller ncr0            # NCR/Symbios Logic
#gwr controller ahb0            # EISA AHA1742 family
#gwr controller ahc0            # AHA2940 and onboard AIC7xxx devices
#gwr controller amd0            # AMD 53C974 (Teckram DC-390(T))
#gwr controller isp0            # Qlogic family
#gwr controller dpt0            # DPT Smartcache - See LINT for options!

#gwr controller adv0    at isa? port ? cam irq ?
#gwr controller adw0
#gwr controller bt0     at isa? port ? cam irq ?
#gwr controller aha0    at isa? port ? cam irq ?
#gwr controller aic0    at isa? port ? cam irq ?

# SCSI peripherals
# Only one of each of these is needed, they are dynamically allocated.
#gwr controller scbus0          # SCSI bus (required)
#gwr device             da0             # Direct Access (disks)
#gwr device             sa0             # Sequential Access (tape etc)
#gwr device             cd0             # CD
#gwr device             pass0           # Passthrough device (direct SCSI)

# Proprietary or custom CD-ROM Interfaces
#gwr device             wt0     at isa? port 0x300 bio irq 5 drq 1
#gwr device             mcd0    at isa? port 0x300 bio irq 10
#gwr device             matcd0  at isa? port 0x230 bio
#gwr device             scd0    at isa? port 0x230 bio

# atkbdc0 controls both the keyboard and the PS/2 mouse
controller      atkbdc0 at isa? port IO_KBD tty
device          atkbd0  at isa? tty irq 1
device          psm0    at isa? tty irq 12

device          vga0    at isa? port ? conflicts

# splash screen/screen saver
pseudo-device   splash

# syscons is the default console driver, resembling an SCO console
device          sc0     at isa? tty

# Enable this and PCVT_FREEBSD for pcvt vt220 compatible console driver
#device         vt0     at isa? tty
#options        XSERVER                 # support for X server
#options        FAT_CURSOR              # start with block cursor
# If you have a ThinkPAD, uncomment this along with the rest of the PCVT lines
#options        PCVT_SCANSET=2          # IBM keyboards are non-std

# Floating point support - do not disable.
device          npx0    at isa? port IO_NPX irq 13

# Power management support (see LINT for more options)
device          apm0    at isa? disable flags 0x31 # Advanced Power Management

# PCCARD (PCMCIA) support
#controller     card0
#device         pcic0   at card?
#device         pcic1   at card?

# Serial (COM) ports
device          sio0    at isa? port "IO_COM1" flags 0x10 tty irq 4
device          sio1    at isa? port "IO_COM2" tty irq 3
device          sio2    at isa? disable port "IO_COM3" tty irq 5
device          sio3    at isa? disable port "IO_COM4" tty irq 9

# Parallel port
device          ppc0    at isa? port? flags 0x40 tty irq 7
controller      ppbus0                  # Parallel port bus (required)
device          lpt0    at ppbus?       # Printer
device          plip0   at ppbus?       # TCP/IP over parallel
device          ppi0    at ppbus?       # Parallel port interface device
#controller     vpo0    at ppbus?       # Requires scbus and da0

# Parallel port 1
device          ppc1    at isa? port? flags 0x40 tty
controller      ppbus1                  # Parallel port bus (required)
device          lpt1    at ppbus1       # Printer

# Parallel port 2
device          ppc2    at isa? port? flags 0x40 tty
controller      ppbus2                  # Parallel port bus (required)
device          lpt2    at ppbus2       # Printer

# PCI Ethernet NICs.
#gwr device             al0             # ADMtek AL981 (``Comet'')
#gwr device             ax0             # ASIX AX88140A
#gwr device             de0             # DEC/Intel DC21x4x (``Tulip'')
#gwr device             fxp0            # Intel EtherExpress PRO/100B (82557, 82558)
#gwr device             mx0             # Macronix 98713/98715/98725 (``PMAC'')
#gwr device             pn0             # Lite-On 82c168/82c169 (``PNIC'')
#gwr device             rl0             # RealTek 8129/8139
#gwr device             sf0             # Adaptec AIC-6915 DuraLAN (``Starfire'')
#gwr device             sis0            # Silicon Integrated Systems SiS 900/SiS 7016
#gwr device             ste0            # Sundance ST201 (D-Link DFE-550)
#gwr device             tl0             # Texas Instruments ThunderLAN
#gwr device             tx0             # SMC 9432TX (83c170 ``EPIC'')
#gwr device             vr0             # VIA Rhine, Rhine II
#gwr device             vx0             # 3Com 3c590, 3c595 (``Vortex'')
#gwr device             wb0             # Winbond W89C840F
#gwr device             xl0             # 3Com 3c90x (``Boomerang'', ``Cyclone'')

# ISA Ethernet NICs.
# Order is important here due to intrusive probes, do *not* alphabetize
# this list of network interfaces until the probes have been fixed.
# Right now it appears that the ie0 must be probed before ep0. See
# revision 1.20 of this file.
#gwr device             ed0     at isa? port 0x280 net irq 10 iomem 0xd8000
device          ed0     at isa? port 0x300 net irq 5 iomem 0xca000
#gwr device             ie0     at isa? port 0x300 net irq 10 iomem 0xd0000
#gwr device             ep0     at isa? port 0x300 net irq 10
#gwr device             ex0     at isa? port? net irq?
#gwr device             fe0     at isa? port 0x300 net irq ?
#gwr device             le0     at isa? port 0x300 net irq 5 iomem 0xd0000
#gwr device             lnc0    at isa? port 0x280 net irq 10 drq 0
#gwr device             cs0     at isa? port 0x300 net irq ?
# requires PCCARD (PCMCIA) support to be activated
#device         xe0     at isa? port? net irq ?

# PCCARD NIC drivers.
# ze and zp take over the pcic and cannot coexist with generic pccard
# support, nor the ed and ep drivers they replace.
#gwr device             ze0     at isa? port 0x300 net irq 10 iomem 0xd8000
#gwr device             zp0     at isa? port 0x300 net irq 10 iomem 0xd8000

# Pseudo devices - the number indicates how many units to allocated.
pseudo-device   loop            # Network loopback
pseudo-device   ether           # Ethernet support
pseudo-device   sl      1       # Kernel SLIP
pseudo-device   ppp     1       # Kernel PPP
pseudo-device   tun     1       # Packet tunnel
pseudo-device   pty     16      # Pseudo-ttys (telnet etc)
pseudo-device   gzip            # Exec gzipped a.out's

# The `bpfilter' pseudo-device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# The number of devices determines the maximum number of
# simultaneous BPF clients programs runnable.
pseudo-device   bpfilter 1      #Berkeley packet filter

Fix: 

A full diagnosis and fix is way beyond me.

I have found two basic workarounds to this instance of the problem:
  1. disable the /etc/periodic/weekly/320.whatis script
  2. recompile the kernel with DDB
How-To-Repeat: Just run /etc/periodic/weekly via cron at any time.