This happens randomly during normal operation with ipv6 over a gif tunnel to freenet6. Here's the stack trace from the memory dump: Unread portion of the kernel message buffer: trap number = 12 panic: page fault (kgdb) bt #0 doadump () at pcpu.h:165 #1 0xc04ee96c in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0xc04eeced in panic (fmt=0xc06ba001 "%s") at /usr/src/sys/kern/kern_shutdown.c:565 #3 0xc068dfae in trap_fatal (frame=0xea26babc, eva=0) at /usr/src/sys/i386/i386/trap.c:837 #4 0xc068dc42 in trap_pfault (frame=0xea26babc, usermode=0, eva=636) at /usr/src/sys/i386/i386/trap.c:745 #5 0xc068d7b0 in trap (frame= {tf_fs = 8, tf_es = 40, tf_ds = 40, tf_edi = 0, tf_esi = 67109631, tf_ebp = -366560480, tf_isp = -366560536, tf_ebx = -946169472, tf_edx = -946099584, tf_ecx = 4, tf_eax = 4, tf_trapno = 12, tf_err = 2, tf_eip = -1068001029, tf_cs = 32, tf_eflags = 66198, tf_esp = -956809216, tf_ss = -941078880}) at /usr/src/sys/i386/i386/trap.c:435 #6 0xc0678e8a in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #7 0xc05798fb in if_delmulti (ifp=0x0, sa=0x40002ff) at atomic.h:146 #8 0xc05e0572 in in6_delmulti (in6m=0xc7419700) at /usr/src/sys/netinet6/mld6.c:649 #9 0xc05cf502 in in6_ifdetach (ifp=0xc6f84000) at /usr/src/sys/netinet6/in6_ifattach.c:806 #10 0xc05769ad in if_detach (ifp=0xc6f84000) at /usr/src/sys/net/if.c:665 #11 0xc057d310 in gif_destroy (sc=0xc7838c80) at /usr/src/sys/net/if_gif.c:209 #12 0xc057d408 in gif_clone_destroy (ifp=0x4) at /usr/src/sys/net/if_gif.c:226 #13 0xc057b287 in ifc_simple_destroy (ifc=0xc06f2060, ifp=0x4) at /usr/src/sys/net/if_clone.c:478 #14 0xc057a552 in if_clone_destroy (name=0x4 <Address 0x4 out of bounds>) at /usr/src/sys/net/if_clone.c:172 #15 0xc0578b3e in ifioctl (so=0xc7d6a858, cmd=2149607801, data=0xc7e84080 "gif0", td=0xc79baa80) at /usr/src/sys/net/if.c:1533 #16 0xc0520bc7 in soo_ioctl (fp=0x4, cmd=2149607801, data=0xc7e84080, active_cred=0xc72c8b00, td=0xc79baa80) at /usr/src/sys/kern/sys_socket.c:214 #17 0xc0519d77 in ioctl (td=0xc79baa80, uap=0xea26bd04) at file.h:265 #18 0xc068e3a0 in syscall (frame= {tf_fs = 59, tf_es = 59, tf_ds = 59, tf_edi = 134533248, tf_esi = -1077941524, tf_ebp = -1077944072, tf_isp = -366559900, tf_ebx = 134578976, tf_edx = 134590397, tf_ecx = 0, tf_eax = 54, tf_trapno = 12, tf_err = 2, tf_eip = 1209393607, tf_cs = 51, tf_eflags = 582, tf_esp = -1077944100, tf_ss = 59}) at /usr/src/sys/i386/i386/trap.c:983 #19 0xc0678edf in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:200 #20 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) Fix: Unknown. How-To-Repeat: Unknown what actually causes it, but it may be related to deletion/ recreation of the gif tunnel.
I am regularly observe problem with smething alike symptoms. I have FreeBSD 6.2-STABLE of Jan 29. I have IPv6 in my kernel, but do not use it actively. In my case it happends with significant probability when mpd4.1 based server trying to destroy several ngX interfaces on shutdown. It does it by shutting down related ng_iface netgraph node. Fatal trap 12: page fault while in kernel mode fault virtual address = 0x100027c fault code = supervisor write, page not present instruction pointer = 0x20:0xc05df5a3 stack pointer = 0x28:0xdce8c94c frame pointer = 0x28:0xdce8c970 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 6089 (mpd4) trap number = 12 panic: page fault Uptime: 4h43m35s Dumping 511 MB (2 chunks) chunk 0: 1MB (159 pages) ... ok chunk 1: 511MB (130800 pages) 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15 #0 doadump () at pcpu.h:165 165 __asm __volatile("movl %%fs:0,%0" : "=r" (td)); (kgdb) bt #0 doadump () at pcpu.h:165 #1 0xc055e046 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0xc055e350 in panic (fmt=0xc0749735 "%s") at /usr/src/sys/kern/kern_shutdown.c:565 #3 0xc0723095 in trap_fatal (frame=0xdce8c90c, eva=0) at /usr/src/sys/i386/i386/trap.c:837 #4 0xc0722db5 in trap_pfault (frame=0xdce8c90c, usermode=0, eva=16777852) at /usr/src/sys/i386/i386/trap.c:745 #5 0xc072299f in trap (frame= {tf_fs = -588775416, tf_es = -1068171224, tf_ds = -588775384, tf_edi = 16777216, tf_esi = 167772927, tf_ebp = -588723856, tf_isp = -588723912, tf_ebx = -1008249152, tf_edx = -1011626624, tf_ecx = -1007975136, tf_eax = 4, tf_trapno = 12, tf_err = 2, tf_eip = -1067584093, tf_cs = 32, tf_eflags = 66194, tf_esp = -1015311360, tf_ss = -2145359566}) at /usr/src/sys/i386/i386/trap.c:435 #6 0xc070fb5a in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #7 0xc05df5a3 in if_delmulti (ifp=0x1000000, sa=0xa0002ff) at atomic.h:146 #8 0xc05f03cd in in_delmulti_locked (inm=0xc3eb8520) at /usr/src/sys/netinet/in.c:1060 #9 0xc05f049b in in_delmulti_ifp (ifp=0xc37b9400) at /usr/src/sys/netinet/in.c:1079 #10 0xc05f0568 in in_ifdetach (ifp=0xc37b9400) at /usr/src/sys/netinet/in.c:1095 #11 0xc05dc82b in if_detach (ifp=0xc37b9400) at /usr/src/sys/net/if.c:655 This looks strange for me: (kgdb) frame 8 #8 0xc05f03cd in in_delmulti_locked (inm=0xc3eb8520) at /usr/src/sys/netinet/in.c:1060 1060 if_delmulti(ifma->ifma_ifp, ifma->ifma_addr); (kgdb) p ifma->ifma_ifp $8 = (struct ifnet *) 0x1000000 (kgdb) p *(ifma->ifma_ifp) Cannot access memory at address 0x1000000 I also have several other alike coredumps: #6 0xc070fb5a in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #7 0xc05df5a3 in if_delmulti (ifp=0x80000, sa=0x0) at atomic.h:146 #8 0xc05f03cd in in_delmulti_locked (inm=0xc4a3e7c0) at /usr/src/sys/netinet/in.c:1060 #9 0xc05f049b in in_delmulti_ifp (ifp=0xc385fc00) at /usr/src/sys/netinet/in.c:1079 #10 0xc05f0568 in in_ifdetach (ifp=0xc385fc00) at /usr/src/sys/netinet/in.c:1095 #11 0xc05dc82b in if_detach (ifp=0xc385fc00) at /usr/src/sys/net/if.c:655 ---- #5 0xc070fb5a in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #6 0xc05839e5 in turnstile_setowner (ts=0xc3a2fcc0, owner=0x4) at /usr/src/sys/kern/subr_turnstile.c:434 #7 0xc0583d11 in turnstile_wait (lock=0xc385e660, owner=0x4) at /usr/src/sys/kern/subr_turnstile.c:593 #8 0xc0553aeb in _mtx_lock_sleep (m=0xc385e660, tid=3286708992, opts=0, file=0x0, line=0) at /usr/src/sys/kern/kern_mutex.c:579 #9 0xc05df5df in if_delmulti (ifp=0xc385e400, sa=0xc3e79b80) at /usr/src/sys/net/if.c:2083 #10 0xc05f03cd in in_delmulti_locked (inm=0x4) at /usr/src/sys/netinet/in.c:1060 #11 0xc05f049b in in_delmulti_ifp (ifp=0xc3855000) at /usr/src/sys/netinet/in.c:1079 #12 0xc05f0568 in in_ifdetach (ifp=0xc3855000) at /usr/src/sys/netinet/in.c:1095 #13 0xc05dc82b in if_detach (ifp=0xc3855000) at /usr/src/sys/net/if.c:655 --- #6 0xc070fb5a in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #7 0xc05df5a3 in if_delmulti (ifp=0x0, sa=0x50001ff) at atomic.h:146 #8 0xc05f03cd in in_delmulti_locked (inm=0xc50901c0) at /usr/src/sys/netinet/in.c:1060 #9 0xc05f049b in in_delmulti_ifp (ifp=0xc4b1a800) at /usr/src/sys/netinet/in.c:1079 #10 0xc05f0568 in in_ifdetach (ifp=0xc4b1a800) at /usr/src/sys/netinet/in.c:1095 #11 0xc05dc82b in if_detach (ifp=0xc4b1a800) at /usr/src/sys/net/if.c:655 If anybody needs additional info, I will be glad to help. -- Alexander Motin
Responsible Changed From-To: freebsd-bugs->freebsd-net Over to maintainer(s).
Responsible Changed From-To: freebsd-net->bms Send over to BMS. He's active in that area and may have fixed the bug already.
Responsible Changed From-To: bms->jinmei jinmei@ has been looking at this
Recent changes to the head and [56] STABLE *may* fix the problem (see, e.g., the commit log for rev 1.71 of sys/netinet6/in6.c at http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/netinet6/in6.c). These just fix memory leak while the symptom rather seems to indicate use-after-free, so I'm not sure if these really solve the problem; however, the fix indeed affects (either good or bad) the same code path that caused the problem shown in this PR, so it may happen to fix this problem via some non trivial side effect. (Except this, I have no idea about this issue even though the responsibility has been given to me)
State Changed From-To: open->feedback replied with a *possible* bug fix.
kern/101553 looks to be a duplicate of this, but contains a extra information that may or may not help diagnose the issue.
I am also getting the identical problem occuring on FreeBSD 6.2-RELEASE, again, I use tspc/freenet6, and again, it seems to be relating to when the tunnel goes down, and the gif0 interface is deleted. It doesn't happen all the time, but I've only had this machine built for about 4 weeks, (after upgrading the server from 4.2) and it happens on average every few days - yesterday it happened twice in 2 hours. Each time it happens at the exact same point. Please help! The many windows users who rely on my webserver have been hearing from me for years that 'freebsd never crashes', and there's only so long I can bluff them now that "I accidently rebooted it" :-) Thanks! And please contact me if I can do anything to help - I have the crashdumps. Dump header from device /dev/ad0s1b Architecture: i386 Architecture Version: 2 Dump Length: 1064173568B (1014 MB) Blocksize: 512 Dumptime: Thu Jul 19 06:24:43 2007 Hostname: catflap.bishopston.net Magic: FreeBSD Kernel Dump Version String: FreeBSD 6.2-RELEASE #0: Wed Jul 4 15:17:58 BST 2007 root@catflap.bishopston.net:/usr/obj/usr/src/sys/CATFLAP Panic String: page fault Dump Parity: 3533894411 Bounds: 0 Dump Status: good 12:59 (1) "crash" root@catflap# bugger vmcore.0 kgdb: kvm_nlist(_stopped_cpus): kgdb: kvm_nlist(_stoppcbs): [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode fault virtual address = 0x27c fault code = supervisor write, page not present instruction pointer = 0x20:0xc0567767 stack pointer = 0x28:0xef7ceafc frame pointer = 0x28:0xef7ceb20 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 96116 (ifconfig) trap number = 12 panic: page fault Uptime: 1d15h33m59s Dumping 1014 MB (2 chunks) chunk 0: 1MB (159 pages) ... ok chunk 1: 1015MB (259648 pages) 999 983 967 951 935 919 903 887 871 855 839 823 807 791 775 759 743 727 711 695 679 663 647 631 615 599 583 567 551 535 519 503 487 471 455 439 423 407 391 375 359 343 327 311 295 279 263 247 231 215 199 18 3 167 151 135 119 103 87 71 55 39 23 7 #0 doadump () at pcpu.h:165 165 __asm __volatile("movl %%fs:0,%0" : "=r" (td)); (kgdb) list *0xc0567767 0xc0567767 is in if_delmulti (atomic.h:149). 144 static __inline int 145 atomic_cmpset_int(volatile u_int *dst, u_int exp, u_int src) 146 { 147 int res = exp; 148 149 __asm __volatile ( 150 " " __XSTRING(MPLOCKED) " " 151 " cmpxchgl %2,%1 ; " 152 " setz %%al ; " 153 " movzbl %%al,%0 ; " 154 "1: " 155 "# atomic_cmpset_int" 156 : "+a" (res), /* 0 (result) */ 157 "=m" (*dst) /* 1 */ 158 : "r" (src), /* 2 */ 159 "m" (*dst) /* 3 */ 160 : "memory"); 161 162 return (res); 163 } 164 165 #endif /* defined(CPU_DISABLE_CMPXCHG) */ 166 167 /* 168 * Atomically add the value of v to the integer pointed to by p and return 169 * the previous value of *p. 170 */ 171 static __inline u_int 172 atomic_fetchadd_int(volatile u_int *p, u_int v) 173 { (kgdb) back #0 doadump () at pcpu.h:165 #1 0xc04e4c56 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0xc04e4f60 in panic (fmt=0xc06d1b0f "%s") at /usr/src/sys/kern/kern_shutdown.c:565 #3 0xc06b04c9 in trap_fatal (frame=0xef7ceabc, eva=0) at /usr/src/sys/i386/i386/trap.c:837 #4 0xc06b01e9 in trap_pfault (frame=0xef7ceabc, usermode=0, eva=636) at /usr/src/sys/i386/i386/trap.c:745 #5 0xc06afdd3 in trap (frame= {tf_fs = 8, tf_es = 40, tf_ds = 40, tf_edi = 0, tf_esi = 0, tf_ebp = -277026016, tf_isp = -277026072, tf_ebx = -987064640, tf_edx = -927535104, tf_ecx = 4, tf_eax = 4, tf_trapno = 12, tf_err = 2, tf_eip = -1068075161, tf_cs = 32, tf_ eflags = 66198, tf_esp = -957961216, tf_ss = -989902176}) at /usr/src/sys/i386/i386/trap.c:435 #6 0xc069c89a in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #7 0xc0567767 in if_delmulti (ifp=0x0, sa=0x0) at atomic.h:146 #8 0xc05cd9ab in in6_delmulti (in6m=0xc708dc80) at /usr/src/sys/netinet6/mld6.c:649 #9 0xc05bf60c in in6_ifdetach (ifp=0xc6e6ac00) at /usr/src/sys/netinet6/in6_ifattach.c:806 #10 0xc05649fa in if_detach (ifp=0xc6e6ac00) at /usr/src/sys/net/if.c:665 #11 0xc056fe3c in gif_destroy (sc=0xc537b480) at /usr/src/sys/net/if_gif.c:209 #12 0xc056ff1b in gif_clone_destroy (ifp=0x4) at /usr/src/sys/net/if_gif.c:226 #13 0xc056dfc2 in ifc_simple_destroy (ifc=0xc070f160, ifp=0x4) at /usr/src/sys/net/if_clone.c:478 #14 0xc056d39a in if_clone_destroy (name=0x4 <Address 0x4 out of bounds>) at /usr/src/sys/net/if_clone.c:172 #15 0xc0566a3a in ifioctl (so=0xc59359bc, cmd=2149607801, data=0xc9169400 "gif99", td=0xc8b6f000) at /usr/src/sys/net/if.c:1533 #16 0xc0513392 in soo_ioctl (fp=0x4, cmd=2149607801, data=0xc9169400, active_cred=0xc4a7ad80, td=0xc8b6f000) at /usr/src/sys/kern/sys_socket.c:214 #17 0xc050ca63 in ioctl (td=0xc8b6f000, uap=0xef7ced04) at file.h:264 #18 0xc06b087f in syscall (frame= {tf_fs = 59, tf_es = 59, tf_ds = 59, tf_edi = -1077941040, tf_esi = -1077940819, tf_ebp = -1077943256, tf_isp = -277025436, tf_ebx = 134571584, tf_edx = 134582782, tf_ecx = 0, tf_eax = 54, tf_trapno = 12, tf_err = 2, tf_eip = 6724844 87, tf_cs = 51, tf_eflags = 582, tf_esp = -1077943284, tf_ss = 59}) at /usr/src/sys/i386/i386/trap.c:983 #19 0xc069c8ef in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:200 #20 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) -- -=-=-=- Virus Scanned by "pacha.mail.bishopston.net" using ClamAv -=-=-=- Database Last Checked: Fri Jul 20 06:38:00 BST 2007 - http://www.clamav.net/ Database Updated : Fri Jul 20 02:38:01 BST 2007 - 138928 viruses scanned
> Recent changes to the head and [56] STABLE *may* fix the problem > (see, e.g., the commit log for rev 1.71 of sys/netinet6/in6.c at > http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/netinet6/in6.c). > These just fix memory leak while the symptom rather seems to indicate > use-after-free, so I'm not sure if these really solve the problem; To be honest, it looks to me more that THIS fix is more relevent, if any. http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/netinet6/mld6.c?r1=1.29#rev1.29 I don't know if it is, though, and whilst it was due to be MFC in April, it doesn't seem like it has been. Any ideas? Cheers, Jamie -- -=-=-=- Virus Scanned by "pacha.mail.bishopston.net" using ClamAv -=-=-=- Database Last Checked: Fri Jul 20 06:38:00 BST 2007 - http://www.clamav.net/ Database Updated : Fri Jul 20 02:38:01 BST 2007 - 138928 viruses scanned
At Fri, 20 Jul 2007 07:02:21 +0100, Jamie Jones <jamie@bishopston.net> wrote: > I am also getting the identical problem occuring on FreeBSD 6.2-RELEASE, > again, I use tspc/freenet6, and again, it seems to be relating to when the > tunnel goes down, and the gif0 interface is deleted. > > It doesn't happen all the time, but I've only had this machine built for about > 4 weeks, (after upgrading the server from 4.2) and it happens on average > every few days - yesterday it happened twice in 2 hours. Each time it happens > at the exact same point. > > Please help! The many windows users who rely on my webserver have been hearing > from me for years that 'freebsd never crashes', and there's only so long I > can bluff them now that "I accidently rebooted it" :-) As noted in the PR page the problem *may* be fixed in 6_STABLE (not 6.2 RELEASE). If you've not tried it could you check it out? Thanks, JINMEI, Tatuya Communication Platform Lab. Corporate R&D Center, Toshiba Corp. jinmei@isl.rdc.toshiba.co.jp
This commit may be a real fix: ups 2008-02-22 19:13:57 UTC FreeBSD src repository Modified files: (Branch: RELENG_6) sys/netinet in.c Log: Fix reference counting for already existing addresses in in_addmulti() Reviewed by: gnn@ Revision Changes Path 1.85.2.10 +0 -1 src/sys/netinet/in.c -- John Baldwin
On Sun, 21 Jan 2007, Nick Johnson wrote: >> Number: 108197 >> Category: kern >> Synopsis: IPv6-related crash if if_delmulti >> Confidential: no >> Severity: critical >> Priority: high >> Responsible: freebsd-bugs >> State: open >> Quarter: >> Keywords: >> Date-Required: >> Class: sw-bug >> Submitter-Id: current-users >> Arrival-Date: Mon Jan 22 01:00:27 GMT 2007 >> Closed-Date: >> Last-Modified: >> Originator: Nick Johnson >> Release: FreeBSD 6.2-STABLE i386 >> Organization: > morons.org >> Environment: > System: FreeBSD turing.morons.org 6.2-STABLE FreeBSD 6.2-STABLE #5: Sun Jan 21 15:19:12 PST 2007 root@turing.morons.org:/usr/obj/usr/src/sys/TURING i386 > >> Description: > This happens randomly during normal operation with ipv6 over a gif tunnel to freenet6. > > Here's the stack trace from the memory dump: > > Unread portion of the kernel message buffer: Hi Nick: I was wondering if the change John pointed you at last year resolved this issue for you? If so, please let us know and I can close this PR. (Details attached below for reference) Thanks, Robert N M Watson Computer Laboratory University of Cambridge From: John Baldwin <jhb@FreeBSD.org> To: bug-followup@FreeBSD.org, freebsd@spatula.net Cc: ups@FreeBSD.org Subject: Re: kern/108197: [panic] [gif] [ipv6] if_delmulti reference counting panic Date: Mon, 25 Feb 2008 14:12:21 -0500 This commit may be a real fix: ups 2008-02-22 19:13:57 UTC FreeBSD src repository Modified files: (Branch: RELENG_6) Modified files: (Branch: RELENG_6) sys/netinet in.c Log: Fix reference counting for already existing addresses in in_addmulti() Reviewed by: gnn@ Revision Changes Path 1.85.2.10 +0 -1 src/sys/netinet/in.c -- John Baldwin
---------- Forwarded message ---------- Date: Sat, 7 Feb 2009 12:43:15 -0800 From: Nicklas Johnson <freebsd@spatula.net> To: Robert Watson <rwatson@freebsd.org> Subject: Re: kern/108197: IPv6-related crash if if_delmulti I can't tell you whether it has or not; I stopped using ipv6 tunnels after reporting this because it made things too crashy, and I never started again. Sorry about that. It should be fairly simple to test though; it doesn't take long after repeatedly adding/removing a tunnel before it would crash. -- "Courage isn't just a matter of not being frightened, you know. It's being afraid and doing what you have to do anyway."  -- Doctor Who - Planet of the Daleks This message has been brought to you by Nick Johnson 2.3b1 and the number 6. http://healerNick.com/    http://morons.org/     http://spatula.net/
State Changed From-To: feedback->feedback commit bit has been taken in for safekeeping. to submitter: is this aging PR still relevant?
Responsible Changed From-To: jinmei->freebsd-net
Just stumbled across this now due to bugzilla. For what it's worth, I've been using/adding/removing gifs on various machines now for years with no problem. Back at the time this PR was raised the panics were often, and replicable.
batch change: For bugs that match the following - Status Is In progress AND - Untouched since 2018-01-01. AND - Affects Base System OR Documentation DO: Reset to open status. Note: I did a quick pass but if you are getting this email it might be worthwhile to double check to see if this bug ought to be closed.
Keyword: crash – in lieu of summary line prefix: [panic] * bulk change for the keyword * summary lines may be edited manually (not in bulk). Keyword descriptions and search interface: <https://bugs.freebsd.org/bugzilla/describekeywords.cgi>