+ Work is still ongoing to this day with major advances in performance + being achieved. +
Continuing work on SMPng in the 6.x branch has included work sweeping +
Continued work on SMPng in the 6.x branch included work sweeping up the "loose ends" that remain under Giant, such as parts of NFS, and less commonly used network stack components.
@@ -126,2676 +124,31 @@ run-time kernel lock order verifier, and wide-spread use of lock assertions and run-time invariants testing. - -This web page contains information relating to the SMPng effort; - because of the immense scope of the work and rapid pace of development, - it captures only a subset of what has been done. As the base SMPng - Project, moving towards fine-grained locking, is complete, this - information is largely historical, and does not attempt to capture - more recent work on locking and multi-processor performance.
- -The task list below is not intended to be complete, but does - represent a set of relevant and/or important components of the - overall work. The "Responsible" field identifies a developer who - has expressed willingness to be responsible for completing the - identified task; this doesn't preclude others working on it, - but suggests that coordination with the responsible party might - be appropriate so as to avoid unnecessary duplication of work, - and to maximize forward progress. If beginning work on a new - area of substantial size, or one that appears unclaimed, it may - be worth dropping an e-mail to the FreeBSD SMP mailing list to - see if any progress has been made.
- -The definition of the date field varies depending on the status - of a task. For completed tasks, it refers to the date completed - or reported completed. For in-progress tasks, it refers to the - date of the last update of the entry. For stalled tasks, it - refers to the date that the task was declared stalled. For - new tasks, it refers to the date the task was added to the list.
- -Locking down of individual device drivers is tracked at - the busdma and - SMPng driver conversion web page.
- -Network stack locking information is available at &a.rwatson;'s - netperf web - page and the FreeBSD.org Netperf - web page. - An SMP network performance - cluster has also been created for the purposes of testing.
- -Tasks are sorted first by status, then by date.
- - -Most SMP-related discussion takes place on the - freebsd-smp - mailing list. You can read more about mailing lists in the - - Resources on the Internet appendix of the - FreeBSD Handbook.
Steve Passe has been maintaining a - SMP - project page that contains additional information, and - goes back further in time than this web page.
&a.rwatson; is maintaining a page for SMP-related network - performance work for the Netperf - project. In addition, he has a a personal - web page with a change log and other information.
OSNews has run an - interview with FreeBSD developers &a.rwatson;, &a.jhb;, and - &a.scottl;, who talk about many features in 6.x, including recent - SMPng work, SMPVFS, and more. The article is - here.
A series of man pages on kernel synchronization and threading - primitives can be found here:
- -In addition, the old SPL man page may be of interest, as it reflects the previous SMP synchronization model.
-- Hiten Pandya's - SMP kernel synchronization rules.
&a.dillon; maintained a - web page - that documented the work he was doing on the SMP project.
- - "Locking in the Multithreaded FreeBSD Kernel" USENIX paper by &a.jhb;. -
-"Reasoning about SMP in FreeBSD" BSDCon'03 paper by &a.hsu;.
"ULE: A Modern Scheduler for FreeBSD" BSDCon'03 paper by &a.jeff;.
This is an incomplete list of high-level kernel subsystems and - current, active staff working on SMP architecture and stability.
- -| Subsystem | -Status | -Last updated | -Staffing | -
|---|---|---|---|
| Newbus | -&status.wip; | -5 October 2003 | -&a.imp; | -
| VM | -&status.wip; | -4 October 2003 | -&a.alc; | -
| Buffer cache | -&status.wip; | -4 October 2003 | -&a.jeff;, &a.phk; | -
| VFS | -&status.done; | -24 January 2005 | -&a.jeff; | -
| Processes and thread operations | -&status.wip; | -5 May 2003 | -&a.jhb; | -
| Scheduler | -&status.done; | -23 April 2004 | -&a.jhb;, &a.jeff; | -
| GEOM | -&status.done; | -5 February 2003 | -&a.phk; | -
| File descriptors | -&status.done; | -5 February 2003 | -&a.alfred;, &a.tanimura;, &a.rwatson; | -
| TTY subsystem | -&status.done; | -20 August 2008 | -&a.ed; | -
| Pipe IPC | -&status.done; | -4 October 2003 | -&a.alfred; | -
| Socket structures and system calls | -&status.done; | -25 November 2004 | -&a.sam;, &a.rwatson; | -
| KQueue | -&status.done; | -24 November 2004 | -&a.jmg;, &a.green; | -
| IPv4 | -&status.done; | -23 April 2004 | -&a.jennifer;, &a.hsu;, &a.sam;, &a.rwatson;, &a.gnn; | -
| IPv6 | -&status.wip; | -01 July 2005 | -&a.rwatson;, &a.gnn; | -
| IPX/SPX | -&status.done; | -09 January 2005 | -&a.rwatson; | -
| netatalk | -&status.done; | -02 February 2005 | -&a.rwatson; | -
| Network stack infrastructure | -&status.done; | -28 August 2004 | -&a.hsu;, &a.sam;, &a.rwatson;, &a.mlaier;, &a.luigi;, Maurycy Pawlowski-Wieronski <maurycy@fouk.org>, &a.brooks;, - &a.rik; | -
| NFS Client | -&status.done; | -19 May 2006 | -&a.mohans; | -
| NFS Server | -&status.wip; | -24 February 2007 | -&a.rwatson;, &a.jeff; | -
Following is an incomplete list of general tasks.
- -| Task | -Responsible | -Last updated | -Status | -
|---|---|---|---|
| Convert the giant lock from spinning to blocking, add the - scheduler lock, add per-CPU idle processes. | -&a.dillon; | -25 June 2000 | -&status.done; | -
| Port the BSD/OS locking primitives (i386). | -&a.jake; | -3 July 2000 | -&status.done; | -
| Implement heavy-weight interrupt threads (i386). | -&a.grog; | -3 August 2000 | -&status.done; | -
| Rewrite the low level interrupt code (i386 UP). | -&a.grog; | -3 August 2000 | -&status.done; | -
| Demonstrated reasonable stability (self-hosted buildworld) - (i386 UP). | --smp developers | -12 August 2000 | -&status.done; | -
| Port the BSD/OS locking primitives (alpha). | -&a.dfr; | -24 August 2000 | -&status.done; | -
| Stub out (disable) spl()s. | -&a.grog; | -30 August 2000 | -&status.done; | -
| Port the BSD/OS ktr code. | -&a.grog;, &a.jhb; | -30 August 2000 | -&status.done; | -
| Rewrite the low level interrupt code (i386 SMP). | -&a.jhb; | -1 September 2000 | -&status.done; | -
| Demonstrated reasonable stability (self-hosted buildworld) - (i386 SMP). | --smp developers | -6 September 2000 | -&status.done; | -
| Demonstrated reasonable stability (self-hosted buildworld) - (alpha). | --smp developers | -6 September 2000 | -&status.done; | -
| Make malloc and friends thread-safe. | -&a.jasone; | -10 September 2000 | -&status.done; | -
| Implement msleep(), make tsleep() an msleep() wrapper. | -&a.jake; | -11 September 2000 | -&status.done; | -
| Make fxp driver thread-safe. | -&a.cp; | -17 September 2000 | -&status.done; | -
| Make mbuf's thread-safe. | -&a.bmilekic; | -29 September 2000 | -&status.done; | -
| Lock manager re-work. | -&a.jasone; | -3 October 2000 | -&status.done; | -
| Implement heavy-weight interrupt threads (alpha). | -&a.jhb;, &a.dfr; | -5 October 2000 | -&status.done; | -
| Rewrite the low level interrupt code (alpha). | -&a.dfr;, &a.jhb; | -5 October 2000 | -&status.done; | -
| Process accounting. | -&a.tegge;, &a.jhb; | -5 October 2000 | -&status.done; | -
| Make ethernet drivers thread-safe. | -&a.wpaul; | -15 October 2000 | -&status.done; | -
| Make the mutex headers mostly machine-independent. | -&a.jhb; | -20 October 2000 | -&status.done; | -
| Rename SMP_DEBUG to MUTEX_DEBUG. | -&a.jhb; | -20 October 2000 | -&status.done; | -
| Give each soft interrupt its own thread. | -&a.cp; | -25 October 2000 | -&status.done; | -
| Make sf_bufs (sendfile(2)) thread-safe. | -&a.bmilekic; | -5 November 2000 | -&status.done; | -
| Make the witness code work correctly. | -&a.jhb; | -18 November 2000 | -&status.done; | -
| Split the ktr-specific code out of db_interface.c. | -&a.jhb; | -15 December 2000 | -&status.done; | -
| Convert the sio driver to using a spin mutex. | -&a.jhb; | -18 December 2000 | -&status.done; | -
| Implement condition variables. | -&a.jake;, &a.jasone; | -15 January 2001 | -&status.done; | -
| Add a flag to mtx_init() (MTX_RECURSE) that denotes - whether a mutex is allowed to recurse. | -&a.bmilekic; | -19 January 2001 | -&status.done; | -
| Make the zone allocator thread-safe. | -&a.des; | -21 January 2001 | -&status.done; | -
| Convert simplelocks to mutexes. | -&a.jasone; | -24 January 2001 | -&status.done; | -
| Make kernel preemptive with respect to interrupts. | -&a.jake; | -31 January 2001 | -&status.done; | -
| Cleanup of mutex API. | -&a.bmilekic; | -8 February 2001 | -&status.done; | -
| Remove COM_LOCK. | -&a.markm; | -11 February 2001 | -&status.done; | -
| Merge various scheduling classes into one run queue. - Modify scheduler to support preemptable kernel. | -&a.jake; | -11 February 2001 | -&status.done; | -
| Make priority propagation work correctly. | -&a.jake; | -11 February 2001 | -&status.done; | -
| Make most of the interrupt thread code MI and shared - between hardware and software interrupts. | -&a.jhb; | -18 February 2001 | -&status.done; | -
| Add protection to struct jail and jail-related functionality. | -&a.rwatson; | -20 February 2001 | -&status.done; | -
| Implement sx (shared/exclusive) locks. | -&a.jasone; | -5 March 2001 | -&status.done; | -
| Generalize/improve witness to handle more complex locking - primitives (mtx, sx). | -&a.jhb; | -28 March 2001 | -&status.done; | -
| Convert the allproc and proctree locks from lockmgr locks - to sx locks. | -&a.jhb; | -28 March 2001 | -&status.done; | -
| Make mbuf system use condition variables instead of - msleep()/wakeup(). | -&a.bmilekic; | -2 April 2001 | -&status.done; | -
| Remove <sys/mutex.h> includes from other kernel - headers such as <vm/vm_zone.h>, <sys/resourcevar.h>, - <sys/ucred.h>, and <sys/mbuf.h>. | -&a.markm; | -15 May 2001 | -&status.done; | -
| Cleanup the various mp_machdep.c's, unify various SMP - API's such as IPI delivery, etc. | -&a.jhb; | -15 May 2001 | -&status.done; | -
| Make most of the forward_* and forwarded_* functions MI. | -&a.jhb; | -15 May 2001 | -&status.done; | -
| Complete the MD support for SMP on the Alpha platform. | -&a.gallatin;, &a.dfr;, &a.jhb; | -15 May 2001 | -&status.done; | -
| Convert select() to use condition variables. | -&a.tanimura; | -15 May 2001 | -&status.done; | -
| Add a "giant" lock around the VM subsystem. | -&a.alfred; | -13 June 2001 | -&status.done; | -
| Introduce a modified slab allocator for the mbuf subsystem. | -&a.bmilekic; | -21 June 2001 | -&status.done; | -
| Add a witness_assert() function to handle lock assertions. | -&a.jhb; | -27 June 2001 | -&status.done; | -
| Extend sx locks to support try lock operations. | -&a.jhb; | -27 June 2001 | -&status.done; | -
| Document KTR. | -&a.jhb; | -28 June 2001 | -&status.done; | -
| Make fork_return, fork_exit, ast, and userret MI. | -&a.jhb; | -29 June 2001 | -&status.done; | -
| Make sched_lock's savecrit a per-process property saved - and restored in mi_switch and initialized in fork_exit. | -&a.jhb; | -30 June 2001 | -&status.done; | -
| Make ast() loop. | -&a.jhb; | -10 August 2001 | -&status.done; | -
| Add upgrade/downgrade sx lock operations. | -&a.kan;, &a.jasone; | -13 August 2001 | -&status.done; | -
| Implement semaphores. | -&a.jasone; | -14 August 2001 | -&status.done; | -
| Add support for upgrade/downgrades in witness. | -&a.jhb; | -23 August 2001 | -&status.done; | -
| Make most of cpu_wait() and cpu_exit() MI. | -&a.peter; | -9 September 2001 | -&status.done; | -
| Split NFS into client and server. | -&a.peter; | -18 Oct 2001 | -&status.done; | -
| Lock taskqueues. | -&a.arr;, &a.jhb; | -25 October 2001 | -&status.done; | -
| Add a per-thread ucred reference. | -&a.jhb; | -25 October 2001 | -&status.done; | -
| Make most of the per-CPU stuff MI. | -&a.jhb; | -11 December 2001 | -&status.done; | -
| Make critical section saved state per-thread instead of per-lock - so that interlocking spin locks work properly. | -&a.jhb; | -17 December 2001 | -&status.done; | -
| Replace the APIC-specific imen_mtx with a MI-named icu_lock to - protect interrupt controllers and associated data within the kernel - for both i386 and alpha. | -&a.jhb; | -20 December 2001 | -&status.done; | -
| Use the per-thread critical section nesting level in the mutex - and interrupt thread code to automatically determine when to not - preempt. This makes the MTX_NOSWITCH, SWI_SWITCH, and - SWI_NOSWITCH flags obsolete as the kernel will be able to figure - out the proper behavior on its own. | -&a.jhb; | -5 January 2002 | -&status.done; | -
| Lock struct filedesc and struct file. | -&a.tanimura;, &a.alfred; | -12 January 2002 | -&status.done; | -
| Lock struct pgrp, struct session, and struct sigio. | -&a.tanimura; | -23 February 2002 | -&status.done; | -
| Lock pipe implementation, but not sigio/fown, VM interactions. | -&a.alfred; | -27 February 2002 | -&status.done; | -
| Move to explicit reference counting for soft vnode references. - | -&a.phk; | -8 March 2002 | -&status.done; | -
| Initialize mutex pools early enough that sx locks can be used - for VM. | -&a.green; | -14 March 2002 | -&status.done; | -
| Place a global lock (sellock) around selinfo structures to fix a - variety of lock order reversals, and make select() MP-safe. | -&a.alfred;, &a.davidc; | -14 March 2002 | -&status.done; | -
| Push down Giant on read, write, pread, pwrite system calls, - acquiring Giant in the per-subsystem fileop layer for sockets, VFS, - etc. | -&a.alfred; | -15 March 2002 | -&status.done; | -
| Lock down kernel module structures. | -&a.arr; | -18 March 2002 | -&status.done; | -
| Lock down kernel linker globals. | -&a.arr; | -18 March 2002 | -&status.done; | -
| Rewrite kernel memory allocator to be a slab allocator that - uses per-cpu caches. | -&a.jeff; | -21 March 2002 | -&status.done; | -
| Replace incorrect use of MD critical section API to disable - interrupts with a specific interrupt disable API. | -&a.imp;, &a.dfr;, &a.benno;, &a.jhb; | -21 March 2002 | -&status.done; | -
| Lock down access to the shared p_args "process arguments" - structure through appropriate protection of that structure and - references to it. | -&a.mini; | -31 March 2002 | -&status.done; | -
| Move from flags/tsleep lock to sx locks to protect sysctl - tree from updates during sysctl operations. | -&a.mini; | -1 April 2002 | -&status.done; | -
| Create/port userland tool to manage KTR event dumps. | -&a.jake; | -1 April 2002 | -&status.done; | -
| Create MTX_SYSINIT and SX_SYSINIT macros that allow for - initializing locks that are subsystem independent. | -&a.arr; | -2 April 2002 | -&status.done; | -
| Lock down the global securelevel variable. | -&a.arr; | -2 April 2002 | -&status.done; | -
| Make grow_stack() MI. Possibly even a macro or inline. | -&a.alc; | -6 April 2002 | -&status.done; | -
| Lock use of p_fd, which otherwise can result in corrupted - p_fd panics during heavy operation. Start with a global, - and move to per-proc locking. | -&a.alfred;, &a.tanimura; | -8 April 2002 | -&status.done; | -
| Lock struct pargs. | -&a.mini; | -9 April 2002 | -&status.done; | -
| Lock sysctl hierarchy. | -&a.mini; | -9 April 2002 | -&status.done; | -
| Make {o,}sigreturn() MPSAFE. | -&a.alc; | -11 April 2002 | -&status.done; | -
| Rewrite kernel memory allocator so that Giant is not required - for malloc() or free(). | -&a.jeff; | -2 May 2002 | -&status.done; | -
| Replace complex shared/exclusive locking scheme in the VM - system with a purely exclusive lockmgr locking scheme, simplifying - locking and removing potential livelock/deadlock scenarios. | -&a.green;, &a.alc; | -3 May 2002 | -&status.done; | -
| Push down Giant into readv/writev system calls in style of - read/write/pread/pwrite once malloc no longer requires Giant in - the handling of iovec structures for uio. | -&a.alc; | -9 May 2002 | -&status.done; | -
| Push down Giant in mprotect(), minherit(), and madvise() so - that it is no longer acquired and released directly. | -&a.alc; | -18 May 2002 | -&status.done; | -
| Update suser() and p_can*() APIs to accept threads instead of - processes. | -&a.jhb; | -18 May 2002 | -&status.done; | -
| Broadly transition to td_ucred from p_ucred once KSE dependencies - are in place. | -&a.jhb; | -18 May 2002 | -&status.done; | -
| Add a witness_sleep() check to uma_zalloc() to catch code - calling malloc() or uma_zalloc() while holding non-sleepable - locks. | -&a.jhb; | -20 May 2002 | -&status.done; | -
| Optimize UP support by changing spin locks to only perform - critical section enter and exits. | -&a.jhb; | -21 May 2002 | -&status.done; | -
| Make sleep mutexes spin if the current lock holder is - executing on another CPU. | -&a.jhb; | -21 May 2002 | -&status.done; | -
| Add support for the IA32 pause instruction to spin loops in - locks. | -&a.jhb; | -21 May 2002 | -&status.done; | -
| Make KTRACE write into tracefiles asynchronously. | -&a.jhb; | -7 June 2002 | -&status.done; | -
| Remove Giant from modnext(2), modfnext(2), - modstat(2),and modfind(2). | -&a.arr; | -25 June 2002 | -&status.done; | -
| Fix synchronization of TLB flushes and invlpg() on x86 SMP. | -&a.peter; | -12 July 2002 | -&status.done; | -
| Add KTR(9) tracing for mutex contention. | -&a.iedowse; | -26 Augist 2002 | -&status.done; | -
| Make cpu_coredump MI. | -&a.peter; | -7 September 2002 | -&status.done; | -
| Add a subsystem lock to the accounting code. | -&a.arr; | -11 September 2002 | -&status.done; | -
| Allow KTR(9) to write trace records to alq(9) record facility. - | -&a.jeff; | -22 September 2002 | -&status.done; | -
| Create mechanism in cdevsw structure to protect - thread-unsafe drivers. | -&a.phk; | -27 September 2002 | -&status.done; | -
| Fix SIGXPU and other #if 0'd things in mi_switch(). | -&a.jhb; | -30 September 2002 | -&status.done; | -
| Lock down TrustedBSD MAC implementation. | -&a.rwatson; | -11 November 2002 | -&status.done; | -
| Lock eventhandlers. | -&a.msmith;, &a.mini;, &a.jhb; | -11 March 2003 | -&status.done; | -
| Fix PHOLD() so that it blocks to guarantee PS_INMEM. | -&a.jhb; | -22 April 2003 | -&status.done; | -
| Fix various procfs_machdep.c to not use sched_lock. | -&a.jhb; | -22 April 2003 | -&status.done; | -
| Lock all references to process credentials and remove Giant - from process credential-related system calls. | -&a.jhb; | -1 May 2003 | -&status.done; | -
| Merge the procsig and sigacts structures, move the new sigacts - structure out of the U-area and add appropriate locking. | -&a.jhb; | -13 May 2003 | -&status.done; | -
| Remove Giant from the kill() and killpg() system calls. | -&a.jhb; | -13 May 2003 | -&status.done; | -
| Enhance the mutex pool implementation to allow creation and - use of multiple, dynamically allocated pools with adjustable pool - sizes and mutex options. | -&a.truckman; | -16 July 2003 | -&status.done; | -
| Create mutex profiling tool for the kernel so as to measure - contention and behavior of kernel mutexes. | -&a.eivind;, &a.des; | -31 March 2002 | -&status.done; | -
| Lock down linker_file_t structures in the kernel linker. | -&a.arr; | -19 June 2002 | -&status.done; | -
| Lock pipe implementation: VM optimizations. | -- | 4 October 2003 | -&status.done; | -
| Reimplement i386 interrupt and SMP code so that SMP kernels - work on UP boxes and SMP can be enabled in GENERIC. | -&a.jhb; | -3 November 2003 | -&status.done; | -
| Implement generic turnstiles to use when blocking on non-sleepable - locks. | -&a.jhb; | -11 November 2003 | -&status.done; | -
| Split witness_lock() into witness_checkorder() and - witness_lock(). witness_checkorder() would be called before - acquiring a lock to increase the changes of detecting and warning - about a reversal prior to deadlocking. witness_lock() would - simply update witness' internal state to note that a lock has - been acquired. | -&a.jhb; | -24 January 2004 | -&status.done; | -
| Lock per-process resource limits. | -&a.mtm;, &a.jhb; | -4 February 2004 | -&status.done; | -
| Implement a sleep queue abstraction to be used by both msleep() - and condition variables. This new abstraction should use a hash - table of sleep queues with a spin lock on each sleep queue chain - similar to turnstile chain locks to make sched_lock finger - grained. | -&a.jhb; | -27 February 2004 | -&status.done; | -
| Remove Giant from jail(2). | -&a.arr;, &a.rwatson; | -23 April 2004 | -&status.done; | -
| Add subsystem locking to NFSv2, NFSv3 server, permitting upcalls - and other network-related elements to run Giant-free. | -&a.rwatson; | -24 July 2004 | -&status.done; | -
| Add KTR(9) tracing for UMA allocation/free events. | -&a.rwatson; | -05 August 2004 | -&status.done; | -
| Add KTR(9) tracing for GEOM I/O events. | -&a.rwatson; | -21 October 2004 | -&status.done; | -
| Add KTR(9) tracing for busdma events. | -&a.rwatson; | -23 October 2004 | -&status.done; | -
| Add KTR(9) tracing for critical sections. | -&a.rwatson; | -07 November 2004 | -&status.done; | -
| Make the kernel fully preemptive. | -&a.jhb; | -24 November 2004 | -&status.done; | -
| Lock pipe implementation: sigio/fown-related evil. | -&a.alfred; | -24 November 2004 | -&status.done; | -
| Lock down the SysV IPC code. | -&a.alfred; | -24 November 2004 | -&status.done; | -
| Lock contention measurement tool to measure heat of various - locks, including Giant, and permit more directed performance and - locking strategy optimization. | -&a.rwatson; | -24 November 2004 | -&status.done; | -
| Add KTR(9) tracing to scheduler run queues. | -&a.jeff; | -26 December 2004 | -&status.done; | -
| Review locking strategy and correctness of VFS operations - and fix up various failure modes associated with enabling - VFS locking assertions. | -&a.jeff; | -01 January 2005 | -&status.done; | -
| Document in-vnode locking strategy, clean it up. | -&a.jeff; | -01 January 2005 | -&status.done; | -
| Run cross-file system VFS without Giant, acquiring Giant - conditionally based on a file system flag. | -&a.jeff; | -01 January 2005 | -&status.done; | -
| Run UFS file system MPSAFE. | -&a.jeff; | -01 January 2005 | -&status.done; | -
| Add KTR(9) tracing for buffer cache events. | -&a.jeff; | -24 January 2005 | -&status.done; | -
| Break out critical section and spin lock APIs, and re-optimize - critical sections to not disable interrupts in hardware due to - the high cost on some hardware architectures. | -&a.jhb; | -04 April 2005 | -&status.done; | -
| Modify uma(9) to use critical sections to protect per-CPU - statistics, instead of mutexdes, in order to optimize access. | -&a.rwatson; | -29 April 2005 | -&status.done; | -
| Migrate malloc(9) to per-CPU statistics, and use critical - sections to optimize access to those statistics. | -&a.rwatson; | -29 May 2005 | -&status.done; | -
| Add KTR(9) support for KTR_VFS to trace additional VFS events, - rather than mechanically inserted KTR_VOP events. | -&a.jeff; | -11 June 2005 | -&status.done; | -
| Push the grabbing of Giant into Linux i386 ABI system calls. | -&a.jhb; | -13 July 2005 | -&status.done; | -
| Push the grabbing of Giant into Linux AXP ABI system calls. | -&a.jhb; | -13 July 2005 | -&status.done; | -
| Push the grabbing of Giant into SVR4 i386 ABI system calls. | -&a.jhb; | -13 July 2005 | -&status.done; | -
| Push the grabbing of Giant into OSF/1 AXP ABI system calls. | -&a.jhb; | -13 July 2005 | -&status.done; | -
| Push the grabbing of Giant into IBCS i386 ABI system calls. | -&a.jhb; | -13 July 2005 | -&status.done; | -
| Add a new witness check for exiting threads to verify that - an exiting thread holds no locks. | -&a.jhb; | -2 September 2005 | -&status.done; | -
| Implement atomic_fetchadd() for ints. | -&a.jhb; | -27 September 2005 | -&status.done; | -
| Implement a simple reference count API using atomic operations and - use this to replace locks that just protect a reference count. | -&a.jhb; | -27 September 2005 | -&status.done; | -
| Split the interrupt handler list out of struct ithread into - its own structure and only start up kthreads for interrupt - vectors that actually have threaded interrupt handlers. | -&a.jhb; | -25 October 2005 | -&status.done; | -
| Lock aio(4). | -&a.davidxu; | -22 January 2006 | -&status.done; | -
| Implement reader/writer locks. | -&a.jhb; | -27 January 2006 | -&status.done; | -
| Remove the MPSAFE syscall flag from the syscall table. | -&a.jhb; | -28 July 2006 | -&status.done; | -
| Add sufficient synchronization to printf to avoid printing - problems but not so much it doesn't work in debugging contexts. | -&a.jb; | -1 November 2006 | -&status.done; | -
| Expand mutex profiling tool to also profile sx locks. | -&a.kmacy; | -11 November 2006 | -&status.done; | -
| Lock down the TTY subsystem. | -&a.ed; | -20 August 2008 | -&status.done; | -
| Lock struct proc. | -&a.jhb; | -20 February 2001 | -&status.wip; | -
| Fix clock locking to be the same on all platforms. | -&a.jhb; | -16 November 2001 | -&status.wip; | -
| Make use of process locking and process reference counting - to protect debugging interfaces (and procfs). | -&a.jhb; | -27 February 2002 | -&status.wip; | -
| Make use of process locking to protect process monitoring - sysctls, including those employed by 'ps' and related tools. - | &a.jhb; | -27 February 2002 | -&status.wip; | -
| Lock down newbus infrastructure to support driver - fine-graining. | -&a.imp; | -28 February 2002 | -&status.wip; | -
| SMPng architecture document. | -&a.jhb;, &a.rwatson; | -28 February 2002 | -&status.wip; | -
| Move to shared lock for VOP_GETATTR() to reduce blocking during - frequent lightweight VFS operations. Modify namei() to provide - a LOOKUP_SHARED flag to indicate when the lock required may be - shared instead of exclusive. | -&a.jeff; | -11 March 2002 | -&status.wip; | -
| Document existing vm_map locking and verify it's correctness. | -&a.alc; | -18 May 2002 | -&status.wip; | -
| Document existing vm_object locking and verify it's - correctness. | -&a.alc; | -4 May 2002 | -&status.wip; | -
| Implement lazy interrupt thread switching (context - stealing) on i386. | -&a.bmilekic;, &a.kan; | -10 December 2002 | -&status.wip; | -
| Implement lazy interrupt thread switching (context - stealing) on sparc64. | -&a.jake; | -10 December 2002 | -&status.wip; | -
| Switch from using lockmgr in VM to using a mutex or exclusive - sxlock. Push down Giant on all VM except for vm_object/VFS and - vm_page/pmap components. | -&a.alc; | -10 December 2002 | -&status.wip; | -
| Modify device driver API to permit drivers to more easily - split "in interrupt context" and "in interrupt thread" - code so as to acknowledge interrupts faster. This will permit - lower latency in interrupt handling. | -&a.piso; | -24 February 2007 | -&status.wip; | -
| Conditionalize atomic ops in the SMP code that are used - for debugging statistics. | -&a.peter; | -15 March 2001 | -&status.new; | -
| Axe schedcpu() in favor of event driven priority updates as - much as possible. | -- | 7 September 2001 | -&status.new; | -
| Fix *hold (e.g. crhold) to return reference to object. | -- | 7 September 2001 | -&status.new; | -
| Add witness checking for lockmgr locks. | -- | 7 September 2001 | -&status.new; | -
| Add ICU spin locks on ia64. | -- | 4 January 2002 | -&status.new; | -
| Add a witness_sleep() check to copyin/out() and s/fuword(). | -&a.jhb; | -7 June 2002 | -&status.new; | -
| Remove Giant from mi_startup() and push Giant down into - individual SYSINIT functions. Many SYSINIT functions probably do - not need Giant anyway. | -- | 21 October 2005 | -&status.new; | -
This table lists the todo subtasks for multithreading the network - stack.
- -| Task | -Responsible | -Last updated | -Status | -
|---|---|---|---|
| Protect network interface queues. | -&a.jlemon; | -24 November 2000 | -&status.done; | -
| Lock up IP. | -&a.jennifer;, &a.hsu; | -10 June 2002 | -&status.done; | -
| Lock up TCP. | -&a.jennifer;, &a.hsu;, &a.sam;, &a.rwatson; | -24 November 2004 | -&status.done; | -
| Lock up UDP. | -&a.jennifer;, &a.hsu;, &a.rwatson; | -24 November 2004 | -&status.done; | -
| Lock ifaddr reference counts. | -&a.hsu; | -18 December 2002 | -&status.done; | -
| Lock up ifnet list. | -&a.hsu; | -21 December 2002 | -&status.done; | -
| Lock radix trees. | -&a.hsu; | -23 December 2002 | -&status.done; | -
| Lock up ARP. | -&a.hsu; | -16 January 2003 | -&status.done; | -
| Lock up raw IP. | -&a.sam;, &a.rwatson; | -24 July 2004 | -&status.done; | -
| Lock divert sockets. | -&a.sam; | -4 October 2003 | -&status.done; | -
| Lock ipfw2. | -&a.sam; | -4 October 2003 | -&status.done; | -
| Lock DUMMYNET. | -&a.sam; | -4 October 2003 | -&status.done; | -
| Lock ethernet bridge. | -&a.sam; | -4 October 2003 | -&status.done; | -
| Lock IP fragment queues. | -&a.rwatson; | -4 October 2003 | -&status.done; | -
| Lock routing entries. | -&a.sam; | -4 October 2003 | -&status.done; | -
| Lock FAST_IPSEC. | -&a.sam; | -4 October 2003 | -&status.done; | -
| Permit parallel entry into isr processing. | -&a.rwatson;, &a.sam; | -11 October 2003 | -&status.done; | -
| Lock if_disc "discard interface". | -&a.rwatson; | -9 March 2004 | -&status.done; | -
| Lock if_faith "IPv6-to-IPv4 TCP relay interface. " | -&a.sam;, &a.rwatson; | -9 March 2004 | -&status.done; | -
| Lock if_gif "generic tunnel interface". | -&a.rwatson; | -9 March 2004 | -&status.done; | -
| Review ECN tunnel support (ip_ecn.c). | -&a.rwatson; | -9 March 2004 | -&status.done; | -
| if_tap global and softc locking. | -&a.rwatson; | -23 April 2004 | -&status.done; | -
| if_tun global and softc locking. | -&a.rwatson; | -23 April 2004 | -&status.done; | -
| netatalk/aarp.c locking. | -&a.rwatson; | -23 April 2004 | -&status.done; | -
| Cache socket MAC label in inpcb label for IPv4 sockets so that - the label can be used safely at the inet layer without socket - locks. | -&a.rwatson; | -23 April 2004 | -&status.done; | -
| IP encapsulation subroutines (ip_encap.c). | -&a.rwatson; | -23 April 2004 | -&status.done; | -
| Lock globals in loopback interface (if_loop.c). | -&a.rwatson; | -23 April 2004 | -&status.done; | -
| Use m_tags in if_gif to limit looping configurations, rather - than a non-MPSAFE static coutner. | -&a.ru; | -23 April 2004 | -&status.done; | -
| netatalk DDP PCB locking. | -&a.rwatson; | -24 July 2004 | -&status.done; | -
| Lock up syncache. | -&a.hsu;, &a.sam; | -10 November 2003 | -&status.done; | -
| Permit IP forwarding path to run Giant-free. | -&a.sam; | -1 December 2003 | -&status.done; | -
| Lock &unix; domain protocols, fifofs. | -&a.sam;, &a.rwatson; | -24 July 2004 | -&status.done; | -
| Giant lock over NFS server to protect against so_upcall() w/o Giant | -&a.rwatson; | -24 July 2004 | -&status.done; | -
| Lock interface cloning meta-data. | -&a.brooks; | -24 July 2004 | -&status.done; | -
| Apply combination of socket and socket buffer locks, label caching - to MAC labels on sockets so that they can be used safely without - Giant. | -&a.rwatson; | -24 July 2004 | -&status.done; | -
| Make routing socket message dispatch use a netisr to avoid - re-entering the socket code from the routing code, resolving lock - order issues. | -&a.rwatson; | -24 July 2004 | -&status.done; | -
| Introduce accept locking to protect accept incomplete and - complete queues on listen sockets. | -&a.rwatson; | -24 July 2004 | -&status.done; | -
| Break out socket buffer wakeup, socket buffer append, socket - state change, socket buffer reserve, flush, etc, calls into - _locked() and unlocked versions, and avoid conditional locking. - | -&a.rwatson; | -24 July 2004 | -&status.done; | -
| Lock down AARP, AppleTalk Address Resolution Protocol. | -&a.rwatson; | -24 July 2004 | -&status.done; | -
| Fix pull/push cache data synchronization issues in sosend(), - soreceive(), allowing them to run Giant-free. | -&a.rwatson; | -24 July 2004 | -&status.done; | -
| Protect socket global counters/limits and generation number - with a mutex. | -&a.rwatson; | -24 July 2004 | -&status.done; | -
| Lock down unit allocation meta-data in interface related - netgraph modules. | -&a.rwatson; | -24 July 2004 | -&status.done; | -
| Lock down socket buffer OOB fields across TCP/IP, IPX. | -&a.rwatson; | -24 July 2004 | -&status.done; | -
| Add MSG_NBIO so that fifofs can avoid frobbing SO_NBIO in - a manner that risks races. | -&a.truckman; | -24 July 2004 | -&status.done; | -
| Protect all use of so_count with socket lock. | -&a.rwatson; | -24 July 2004 | -&status.done; | -
| Move socket buffer related state from so_state to sb_state so - it can be properly locked by the socket buffer mutex. | -&a.rwatson; | -24 July 2004 | -&status.done; | -
| Introduce a temporary global lock to lock the if_label field - used by the MAC Framework. | -&a.rwatson; | -24 July 2004 | -&status.done; | -
| Push VFS-specific behavior out of fdrop_locked() and acquire - Giant in the fo_close per-object methods rather than - fdrop_locked(), so that pipes and sockets can run fo_close() - Giant-free. | -&a.rwatson; | -24 July 2004 | -&status.done; | -
| Push Giant acquisition into fo_stat() file descriptor stat - operation, rather than acquiring it in fstat(), so that fstat() - on sockets and pipes can run Giant-free. | -&a.rwatson; | -24 July 2004 | -&status.done; | -
| Don't hold socket locks over entry to protocol switch methods, - allowing protocol methods to acquire socket locks after protocol - locks in the lock order. | -&a.rwatson; | -24 July 2004 | -&status.done; | -
| Port inpcb mutex locking, assertions from IPv4 to IPv6. | -&a.rwatson; | -8 August 2004 | -&status.done; | -
| Add IFF_NEEDSGIANT to allow if_start to run with Giant for - specific interfaces. Defer if_start to task queue. | -&a.rwatson; | -8 August 2004 | -&status.done; | -
| Push down Giant in stat(), fo_stat() to allow Giant-free stat - of pipes, sockets. | -&a.rwatson; | -8 August 2004 | -&status.done; | -
| Add TCP lock assertions. | -&a.rwatson; | -24 November 2004 | -&status.done; | -
| Lock socket layer. | -&a.sam;, &a.rwatson; | -24 November 2004 | -&status.done; | -
| Review TCP timer code. | -&a.rwatson; | -24 November 2004 | -&status.done; | -
| Analyze and reduce cost of entropy gathering in network critical - paths. | -&a.rwatson;, &a.markm; | -24 November 2004 | -&status.done; | -
| Allow code to declare NET_NEEDS_GIANT(), forcing Giant over the - network stack if that code is compiled into the kernel. | -&a.rwatson; | -28 August 2004 | -&status.done; | -
| Disable Giant over the network stack in the default - configuration. | -&a.rwatson; | -28 August 2004 | -&status.done; | -
| Additional KTR tracing for UMA, callouts, interrupts, etc. | -&a.rwatson; | -07 November 2004 | -&status.done; | -
| Move to using file descriptor reference counts instead of - socket reference counts for socket system calls, avoiding extra - reference couht operations. | -&a.rwatson; | -24 October 2004 | -&status.done; | -
| Lock down netnatm. | -&a.rwatson; | -April 2006 | -&status.done; | -
| Default to direct dispatch from ithreads | -&a.rwatson; | -28 November 2006 | -&status.done; | -
| Lock IPv6. | -&a.sam;, &a.rwatson;, &a.ume;, &a.mlaier; | -8 August 2004 | -&status.wip; | -
| if_ppp global, per-softc locking. | -&a.rwatson;, Maurycy Pawlowski-Wieronski <maurycy@fouk.org> | -23 April 2004 | -&status.wip; | -
| Lock struct ifnet. | -&a.mlaier;, &a.luigi;, Maurycy Pawlowski-Wieronski <maurycy@fouk.org> | -23 April 2004 | -&status.wip; | -
| Lock IPv4, IPv6, atalk interface address lists. | -&a.mlaier;, &a.rwatson; | -8 August 2004 | -&status.wip; | -
| Lock consumers of BSD compress (bsd_comp.c) code to protect - compression state. | -&a.rwatson; | -23 April 2004 | -&status.wip; | -
| Lock global and softc state for six-to-four converter (if_stf.c). | -&a.rwatson; | -23 April 2004 | -&status.wip; | -
| Lock down global and softc state for SLIP (if_sl.c). | -&a.rwatson; | -23 April 2004 | -&status.wip; | -
| Lock global and softc state for SPPP (if_sppsubr.c). | -&a.rik;, &a.rwatson; | -23 April 2004 | -&status.wip; | -
| IGMP locking. | -&a.rwatson; | -23 April 2004 | -&status.wip; | -
| IP ID locking. | -&a.ups; | -24 June 2005 | -&status.wip; | -
| Locking for polling(4). | -&a.pjd;, &a.glebius; | -19 October 2005 | -&status.wip; | -
| BPF locking needs some cleanup, there are some race conditions there relating to interface removal. | -&a.csjp; | -2 June 2006 | -&status.wip; | -
| Reduce contention upon locking a socket buffer by replacing - tsleep() and wakeup() with a condvar. | -&a.tanimura; | -21 April 2002 | -&status.new; | -
| Lock if_ef "ethernet frame" driver. | -- | 9 March 2004 | -&status.new; | -
| Further cleanup of socket state machine in order to facilitate finishing socking locking of state transitions. | -&a.rwatson; | -19 October 2005 | -&status.new; | -
| Lock KAME IPSEC. | -- | 19 October 2005 | -&status.new; | -
| Only one of our ATM stacks is MPSAFE, the other two should be deleted or fixed. | -- | 19 October 2005 | -&status.new; | -
| Lock ND6 (IPv6 Neighbor Discovery). | -&a.gnn; | -19 October 2005 | -&status.new; | -
| Lock IPv6 multicast address lists. | -- | 19 October 2005 | -&status.new; | -
| Lock IPv4 and IPv6 global address lists. | -- | 19 October 2005 | -&status.new; | -
| Continued cleanup of the stack/device driver ownership and locking for struct ifnet needs to be done. Most fields are now either locked or assigned ownership. Some fields, such as if_flags, need a bit more cleanup due to device drivers modifying stack-owned fields. | -&a.rwatson; | -19 October 2005 | -&status.new; | -
| When interfaces are torn down, there are a number of races (not all associated with SMPng) that need to be thought about. | -- | 19 October 2005 | -&status.new; | -
| Lock if_vlan and inter-layer multicast address manipulation and synchronization in if_vlan. | -&a.glebius;, &a.yar; | -19 October 2005 | -&status.new; | -
| Further investigate locking in in_gif and in6_gif. | -- | 19 October 2005 | -&status.new; | -
| FAST_IPSEC and KAME IPSEC's PF_KEY support likely needs an asynchronous dispatch to prevent socket lock ordering issues similar to what was done for PF_ROUTE. | -&a.rwatson; | -19 October 2005 | -&status.new; | -
| Investigate how to eliminate the use of ACCEPT_LOCK(), which currently prevents races in the tear-down of sockets. | -&a.rwatson; | -19 October 2005 | -&status.new; | -
| Fix SMP problems with netgraph restructuring. | -&a.glebius; | -19 October 2005 | -&status.new; | -
| Verify locking in netgraph nodes and improve where necessary. | -&a.glebius; | -19 October 2005 | -&status.new; | -
| More finely-grained locking for pf(4). | -&a.glebius; | -19 October 2005 | -&status.new; | -
| Issue | -Last updated | -Status | -
|---|---|---|
| Idle processor time is not charged to the idle processes. | -20 September 2000 | -&status.resolved; | -
| microuptime creeps backwards. | -4 October 2000 | -&status.resolved; | -
| microuptime() went backwards | -4 October 2000 | -&status.resolved; | -
| Process accounting is not accurate (the more CPUs, the - closer to correct it is). | -5 October 2000 | -&status.resolved; | -
| M_DEVBUF is probably the wrong memory pool for interrupt - stuff and we should think about creating a new malloc pool for - that stuff. | -9 February 2001 | -&status.resolved; | -
| PC card eject panics due to a race condition in the - interrupt thread code. | -15 March 2001 | -&status.resolved; | -
| SMP x86 boxes are seeing NCPU * 100 clk interrupts and - NCPU * 128 rtc interrupts. | -15 May 2001 | -&status.resolved; | -
| Witness will infinitely recurse when it acquires Giant after - sleeping with a sleepable lock. | -27 June 2001 | -&status.resolved; | -
| Serial gdb does not work if boot_ddb and boot_gdb options - are specified. | -14 July 2002 | -&status.resolved; | -
| Serial gdb does not work at 115200 baud. | -14 July 2002 | -&status.resolved; | -
| Serial gdb never regains control once 'cont' has been - entered. | -14 July 2002 | -&status.resolved; | -
| Profiling is broken. | -20 February 2001 | -&status.unresolved; | -
The remainder of this page is structured as a - reverse-chronological log.
- - - 28 August 2004 -&a.rwatson; threw the switch to change the network stack to run - without the Giant lock by default, permitting the network stack to be - run on multiple CPUs at a time, as well as to preempt and be - preempted by other code.
-&a.grog; submitted a - FreeBSD SMPng paper - to the Asian Enterprise Open Source - Conference in Singapore. The paper presents a historical view of - SMPng development through 2001, but omits discussion of more recent - progress on the SMPng project, such as substantial performance enhancements - resulting from extensive lock pushdown in the storage subsystem, VM - subsystem, and major IPC subsystems.
-A - status report was sent to the -smp mailing list.
-&a.grog; has made his USENIX paper available, - which he will present in Boston at the end of June.
-A - status report was sent to the -smp mailing list.
-A - status report was sent to the -smp mailing list.
-A - status report was sent to the -smp mailing list.
-A - status report was sent to the -smp mailing list.
-&a.jhb; and &a.cp; came up with a preliminary list of - rules - that should be followed when working on kernel - synchronization.
-The SMP code has been committed. All further work is being - done in cvs rather than with patches.
-An updated patch - is available for download. This patch is probably what will - actually get committed.
-An updated patch - is available for download. This patch makes rtc a fast - interrupt, uses locked instructions for mutexes in MP kernels, - and corrects mtx_*() linkage within modules.
-The code is working for the most part now on i386 (UP and MP). - Some additional coding is still necessary for the alpha, - which is being done now.
-Updated patches for i386 and alpha are available here.
-Updated patches for i386 are available here. - Process accounting still doesn't work correctly, but a number - of other improvements have been made.
-Patches with functional heavy-weight threads for the i386 - platform are available here. - There are a couple of minor issues with this patch set. - Specifically, process accounting doesn't work correctly.
-Sheldon Hearn has prepared a mutex(9) man page based on the - BSD/OS one, which is available here.
-&a.jake; put an updated patch here.
-&a.jake; has the BSD/OS lock code working now, and has - incorporated the pertinent portions of &a.dillon;'s patches - (idle processes, some of the schedlock changes, etc.). - His patch set is available here.
-&a.cp; has provided the PostScript versions of his - presentation slides for the - first day and - second day of the - SMP meeting.
-Here's a copy of the - SMP meeting summary - that was posted to the -smp mailing list.
-Here's a copy of the +
SMP project announcement - that was posted to the -current mailing list.
+ that was posted to the -current mailing list. +