Transcription

1

2 Light Weight Kernel Threading and User Processes LWKT SUBSYSTEM USERLAND PROCESS SCHEDULER CPU #1 BGL PROCESS SCHEDULER RUNQ LWKT SCHEDULER (FIXED PRI) BGL PROCESS2 CURPROC ON CPU1 PROCESS1 PROCESS2 CPU #2 PROCESS3 BGL PROCESS4 LWKT SCHEDULER (FIXED PRI) PROCESS5 CURPROC ON CPU2 PROCESS5 BGL PROCESS6

3 IPI Messaging Abstraction promotes cpu isolative algorithms. IPI Messages avoid mutexes. Software crossbar approach. Critical sections to interlock against interrupts and IPIs. Many IPIs can be passive. Pipelining can be made optimal. CPU #1 CPU #2 SCHEDULER SCHEDULER 1 SCHEDULE 2 ASYNC IPI MESSAGE SCHEDULE ALLOCATOR ALLOCATE MEMORY ON CPU#1 ALLOCATOR FREE MEMORY ON CPU #1 PASSIVE IPI MESSAGE ALLOCATOR FREE MEMORY OWNED BY CPU #1 2

4 Light Weight Kernel Thread Messaging Amiga style message and port API. Highly Flexible. Semisynchronous: direct call to target's port agent. UP/SMP optimized Very fast synchronous path. #1 TARGET PORT (OWNED BY T2) APP SEND MSG SYNC RETURN PORT AGENT PROCESSES MSG APP #1 #2 TARGET PORT (OWNED BY T2) APP SEND MSG EASYNC PORT AGENT QUEUES MSG REPLY PORT (WAKEUP T2) GETMSG APP WAITMSG RETURN (WAKEUP T1) REPLY MSG

5 Per-CPU localization. Backed by kernel_map. No more kmem_map. Slab Allocator CPU #1 CPU #2 PER-CPU SLAB CACHE REMOTE FREE (PASSIVE IPI) PER-CPU SLAB CACHE BULK ALLOCATION (LOCK) BULK FREE (LOCK) BULK BULK=128K TYP ALLOCATION HYSTERESIS W/FREE (LOCK) BULK FREE (LOCK) KERNEL_MAP

6 Threaded Network Protocol Stacks Work being done by Jeffrey Hsu Utilizes LWKT messages and ports 1 thread per cpu per protocol typ. UP is a degenerate case of SMP. NETISR TCP A NETISR TCP B PCB PCB PCB PCB PCB PCB NETIF MB UF ALLOC PROTO SWITCH PACKETS (IPI) Etc NETISR UDP A NETISR UDP B PCB PCB PCB PCB PCB PCB Etc CPU #0 CPU #1 MBUF FREE (PASSIVE IPI) USER READ Mbuf free path expected to cross cpus. Cache and return, or cache and reuse?

7 Threaded Network Protocol Stacks Wildcard PCBs for TCP Hash on (srcaddr, srcport, destaddr, destport). Replicate wildcard listen sockets. Create normal PCB on accept(). CPU #1 (TCP ) CPU #2 (TCP ) CPU #3 (TCP ) CPU #4 (TCP ) PCB 1 PCB 2 PCB 3 PCB 4 PCB 5 PCB 6 PCB 7 PCB 8 PCB 9 PCB 10 PCB 11 PCB 12 PCB #30 PCB #30 PCB #30 PCB #30 PCB #31 PCB #31 PCB #31 PCB #31

8 PROGRAM Threaded Network Protocol Stacks Wildcard PCBs CLOSE MSG Reference counters require synchronization. So don't use reference counters.. CPU #1 (TCP ) CPU #2 (TCP ) CPU #3 (TCP ) CPU #4 (TCP ) PCB #30 PCB #30 PCB #30 PCB #30 FREE PCB

9 The Namecache FREEBSD optional, disconnected namespace vnode locked DRAGONFLY mandatory, connected namespace namecache locked Struct vnode LIST_HEAD(, ncp) v_cache_src; TAILQ_HEAD(, ncp) v_cache_dst; struct vnode *v_dd; Struct vnode Struct ncp_list v_namecache; Struct namecache (ncp) LIST_ENTRY(ncp) nc_src; TAILQ_ENTRY(ncp) nc_dst; struct vnode *nc_dvp; struct vnode *nc_vp; Struct namecache (ncp) TAILQ_ENTRY(ncp) nc_entry; TAILQ_ENTRY(ncp) nc_vnode; struct ncp_list nc_list; struct vnode *nc_vp;

10 The Namecache Namespace locked by namecache, not by vnode. Negative namecache entries can be created and locked. Aliased vnodes do not confuse the kernel. VOP_RENAME *BSD: VOP_RENAME(dvp, fvp, fcnp, tdvp, tvp, tncp) DRAGONFLY: VOP_NRENAME(fncp, tncp) Struct filedesc struct vnode *fd_cdir; struct vnode *fd_rdir; struct vnode *fd_jdir; struct namecache *fd_ncdir; struct namecache *fd_nrdir; struct namecache *fd_njdir;

11 The Namecache Simplified VFS interface: VOP_NRESOLVE(ncp, cred) Simplified kernel interface: nlookup(). No more namei(), no more lookup() No more componentnames, except in compatibility shims. Separate NLOOKUPDOTDOT for.. lookups (ala Linux). Topological guarentees for referenced namecache records. Node remains intact in the topology even if file deleted. Traversal to root is guarenteed (getcwd, fstat, and more) Most dvp references replaced by ncp references. VFS involvement only for attribute access (for now).

12 The Namecache Future work Intended to be hooked into a cache coherency layer (aliasing, SSI, NFS). Direct Attribute Caching. Good place for a MAC implementation. Clean up VFSs and remove shims supporting obsolete VOP's.

13 Big Giant Lock Removal Matthew Dillon DragonFly BSD Project April 2005

14 Current BGL Coverage FASTINT SERIAL INTERRUPT S CLOCK LWKT SUBSYSTEM -- (scheduler) (messaging) SOFTINT S IDLE NETIF DISK S (IN KERNEL) TIMER IPI MSGS S (IN USER) NETISR ROUTE TBL SYSCALLS VFS MEMORY SUB SLAB CACHE MEMORY SUB KMEM ALLOC =BGL NOT REQUIRED =BGL REQUIRED

15 Next Stage BGL Removal FASTINT SERIAL INTERRUPT S CLOCK LWKT SUBSYSTEM SOFTINT S IDLE NETIF DISK S (IN KERNEL) TIMER IPI MSGS S (IN USER) NETISR ROUTE TBL SYSCALLS MEMORY SUB SLAB CACHE VFS MEMORY SUB KMEM ALLOC =BGL NOT REQUIRED =BGL REQUIRED

16 BGL Removal Network Detail WORK BEING DONE BY JEFFREY HSU UTILIZES LWKT MSGS AND PORTS MULTIPLE S PER PROTOCOL NO UNI-PROCESSOR PENALTY NETISR TCP A NETISR TCP B PCB PCB PCB PCB PCB PCB NETIF PACKETS PROTO SWITCH Etc NETISR UDP A NETISR UDP B PCB PCB PCB PCB PCB PCB Etc =BGL NOT REQUIRED

17 The DragonFly BSD Project April 2005 Joe Angerson David Xu Matthew Dillon Craig Dooley Liam J. Foy Robert Garrett Jeffrey Hsu Douwe Kiela Sascha Wildner Emiel Kollof Kip Macy Andre Nathan Erik Nygaard Max Okumoto Hiten Pandya Chris Pressey David Rhodus Galen Sampson YONETANI Tomokazu Hiroki Sato Simon Schubert Joerg Sonnenberger Justin Sherrill Scott Ullrich Jeroen Ruigrok van der Werven Todd Willey and Fred -->