opnsense-src/sys/kern
Robert Watson d4b5cae49b Reimplement the netisr framework in order to support parallel netisr
threads:

- Support up to one netisr thread per CPU, each processings its own
  workstream, or set of per-protocol queues.  Threads may be bound
  to specific CPUs, or allowed to migrate, based on a global policy.

  In the future it would be desirable to support topology-centric
  policies, such as "one netisr per package".

- Allow each protocol to advertise an ordering policy, which can
  currently be one of:

  NETISR_POLICY_SOURCE: packets must maintain ordering with respect to
    an implicit or explicit source (such as an interface or socket).

  NETISR_POLICY_FLOW: make use of mbuf flow identifiers to place work,
    as well as allowing protocols to provide a flow generation function
    for mbufs without flow identifers (m2flow).  Falls back on
    NETISR_POLICY_SOURCE if now flow ID is available.

  NETISR_POLICY_CPU: allow protocols to inspect and assign a CPU for
    each packet handled by netisr (m2cpuid).

- Provide utility functions for querying the number of workstreams
  being used, as well as a mapping function from workstream to CPU ID,
  which protocols may use in work placement decisions.

- Add explicit interfaces to get and set per-protocol queue limits, and
  get and clear drop counters, which query data or apply changes across
  all workstreams.

- Add a more extensible netisr registration interface, in which
  protocols declare 'struct netisr_handler' structures for each
  registered NETISR_ type.  These include name, handler function,
  optional mbuf to flow ID function, optional mbuf to CPU ID function,
  queue limit, and ordering policy.  Padding is present to allow these
  to be expanded in the future.  If no queue limit is declared, then
  a default is used.

- Queue limits are now per-workstream, and raised from the previous
  IFQ_MAXLEN default of 50 to 256.

- All protocols are updated to use the new registration interface, and
  with the exception of netnatm, default queue limits.  Most protocols
  register as NETISR_POLICY_SOURCE, except IPv4 and IPv6, which use
  NETISR_POLICY_FLOW, and will therefore take advantage of driver-
  generated flow IDs if present.

- Formalize a non-packet based interface between interface polling and
  the netisr, rather than having polling pretend to be two protocols.
  Provide two explicit hooks in the netisr worker for start and end
  events for runs: netisr_poll() and netisr_pollmore(), as well as a
  function, netisr_sched_poll(), to allow the polling code to schedule
  netisr execution.  DEVICE_POLLING still embeds single-netisr
  assumptions in its implementation, so for now if it is compiled into
  the kernel, a single and un-bound netisr thread is enforced
  regardless of tunable configuration.

In the default configuration, the new netisr implementation maintains
the same basic assumptions as the previous implementation: a single,
un-bound worker thread processes all deferred work, and direct dispatch
is enabled by default wherever possible.

Performance measurement shows a marginal performance improvement over
the old implementation due to the use of batched dequeue.

An rmlock is used to synchronize use and registration/unregistration
using the framework; currently, synchronized use is disabled
(replicating current netisr policy) due to a measurable 3%-6% hit in
ping-pong micro-benchmarking.  It will be enabled once further rmlock
optimization has taken place.  However, in practice, netisrs are
rarely registered or unregistered at runtime.

A new man page for netisr will follow, but since one doesn't currently
exist, it hasn't been updated.

This change is not appropriate for MFC, although the polling shutdown
handler should be merged to 7-STABLE.

Bump __FreeBSD_version.

Reviewed by:	bz
2009-06-01 10:41:38 +00:00
..
bus_if.m Allow device hints to wire the unit numbers of devices. 2008-11-18 21:01:54 +00:00
clock_if.m /* -> /*- for copyright notices, minor format tweaks as necessary 2005-01-06 23:35:40 +00:00
cpufreq_if.m Introduce a new method, cpufreq_drv_type(), that returns the type of the 2005-02-18 00:23:36 +00:00
device_if.m /* -> /*- for copyright notices, minor format tweaks as necessary 2005-01-06 23:35:40 +00:00
genassym.sh refactor code so it can run in a chroot without having to have /dev/mounted 2008-01-18 17:02:14 +00:00
imgact_aout.c Add sv_flags field to struct sysentvec with intention to provide description 2008-11-22 12:36:15 +00:00
imgact_elf.c Fix KBI breakage by r190520 which affects older linux.ko binaries: 2009-04-05 09:27:19 +00:00
imgact_elf32.c Use __FBSDID(). 2003-06-11 00:56:59 +00:00
imgact_elf64.c Use __FBSDID(). 2003-06-11 00:56:59 +00:00
imgact_gzip.c VOP_LOCK1() (and so VOP_LOCK()) and VOP_UNLOCK() are only used in 2008-01-13 14:44:15 +00:00
imgact_shell.c Decontextualize the couplet VOP_GETATTR / VOP_SETATTR as the passed thread 2008-08-28 15:23:18 +00:00
inflate.c Normalize a significant number of kernel malloc type names: 2005-10-31 15:41:29 +00:00
init_main.c Add hierarchical jails. A jail may further virtualize its environment 2009-05-27 14:11:23 +00:00
init_sysent.c Regen for new jail system calls in r191673. 2009-04-29 21:50:13 +00:00
kern_acct.c Remove the thread argument from the FSD (File-System Dependent) parts of 2009-05-11 15:33:26 +00:00
kern_alq.c Remove VOP_LEASE and supporting functions. This hasn't been used since 2009-04-10 10:52:19 +00:00
kern_clock.c Mark the clock sysctls as MPSAFE. 2009-05-18 12:03:43 +00:00
kern_condvar.c Remove unused variables p' and unneeded assignments of rval'. 2009-02-26 13:00:13 +00:00
kern_conf.c Move the M_WAITOK flag in notify() into an M_NOWAIT one in order to match 2009-05-21 13:22:07 +00:00
kern_cons.c Remove unneeded variable `ocn_mute'. 2009-02-26 13:01:45 +00:00
kern_context.c Further system call comment cleanup: 2007-03-05 13:10:58 +00:00
kern_cpu.c Provide a new CPU device driver ivar to report the nominal speed of the 2009-05-31 08:59:15 +00:00
kern_cpuset.c Add hierarchical jails. A jail may further virtualize its environment 2009-05-27 14:11:23 +00:00
kern_ctf.c Add the CTF source file which gets shared with link_elf.c and link_elf_obj.c. 2008-05-23 03:04:27 +00:00
kern_descrip.c Add hierarchical jails. A jail may further virtualize its environment 2009-05-27 14:11:23 +00:00
kern_dtrace.c Remove code that isn't required. It actually breaks the case where KDTRACE_HOOKS 2008-06-16 04:44:29 +00:00
kern_environment.c Correctly sanity-check timer IDs. [SA-09:06] 2009-03-23 00:00:50 +00:00
kern_event.c Fix a number of style issues in the MALLOC / FREE commit. I've tried to 2008-10-23 20:26:15 +00:00
kern_exec.c Supply AT_EXECPATH auxinfo entry to the interpreter, both for native and 2009-03-17 12:53:28 +00:00
kern_exit.c Add hierarchical jails. A jail may further virtualize its environment 2009-05-27 14:11:23 +00:00
kern_fail.c fail(9) support: 2009-05-27 16:36:54 +00:00
kern_fork.c Add hierarchical jails. A jail may further virtualize its environment 2009-05-27 14:11:23 +00:00
kern_idle.c In keeping with style(9)'s recommendations on macros, use a ';' 2008-03-16 10:58:09 +00:00
kern_intr.c Binding interrupts to a CPU consists of two parts: setting up CPU 2009-05-18 14:02:55 +00:00
kern_jail.c Place hostnames and similar information fully under the prison system. 2009-05-29 21:27:12 +00:00
kern_kthread.c Kill a dead variable 2008-08-03 21:07:19 +00:00
kern_ktr.c Remove slightly oddly placed suser() call from the KTR/ALQ setup sysctl: 2006-09-09 16:09:01 +00:00
kern_ktrace.c Remove VOP_LEASE and supporting functions. This hasn't been used since 2009-04-10 10:52:19 +00:00
kern_linker.c Add hierarchical jails. A jail may further virtualize its environment 2009-05-27 14:11:23 +00:00
kern_lock.c Add the OpenSolaris dtrace lockstat provider. The lockstat provider 2009-05-26 20:28:22 +00:00
kern_lockf.c The advisory lock may be activated or activated and removed during the 2009-05-24 12:39:38 +00:00
kern_lockstat.c Add the OpenSolaris dtrace lockstat provider. The lockstat provider 2009-05-26 20:28:22 +00:00
kern_malloc.c Retire kern.vm.kmem.size. It was marked as obsolete prior to 5.2, so 2009-05-09 19:00:47 +00:00
kern_mbuf.c Temporary workaround for the limitations of the mbuf flowid field: zero 2009-01-01 20:03:01 +00:00
kern_mib.c Place hostnames and similar information fully under the prison system. 2009-05-29 21:27:12 +00:00
kern_module.c When the SYSINIT() to load a module invokes the MOD_LOAD event successfully, 2008-12-05 16:47:30 +00:00
kern_mtxpool.c Fix a number of style issues in the MALLOC / FREE commit. I've tried to 2008-10-23 20:26:15 +00:00
kern_mutex.c Remove extra cpu_spinwait() invocations. This should really only be used 2009-05-29 14:03:34 +00:00
kern_ntptime.c In keeping with style(9)'s recommendations on macros, use a ';' 2008-03-16 10:58:09 +00:00
kern_osd.c Make the rmlock(9) interface a bit more like the rwlock(9) interface: 2009-05-29 10:52:37 +00:00
kern_physio.c /* -> /*- for copyright notices, minor format tweaks as necessary 2005-01-06 23:35:40 +00:00
kern_pmc.c Support sparsely numbered CPUs. 2008-09-22 10:37:02 +00:00
kern_poll.c Reimplement the netisr framework in order to support parallel netisr 2009-06-01 10:41:38 +00:00
kern_priv.c Reduce the verbosity of SDT trace points for DTrace by defining several 2009-03-03 17:15:05 +00:00
kern_proc.c Add hierarchical jails. A jail may further virtualize its environment 2009-05-27 14:11:23 +00:00
kern_prot.c Introduce an interm userland-kernel API for creating vnets and 2009-05-31 12:10:04 +00:00
kern_resource.c Don't rearm callout if the process is exiting, it may leak a callout 2008-10-24 01:09:24 +00:00
kern_rmlock.c Minor style tweak. 2009-05-29 14:25:51 +00:00
kern_rwlock.c Remove extra cpu_spinwait() invocations. This should really only be used 2009-05-29 14:03:34 +00:00
kern_sdt.c Add kernel support for the Statically Defined Trace provider. 2008-05-18 19:32:36 +00:00
kern_sema.c /* -> /*- for copyright notices, minor format tweaks as necessary 2005-01-06 23:35:40 +00:00
kern_shutdown.c Place hostnames and similar information fully under the prison system. 2009-05-29 21:27:12 +00:00
kern_sig.c Remove VOP_LEASE and supporting functions. This hasn't been used since 2009-04-10 10:52:19 +00:00
kern_subr.c Make ureadc() warn when holding any locks, just like uiomove(). 2008-08-28 19:34:58 +00:00
kern_switch.c fix typo in runz_fuzz 2008-05-12 06:42:06 +00:00
kern_sx.c The patch for r193011 was partially rejected when applied, complete it. 2009-05-29 08:01:48 +00:00
kern_synch.c Remove even more unneeded variable assignments. 2009-02-26 15:51:54 +00:00
kern_syscalls.c Various style fixes. 7 space indent is just odd. 2008-09-18 20:10:11 +00:00
kern_sysctl.c Remove do-nothing code that was required to dirty the old buffer on Alpha. 2009-05-15 21:34:58 +00:00
kern_tc.c Remove conditionally compiled time counter statistics; tools like 2009-04-11 22:01:40 +00:00
kern_thr.c Remove even more unneeded variable assignments. 2009-02-26 15:51:54 +00:00
kern_thread.c - Implement a new mechanism for resetting lock profiling. We now 2009-03-15 06:41:47 +00:00
kern_time.c Remove VOP_LEASE and supporting functions. This hasn't been used since 2009-04-10 10:52:19 +00:00
kern_timeout.c Add explicit static DTrace tracing to the callout mechanism, capturing 2009-01-24 10:22:49 +00:00
kern_umtx.c Make UMTX_OP_WAIT_UINT actually wait for an unsigned integer on 64-bits 2009-04-13 05:21:17 +00:00
kern_uuid.c Lock the interface address list while iterating a network interface's 2009-04-19 21:36:18 +00:00
kern_vimage.c Introduce an interm userland-kernel API for creating vnets and 2009-05-31 12:10:04 +00:00
kern_xxx.c Place hostnames and similar information fully under the prison system. 2009-05-29 21:27:12 +00:00
ksched.c Commit 14/14 of sched_lock decomposition. 2007-06-05 00:00:57 +00:00
link_elf.c Add the ksyms(4) pseudo driver. The ksyms driver allows a process to 2009-05-26 21:39:09 +00:00
link_elf_obj.c Add the ksyms(4) pseudo driver. The ksyms driver allows a process to 2009-05-26 21:39:09 +00:00
linker_if.m Add the ksyms(4) pseudo driver. The ksyms driver allows a process to 2009-05-26 21:39:09 +00:00
Make.tags.inc Catch up with the disappearance of sys/dev/hfa. 2008-12-01 14:34:42 +00:00
Makefile style.Makefile(5) 2007-12-14 21:30:51 +00:00
makesyscalls.sh Tidy up a few things with syscall generation: 2008-09-25 20:07:42 +00:00
md4c.c /* -> /*- for copyright notices, minor format tweaks as necessary 2005-01-06 23:35:40 +00:00
md5c.c Fix a panic on sparc64 related to inproper aligment - we cannot assume, 2006-03-30 18:45:50 +00:00
p1003_1b.c Remove kernel support for M:N threading. 2008-03-12 10:12:01 +00:00
posix4_mib.c Fix mispatch of includes list; allows my kernel to build successfully. 2006-11-12 03:34:03 +00:00
sched_4bsd.c - Use __XSTRING where I want the define to be expanded. This resulted in 2009-01-25 07:35:10 +00:00
sched_ule.c - Fix non-SMP build by encapsulating idle spin logic in a macro. 2009-04-29 23:04:31 +00:00
serdev_if.m MFp4: Add the ipend() method to the serdev I/F to allow umbrella 2006-04-23 22:12:39 +00:00
stack_protector.c Fix a chicken-and-egg problem: this files implements SSP support, 2008-06-26 07:52:45 +00:00
subr_acl_posix1e.c Make 'struct acl' larger, as required to support NFSv4 ACLs. Provide 2009-05-22 15:56:43 +00:00
subr_autoconf.c Prefer ANSI function definitions to K&R ones. 2009-02-03 07:52:07 +00:00
subr_blist.c Retire the MALLOC and FREE macros. They are an abomination unto style(9). 2008-10-23 15:53:51 +00:00
subr_bufring.c - bump __FreeBSD version to reflect added buf_ring, memory barriers, 2008-11-22 05:55:56 +00:00
subr_bus.c Fix a typo. 2009-05-20 17:19:30 +00:00
subr_clist.c Remove a stale comment from the clists code. 2009-02-09 11:27:56 +00:00
subr_clock.c Now that all platforms use genclock, shuffle things around slightly 2008-04-22 19:38:30 +00:00
subr_devstat.c Use NULL in preference to 0 in pointer contexts. 2009-02-03 07:54:42 +00:00
subr_disk.c Clarify and reimplement the bioq API so that bioq_disksort() has 2009-02-13 11:36:32 +00:00
subr_eventhandler.c In keeping with style(9)'s recommendations on macros, use a ';' 2008-03-16 10:58:09 +00:00
subr_fattime.c Better naming of fattime conversion functions, they do convert to timespec 2006-10-24 10:27:23 +00:00
subr_firmware.c Use NULL in preference to 0 for pointers. 2009-02-03 07:51:11 +00:00
subr_hints.c Use a sleep mutex instead of an sx lock for the kernel environment. This 2006-07-09 21:42:58 +00:00
subr_kdb.c Spell NULL properly, use (void) rather than () for functions with no 2009-05-09 19:08:22 +00:00
subr_kobj.c Use NULL in preference to 0 in pointer contexts. 2009-02-03 07:54:42 +00:00
subr_lock.c - Implement a new mechanism for resetting lock profiling. We now 2009-03-15 06:41:47 +00:00
subr_log.c In keeping with style(9)'s recommendations on macros, use a ';' 2008-03-16 10:58:09 +00:00
subr_mbpool.c Add parens around *free in *free++ in mbp_count() so that mbp_count() 2007-05-27 17:38:36 +00:00
subr_mchain.c Replaced the misleading uses of a historical artefact M_TRYWAIT with M_WAIT. 2008-03-25 09:39:02 +00:00
subr_module.c Use __FBSDID(). 2003-06-11 00:56:59 +00:00
subr_msgbuf.c /* -> /*- for copyright notices, minor format tweaks as necessary 2005-01-06 23:35:40 +00:00
subr_param.c Improve the description of a few sysctls. 2009-03-23 20:18:06 +00:00
subr_pcpu.c Change the curvnet variable from a global const struct vnet *, 2009-05-05 10:56:12 +00:00
subr_power.c General consensus is that it would be even better to run this in a 2005-11-09 16:22:56 +00:00
subr_prf.c Remove redundant code in printf() and vprintf(). 2009-02-27 13:28:54 +00:00
subr_prof.c Use ANSI function definition for profil. 2009-02-03 07:52:36 +00:00
subr_rman.c sysctl_rman: report shared resources to devinfo 2009-05-19 14:08:21 +00:00
subr_rtc.c Clean up MI inittodr(9) and kill noop code. 2009-03-23 21:16:21 +00:00
subr_sbuf.c Switch to simplified BSD license (with phk's approval), plus whitespace 2008-08-09 10:26:21 +00:00
subr_scanf.c Remove advertising clause from University of California Regent's license, 2004-04-05 21:03:37 +00:00
subr_sleepqueue.c Revision 184199 had not been fully reverted, add missing piece. 2008-12-01 01:54:55 +00:00
subr_smp.c - Remove the bogus idle thread state code. This may have a race in it 2009-04-29 03:15:43 +00:00
subr_stack.c Make it possible to compile kernel with KTR but without DDB. 2008-10-30 21:48:28 +00:00
subr_taskqueue.c Remove semicolon left in the last commit 2009-02-13 18:51:39 +00:00
subr_trap.c - Bug fix: prevent a thread from migrating between CPUs between the 2008-12-13 13:07:12 +00:00
subr_turnstile.c Make ddb command registration dynamic so modules can extend 2008-09-15 22:45:14 +00:00
subr_unit.c Since cdev mutex is after system map mutex in global lock order, free() 2007-07-04 06:56:58 +00:00
subr_witness.c Add minimal ZFS lock hierarchy 2009-05-20 02:51:48 +00:00
sys_generic.c - Implement a lockless file descriptor lookup algorithm in 2009-05-14 03:24:22 +00:00
sys_pipe.c - Make maxpipekva a signed long rather than an unsigned long as overflow 2009-03-10 21:28:43 +00:00
sys_process.c Use the p_sysent->sv_flags flag SV_ILP32 to detect 32bit process 2009-03-02 18:43:50 +00:00
sys_socket.c Change the curvnet variable from a global const struct vnet *, 2009-05-05 10:56:12 +00:00
syscalls.c Regen for new jail system calls in r191673. 2009-04-29 21:50:13 +00:00
syscalls.master Introduce the extensible jail framework, using the same "name=value" 2009-04-29 21:14:15 +00:00
systrace_args.c Regen for new jail system calls in r191673. 2009-04-29 21:50:13 +00:00
sysv_ipc.c Eliminate now-unused SUSER_ALLOWJAIL arguments to priv_check_cred(); in 2007-06-12 00:12:01 +00:00
sysv_msg.c Add hierarchical jails. A jail may further virtualize its environment 2009-05-27 14:11:23 +00:00
sysv_sem.c Add hierarchical jails. A jail may further virtualize its environment 2009-05-27 14:11:23 +00:00
sysv_shm.c Add hierarchical jails. A jail may further virtualize its environment 2009-05-27 14:11:23 +00:00
tty.c Last minute TTY API change: remove mutex argument from tty_alloc(). 2009-05-29 06:41:23 +00:00
tty_compat.c Fix an awful bug inside our COMPAT_43TTY code. 2008-09-04 16:30:53 +00:00
tty_info.c Print an extra newline when not at the first column already. 2009-05-17 16:17:48 +00:00
tty_inq.c Enable secure TTY input buffer flushing by default. 2009-05-21 16:48:06 +00:00
tty_outq.c Use unsigned longs for the TTY's sysctl stats. 2009-02-26 10:28:32 +00:00
tty_pts.c Last minute TTY API change: remove mutex argument from tty_alloc(). 2009-05-29 06:41:23 +00:00
tty_pty.c Don't use PTY name as format string, even though it isn't insecure here. 2009-02-26 10:14:10 +00:00
tty_tty.c Remove unneeded Giant locking of /dev/tty. 2008-06-03 12:38:00 +00:00
tty_ttydisc.c If we have a regular rint handler, never go into rint_bypass mode. 2009-05-07 17:39:23 +00:00
uipc_accf.c Retire the MALLOC and FREE macros. They are an abomination unto style(9). 2008-10-23 15:53:51 +00:00
uipc_cow.c Extend the struct vm_page wire_count to u_int to avoid the overflow 2009-01-03 13:24:08 +00:00
uipc_debug.c Add missing socket options. 2009-05-26 09:19:21 +00:00
uipc_domain.c Introduce vnet module registration / initialization framework with 2009-04-11 05:58:58 +00:00
uipc_mbuf.c Teach m_copyback() to use trailing space of the last mbuf in chain. 2009-01-18 20:19:55 +00:00
uipc_mbuf2.c Merge first in a series of TrustedBSD MAC Framework KPI changes 2007-10-24 19:04:04 +00:00
uipc_mqueue.c Remove the thread argument from the FSD (File-System Dependent) parts of 2009-05-11 15:33:26 +00:00
uipc_sem.c Ensure that the semaphore value is re-checked after sem_lock 2009-03-12 10:36:39 +00:00
uipc_shm.c Shared memory objects that have size which is not necessarily equal to 2008-12-01 22:33:50 +00:00
uipc_sockbuf.c Fix sbappendrecord_locked(). 2009-04-21 19:14:13 +00:00
uipc_socket.c A NOP change: style / whitespace cleanup of the noise that slipped 2009-05-08 14:34:25 +00:00
uipc_syscalls.c Split native socketpair() syscall onto kern_socketpair() which should 2009-05-31 12:12:38 +00:00
uipc_usrreq.c Change the curvnet variable from a global const struct vnet *, 2009-05-05 10:56:12 +00:00
vfs_acl.c Make 'struct acl' larger, as required to support NFSv4 ACLs. Provide 2009-05-22 15:56:43 +00:00
vfs_aio.c Use the correct type for the timeout parameter to the 32-bit 2009-01-23 13:23:17 +00:00
vfs_bio.c Eliminate a comment describing code that was deleted over eight years ago. 2009-06-01 06:12:08 +00:00
vfs_cache.c Unbreak the build. Add missed probes. 2009-05-31 20:16:06 +00:00
vfs_cluster.c - Complete part of the unfinished bufobj work by consistently using 2008-03-22 09:15:16 +00:00
vfs_default.c Add VOP_ACCESSX, which can be used to query for newly added V* 2009-05-30 13:59:05 +00:00
vfs_export.c Remove the thread argument from the FSD (File-System Dependent) parts of 2009-05-11 15:33:26 +00:00
vfs_extattr.c Remove the thread argument from the FSD (File-System Dependent) parts of 2009-05-11 15:33:26 +00:00
vfs_hash.c In keeping with style(9)'s recommendations on macros, use a ';' 2008-03-16 10:58:09 +00:00
vfs_init.c Expand the scope of the sysctllock sx lock to protect the sysctl tree itself. 2009-02-06 14:51:32 +00:00
vfs_lookup.c Let vfs_lookup() return ENOTDIR if the path has a trailing slash and 2009-05-29 10:02:44 +00:00
vfs_mount.c sys/boot/common.c 2009-06-01 01:02:30 +00:00
vfs_subr.c Remove the now invalid (and possibly unused) debug.mpsafevfs 2009-05-30 23:52:23 +00:00
vfs_syscalls.c Add hierarchical jails. A jail may further virtualize its environment 2009-05-27 14:11:23 +00:00
vfs_vnops.c Remove the thread argument from the FSD (File-System Dependent) parts of 2009-05-11 15:33:26 +00:00
vnode_if.src Add VOP_ACCESSX, which can be used to query for newly added V* 2009-05-30 13:59:05 +00:00