Setting so_snd.sb_lowat to at least 1/8 of the socket buffer size allows
send thread more actively use PDUs coalescing, that dramatically reduces
TCP lock congestion and number of context switches, when the socket is
full and PDUs are small.
MFC after: 1 week
Sponsored by: iXsystems, Inc.
r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are
still not MPSAFE (or already are but aren’t properly marked). Use it in
preparation for a general review of all nodes.
This is non-functional change that adds annotations to SYSCTL_NODE and
SYSCTL_PROC nodes using one of the soon-to-be-required flags.
Reviewed by: kib, trasz
Approved by: kib (mentor)
Differential Revision: https://reviews.freebsd.org/D23640
rename the source to gsb_crc32.c.
This is a prerequisite of unifying kernel zlib instances.
PR: 229763
Submitted by: Yoshihiro Ota <ota at j.email.ne.jp>
Differential Revision: https://reviews.freebsd.org/D20193
This allows replacing "sys/eventfilter.h" includes with "sys/_eventfilter.h"
in other header files (e.g., sys/{bus,conf,cpu}.h) and reduces header
pollution substantially.
EVENTHANDLER_DECLARE and EVENTHANDLER_LIST_DECLAREs were moved out of .c
files into appropriate headers (e.g., sys/proc.h, powernv/opal.h).
As a side effect of reduced header pollution, many .c files and headers no
longer contain needed definitions. The remainder of the patch addresses
adding appropriate includes to fix those files.
LOCK_DEBUG and LOCK_FILE_LINE_ARG are moved to sys/_lock.h, as required by
sys/mutex.h since r326106 (but silently protected by header pollution prior
to this change).
No functional change (intended). Of course, any out of tree modules that
relied on header pollution for sys/eventhandler.h, sys/lock.h, or
sys/mutex.h inclusion need to be fixed. __FreeBSD_version has been bumped.
this check on open, but "iscsictl -M", or an iSCSI redirect received by
iscsid(8) could end up with two sessions with the same target name and
portal.
MFC after: 2 weeks
Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.
The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.
can be used prior to the ISCSIDHANDOFF IOCTL which set the negotiated values.
Else the login PDU will fail when passing the "-r" option to "iscsictl" which
means iSCSI over RDMA instead of TCP/IP.
Discussed with: np@ and trasz@
Sponsored by: Mellanox Technologies
MFC after: 1 week
CAM_REQ_ABORTED sounds natural for aborting outstanding requests when
tearing down a session, but that status actually causes eligible
requests to be tried again. That's completely useless, so let's use
CAM_DEV_NOT_THERE instead. Perhaps there is a better status, but this
should be good enough. The change should affect only the session
termination.
Tested by: Ben RUBSON <ben.rubson@gmail.com>
Reviewed by: mav, trasz
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D12653
It should be sufficient to hold the lock just for removing the session
from the session list. Everything else should be covered by the session
specific lock.
On top of that, at present we can get a deadlock caused by waiting on
the CAM SIM reference count while holding the global lock. A specific
scenario involving ZFS is this:
- concurrent termination of two sessions, S1 and S2
- session S1 completed all I/Os and sleeps in CAM waiting for device
close by ZFS;
- session S2 is also dead now, but can not forcefully complete
outstanding requests by calling iscsi_session_cleanup() from
iscsi_maintenance_thread_terminate(), since it can't get the same
global sc_lock;
- as soon as there are unfinished requests, ZFS can not do
spa_config_enter() as writer, and so can not close the device for
session S1;
- deadlock.
Reported by: Ben RUBSON <ben.rubson@gmail.com>
Tested by: Ben RUBSON <ben.rubson@gmail.com>
Reviewed by: mav, trasz
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D12652
o Separate fields of struct socket that belong to listening from
fields that belong to normal dataflow, and unionize them. This
shrinks the structure a bit.
- Take out selinfo's from the socket buffers into the socket. The
first reason is to support braindamaged scenario when a socket is
added to kevent(2) and then listen(2) is cast on it. The second
reason is that there is future plan to make socket buffers pluggable,
so that for a dataflow socket a socket buffer can be changed, and
in this case we also want to keep same selinfos through the lifetime
of a socket.
- Remove struct struct so_accf. Since now listening stuff no longer
affects struct socket size, just move its fields into listening part
of the union.
- Provide sol_upcall field and enforce that so_upcall_set() may be called
only on a dataflow socket, which has buffers, and for listening sockets
provide solisten_upcall_set().
o Remove ACCEPT_LOCK() global.
- Add a mutex to socket, to be used instead of socket buffer lock to lock
fields of struct socket that don't belong to a socket buffer.
- Allow to acquire two socket locks, but the first one must belong to a
listening socket.
- Make soref()/sorele() to use atomic(9). This allows in some situations
to do soref() without owning socket lock. There is place for improvement
here, it is possible to make sorele() also to lock optionally.
- Most protocols aren't touched by this change, except UNIX local sockets.
See below for more information.
o Reduce copy-and-paste in kernel modules that accept connections from
listening sockets: provide function solisten_dequeue(), and use it in
the following modules: ctl(4), iscsi(4), ng_btsocket(4), ng_ksocket(4),
infiniband, rpc.
o UNIX local sockets.
- Removal of ACCEPT_LOCK() global uncovered several races in the UNIX
local sockets. Most races exist around spawning a new socket, when we
are connecting to a local listening socket. To cover them, we need to
hold locks on both PCBs when spawning a third one. This means holding
them across sonewconn(). This creates a LOR between pcb locks and
unp_list_lock.
- To fix the new LOR, abandon the global unp_list_lock in favor of global
unp_link_lock. Indeed, separating these two locks didn't provide us any
extra parralelism in the UNIX sockets.
- Now call into uipc_attach() may happen with unp_link_lock hold if, we
are accepting, or without unp_link_lock in case if we are just creating
a socket.
- Another problem in UNIX sockets is that uipc_close() basicly did nothing
for a listening socket. The vnode remained opened for connections. This
is fixed by removing vnode in uipc_close(). Maybe the right way would be
to do it for all sockets (not only listening), simply move the vnode
teardown from uipc_detach() to uipc_close()?
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D9770
Renumber cluase 4 to 3, per what everybody else did when BSD granted
them permission to remove clause 3. My insistance on keeping the same
numbering for legal reasons is too pedantic, so give up on that point.
Submitted by: Jan Schaumann <jschauma@stevens.edu>
Pull Request: https://github.com/freebsd/freebsd/pull/96
I found that at least with Chelsio NICs TOE sockets quite often report
negative sbspace() values. Using unsigned variable to store it resulted
in attempts to aggregate too much data in one sosend() call, that caused
errors and following connection termination.
MFC after: 2 weeks
All this code is based on assumption that data will be stored in one piece,
and since buffer size if known and fixed, it is easier to hardcode it.
MFC after: 2 weeks
In general case m_pullup() does not really guarantee any data alignment.
Instead of depenting on side effects caused by data being always copied
out of mbuf cluster (which is probably a bug by itself), always allocate
aligned BHS buffer and read data there directly from socket.
While there, reuse new icl_conn_receive_buf() function to read digests.
The code could probably be even more optimized to aggregate those reads,
but until that done, this is still easier then the way it was before.
MFC after: 2 weeks
ip_data_mbuf is always appended to ip_bhs_mbuf, so it does not need own
packet header. This change first avoids allocation/initialization of the
header, and then avoids dropping one when it later gets to socket buffer.
MFC after: 2 weeks
Decouple the send and receive limits on the amount of data in a single
iSCSI PDU. MaxRecvDataSegmentLength is declarative, not negotiated, and
is direction-specific so there is no reason for both ends to limit
themselves to the same min(initiator, target) value in both directions.
Allow iSCSI drivers to report their send, receive, first burst, and max
burst limits explicitly instead of using hardcoded values or trying to
derive all of them from the receive limit (which was the only limit
reported by the drivers prior to this change).
Display the send and receive limits separately in the userspace iSCSI
utilities.
Reviewed by: jpaetzel@ (earlier version), trasz@
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D7279
discovery without attaching to the targets ("iscsictl -Ad ... -e off"),
and then attach to selected ones ("iscsictl -Mi ... -e on").
PR: 204129
MFC after: 1 month
Relnotes: yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D6633
method. This is required for upcoming iSER support.
Obtained from: Mellanox Technologies (earlier version)
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Ensure that all iSCSI sessions are correctly terminated during shutdown.
* Enhances the changes done by r286226 (D3052).
* Add shutdown post sync event to run after filesystem shutdown
(SHUTDOWN_PRI_FIRST) but before CAM shutdown (SHUTDOWN_PRI_DEFAULT).
* Changes iscsi_maintenance_thread to processes terminate in preference to
reconnect.
Reviewed by: trasz
MFC after: 2 weeks
Sponsored by: Multiplay
Differential Revision: https://reviews.freebsd.org/D4429
r291227:
s/is->is_conn/ic to shorten things a bit.
r291228:
Do not generate PDUs with payload greater than max_data_segment_length.
It is perhaps preferable to have a separate limit for send instead of
reusing the receive limit. I'll discuss with trasz@ and mav@ before
pulling this into head.
r292618:
Add comment to go with r291228.
iscsi's shutdown_pre_sync prio was SHUTDOWN_PRI_FIRST which caused it to
run before other high priority handlers such as filesystems e.g. ZFS.
This meant the iscsi sessions where removed before the ZFS geom consumer
was closed, resulting in a panic from g_access calls on debug kernels
due to negative acr.
Instead use the same as the old iscsi_initiator SHUTDOWN_PRI_DEFAULT-1
which allows it to run before dashutdown etc but after filesystems.
MFC after: 2 weeks
Sponsored by: Multiplay
It is perhaps preferable to have a separate limit for send instead of
reusing the receive limit. I'll discuss with trasz@ and mav@ before
pulling this into head.
hangs on shutdown with LUNs with mounted filesystems over a disconnected
iSCSI session.
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3052
initiator iSCSI offload. Pass maximum data segment size supported by
chosen offload module to iscsid(8), and make iscsid(8) not try to negotiate
anything larger than that.
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
While we don't support MCS, hole in received sequence numbers may mean
only PDU loss. While we don't support lost PDU recovery, terminate the
connection to avoid stuck commands.
While there, improve handling of sequence numbers wrap after 2^32 PDUs.
MFC after: 2 weeks
establishing connection.
This is a workaround for Chelsio TOE driver, that does not update socket
buffer size in hardware after connection established, and unless that is
done beforehand, kernel code will stuck, attempting to send/receive full
PDU at once.
MFC after: 1 week
I've missed that iscsi_outstanding_remove() frees the second pointer,
so it should no longer be used. And in fact we don't really need to.
MFC after: 2 weeks
During heavy reads data copying in icl_pdu_get_data() may consume large
percent of CPU time. Moving it out of the lock significantly reduces
lock hold time and respectively lock congestion on read operations.
MFC after: 2 weeks
sb_cc member of struct sockbuf to a couple of inline functions:
sbavail() and sbused()
Right now they are equal, but once notion of "not ready socket buffer data",
will be checked in, they are going to be different.
Sponsored by: Netflix
Sponsored by: Nginx, Inc.
It allows to push out some final data from the send queue to the socket
before its close. In particular, it increases chances for logout response
to be delivered to the initiator.
in r263741. At least with CTL (slightly modified to report SPC2) there
is still some problem: it doesn't seem to find LUNs higher than 7.
Sponsored by: The FreeBSD Foundation
Previously ISID was changed every time, that made impossible correct
persistent reservation, because reconnected session was identified as
completely new one.
Reviewed by: trasz
MFC after: 1 week
These changes prevent sysctl(8) from returning proper output,
such as:
1) no output from sysctl(8)
2) erroneously returning ENOMEM with tools like truss(1)
or uname(1)
truss: can not get etype: Cannot allocate memory
there is an environment variable which shall initialize the SYSCTL
during early boot. This works for all SYSCTL types both statically and
dynamically created ones, except for the SYSCTL NODE type and SYSCTLs
which belong to VNETs. A new flag, CTLFLAG_NOFETCH, has been added to
be used in the case a tunable sysctl has a custom initialisation
function allowing the sysctl to still be marked as a tunable. The
kernel SYSCTL API is mostly the same, with a few exceptions for some
special operations like iterating childrens of a static/extern SYSCTL
node. This operation should probably be made into a factored out
common macro, hence some device drivers use this. The reason for
changing the SYSCTL API was the need for a SYSCTL parent OID pointer
and not only the SYSCTL parent OID list pointer in order to quickly
generate the sysctl path. The motivation behind this patch is to avoid
parameter loading cludges inside the OFED driver subsystem. Instead of
adding special code to the OFED driver subsystem to post-load tunables
into dynamically created sysctls, we generalize this in the kernel.
Other changes:
- Corrected a possibly incorrect sysctl name from "hw.cbb.intr_mask"
to "hw.pcic.intr_mask".
- Removed redundant TUNABLE statements throughout the kernel.
- Some minor code rewrites in connection to removing not needed
TUNABLE statements.
- Added a missing SYSCTL_DECL().
- Wrapped two very long lines.
- Avoid malloc()/free() inside sysctl string handling, in case it is
called to initialize a sysctl from a tunable, hence malloc()/free() is
not ready when sysctls from the sysctl dataset are registered.
- Bumped FreeBSD version to indicate SYSCTL API change.
MFC after: 2 weeks
Sponsored by: Mellanox Technologies
and normal mode; this makes it possible to compile with the former
by default, but use it only when neccessary. That's especially
important for the userland part.
Sponsored by: The FreeBSD Foundation
