Instead of collecting statistics for each combination of ports and logical
units, that consumed ~45KB per LU with present number of ports, collect
separate statistics for every port and every logical unit separately, that
consume only 176 bytes per each single LU/port. This reduces struct
ctl_lun size down to just 6KB.
Also new IOCTL API/ABI does not hardcode number of LUs/ports, and should
allow handling of very large quantities.
MFC after: 2 weeks (probably keeping old API enabled for some time)
This array takes 64KB of RAM now, that was more then half of struct ctl_lun
size. If at some point we support more ports, this may need another tune.
MFC after: 2 weeks
The sim_vid, hba_vid, and dev_name fields of struct ccb_pathinq are
fixed-length strings. AFAICT the only place they're read is in
sbin/camcontrol/camcontrol.c, which assumes they'll be null-terminated.
However, the kernel doesn't null-terminate them. A bunch of copy-pasted code
uses strncpy to write them, and doesn't guarantee null-termination. For at
least 4 drivers (mpr, mps, ciss, and hyperv), the hba_vid field actually
overflows. You can see the result by doing "camcontrol negotiate da0 -v".
This change null-terminates those fields everywhere they're set in the
kernel. It also shortens a few strings to ensure they'll fit within the
16-character field.
PR: 215474
Reported by: Coverity
CID: 1009997 1010000 1010001 1010002 1010003 1010004 1010005
CID: 1331519 1010006 1215097 1010007 1288967 1010008 1306000
CID: 1211924 1010009 1010010 1010011 1010012 1010013 1010014
CID: 1147190 1010017 1010016 1010018 1216435 1010020 1010021
CID: 1010022 1009666 1018185 1010023 1010025 1010026 1010027
CID: 1010028 1010029 1010030 1010031 1010033 1018186 1018187
CID: 1010035 1010036 1010042 1010041 1010040 1010039
Reviewed by: imp, sephe, slm
MFC after: 4 weeks
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D9037
Differential Revision: https://reviews.freebsd.org/D9038
observed to fix any actual error, but it's the right thing to do
from the correctness point of view.
Tested by: Eugene M. Zheganin <emz at norma.perm.ru>
MFC after: 1 month
- Since I/Os are allocates from per-port pools, make allocations store
pointer to CTL softc there, and use it where needed instead of global.
- Created bunch of helper macros to access LUN, port and CTL softc.
MFC after: 2 weeks
We do not have non-volatile memory to really save those values, so we
neither report nor support this capability. Also saved mode pages are
not replicated between HA peers now.
MFC after: 2 weeks
Those two values are not directly related, so make them independent.
This does not change any limits immediately, but makes number of LUNs
per port controllable via tunable/sysctl kern.cam.ctl.lun_map_size.
After this change increasing CTL_MAX_LUNS should be pretty cheap,
and even making it tunable should be easy.
MFC after: 2 weeks
For EXTENDED COPY:
- improve parameters checking to report some errors before copy start;
- forward sense data from copy target as descriptor in case of error;
- report which CSCD reported error in sense key specific information.
For WRITE USING TOKEN:
- pass through real sense data from copy target instead of reporting
our copy error, since for initiator its a "simple" write, not a copy.
MFC after: 2 weeks
- Allow maximal sense size limitation via Control Extension mode page.
- When sense size limited, include descriptors atomically: whole or none.
- Set new SDAT_OVFL bit if some descriptors don't fit the limit.
- Report real written sense length instead of static maximal 252 bytes.
MFC after: 2 weeks
VMware tries to enable this bit to avoid multiple threshold notifications
in case of multiple initiators connected to the same LUN. Unfortunately
their code sends MODE SELECT(6) request with parameter length hardcoded
for the page without any thresholds. Since we have four threshold and our
page is bigger, this attempt fails, that is correct in my understanding.
So all we can do about this now is to report proper error code and hope
VMware fix their code one day.
MFC after: 2 weeks
While CTL still has no real events to report in this way (like SMART),
it is possible to trigger false event by manually setting TEST bit in
Informational Exceptions Control mode page, that can be useful for
initiator testing. This code supports all flavours of IE reporting:
UNIT ATTENTION, RECOVERED ERROR and NO SENSE sense keys, REQUEST SENSE
command and Informational Exceptions log page.
MFC after: 2 weeks
Sponsored by: iXsystems, Inc.
be accessed by root. It uses unsigned ints instead of size_t
to preserve the ABI.
PR: 207627
Submitted by: ryan@ryanday.net (with slight tweaks)
MFC after: 1 month
CTL itself has no limits on on UNMAP and WRITE SAME sizes. But depending
on backends large requests may take too much time. To avoid that new
configuration options allow to hint initiator maximal sizes it should not
exceed.
MFC after: 2 weeks
Decouple the send and receive limits on the amount of data in a single
iSCSI PDU. MaxRecvDataSegmentLength is declarative, not negotiated, and
is direction-specific so there is no reason for both ends to limit
themselves to the same min(initiator, target) value in both directions.
Allow iSCSI drivers to report their send, receive, first burst, and max
burst limits explicitly instead of using hardcoded values or trying to
derive all of them from the receive limit (which was the only limit
reported by the drivers prior to this change).
Display the send and receive limits separately in the userspace iSCSI
utilities.
Reviewed by: jpaetzel@ (earlier version), trasz@
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D7279
utilizing previously unused arg field of struct ccb_notify_acknowledge.
This makes new QUERY TASK, QUERY TASK SET and QUERY ASYNC EVENT requests
really functional for CAM target mode drivers.
- Use SDT_PROBE<N>() instead of SDT_PROBE(). This has no functional effect
at the moment, but will be needed for some future changes.
- Don't hardcode the module component of the probe identifier. This is
set automatically by the SDT framework.
MFC after: 1 week
- Introduce "ha_shared" port option, which being set to "on" moves the
port into separate port group, shared between HA nodes. This allows to
better handle cases when iSCSI portals are bound to CARP address that can
dynamically move between nodes. Some initiators (at least VMware) don't
detect that after iSCSI reconnect they've attached to different SCSI port
from different port group, that totally breakes ALUA status parsing.
In theory, I believe, it should be enough to have different iSCSI portal
group tags on different nodes to make initiators detect this condition,
but it seems like VMware ignores those values, and even full LUN retaste
forced by UA does not help.
- Make CTL report up to three port groups: 1 -- non-HA mode or ports
with "ha_shared" option set, 2 -- HA node 1, 3 -- HA node 2.
- Report Transitioning state for all port groups when HA interlink is
connected, but neither of nodes is primary for the LUN.
MFC after: 2 weeks
This change allows to decode respective functions in isp(4) in target mode
and pass them through CAM to CTL. Unfortunately neither CAM nor isp(4)
support returning response info for those task management functions now.
On the other side I just have no initiator to test this functionality.
I am not sure what for it was done. Now open routine should automatically
fall back to read-only if open for writing is impossible. In such case
attempt to upgrade to write sounds strange.
MFC after: 1 week
We allow to modify only few fields in mode pages now, but still it is
not good if they unexpectedly change during failover. Also this fixes
reporting of "Mode parameters changed" UAs on secondary node.
HA protocol requires strict version, parameters and configuration match.
Differences there may cause full set of problems up to kernel panic.
To avoid that, validate peer parameters on connect, and abort connection
immediately if some mismatch detected.
REPORT LUNS command is more related to target rather then specific LUN.
This node may be primary for LUNs for some reason unknown to another,
and command forwarded to another node won't be able to report them.
REQUEST SENSE is related to LUN, but in our implementation it reports
only UAs and CAs, that are stored locally rather then on primary node.
Previously, with serseq enabled, next command was unblocked only after
previous completed. With this change, for read operations, next command
is unblocked as soon as last media read completed. This is important
for frontends that actually wait for data move completion (like camtgt),
or when data are moved through the HA link, or especially when both.
All requests arriving for processing after OFFLINE flag set are rejected
with BUSY status. Races around OFFLINE flag setting are closed by calling
taskqueue_drain_all().
CTL HA functionality was originally implemented by Copan many years ago,
but large part of the sources was never published. This change includes
clean room implementation of the missing code and fixes for many bugs.
This code supports dual-node HA with ALUA in four modes:
- Active/Unavailable without interlink between nodes;
- Active/Standby with second node handling only basic LUN discovery and
reservation, synchronizing with the first node through the interlink;
- Active/Active with both nodes processing commands and accessing the
backing storage, synchronizing with the first node through the interlink;
- Active/Active with second node working as proxy, transfering all
commands to the first node for execution through the interlink.
Unlike original Copan's implementation, depending on specific hardware,
this code uses simple custom TCP-based protocol for interlink. It has
no authentication, so it should never be enabled on public interfaces.
The code may still need some polishing, but generally it is functional.
Relnotes: yes
Sponsored by: iXsystems, Inc.
