Chacha20+Poly1305 doesn't use an ossl_cipher instance the way AES-GCM
does, so ossl_lookup_cipher() failed causing ossl_newsession() to
always fail for Chacha20+Poly1305 sessions.
Reported by: gallatin (ktls_test fails with ossl.ko loaded)
Fixes: 9a3444d91c ossl: Add a VAES-based AES-GCM implementation for amd64
Tested by: gallatin
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D40580
This is required on i386. The patch has no functional change, since
AES-GCM isn't implemented for that platform.
Fixes: 9a3444d91c ("ossl: Add a VAES-based AES-GCM implementation for amd64")
Reported by: Jenkins
This lets one use ossl(4) for AES-GCM operations on contemporary amd64
platforms. A kernel benchmark indicates that this gives roughly
equivalent throughput to aesni(4) for various buffer sizes.
Bulk processing is done in aesni-gcm-x86_64.S, the rest is handled in a
C wrapper ported from OpenSSL's gcm128.c.
Sponsored by: Stormshield
Sponsored by: Klara, Inc.
Reviewed by: jhb
MFC after: 3 months
Differential Revision: https://reviews.freebsd.org/D39967
aes-gcm-avx512.S is generated from OpenSSL 3.1 and implements AES-GCM.
ossl_x86.c detects whether the CPU implements the required AVX512
instructions; if not, the ossl(4) module does not provide an AES-GCM
implementation. The VAES implementation increases throughput for all
buffer sizes in both directions, up to 2x for sufficiently large
buffers.
The "process" implementation is in two parts: a generic OCF layer in
ossl_aes.c that calls a set of MD functions to do the heavy lifting.
The intent there is to make it possible to add other implementations for
other platforms, e.g., to reduce the diff required for D37421.
A follow-up commit will add a fallback path to legacy AES-NI, so that
ossl(4) can be used in preference to aesni(4) on all amd64 platforms.
In the long term we would like to replace aesni(4) and armv8crypto(4)
with ossl(4).
Note, currently this implementation will not be selected by default
since aesni(4) and ossl(4) return the same probe priority for crypto
sessions, and the opencrypto framework selects the first registered
implementation to break a tie. Since aesni(4) is compiled into the
kernel, aesni(4) wins. A separate change may modify ossl(4) to have
priority.
Sponsored by: Stormshield
Sponsored by: Klara, Inc.
Reviewed by: jhb
MFC after: 3 months
Differential Revision: https://reviews.freebsd.org/D39783
This is needed to let OpenSSL 3.1 routines detect VAES and VPCLMULQDQ
extensions. The intent is to import ASM routines which implement
AES-GCM using VEX-prefixed AES-NI instructions.
No functional change intended.
Sponsored by: Stormshield
Sponsored by: Klara, Inc.
MFC after: 3 months
Differential Revision: https://reviews.freebsd.org/D39782
The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.
Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix
All the other functions used pointers for from/to instead of
fixed-size array parameters. More importantly, this function can
accept pointers to buffers of multiple blocks, not just a single
block.
Reported by: GCC -Warray-parameter
Reviewed by: imp, emaste
Differential Revision: https://reviews.freebsd.org/D37547
OpenZFS changed the define _OPENSOLARIS_SYS_TYPES_H_ to
_SPL_SYS_TYPES_H_ to guard the sys/types.h compatibility file
inclusion. Follow the change here. The only place in the tree
_OPENSOLARIS_SYS_TYPES_H_ is mentioned is in the /*
_OPENSOLARIS_SYS_TYPES_H_ */ at the end of sys/types.h. That needs to be
changed upstream in OpenZFS since we don't like changing things in
FreeBSD's tree.
Sponsored by: Netflix
Reviewed by: tsoome, delphij
Differential Revision: https://reviews.freebsd.org/D35891
All supported compilers (modern versions of GCC and clang) support
this.
Many places didn't have an #else so would just silently do the wrong
thing. Ancient versions of icc (the original motivation for this) are
no longer a compiler FreeBSD supports.
PR: 263102 (exp-run)
Reviewed by: brooks, imp
Differential Revision: https://reviews.freebsd.org/D34797
Currently armv8crypto copies the scheme used in aesni(9), where payload
data and output buffers are allocated on the fly if the crypto buffer is
not virtually contiguous. This scheme is simple but incurs a lot of
overhead: for an encryption request with a separate output buffer we
have to
- allocate a temporary buffer to hold the payload
- copy input data into the buffer
- copy the encrypted payload to the output buffer
- zero the temporary buffer before freeing it
We have a handy crypto buffer cursor abstraction now, so reimplement the
armv8crypto routines using that instead of temporary buffers. This
introduces some extra complexity, but gallatin@ reports a 10% throughput
improvement with a KTLS workload without additional CPU usage. The
driver still allocates an AAD buffer for AES-GCM if necessary.
Reviewed by: jhb
Tested by: gallatin
Sponsored by: Ampere Computing LLC
Submitted by: Klara Inc.
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D28950
This is in preparation for using buffer cursors. No functional change
intended.
Reviewed by: jhb
Sponsored by: Ampere Computing LLC
Submitted by: Klara Inc.
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D28948
This adds a wrapper around libsodium's curve25519 support matching
Linux's curve25519 API. The intended use case for this is WireGuard.
Note that this is not integrated with OCF as it is not related to
symmetric operations on data.
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33935
This is a synchronous software API which wraps the existing software
implementation shared with OCF. Note that this will not currently
use optimized backends (such as ossl(4)) but may be appropriate for
operations on small buffers.
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33524
These callbacks allow multiple contiguous blocks to be manipulated in
a single call. Note that any trailing partial block for a stream
cipher must still be passed to encrypt/decrypt_last.
While here, document the setkey and reinit hooks and reorder the hooks
in 'struct enc_xform' to better reflect the life cycle.
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33529
This is the same as 4285655adb ("aesni: Avoid a potential
out-of-bounds load in AES_GCM_encrypt()") except for the decryption
path.
Reported by: Jenkins (KASAN job)
Reviewed by: cem
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33474
As with sha256 add support for accelerated sha512 support to libmd on
arm64. This depends on clang 13+ to build as this is the first release
with the needed intrinsics. Gcc should also support them, however from
a currently unknown release.
Reviewed by: cem
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33373
Only pre-allocate auth contexts when a session-wide key is provided or
for sessions without keys. For sessions with per-operation keys,
always initialize the on-stack context directly rather than
initializing the session context in swcr_authprepare (now removed) and
then copying that session context into the on-stack context.
This approach permits parallel auth operations without needing a
serializing lock. In addition, the previous code assumed that auth
sessions always provided an initial key unlike cipher sessions which
assume either an initial key or per-op keys.
While here, fix the Blake2 auth transforms to function like other auth
transforms where Setkey is invoked after Init rather than before.
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33316
This centralizes the check for valid nonce lengths for AES-GCM.
While here, remove some duplicate checks for valid AES-GCM tag lengths
from ccp(4) and ccr(4).
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33194
GCC 9 doesn't define a SHA256_Transform symbol when the stub just wraps
SHA256_Transform_c resulting in an undefined symbol for
_libmd_SHA256_Transform in libmd.so.
Discussed with: andrew, jrtc27
Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D31945
Now that the AES-CBC is supported we can handle ETA requests.
Sponsored by: Stormshield
Obtained from: Semihalf
Reviewed by: jhb
Differential revision: https://reviews.freebsd.org/D32100
AES-CBC OpenSSL assembly is used underneath.
The glue layer(ossl_aes.c) is based on CHACHA20 implementation.
Contrary to the SHA and CHACHA20, AES OpenSSL assembly logic
does not have a fallback implementation in case CPU doesn't
support required instructions.
Because of that CPU caps are checked during initialization and AES
support is advertised only if available.
The feature is available on all architectures that ossl supports:
i386, amd64, arm64.
The biggest advantage of this patch over existing solutions
(aesni(4) and armv8crypto(4)) is that it supports SHA,
allowing for ETA operations.
Sponsored by: Stormshield
Obtained from: Semihalf
Reviewed by: jhb (previous version)
Differential revision: https://reviews.freebsd.org/D32099
Now that the AES-CBC is supported we can handle ETA requests.
Sponsored by: Stormshield
Obtained from: Semihalf
Reviewed by: jhb(previous version)
Differential revision: https://reviews.freebsd.org/D32100
AES-CBC OpenSSL assembly is used underneath.
The glue layer(ossl_aes.c) is based on CHACHA20 implementation.
Contrary to the SHA and CHACHA20, AES OpenSSL assembly logic
does not have a fallback implementation in case CPU doesn't
support required instructions.
Because of that CPU caps are checked during initialization and AES
support is advertised only if available.
The feature is available on all architectures that ossl supports:
i386, amd64, arm64.
The biggest advantage of this patch over existing solutions
(aesni(4) and armv8crypto(4)) is that it supports SHA,
allowing for ETA operations.
Sponsored by: Stormshield
Obtained from: Semihalf
Reviewed by: jhb
Differential revision: https://reviews.freebsd.org/D32099
This is useful for WireGuard which uses a nonce of 8 bytes rather
than the 12 bytes used for IPsec and TLS.
Note that this also fixes a (should be) harmless bug in ossl(4) where
the counter was incorrectly treated as a 64-bit counter instead of a
32-bit counter in terms of wrapping when using a 12 byte nonce.
However, this required a single message (TLS record) longer than 64 *
(2^32 - 1) bytes (about 256 GB) to trigger.
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32122
Add a 'len' argument to the reinit hook in 'struct enc_xform' to
permit support for AEAD ciphers such as AES-CCM and Chacha20-Poly1305
which support different nonce lengths.
Reviewed by: markj
Sponsored by: Chelsio Communications, The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32105
While here, use crypto_read_iv() in a few more places in ccr(4) that I
missed previously.
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D32104
