crypto_unregister_all already disables new sessions and waits for
existing sessions to be destroyed before returning.
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D41581
(cherry picked from commit 79aeecc89f5c740f2a3b0c30238233d2a6bb7ec7)
Now armv8_crypto is using FPU_KERN_NOCTX, this results in a kernel panic
in armv8_crypto.c:armv8_crypto_cipher_setup:
panic: recursive fpu_kern_enter while in PCB_FP_NOSAVE state
This is because in armv8_crypto.c:armv8_crypto_cipher_process,
directly after calling fpu_kern_enter() a call is made to
armv8_crypto_cipher_setup(), resulting in nested calls to
fpu_kern_enter() without the required fpu_kern_leave() in between.
Move fpu_kern_enter() in armv8_crypto_cipher_process() after the
call to armv8_crypto_cipher_setup() to resolve this.
Reviewed by: markj, andrew
Fixes: 6485286f536f ("armv8_crypto: Switch to using FPU_KERN_NOCTX")
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D41671
(cherry picked from commit 565c887a775284bfb1a15eadee2c3f312d288c01)
crypto_unregister_all already disables new sessions and waits for
existing sessions to be destroyed before returning.
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D41579
(cherry picked from commit fdd51760c278302773edbd2466d35b35b865f5d9)
After commit 937b4473be21 aesni_cipher_crypt() and aesni_cipher_mac()
execute in a FPU_KERN_NOCTX section, which means that they must run with
preemption disabled. These functions handle discontiguous I/O buffers
by allocating a contiguous buffer and copying as necessary, but this
allocation cannot happen with preemption disabled. Fix the problem by
pushing the FPU section down into aesni_cipher_crypt() and
aesni_cipher_mac(). In particular, encrypt-then-auth transforms need
not be handled with a single FPU section.
Reported by: syzbot+78258dbb02eb92157357@syzkaller.appspotmail.com
Discussed with: jhb
Fixes: 937b4473be21 ("aesni: Switch to using FPU_KERN_NOCTX.")
(cherry picked from commit 6b635c74fd4135eaae68970bfc5aad9ae905fec7)
gcm_*_aesni() are used when the AVX512 implementation is not available.
Fix two bugs which manifest when handling operations spanning multiple
segments:
- Avoid underflow when the length of the input is smaller than the
residual.
- In gcm_decrypt_aesni(), ensure that we begin the operation at the
right offset into the input and output buffers.
Reviewed by: jhb
Fixes: 9b1d87286c ("ossl: Add a fallback AES-GCM implementation using AES-NI")
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D42838
(cherry picked from commit 47d767dab54895f3ba8abac6ab2295797394659e)
ossl(4)'s AES-GCM implementation keeps mutable state in the session
structure, together with the key schedule. This was done for
convenience, as both are initialized together. However, some OCF
consumers, particularly ZFS, assume that requests may be dispatched to
the same session in parallel. Without serialization, this results in
incorrect output.
Fix the problem by explicitly copying per-session state onto the stack
at the beginning of each operation.
PR: 275306
Reviewed by: jhb
Fixes: 9a3444d91c ("ossl: Add a VAES-based AES-GCM implementation for amd64")
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D42783
(cherry picked from commit 5c0dac0b7a012f326edab06ad85aee5ad68ff120)
Consumers may optionally provide a reference to a separate buffer
containing AAD, but ossl_aes_gcm() didn't handle this and would thus
compute an incorrect digest.
Fixes: 9a3444d91c ("ossl: Add a VAES-based AES-GCM implementation for amd64")
Reviewed by: jhb
MFC after: 3 days
Sponsored by: Klara, Inc.
Sponsored by: Stormshield
Differential Revision: https://reviews.freebsd.org/D42736
(cherry picked from commit 87826c87c63b2995cb69e9c1a624c7e20dd70fcd)
OpenSSL 3.0.12 addresses:
* Fix incorrect key and IV resizing issues when calling
EVP_EncryptInit_ex2(), EVP_DecryptInit_ex2() or EVP_CipherInit_ex2()
with OSSL_PARAM parameters that alter the key or IV length
([CVE-2023-5363]).
Relnotes: Yes
Sponsored by: The FreeBSD Foundation
(cherry picked from commit ad991e4c142ebabad7aef488ad97b189ecabb270)
(cherry picked from commit 575878a533823aa3e5bab715928d9cdffbc4dcbc)
There is a bug in the OpenSSL script which generates this file; the bug
is in the process of being fixed upstream.
Specifically, when generating the output, bsaes-armv7.pl strips some
labels that are used when the output asm is compiled with __KERNEL__
defined, resulting in a build error. As a step towards adding armv7
support to ossl(4), manually patch the generated asm. The upstream fix
will be imported later.
Reviewed by: andrew, jhb, emaste
MFC after: 1 week
Sponsored by: Klara, Inc.
Sponsored by: Stormshield
Differential Revision: https://reviews.freebsd.org/D41303
Chacha20+Poly1305 doesn't use an ossl_cipher instance the way AES-GCM
does, so ossl_lookup_cipher() failed causing ossl_newsession() to
always fail for Chacha20+Poly1305 sessions.
Reported by: gallatin (ktls_test fails with ossl.ko loaded)
Fixes: 9a3444d91c ossl: Add a VAES-based AES-GCM implementation for amd64
Tested by: gallatin
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D40580
This is required on i386. The patch has no functional change, since
AES-GCM isn't implemented for that platform.
Fixes: 9a3444d91c ("ossl: Add a VAES-based AES-GCM implementation for amd64")
Reported by: Jenkins
This lets one use ossl(4) for AES-GCM operations on contemporary amd64
platforms. A kernel benchmark indicates that this gives roughly
equivalent throughput to aesni(4) for various buffer sizes.
Bulk processing is done in aesni-gcm-x86_64.S, the rest is handled in a
C wrapper ported from OpenSSL's gcm128.c.
Sponsored by: Stormshield
Sponsored by: Klara, Inc.
Reviewed by: jhb
MFC after: 3 months
Differential Revision: https://reviews.freebsd.org/D39967
aes-gcm-avx512.S is generated from OpenSSL 3.1 and implements AES-GCM.
ossl_x86.c detects whether the CPU implements the required AVX512
instructions; if not, the ossl(4) module does not provide an AES-GCM
implementation. The VAES implementation increases throughput for all
buffer sizes in both directions, up to 2x for sufficiently large
buffers.
The "process" implementation is in two parts: a generic OCF layer in
ossl_aes.c that calls a set of MD functions to do the heavy lifting.
The intent there is to make it possible to add other implementations for
other platforms, e.g., to reduce the diff required for D37421.
A follow-up commit will add a fallback path to legacy AES-NI, so that
ossl(4) can be used in preference to aesni(4) on all amd64 platforms.
In the long term we would like to replace aesni(4) and armv8crypto(4)
with ossl(4).
Note, currently this implementation will not be selected by default
since aesni(4) and ossl(4) return the same probe priority for crypto
sessions, and the opencrypto framework selects the first registered
implementation to break a tie. Since aesni(4) is compiled into the
kernel, aesni(4) wins. A separate change may modify ossl(4) to have
priority.
Sponsored by: Stormshield
Sponsored by: Klara, Inc.
Reviewed by: jhb
MFC after: 3 months
Differential Revision: https://reviews.freebsd.org/D39783
This is needed to let OpenSSL 3.1 routines detect VAES and VPCLMULQDQ
extensions. The intent is to import ASM routines which implement
AES-GCM using VEX-prefixed AES-NI instructions.
No functional change intended.
Sponsored by: Stormshield
Sponsored by: Klara, Inc.
MFC after: 3 months
Differential Revision: https://reviews.freebsd.org/D39782
The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.
Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix
All the other functions used pointers for from/to instead of
fixed-size array parameters. More importantly, this function can
accept pointers to buffers of multiple blocks, not just a single
block.
Reported by: GCC -Warray-parameter
Reviewed by: imp, emaste
Differential Revision: https://reviews.freebsd.org/D37547
OpenZFS changed the define _OPENSOLARIS_SYS_TYPES_H_ to
_SPL_SYS_TYPES_H_ to guard the sys/types.h compatibility file
inclusion. Follow the change here. The only place in the tree
_OPENSOLARIS_SYS_TYPES_H_ is mentioned is in the /*
_OPENSOLARIS_SYS_TYPES_H_ */ at the end of sys/types.h. That needs to be
changed upstream in OpenZFS since we don't like changing things in
FreeBSD's tree.
Sponsored by: Netflix
Reviewed by: tsoome, delphij
Differential Revision: https://reviews.freebsd.org/D35891
All supported compilers (modern versions of GCC and clang) support
this.
Many places didn't have an #else so would just silently do the wrong
thing. Ancient versions of icc (the original motivation for this) are
no longer a compiler FreeBSD supports.
PR: 263102 (exp-run)
Reviewed by: brooks, imp
Differential Revision: https://reviews.freebsd.org/D34797
Currently armv8crypto copies the scheme used in aesni(9), where payload
data and output buffers are allocated on the fly if the crypto buffer is
not virtually contiguous. This scheme is simple but incurs a lot of
overhead: for an encryption request with a separate output buffer we
have to
- allocate a temporary buffer to hold the payload
- copy input data into the buffer
- copy the encrypted payload to the output buffer
- zero the temporary buffer before freeing it
We have a handy crypto buffer cursor abstraction now, so reimplement the
armv8crypto routines using that instead of temporary buffers. This
introduces some extra complexity, but gallatin@ reports a 10% throughput
improvement with a KTLS workload without additional CPU usage. The
driver still allocates an AAD buffer for AES-GCM if necessary.
Reviewed by: jhb
Tested by: gallatin
Sponsored by: Ampere Computing LLC
Submitted by: Klara Inc.
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D28950
This is in preparation for using buffer cursors. No functional change
intended.
Reviewed by: jhb
Sponsored by: Ampere Computing LLC
Submitted by: Klara Inc.
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D28948
This adds a wrapper around libsodium's curve25519 support matching
Linux's curve25519 API. The intended use case for this is WireGuard.
Note that this is not integrated with OCF as it is not related to
symmetric operations on data.
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33935
This is a synchronous software API which wraps the existing software
implementation shared with OCF. Note that this will not currently
use optimized backends (such as ossl(4)) but may be appropriate for
operations on small buffers.
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33524
These callbacks allow multiple contiguous blocks to be manipulated in
a single call. Note that any trailing partial block for a stream
cipher must still be passed to encrypt/decrypt_last.
While here, document the setkey and reinit hooks and reorder the hooks
in 'struct enc_xform' to better reflect the life cycle.
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33529
This is the same as 4285655adb ("aesni: Avoid a potential
out-of-bounds load in AES_GCM_encrypt()") except for the decryption
path.
Reported by: Jenkins (KASAN job)
Reviewed by: cem
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33474
