An overread condition in memccpy(dst, src, c, len) would occur if
src does not cross a 16 byte boundary and there is no instance of
c between *src and the next 16 byte boundary. This could cause a
read fault if src is just before the end of a page and the next page
is unmapped or unreadable.
The bug is a consequence of basing memccpy() on the strlcpy() code:
whereas strlcpy() assumes that src is a nul-terminated string and
hence a terminator is always present, c may not be present at all in
the source string. It was not caught earlier due to insufficient
unit test design.
As a part of the fix, the function is refactored such that the runt
case (buffer length from last alignment boundary between 1 and 32 B)
is handled separately. This reduces the number of conditional
branches on all code paths and simplifies the handling of early
matches in the non-runt case. Performance is improved slightly.
os: FreeBSD
arch: amd64
cpu: 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz
│ memccpy.unfixed.out │ memccpy.fixed.out │
│ sec/op │ sec/op vs base │
Short 66.76µ ± 0% 62.45µ ± 1% -6.44% (p=0.000 n=20)
Mid 7.938µ ± 0% 7.967µ ± 0% +0.36% (p=0.001 n=20)
Long 3.577µ ± 0% 3.577µ ± 0% ~ (p=0.429 n=20)
geomean 12.38µ 12.12µ -2.08%
│ memccpy.unfixed.out │ memccpy.fixed.out │
│ B/s │ B/s vs base │
Short 1.744Gi ± 0% 1.864Gi ± 1% +6.89% (p=0.000 n=20)
Mid 14.67Gi ± 0% 14.61Gi ± 0% -0.36% (p=0.001 n=20)
Long 32.55Gi ± 0% 32.55Gi ± 0% ~ (p=0.429 n=20)
geomean 9.407Gi 9.606Gi +2.12%
Reported by: getz
Reviewed by: getz
Approved by: mjg (blanket, via IRC)
See also: D46051
MFC: stable/14
Event: GSoC 2024
Differential Revision: https://reviews.freebsd.org/D46052
The code in this file runs before the sanitizer can initialize its
shadow map.
Fixes: ad2fac552c ("lib/libc/amd64: add archlevel-based simd dispatch framework")
(cherry picked from commit 4dedcb1bb54cbbe8043c79ad733f966b6ffc6972)
The scalar implementation is fairly simplistic and only performs
slightly better than the generic C implementation. It could be
improved by using the same algorithm as for memchr, but it would
have been a lot more complicated.
The baseline implementation is similar to timingsafe_memcmp. It's
slightly slower than memchr() due to the more complicated main
loop, but I don't think that can be significantly improved.
Tested by: developers@, exp-run
Approved by: mjg
MFC after: 1 month
MFC to: stable/14
PR: 275785
Differential Revision: https://reviews.freebsd.org/D42925
(cherry picked from commit fb197a4f7751bb4e116989e57ba7fb12a981895f)
Based on the strlcpy code from D42863, this patch adds a SIMD-enhanced
implementation of memccpy for amd64. A scalar implementation calling
into memchr and memcpy to do the job is provided, too.
Please note that this code does not behave exactly the same as the C
implementation of memccpy for overlapping inputs. However, overlapping
inputs are not allowed for this function by ISO/IEC 9899:1999 and neither
has the C implementation any code to deal with the possibility. It just
proceeds byte-by-byte, which may or may not do the expected thing for
some overlaps. We do not document whether overlapping inputs are
supported in memccpy(3).
Tested by: developers@, exp-run
Approved by: mjg
MFC after: 1 month
MFC to: stable/14
PR: 275785
Differential Revision: https://reviews.freebsd.org/D42902
(cherry picked from commit fc0e38a7a67a6d43095efb00cf19ee5f95dcf710)
This should pick up our optimised memchr(), strlen(), and strlcpy()
when strlcat() is called.
Tested by: developers@, exp-run
Approved by: mjg
MFC after: 1 month
MFC to: stable/14
PR: 275785
Differential Revision: https://reviews.freebsd.org/D42863
(cherry picked from commit 2b7b03b7ae179db465c1ef19a5007f729874916a)
Somewhat similar to stpncpy, but different in that we need to compute
the full source length even if the buffer is shorter than the source.
strlcat is implemented as a simple wrapper around strlcpy. The scalar
implementation of strlcpy just calls into strlen() and memcpy() to do
the job.
Perf-wise we're very close to stpncpy. The code is slightly slower as
it needs to carry on with finding the source string length even if the
buffer ends before the string.
Sponsored by: The FreeBSD Foundation
Tested by: developers@, exp-run
Approved by: mjg
MFC after: 1 month
MFC to: stable/14
PR: 275785
Differential Revision: https://reviews.freebsd.org/D42863
(cherry picked from commit 74d6cfad54d676299ee5e4695139461876dfd757)
strcat has a bespoke scalar assembly implementation we
inherited from NetBSD. While it performs well, it is
better to call into our SIMD implementations if any SIMD
features are available at all. So do that and implement
strcat() by calling into strlen() and strcpy() if these
are available.
Sponsored by: The FreeBSD Foundation
Tested by: developers@, exp-run
Approved by: mjg
MFC after: 1 month
MFC to: stable/14
PR: 275785
Differential Reviison: https://reviews.freebsd.org/D42600
(cherry picked from commit aff9143a242c0012b0195b3666e03fa3b7cd33e8)
This was surprisingly annoying to get right, despite being such a simple
function. A scalar implementation is also provided, it just calls into
our optimised memchr(), memcpy(), and memset() routines to carry out its
job.
I'm quite happy with the performance. glibc only beats us for very long
strings, likely due to the use of AVX-512. The scalar implementation
just calls into our optimised memchr(), memcpy(), and memset() routines,
so it has a high overhead to begin with but then performs ok for the
amount of effort that went into it. Still beats the old C code, except
for very short strings.
Sponsored by: The FreeBSD Foundation
Tested by: developers@, exp-run
Approved by: mjg
MFC after: 1 month
MFC to: stable/14
PR: 275785
Differential Revision: https://reviews.freebsd.org/D42519
(cherry picked from commit 90253d49db09a9b1490c448d05314f3e4bbfa468)
The strsep() function is basically strcspn() with extra steps.
On amd64, we now have an optimised implementation of strcspn(),
so instead of implementing the inner loop manually, just call
into the optimised routine.
Sponsored by: The FreeBSD Foundation
Tested by: developers@, exp-run
Approved by: mjg
MFC after: 1 month
MFC to: stable/14
PR: 275785
Differential Revision: https://reviews.freebsd.org/D42346
(cherry picked from commit fd2ecd91aeeeab579c769c9a39f90b4bd4a493a9)
The baseline implementation is very straightforward, while the scalar
implementation suffers from register pressure and the need to use SWAR
techniques similar to those used for strchr().
Sponsored by: The FreeBSD Foundation
Tested by: developers@, exp-run
Approved by: mjg
MFC after: 1 month
MFC to: stable/14
PR: 275785
Differential Revision: https://reviews.freebsd.org/D42217
(cherry picked from commit 2ed514a220edbac6ca5ec9f40a3e0b3f2804796d)
The scalar implementation is fairly straightforward and merely unrolled
four times. The baseline implementation closely follows D41971 with
appropriate extensions and extra code paths to pay attention to string
length.
Performance is quite good. We beat both glibc (except for very long
strings, but they likely use AVX which we don't) and Bionic (except for
medium-sized aligned strings, where we are still in the same ballpark).
Sponsored by: The FreeBSD Foundation
Tested by: developers@, exp-run
Approved by: mjg
MFC after: 1 month
MFC to: stable/14
PR: 275785
Differential Revision: https://reviews.freebsd.org/D42122
(cherry picked from commit 14289e973f5c941e4502cc2b11265e4b3072839a)
This lets us use our optimised strcspn() routine for strpbrk() calls.
Sponsored by: The FreeBSD Foundation
Tested by: developers@, exp-run
Approved by: mjg
MFC after: 1 month
MFC to: stable/14
PR: 275785
Differential Revision: https://reviews.freebsd.org/D41980
(cherry picked from commit f4fc317c364f2c81ad3d36763d8e5a60393ddbd1)
This is the most complicated one so far. The basic idea is to process
the bulk of the string in aligned blocks of 16 bytes such that one
string runs ahead and the other runs behind. The string that runs ahead
is checked for NUL bytes, the one that runs behind is compared with the
corresponding chunk of the string that runs ahead. This trades an extra
load per iteration for the very complicated block-reassembly needed in
the other implementations (bionic, glibc). On the flip side, we need
two code paths depending on the relative alignment of the two buffers.
The initial part of the string is compared directly if it is known not
to cross a page boundary. Otherwise, a complex slow path to avoid
crossing into unmapped memory commences.
Performance-wise we beat bionic for misaligned strings (i.e. the strings
do not share an alignment offset) and reach comparable performance for
aligned strings. glibc is a bit better as it has a special kernel for
AVX-512, where this stuff is a bit easier to do.
Sponsored by: The FreeBSD Foundation
Tested by: developers@, exp-run
Approved by: mjg
MFC after: 1 month
MFC to: stable/14
PR: 275785
Differential Revision: https://reviews.freebsd.org/D41971
(cherry picked from commit bca25680b91b3bea7faef615765806a04634eb23)
Conceptually very similar to timingsafe_bcmp(), but with comparison
logic inspired by Elijah Stone's fancy memcmp. A baseline (SSE)
implementation was omitted this time as I was not able to get it to
perform adequately. Best I got was 8% over the scalar version for
long inputs, but slower for short inputs.
Sponsored by: The FreeBSD Foundation
Approved by: security (cperciva)
Inspired by: https://github.com/moon-chilled/fancy-memcmp
Differential Revision: https://reviews.freebsd.org/D41696
(cherry picked from commit 5048c1b85506c5e0f441ee7dd98dd8d96d0a4a24)
Very straightforward and similar to memcmp(3). The code has
been written to use only instructions specified as having
data operand independent timing by Intel.
Sponsored by: The FreeBSD Foundation
Approved by: security (cperciva)
Differential Revision: https://reviews.freebsd.org/D41673
(cherry picked from commit 76c2b331bcd9f73c5c8c43a06e328fa0c7b8c39a)
When matching against a set of 17--32 characters, strcspn() uses two
invocations of PCMPISTRI to match against the first 16 characters
of the set and then the remaining characters. If a match was found in
the first half of the set, the code originally immediately returned
that match. However, it is possible for a match in the second half of
the set to occur earlier in the vector, leading to that match being
overlooked.
Fix the code by checking if there is a match in the second half of the
set and taking the earlier of the two matches.
The correctness of the function has been verified with extended unit
tests and test runs against the glibc test suite.
Approved by: mjg (implicit, via IRC)
MFC after: 1 week
MFC to: stable/14
(cherry picked from commit c91cd7d03a9dee649ba3a1b9b4014df9de111bb8)
These sys/cdefs.h are not needed. Purge them. They are mostly left-over
from the $FreeBSD$ removal. A few in libc are still required for macros
that cdefs.h defines. Keep those.
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D42385
(cherry picked from commit 559a218c9b257775fb249b67945fe4a05b7a6b9f)
When memcmp(a, b, len) (or equally, bcmp) is called with a phony length
such that a + len < a, the code would malfunction and not compare the
two buffers correctly. While such arguments are illegal (buffers do not
wrap around the end of the address space), it is neverthless conceivable
that people try things like memcmp(a, b, SIZE_MAX) to compare a and b
until the first mismatch, in the knowledge that such a mismatch exists,
expecting memcmp() to stop comparing somewhere around the mismatch.
While memcmp() is usually written to confirm to this assumption, no
version of ISO/IEC 9899 guarantees this behaviour (in contrast to
memchr() for which it is).
Neverthless it appears sensible to at least not grossly misbehave on
phony lengths. This change hardens memcmp() against this case by
comparing at least until the end of the address space if a + len
overflows a 64 bit integer.
Sponsored by: The FreeBSD Foundation
Approved by: mjg (blanket, via IRC)
See also: b2618b651b28fd29e62a4e285f5be09ea30a85d4
MFC after: 1 week
(cherry picked from commit 953b93cf24d8871c62416c9bcfca935f1f1853b6)
Now that we have an optimised memchr(3), we can use it to implement
strnlen(3) with better perofrmance.
Sponsored by: The FreeBSD Foundation
Approved by: mjg
MFC after: 1 week
MFC to: stable/14
Differential Revision: https://reviews.freebsd.org/D41598
(cherry picked from commit 331737281c1929c29e679e48783055351ac4fbd9)
This is conceptually similar to strchr(3), but there are
slight changes to account for the buffer having an explicit
buffer length.
this includes the bug fix from b2618b6.
Sponsored by: The FreeBSD Foundation
Reported by: yuri, des
Tested by: des
Approved by: mjg
MFC after: 1 week
MFC to: stable/14
PR: 273652
Differential Revision: https://reviews.freebsd.org/D41598
(cherry picked from commit de12a689fad271f5a2ba7c188b0b5fb5cabf48e7)
(cherry picked from commit b2618b651b28fd29e62a4e285f5be09ea30a85d4)
This is conceptually very similar to the strcspn(3) implementations
from D41557, but we can't do the fast paths the same way.
Sponsored by: The FreeBSD Foundation
Approved by: mjg
MFC after: 1 week
MFC to: stable/14
Differential Revision: https://reviews.freebsd.org/D41567
(cherry picked from commit 7084133cde6a58412d86bae9f8a55b86141fb304)
This changeset adds both a scalar and an x86-64-v2 implementation
of the strcspn(3) function to libc. A baseline implementation does not
appear to be feasible given the requirements of the function.
The scalar implementation is similar to the generic libc implementation,
but expands the bit set into a byte set to reduce latency, improving
performance. This approach could probably be backported to the generic
C version to benefit other platforms.
The x86-64-v2 implementation is built around the infamous pcmpistri
instruction. An alternative implementation based on the Muła/Langdale
algorithm [1] was prototyped, but performed worse than the pcmpistri
approach except for sets of more than 16 characters with long input
strings.
All implementations provide special cases for the empty set (reduces to
strlen as well as single-character sets (reduces to strchr). The
x86-64-v2 kernel falls back to the scalar implementation for sets of
more than 32 characters. This limit could be raised by additional
multiples of 16 through the use of additional pcmpistri code paths, but
I consider this case to be too rare to be of importance.
This includes the bug fix from 52d4a4d.
[1]: http://0x80.pl/articles/simd-byte-lookup.html
Sponsored by: The FreeBSD Foundation
Approved by: mjg
MFC after: 1 week
MFC to: stable/14
Differential Revision: https://reviews.freebsd.org/D41557
(cherry picked from commit 474408bb7933f0383a0da2b01e717bfe683ae77c)
(cherry picked from commit 52d4a4d4e0dedc72bc33082a3f84c2d0fd6f2cbb)
When the buffer is immediately preceeded by the character we
are looking for and begins with one higher than that character,
and the buffer is misaligned, a match was errorneously detected
in the first character. Fix this by changing the way we prevent
matches before the buffer from being detected: instead of
removing the corresponding bit from the 0x80..80 mask, set the
LSB of bytes before the buffer after xoring with the character we
look for.
The bug only affects amd64 with ARCHLEVEL=scalar (cf. simd(7)).
The change comes at a 2% performance impact for short strings
if ARCHLEVEL is set to scalar. The default configuration is not
affected.
os: FreeBSD
arch: amd64
cpu: 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz
│ strchrnul.scalar.0.out │ strchrnul.scalar.2.out │
│ sec/op │ sec/op vs base │
Short 57.89µ ± 2% 59.08µ ± 1% +2.07% (p=0.030 n=20)
Mid 19.24µ ± 0% 19.73µ ± 0% +2.53% (p=0.000 n=20)
Long 11.03µ ± 0% 11.03µ ± 0% ~ (p=0.547 n=20)
geomean 23.07µ 23.43µ +1.53%
│ strchrnul.scalar.0.out │ strchrnul.scalar.2.out │
│ B/s │ B/s vs base │
Short 2.011Gi ± 2% 1.970Gi ± 1% -2.02% (p=0.030 n=20)
Mid 6.049Gi ± 0% 5.900Gi ± 0% -2.47% (p=0.000 n=20)
Long 10.56Gi ± 0% 10.56Gi ± 0% ~ (p=0.547 n=20)
geomean 5.045Gi 4.969Gi -1.50%
MFC to: stable/14
MFC after: 3 days
Approved by: mjg (blanket, via IRC), re (gjb)
Sponsored by: The FreeBSD Foundation
(cherry picked from commit 3d8ef251aa9dceabd57f7821a0e6749d35317db3)
This changeset adds a baseline implementation of memcmp and bcmp
for amd64. The same code is used for both functions with conditional
code were the behaviour differs (we need more precise output for the
memcmp case).
FreeBSD documents that memcmp returns the difference between the
mismatching characters. Slightly faster code would be possible could
we relax this requirement to the ISO/IEC 9899:1999 requirement of
merely returning a negative/positive integer or zero.
Performance is better than bionic and glibc, except for long strings
were the two are 13% faster. This could be because they use SSE4
ptest which we cannot use in a baseline kernel.
Sponsored by: The FreeBSD Foundation
Approved by: mjg
Differential Revision: https://reviews.freebsd.org/D41442
This commit adds a baseline implementation of stpcpy(3) for amd64.
It performs quite well in comparison to the previous scalar implementation
as well as agains bionic and glibc (though glibc is faster for very long
strings). Fiddle with the Makefile to also have strcpy(3) call into the
optimised stpcpy(3) code, fixing an oversight from D9841.
Sponsored by: The FreeBSD Foundation
Reviewed by: imp ngie emaste
Approved by: mjg kib
Fixes: D9841
Differential Revision: https://reviews.freebsd.org/D41349
This performs very well. x86-64-v3 and x86-64-v4 kernels were written,
too, but performed worse than the baseline kernel on short strings.
These may be added at a future point in time if the performance issues
can be fixed.
os: FreeBSD
arch: amd64
cpu: 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz
│ strlen_scalar.out │ strlen_baseline.out │
│ B/s │ B/s vs base │
Short 1.667Gi ± 1% 2.676Gi ± 1% +60.55% (p=0.000 n=20)
Mid 5.459Gi ± 1% 8.756Gi ± 1% +60.39% (p=0.000 n=20)
Long 15.34Gi ± 0% 52.27Gi ± 0% +240.64% (p=0.000 n=20)
geomean 5.188Gi 10.70Gi +106.24%
Sponsored by: The FreeBSD Foundation
Approved by: kib
Reviewed by: mjg jrtc27
Differential Revision: https://reviews.freebsd.org/D40693
Add a framework for selecting from one of multiple implementations
of a function based on amd64 architecture level (cf. amd64 SysV
ABI supplement).
Sponsored by: The FreeBSD Foundation
Approved by: kib
Reviewed by: jrtc27
Differential Revision: https://reviews.freebsd.org/D40693
The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.
Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix
Enhanced REP MOVSB feature of CPUs starting from Ivy Bridge makes
REP MOVSB the fastest way to copy memory in most of cases. However
Intel Optimization Reference Manual says: "setting the DF to force
REP MOVSB to copy bytes from high towards low addresses will expe-
rience significant performance degradation". Measurements on Intel
Cascade Lake and Alder Lake, same as on AMD Zen3 show that it can
drop throughput to as low as 2.5-3.5GB/s, comparing to ~10-30GB/s
of REP MOVSQ or hand-rolled loop, used for non-ERMS CPUs.
This patch keeps ERMS use for forward ordered memory copies, but
removes it for backward overlapped moves where it does not work.
This is just a cosmetic sync with kernel, since libc does not use
ERMS at this time.
Reviewed by: mjg
MFC after: 2 weeks
Turns out clang converts "memcmp(foo, bar, len) == 0" and similar to
bcmp calls.
Reviewed by: emaste (previous version), jhb (previous version)
Differential Revision: https://reviews.freebsd.org/D34673
Preferably bcmp would just alias memcmp but there is build magic which
makes this problematic.
Reviewed by: jhb
Differential Revision: https://reviews.freebsd.org/D28846
This is a tradeoff which saves jumps for smaller sizes while making
the 8-16 range slower (roughly in line with the other cases).
Tested with glibc test suite.
For example size 3 (most common with vfs namecache) (ops/s):
before: 407086026
after: 461391995
The regressed range of 8-16 (with 8 as example):
before: 540850489
after: 461671032
See the review for sample test results.
Reviewed by: kib (kernel part)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D18401
Handling sizes of > 32 backwards will be updated later.
Reviewed by: kib (kernel part)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D18387
