LSan wants to be able to register atfork handlers at __lsan_init time,
which can happen either at the first intercepted function call or in a
.preinit_array function. Both of these end up being very early in rtld
and executed with the bind lock held, which ends up causing problems
when we go to _libpthread_init().
Instead of requiring libpthread to be initialized, just insert the new
atfork handler straight into the list if it's not ready yet. The
critical section and locking should not be necessary if we're really
executing this early, as there won't be any threads to contend with.
Reviewed by: kib (earlier version), markj
(cherry picked from commit 4b202f4faf40fd7af8b84491360186aed8ce5733)
These ones were unambiguous cases where the Foundation was the only
listed copyright holder.
Sponsored by: The FreeBSD Foundation
(cherry picked from commit 5c2bc3db201a4fe8d7911cf816bea104d5dc2138)
Add the ability to pre-resolve architecture-specific EABI symbols and
use it on arm for selected EABI functions. These functions can be called
with rtld bind lock write-locked, so they should be resolved in forward.
Reported by: Mark Millard <marklmi@yahoo.com>, John F Carr <jfc@mit.edu>
Reviewed by: kib, imp
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D46104
(cherry picked from commit 5670b8cc3672d5a6bc2c41eb48d7d01343c43ad0)
Otherwise the lock upgrade performed by rtld's load_filtees() can result
in infinite recursion, wherein:
1. _rtld_bind() acquires the bind read lock,
2. the source DSO's filtees haven't been loaded yet, so the lock upgrade
in load_filtees() cause rtld to jump to _rtld_bind() and release the
bind lock,
3. _thr_rtld_lock_release() calls _thr_ast(), which calls thr_wake(),
which hasn't been resolved yet,
4. _rtld_bind() acquires the bind read lock in order to resolve
thr_wake(),
5. ...
See the linked pull request for an instance of this problem arising with
libsys. That particular instance is also worked around by commit
e7951d0b04e6.
Reported by: brooks
Reviewed by: kib
Pull Request: https://github.com/freebsd/freebsd-src/pull/908
MFC after: 1 week
Sponsored by: Innovate UK
(cherry picked from commit 36f0a34ca645d49ec79d60ea7e773374ef0991ea)
Similarly as in the previous commit, using calloc() instead of malloc()
is useless here in the regular case since the subsequent call to
cpuset_getaffinify() is going to completely fill the allocated memory.
However, there is an additional complication. This function tries to
allocate memory to hold the cpuset if it previously wasn't, and does so
before the thread lock is acquired, which can fail on a bad thread ID.
In this case, it is necessary to deallocate the memory allocated in this
function so that the attributes object appears unmodified to the caller
when an error is returned. Without this, a subsequent call to
pthread_attr_getaffinity_np() would expose uninitialized memory (not
a security problem per se, since it comes from the same process) instead
of returning a full mask as it would before the failing call to
pthread_attr_get_np(). So the caller would be able to notice a change
in the state of the attributes object even if pthread_attr_get_np()
reported failure, which would be quite surprising. A similar problem
that could occur on failure of cpuset_setaffinity() has been fixed.
Finally, we shall always report memory allocation failure. This already
goes for pthread_attr_init(), so, if for nothing else, just be
consistent.
Reviewed by: emaste, kib
Approved by: emaste (mentor)
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D43329
(cherry picked from commit aadb4a1b3fd185d547087f6eafca6ce0b4df3291)
Approved by: markj (mentor)
Using calloc() instead of malloc() is useless here since the allocated
memory is to be wholly crushed by the memcpy() call that follows.
Suggested by: kib
Reviewed by: emaste, kib
Approved by: emaste (mentor)
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D43328
(cherry picked from commit 15bb0300eeca233827a6a13cd2ed883e0034979d)
Approved by: markj (mentor)
The change of argument for sizeof() (from a type to an object) is to be
consistent with the change done for the malloc() code just above in the
preceding commit touching this file.
Consider bit flags as integers and test whether they are set with an
explicit comparison with 0.
Use an explicit flag value (PTHREAD_SCOPE_SYSTEM) in place of a variable
that has this value at point of substitution.
All other changes are straightforward.
Suggested by: kib
Reviewed by: kib
Approved by: emaste (mentor)
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D43327
(cherry picked from commit 4d312aa051414ad77d3515f258230d01ad11d6dc)
Approved by: markj (mentor)
On first read, POSIX may seem ambiguous about the return code for some
scheduling-related pthread functions on invalid arguments. But a more
thorough reading and a bit of standards archeology strongly suggests
that this case should be handled by EINVAL and that ENOTSUP is reserved
for implementations providing only part of the functionality required by
the POSIX option POSIX_PRIORITY_SCHEDULING (e.g., if an implementation
doesn't support SCHED_FIFO, it should return ENOTSUP on a call to, e.g.,
sched_setscheduler() with 'policy' SCHED_FIFO).
This reading is supported by the second sentence of the very definition
of ENOTSUP, as worded in CAE/XSI Issue 5 and POSIX Issue 6: "The
implementation does not support this feature of the Realtime Feature
Group.", and the fact that an additional ENOTSUP case was added to
pthread_setschedparam() in Issue 6, which introduces SCHED_SPORADIC,
saying that pthread_setschedparam() may return it when attempting to
dynamically switch to SCHED_SPORADIC on systems that doesn't support
that.
glibc, illumos and NetBSD also support that reading by always returning
EINVAL, and OpenBSD as well, since it always returns EINVAL but the
corresponding code has a comment suggesting returning ENOTSUP for
SCHED_FIFO and SCHED_RR, which it effectively doesn't support.
Additionally, always returning EINVAL fixes inconsistencies where EINVAL
would be returned on some out-of-range values and ENOTSUP on others.
Reviewed by: markj
Approved by: markj (mentor)
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D43006
(cherry picked from commit 0eccb45979a8ee3129e11b638ebc4cfa00942b80)
pthread_getname_np needs to be provided by libc in order to import
jemalloc 5.3.0.
A stub implementation for libc pthread_getname_np() is added for
_pthread_stubs.c, which always reports empty name for the main thread.
Internal _pthread_getname_np() is not exported, but provided for libc
own use.
Reviewed by: kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D41461
The acquisition and release of an uncontended default/normal pthread
mutex on FreeBSD is suprisingly slow, e.g., pthread wrlocks and binary
semaphores both exhibit roughly 33% lower latency, while default/normal
mutexes on Linux exhibit roughly 67% lower latency than FreeBSD. This is
likely explained by the fact that AFAICT in the best case to acquire an
uncontended mutex on Linux one need touch only 1 page and read+modify
only 1 cacheline, whereas on FreeBSD we need to touch at least 4 pages,
read 6 cachelines, and modify at least 4 cachelines.
This patch does not address the pthread mutex architecture. Instead,
it improves performance by adding the __always_inline attribute to
mutex_lock_common() and mutex_unlock_common() to encourage constant
folding and propagation, thereby lowering the latency to acquire and
release a mutex due to a shorter code path with fewer compares, jumps,
and mispredicts.
With this patch on a stock build I see a reduction in latency of roughly
7% for default/normal mutexes, and 17% for robust mutexes. When built
without PTHREADS_ASSERTIONS enabled I see a reduction in latency of
roughly 15% and 26%, respectively. Suprisingly, I see similar reductions
in latency for heavily contended mutexes.
By default, this patch increases the size of libthr.so.3 by 2448 bytes,
but when built without PTHREAD_ASSERTIONS enabled it only increases by
448 bytes.
Reviewed by: jhb (previous version), kib
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D40912
This patch fixes a bug which prevents building libthr without
_PTHREADS_INVARIANTS defined. The default remains to build libthr
with -D_PTHREADS_INVARIANTS. However, with this patch, if one builds
libthr with WITHOUT_PTHREADS_ASSERTIONS=true then the latency to
acquire+release a default pthread mutex is reduced by roughly 5%, and a
robust mutex by roughly 18% (as measured by a simple synthetic test on a
Xeon E5-2697a based machine).
Reviewed by: jhb, kib, mjg
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D40900
Reduces severe performance degradation due to false-sharing. Note that this
does not account for hardware which can perform adjacent cacheline prefetch.
[mjg: massaged the commit message and the patch to use aligned_alloc
instead of malloc]
PR: 272238
MFC after: 1 week
Since there is only the current thread in the child, no pending readers
exist. Clear the bit, since it confuses future attempts to acquire
write ownership of the rtld locks, due to URWLOCK_PREFER_READERS flag.
To be future-proof, clear all state about pending writers and readers.
PR: 271490
Reported and tested by: KJ Tsanaktsidis <kj@kjtsanaktsidis.id.au>
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D40178
The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.
Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix
When __thr_pshared_offpage() is called for allocation, it must not use
the cached offpage for the key. Instead, the cached offpage must be
unmapped and removed from the cache, if any.
It is legitimate for the user code to unmap the shared lock object without
destroying it, and then mapping something over the freed VA to carry
another shared lock. In this case the cached offpage must be un-cached.
PR: 269277
Reported by: rau8344@gmail.com
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D38345
Since f35093f8 semantics of a thread affinity functions is changed to be a
compatible with Linux:
In case of getaffinity(), the minimum cpuset_t size that the kernel permits is
the maximum CPU id, present in the system, / NBBY bytes, the maximum size is not
limited.
In case of setaffinity(), the kernel does not limit the size of the user-provided
cpuset_t, internally using only the meaningful part of the set, where the upper
bound is the maximum CPU id, present in the system, no larger than the size of
the kernel cpuset_t.
To match pthread_attr_[g|s]etaffinity_np checks of the user-provided cpusets to
the kernel behavior export the minimum cpuset_t size allowed by running kernel
via new sysctl kern.sched.cpusetsizemin and use it in checks.
Reviewed by:
Differential Revision: https://reviews.freebsd.org/D38112
MFC after: 1 week
In libthr we use PAGE_SIZE when allocating memory with mmap and to check
various structs will fit into a single page so we can use this allocator
for them.
Ask the kernel for the page size on init for use by the page allcator
and add a new machine dependent macro to hold the smallest page size
the architecture supports to check the structure is small enough.
This allows us to use the same libthr on arm64 with either 4k or 16k
pages.
Reviewed by: kib, markj, imp
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34984
Rather than calling getpagesize() twice use the value saved after the
first call to size a mmap allocation.
Reviewed by: kib, markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34983
This matches the type in other unwind headers (LLVM libunwind,
libcxxrt, glibc).
NB: include/unwind.h is not installed but is only used by libthr
Reviewed by: imp, dim, emaste
Differential Revision: https://reviews.freebsd.org/D34049
This matches libc and rtld in using the alignment (TLS_TCB_ALIGN) from
machine/tls.h instead of hardcoding 16.
Reviewed by: kib
Sponsored by: The University of Cambridge, Google Inc.
Differential Revision: https://reviews.freebsd.org/D34023
The current ASLR stack gap feature will be removed, and with that the
need for this change, and the kern.stactop sysctl, is gone. Moreover,
the approach taken in this revision does not provide compatibility for
old copies of libthr.so, and the revision should have also updated
__libc_map_stacks_exec().
This reverts commit 78df56ccfc.
Reviewed by: kib
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33704
This macro is confusing as it is not related to the similarly named
TLS_DTV_OFFSET. Instead, replace its one use with the desired
expression which is the same on all platforms.
Reviewed by: kib, emaste, jrtc27
Sponsored by: The University of Cambridge, Google Inc.
Differential Revision: https://reviews.freebsd.org/D33345
Use the new kern.stacktop sysctl to retrieve the address of stack top
instead of kern.usrstack. kern.usrstack does not have any knowledge
of the stack gap, so this can cause problems with thread stacks.
Using kern.stacktop sysctl should fix most of those problems.
kern.usrstack is used as a fallback when kern.stacktop cannot be read.
Rename usrstack variables to stacktop to reflect this change.
Fixes problems with firefox and thunderbird not starting with
stack gap enabled.
PR: 239873
Reviewed by: kib
Obtained from: Semihalf
Sponsored by: Stormshield
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D31898
Right now, libthr does not initialize RtldLockInfo.rtli_version when calling
_rtld_thread_init(), which makes versioning the interface troublesome.
Add a workaround: if the calling object of _rtld_thread_init() exports
the "_pli_rtli_version" symbol, then consider rtli_version initialized.
Otherwise, forcibly set it to RTLI_VERSION_ONE, currently defined as
RTLI_VERSION.
Export "_pli_rtli_version" from libthr and properly initialize rtli_version.
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D29633
One possible way the recursion can happen is during fork: suppose
that fork is called from early code that did not triggered
jemalloc(3) initialization yet. Then we lock thr_malloc lock, and
call malloc_prefork() that might require initialization of jemalloc
pthread_mutexes, calling into libthr malloc. It is safe to allow
recursion for this occurence.
PR: 252579
Reported by: Vasily Postnicov <shamaz.mazum@gmail.com>
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Without wrapping, rtld services and malloc(3) are not guaranteed
to operate correctly in the forked child.
Reviewed by: markj
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D28088
We need at least thr_malloc ready. The situation is possible e.g. in case
of libthr being listed in DT_NEEDED before some of its consumers.
Reported and tested by: lev
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
This seems to be required by recent clang asan.
I do not see other way than put the symbol under FBSD_1.0 version.
PR: 251112
Reported by: Andrew Stitcher <astitcher@apache.org>
Reviewed by: emaste
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D27389
pthread_get_name_np() and pthread_set_name_np().
This re-applies r361770 after compatibility fixes.
Reviewed by: antoine, jkim, markj
Tested by: antoine (exp-run)
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D25117
