and this is more space efficient.
Stop queueing recently used buckets to the head of the list. If the bucket
goes to a different processor the cache coherency will be more expensive.
We already try to encourage cache-hot behavior in the per-cpu layer.
Reviewed by: rlibby
Differential Revision: https://reviews.freebsd.org/D23493
This simplifies the driver's rx fast path as well as the bookkeeping
code that tracks various rx buffer sizes and layouts.
MFC after: 1 week
Sponsored by: Chelsio Communications
This allows us to boot FreeBSD RISCV on QEMU using the -kernel command line
options. When using that option, QEMU maps the kernel ELF file to the
addresses specified in the LMAs in the program headers.
Since version 4.2 QEMU ships with OpenSBI fw_jump by default so this allows
booting FreeBSD using the following command line:
qemu-system-riscv64 -bios default -kernel /.../boot/kernel/kernel -nographic -M virt
Without this change the -kernel option cannot be used since the LMAs start
at address zero and QEMU already maps a ROM to these low physical addresses.
For targets that require a different kernel LMA the make variable
KERNEL_LMA can be overwritten in the config file. For example, adding
`makeoptions KERNEL_LMA=0xc0200000` will create an ELF file that will be
loaded at 0xc0200000.
Before:
There are 4 program headers, starting at offset 64
Program Headers:
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
LOAD 0x001000 0xffffffc000000000 0x0000000000000000 0x75e598 0x8be318 RWE 0x1000
DYNAMIC 0x71fb20 0xffffffc00071eb20 0x000000000071eb20 0x000100 0x000100 RW 0x8
GNU_STACK 0x000000 0x0000000000000000 0x0000000000000000 0x000000 0x000000 RW 0x0
NOTE 0x693400 0xffffffc000692400 0x0000000000692400 0x000024 0x000024 R 0x4
After:
There are 4 program headers, starting at offset 64
Program Headers:
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
LOAD 0x001000 0xffffffc000000000 0x0000000080200000 0x734198 0x893e18 RWE 0x1000
DYNAMIC 0x6f7810 0xffffffc0006f6810 0x00000000808f6810 0x000100 0x000100 RW 0x8
GNU_STACK 0x000000 0x0000000000000000 0x0000000000000000 0x000000 0x000000 RW 0x0
NOTE 0x66ca70 0xffffffc00066ba70 0x000000008086ba70 0x000024 0x000024 R 0x4
Reviewed By: br, mhorne (earlier version)
Differential Revision: https://reviews.freebsd.org/D23436
ext_arg2 is the only item in the third cacheline in an mbuf and could be
cold by the time rxb_free runs. Put the information needed by rxb_free
in the same line as the refcount, which is very likely to be hot given
that rxb_free runs when the refcount is decremented and reaches 0.
MFC after: 1 week
Sponsored by: Chelsio Communications
requested.
This is a tradeoff between PCIe efficiency during large packet rx and
packing efficiency during small packet rx.
MFC after: 1 week
Sponsored by: Chelsio Communications
If the thread's lock is already that of the runqueue, don't recurse on
the queue lock.
Reviewed by: jeff, kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D23492
clang has the unfortunate property of paying little attention to prediction
hints when faced with a loop spanning the majority of the rotuine.
In particular fget_unlocked has an unlikely corner case where it starts almost
from scratch. Faced with this clang generates a maze of taken jumps, whereas
gcc produces jump-free code (in the expected case).
Work around the problem by providing a variant which only tries once and
resorts to calling the original code if anything goes wrong.
While here note that the 'seq' parameter is almost never passed, thus the
seldom users are redirected to call it directly.
This eliminates a branch from its consumers trading it for an extra call
if ktrace is enabled for curthread. Given that this is almost never true,
the tradeoff is worth it.
With r357314, sizeof(struct uma_bucket) grew to 16 bytes on 32-bit
platforms, so BUCKET_SIZE(4) is 0. This resulted in the creation of a
bucket zone for buckets with zero capacity. A more general fix is
planned, but for now this bandaid allows 32-bit platforms to boot again.
PR: 243837
Discussed with: jeff
Reported by: pho, Jenkins via lwhsu
Tested by: pho
Sponsored by: The FreeBSD Foundation
Most notably, we want to make sure we don't clobber any capabilities-related
errors. This is a regression from r357412 (O_SEARCH) that was picked up by
the capsicum tests.
PR: 243839
Reviewed by: kib (committed form recommended by)
Tested by: lwhsu
Differential Revision: https://reviews.freebsd.org/D23479
Once all CPUs are online, determine if they all support LSE atomics and
set lse_supported to indicate this. For now the atomic(9)
implementations are still always inlined, though it would be preferable
to create out-of-line functions to avoid text bloat. This was not done
here since big.little systems exist in which some CPUs implement LSE
while others do not, and ifunc resolution must occur well before this
scenario can be detected. It does seem unlikely that FreeBSD will
ever run on such platforms, however, so converting atomic(9) to use
ifuncs is probably a good next step.
Add a LSE_ATOMICS arm64 kernel configuration option to unconditionally
select LSE-based atomic(9) implementations when the target system is
known.
Reviewed by: andrew, kib
MFC after: 1 month
Sponsored by: The FreeBSD Foundation, Amazon (hardware)
Differential Revision: https://reviews.freebsd.org/D23325
These make use of the cas*, ld* and swp instructions added in ARMv8.1.
Testing shows them to be significantly more performant than LL/SC-based
implementations.
No functional change here since the wrappers still unconditionally
select the _llsc variants.
Reviewed by: andrew, kib
MFC after: 1 month
Submitted by: Ali Saidi <alisaidi@amazon.com> (original version)
Differential Revision: https://reviews.freebsd.org/D23324
Add a _llsc suffix for the existing LL/SC-based implementations and add
trivial wrappers. This is in preparation for supporting LSE-based
atomic(9) implementations.
No functional change intended.
Reviewed by: andrew, kib
MFC after: 1 month
Sponsored by: The FreeBSD Foundation, Amazon (hardware)
Differential Revision: https://reviews.freebsd.org/D23323
Parameterize the macros by type width as well as acq/rel semantics.
This makes modifying the implementations much less tedious and
error-prone and makes it easier to support alternate LSE-based
implementations. No functional change intended.
Reviewed by: andrew, kib
MFC after: 1 month
Sponsored by: The FreeBSD Foundation, Amazon (hardware)
Differential Revision: https://reviews.freebsd.org/D23322
Instead of doing a 2 iteration loop (determined at runeimt), take advantage
of the fact that the size is already known.
While here provdie cap_check_inline so that fget_unlocked does not have to
do a function call.
Verified with the capsicum suite /usr/tests.
It was generated to be just a jumping off point to tmpfs_itimes.
While here provide a dedicated variant for getattr since we normally don't
expect to need to the update from that caller.
In r357324 most of the use of gi_irq was moved to gi_lpi. Complete this
with the last few places we need the IRQ value and create gi_id for the
per-device value we need.
MFC after: 2 weeks
Sponsored by: DARPA, AFRL
The code was using a hand-rolled fcmpset loop, while in other places the same
count is manipulated with the refcount API.
This transferred from a stylistic issue into a bug after the API got extended
to support flags. As a result the hand-rolled loop could bump the count high
enough to set the bit flag. Another bump + refcount_release would then free
the file prematurely.
The bug is only present in -CURRENT.
When there are multiple GICv3 ITS devices we don't know which vmem is for
which device. Use device_get_nameunit to get a per-device name.
MFC after: 2 weeks
Sponsored by: DARPA, AFRL
This uses UMA to allocate space. It causes issues when there are multiple
ITS devices in the system where interrupts are not allocated from a low
address on some interrupt controllers. Disabling the quantum cache fixes
this on the Neoverse N1 SDP.
MFC after: 2 weeks
Sponsored by: DARPA, AFRL
fonts. As a workaround, remove the static. vt is default on powerpc, but there's
a few old macs that still fail with vt. sc is used as a work arouond for those
machines, and the kernel fails to build w/o it.
O_SEARCH is defined by POSIX [0] to open a directory for searching, skipping
permissions checks on the directory itself after the initial open(). This is
close to the semantics we've historically applied for O_EXEC on a directory,
which is UB according to POSIX. Conveniently, O_SEARCH on a file is also
explicitly undefined behavior according to POSIX, so O_EXEC would be a fine
choice. The spec goes on to state that O_SEARCH and O_EXEC need not be
distinct values, but they're not defined to be the same value.
This was pointed out as an incompatibility with other systems that had made
its way into libarchive, which had assumed that O_EXEC was an alias for
O_SEARCH.
This defines compatibility O_SEARCH/FSEARCH (equivalent to O_EXEC and FEXEC
respectively) and expands our UB for O_EXEC on a directory. O_EXEC on a
directory is checked in vn_open_vnode already, so for completeness we add a
NOEXECCHECK when O_SEARCH has been specified on the top-level fd and do not
re-check that when descending in namei.
[0] https://pubs.opengroup.org/onlinepubs/9699919799/
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D23247
If we come from VOP_CACHEDLOOKUP, we must skip the VEXEC check as it will
have been done in the caller (vfs_cache_lookup). This is a part of D23247,
which may skip the earlier VEXEC check as well if the root fd was opened
with O_SEARCH.
This one required slightly more work as zfs_lookup may also be called
indirectly as VOP_LOOKUP or a couple of other places where we must do the
check.
clang inlines fget -> _fget into kern_fstat and eliminates several checkes,
but prior to this change it would assume fget_unlocked was likely to fail
and consequently avoidable jumps got generated.
