When checking if a physical address is in the DMAP region we assume
all physical addresses between DMAP_MIN_PHYSADDR and DMAP_MAX_PHYSADDR
are able to be accesses through the DMAP. It may be the case that
there is device memory in this range that shouldn't be accessed through
the DMAP mappings.
Add a check to PHYS_IN_DMAP that the translated virtual address is a
valid kernel address. To support code that already checks the address
is valid add PHYS_IN_DMAP_RANGE.
PR: 278233
Reviewed by: alc, markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D44677
(cherry picked from commit 9d40492efa467095340cf3dca5860880aa441472)
Part numbers are sourced from Linux (arch/arm64/include/asm/cputype.h).
MFC after: 2 weeks
Pull Request: https://github.com/freebsd/freebsd-src/pull/1144
(cherry picked from commit da2b732288c72b347b128e1fa9a169c7bba3beb8)
When booting a KMSAN kernel on an Ampere Altra, I've seen some boot time
hangs when the XHCI controller driver attempts to allocate memory for
32-bit DMA. The system boots fine with a GENERIC kernel; I believe that
the additional memory requirements of KMSAN push it over the edge. The
system has a bit less than 2GB of RAM below the 4GB boundary.
Allocate a new freelist to segregate memory below 4GB, as we do on
amd64, so that such memory allocation failures are less likely to occur.
Reviewed by: alc
MFC after: 1 month
Sponsored by: Klara, Inc.
Sponsored by: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D43503
(cherry picked from commit 48d5dab7bace6bae938672784ca3aa8733d72eb9)
Use pmap_early_vtophys to translate from a virtual to physical where
we were previously using the calculated delta. This means that, while
we still calculate it, we don't need to pass it to initarm or either
pmap bootstrap functions.
While here remove an unneeded printf that indirectly used it or was
related to the previous printf.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42567
(cherry picked from commit ba31362694fe465b88a025ac7c35ad7f378efc86)
(cherry picked from commit f3a83b3a6257ea0fd5aba66ff42218350228b321)
Use pmap_early_vtophys to find the physical address of the kernel base
rather than using the calculated offset as it will be removed in a
latter commit.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42566
(cherry picked from commit 5fae53589eacbf65855419922c881b3a25a68e03)
Update the HWCAP2 values to align with Linux v6.6-rc7
Reviewed by: manu, imp, emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42379
(cherry picked from commit 2ee649c1dd5c41e93f02fdca8bba9279371e4efd)
There are more than 32 arm64 HWCAP2 values. Expand the macros to
include leading zeros and mark them as unsigned long.
Reviewed by: manu, imp, emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42378
(cherry picked from commit 0f0938800989dbf84a56c370db0b9379ee730a39)
When adding support for new hardware extensions we may not want to
enable support for the FreeBSD and Linux ABIs at the same time. To
support this split the Linux ID register and hwcaps so they can be
configured separately.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42372
(cherry picked from commit e6dbc99d47ddb254d75822817592bb82b5ce4d97)
The .mcount function needs a BTI branch target. As we can't rely on
asm.h being included use the hint version of a "bti c" instruction.
This is a nop when BTI is not supported or not used.
Reviewed by: markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42230
(cherry picked from commit 0590ed0967e025e1d155da9e06509f3fe9686feb)
To support Pointer Authentication (PAC) in assembly files add a pair of
macros that sign the link register. When used before storing to the
stack it will allow hardware to detect if it has changed before using
it in the return instruction.
Reviewed by: markj, emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42226
(cherry picked from commit 2b39a6f68c3839454cd255b91978202cd995182a)
To detect when an object file is build with the Branch Target
Identification (BTI) and Pointer Authentication Code (PAC) extensions
there is an elf note the compiler will insert. It will only do so from
a high level language, e.g. C or C++.
To get the not in assembly add the GNU_PROPERTY_AARCH64_FEATURE_1_NOTE
macro that can be used to create it, and the
GNU_PROPERTY_AARCH64_FEATURE_1_VAL macro to insert the correct value
based on which combination of BTI and PAC are enabled.
Reviewed by: markj (earlier version), emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42225
(cherry picked from commit 82597d2102a02c4e9e9355717c32867d34b77ef0)
When we enable BTI iboth the first instruction in a function that could
be called indirectly, and a branch within a function need a valid
landing pad instruction.
There are three options for these instructions:
1. A breakpoint instruction
2. A pointer authentication PACIASP/PACIBSP
3. A BTI instruction
Option 1 will raise a breakpoint exception so isn't useable in either
cases. Option 2 could be used in some function entry cases, but needs
to be paired with an authentication instruction, and is normally only
used in non-leaf functions we can't use it in this case. This leaves
option 3.
There are four variants of the instruction, the C variant is used on
function entry and the J variant is for jumping within a function.
There is also a JC that works with both and one with no target that
works with neither.
Reviewed by: markj
Sponsored by: Arm Ltd
Sponsored by: The FreeBSD Foundation (earlier version)
Differential Revision: https://reviews.freebsd.org/D42078
(cherry picked from commit e340882d3e49a98aa39b13041a2bf714c30dccdf)
We could hit these when executing code marked as using BTI but jumps
to a non-branch target instruction.
Sponsored by: Arm Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39450
(cherry picked from commit 450f731b7f2351d475b43bbc6234eea28adfd2b0)
These will be used by bhyve to implement page table walking.
Sponsored by: Arm Ltd
(cherry picked from commit a35e47369f88e8c2e14b7371a1a5e1152680551f)
This needs to be 1UL to ensure the type is large enough for the shift.
Sponsored by: Arm Ltd
(cherry picked from commit aea540538991dd9e8cf21c6ca06e09f706bf2d18)
pmap_san_bootstrap() doesn't really do much, and it was hard-coding the
the bootstrap stack size defined in locore.S. Moreover, the name is a
bit confusing given the existence of pmap_bootstrap_san(). Just remove
it and call kasan_init_early() directly like we do on amd64. It will
not be used by KMSAN in a forthcoming patch series.
No functional change intended.
MFC after: 1 week
Sponsored by: Klara, Inc.
Sponsored by: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D43403
(cherry picked from commit 90372a9e3cda838914bc8518dbc4340906351e98)
This is a direct commit to stable/14 to preserve the ABI of the
the pci_cfgregread and pci_cfgregwrite functions. The new routines
are renamed to add a _domain suffix and macros map the new API to
the new functions.
Note: No API compatibility has been provided as modules in ports
should not be using this internal API (normal PCI drivers use
pci_read_config and pci_write_config with a device_t).
This commit changes the API of pci_cfgreg(read|write) to add a domain
argument (referred to as a segment in ACPI parlance) (note that this
is not the same as a NUMA domain, but something PCI-specific). This
does not yet enable access to domains other than 0, but updates the
API to support domains.
Places that use hard-coded bus/slot/function addresses have been
updated to hardcode a domain of 0. A few places that have the PCI
domain (segment) available such as the acpi_pcib_acpi.c Host-PCI
bridge driver pass the PCI domain.
The hpt27xx(4) and hptnr(4) drivers fail to attach to a device not on
domain 0 since they provide APIs to their binary blobs that only
permit bus/slot/function addressing.
The x86 non-ACPI PCI bus drivers all hardcode a domain of 0 as they do
not support multiple domains.
Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D42827
(cherry picked from commit 1587a9db92c03c738bb3f0fc5874b43c961e7c99)
This grows the block enough to fit a 4K 32-bit depth framebuffer; some
firmware would present smaller GOP modes to be able to boot with a
smaller framebuffer on these devices, but the Windows Devkit firmware
is simply not that nice. Instead, it offers exactly one GOP mode that
matches the current resolution of the attached display, so with limited
control over resolution on most of my displays it'd be nice if we could
Just Work(TM) at 4K.
andrew notes that he has some ideas for removing PMAP_MAPDEV_EARLY_SIZE
entirely, so this limitation could end up removed altogether in the
future.
Reviewed by: andrew, emaste
(cherry picked from commit a3ceeef26bc880b86f4e181bddd9924a2b5e0691)
This is needed to support the bhyve gdb stub implementation on arm64.
Reviewed by: andrew
MFC after: 1 week
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D42867
(cherry picked from commit 4f12883c360dbb03562f11713e711e3a3a330a0f)
Background:
The pm_ev field of struct pmc_op_pmcallocate and struct pmc
traditionally contains the index of the chosen event, corresponding to
the __PMC_EVENTS array in pmc_events.h. This is a static list of events,
maintained by FreeBSD.
In the usual case, libpmc translates the user supplied event name
(string) into the pm_ev index, which is passed as an argument to the
allocation syscall. On the kernel side, the allocation method for the
relevant hwpmc class translates the given index into the event code that
will be written to an event selection register.
In 2018, a new source of performance event definitions was introduced:
the pmu-events json files, which are maintained by the Linux kernel. The
result was better coverage for newer Intel processors with a reduced
maintenance burden for libpmc/hwpmc. Intel and AMD CPUs were
unconditionally switched to allocate events from pmu-events instead of
the traditional scheme (959826ca1b, 81eb4dcf9e).
Under the pmu-events scheme, the pm_ev field contains an index
corresponding to the selected event from the pmu-events table, something
which the kernel has no knowledge of. The configuration for the
performance counting registers is instead passed via class-dependent
fields (struct pmc_md_op_pmcallocate).
In 2021 I changed the allocation logic so that it would attempt to
pull from the pmu-events table first, and fall-back to the traditional
method (dfb4fb4116). Later, pmu-events support for arm64 and power8
CPUs was added (28dd6730a5 and b48a2770d4).
The problem that remains is that the pm_ev field is overloaded, without
a definitive way to determine whether the event allocation came from the
pmu-events table or FreeBSD's statically-defined PMC events. This
resulted in a recent fix, 21f7397a61.
Change:
To disambiguate these two supported but separate use-cases, add a new
flag, PMC_F_EV_PMU, to be set as part of the allocation, indicating that
the event index came from pmu-events.
This is useful in two ways:
1. On the kernel side, we can validate the syscall arguments better.
Some classes support only the traditional event scheme (e.g.
hwpmc_armv7), while others support only the pmu-events method (e.g.
hwpmc_core for Intel). We can now check for this. The hwpmc_arm64
class supports both methods, so the new flag supersedes the existing
MD flag, PM_MD_EVENT_RAW.
2. The flag will be tracked in struct pmc for the duration of its
lifetime, meaning it is communicated back to userspace. This allows
libpmc to perform the reverse index-to-event-name translation
without speculating about the meaning of the index value.
Adding the flag is a backwards-incompatible ABI change. We recently
bumped the major version of the hwpmc module, so this breakage is
acceptable.
Reviewed by: jkoshy
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40753
(cherry picked from commit c190fb35f35cc163b61e582a49115680b0d49dcc)
For amd64, i386, arm, and riscv, i.e. all architectures except arm64,
the custom implementation is provided since we maintain the bitmask of
active CPUs anyway.
Arm64 uses somewhat naive iteration over CPUs and match current vmspace'
pmap with the argument. It is not guaranteed that vmspace->pmap is the
same as the active pmap, but the inaccuracy should be toleratable.
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D32360
As in commit 9051987e40 for amd64, support up to 1024 CPU cores.
arm64 hardware with more than 256 CPU cores is currently available and
will become increasingly common over FreeBSD 14's lifetime.
PR: 269572
Reviewed by: andrew
Relnotes: Yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D41319
WFI and WFIT trap to EL2 when executed in a vmm guest. (Currently
WFE/WFET are not configured to trap.) We only handle WFI at the moment,
so these constants are useful when handling the exception.
Reviewed by: andrew
MFC after: 1 week
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D41199
No fields have been defined, but it has been documented in the
Architecture Reference Manual.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40897
While here move to decimal for the _op and _CR definitions to be used
by a future macro to define the register when the assembler doesn't
know about it.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40896
While here move to decimal for the _op and _CR definitions to be used
by a future macro to define the register when the assembler doesn't
know about it.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40895
No fields have been defined, but it has been documented in the
Architecture Reference Manual.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40894
It breaks a future macro that creates the alternative register name
for old compilers.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40892
While here move to decimal for the _op and _CR definitions to be used
by a future macro to define the register when the assembler doesn't
know about it.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40890
While here move to decimal for the _op and _CR definitions to be used
by a future macro to define the register when the assembler doesn't
know about it.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40889
While here move to decimal for the _op and _CR definitions to be used
by a future macro to define the register when the assembler doesn't
know about it.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40888
While here move to decimal for the _op and _CR definitions to be used
by a future macro to define the register when the assembler doesn't
know about it.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40887
In order to compile lib32 libraries and other 32-bit code on arm64,
<machine/foo.h> needs to be redirected to an arm header rather
than arm64 when building with -m32. Ifdef the arm64 headers that
are installed in /usr/include/machine and used by user-level software
(including references from /usr/include/*.h) so that if __arm__ is
defined when including the arm64 version, <arm/foo.h> is included
rather than using the rest of the file's contents. Some arm headers
had no arm64 equivalent; headers were added just to do the redirection.
These files use #error if __arm__ is not defined to guard against
confusion. Also add an include/arm Makefile, and modify Makefiles
as needed to install everything, including the arm files in
/usr/include/arm. fenv.h comes from lib/msun/arm/fenv.h.
The new arm64 headers are:
acle-compat.h
cpuinfo.h
sysreg.h
Reviewed by: jrtc27, imp
Differential Revision: https://reviews.freebsd.org/D40944
Use of compiler builtin ffs/ctz functions will result in optimized
instruction sequences when possible, and fall back to calling a function
provided by the compiler run-time library. We have slowly shifted our
platforms to take advantage of these builtins in 60645781d6 (arm64),
1c76d3a9fb (arm), 9e319462a0 (powerpc, partial).
Some platforms still rely on the libkern implementations of these
functions provided by libkern, namely riscv, powerpc (ffs*, flsll), and
i386 (ffsll and flsll). These routines are slow, as they perform a
linear search for the bit in question. Even on platforms lacking
dedicated bit-search instructions, such as riscv, the compiler library
will provide better-optimized routines, e.g. by using binary search.
Consolidate all definitions of these functions (whether currently using
builtins or not) to libkern.h. This should result in equivalent or
better performing routines in all cases.
One wart in all of this is the existing HAVE_INLINE_F*** macros, which
we use in a few places to conditionally avoid the slow libkern routines.
These aren't easily removed in one commit. For now, provide these
defines unconditionally, but marked for removal after subsequent
cleanup.
Removal of the now unused libkern routines will follow in the next
commit.
Reviewed by: dougm, imp (previous version)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40698
This will be used by bhyve and will allow the size to change, e.g. for SVE.
Reviewed by: markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40131
The setting of VM_NFREEORDER and the comment describing it were copied
from sparc64 where both the page size and the number of page table
entries that fit in a cache line are different from arm64.
Reviewed by: andrew, kib, markj
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D40782
This lets consumers fetch the value of a system register and apply a
mask over individual fields. That is, each field in the returned value
will be the "smaller" of the two provided by "mask" and the value saved
in kern_cpu_desc. This will be used by vmm to sanitize host system
register fields.
Reviewed by: andrew
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Sponsored by: Klara, Inc. (hardware)
Differential Revision: https://reviews.freebsd.org/D40500
It was used in one place and was added specifically to support dtrace
stack unwinding code. Write an equivalent expression using struct
unwind_state instead. No functional change intended.
Reviewed by: andrew
MFC after: 1 week
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D40538
We only need this during boot. Allocate the array before starting CPUs
to reduce the memory usage.
Reviewed by: Zach Leaf <zachary.leaf@arm.com>
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40433
