While here make it return void, we don't set any useful return value
and nothing checks for it.
Sponsored by: Arm Ltd
(cherry picked from commit a7c5c88cb29ff512467a6599f1b4d8e2b8817480)
Report when SVE is present and allow it to be used by calling
sve_restore_state on an SVE exception from userspace.
Reviewed by: kib
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43310
(cherry picked from commit 205c1007be6d57b2eb8620f79384cb2299dde40e)
As with floating point instructions don't trap SVE instructions to the
hypervisor. This lets us handle then in the kernel.
Reviewed by: imp (earlier version)
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43303
(cherry picked from commit fe5ed2496e44aec018a6215175bba225b20d81fd)
Add the NT_ARM_SVE note type and use it to access the SVE registers
from ptrace. This allows userspace to modify the full SVE register
values.
Try to follow the Linux semantics to allow debuggers to use this with
minimal changes.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43309
(cherry picked from commit aab60068943d733b0b4573e5481c543ab3d45a00)
To allow for user space to read the SVE vector length add a sysarch
handler to return the value to userspace.
Reviewed by: imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43308
(cherry picked from commit 87a58d931db7b40d8d225a7edb82b78937078b7f)
Use the new extended register support in the arm64 signal frame to
handle the SVE registers.
As these registers alias the VFP registers we use the floating point
register values to restore the lower bits of the SVE registers. This
is to support software that doesn't understand SVE to continue working.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43307
(cherry picked from commit e45132e978030f0aa964d975a9fbad5614f260bd)
Add initial kernel support for SVE. This detects if SVE is present on
all CPUs, and if so allows for the use of SVE in the future.
As the SVE registers are a superset of the VFP registers we don't need
to restore the VFP registers when SVE is enabled.
Ths interface to enable SVE is provided, but not used until SVE is
supported in signals and with ptrace.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43306
(cherry picked from commit 332c426328dbb30a6b2e69d9b1e8298d77d85bd1)
This will be used when we add SVE support to reduce the registers
needed to be saved on context switch.
Reviewed by: imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43305
(cherry picked from commit a30149b2a9c6ac5280523eea9570e5b5e5f1fdf8)
Rather than try to detect when vfp_save_state is called by savectx use
a separate function that sets up the pcb as needed.
Reviewed by: imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43304
(cherry picked from commit 12257233e8fd94ab24e1a84ad87126af2a7be33b)
This will be used to support FEAT_LPA2 to allow more than 48 bits of
physical address space.
Reviewed by: alc, kib, markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46392
(cherry picked from commit 7be11454edd4eca6cbd299cb5b486294f912ffbd)
This leads to an unallocated register. Fix the value to point to
ESR_EL1.
Sponsored by: Arm Ltd
(cherry picked from commit 8b017284319c46d349abee5573d4656bb377b0fd)
When adding VHE support we may need to access these registers using
the _EL12 or _EL1 name depending on if VHE is enabled or not.
Add the registers we will likely need to access from VHE so these can
be updated in the guest switcher code.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45525
(cherry picked from commit 47361851bc2e292bd7c723ba5f57792a7794f1b4)
CONTEXTIDR_EL1 is used in debug and trace features to identify the
current process or context.
Reviewed by: andrew
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45173
(cherry picked from commit 4f8ba1c9dd00e0ae20fa161e166715746b6d1c04)
We have the mechanism in place to support encoding system registers
explicitly, so use that rather than requiring LLVM 13+, which breaks our
current set of GitHub CI builds.
Fixes: 9eecef052155 ("Add an Armv8 rndr random number provider")
(cherry picked from commit 9560ac4b638edf688566f576adc65d3654f2240c)
The arm and arm64 implementations of dispatching IPIs via PIC_IPI_SEND
are almost identical, and entirely MI with the lone exception of a
single store barrier on arm64 (that is likely either redundant or needed
on arm too). Thus, de-duplicate this code by moving it to INTRNG as a
generic IPI glue framework. The ipi_* functions remain declared in MD
smp.h headers and implemented in MD code, but are trivial wrappers
around intr_ipi_send that could be made MI, at least for INTRNG ports,
at a later date.
Note that, whilst both arm and arm64 had an ii_send member in intr_ipi
to abstract over how to send interrupts,, they were always ultimately
using PIC_IPI_SEND, and so this complexity has been removed. A follow-up
commit will re-introduce the same flexibility by instead allowing a
device other than the root PIC to be registered as the IPI sender.
As part of this, strengthen a MAXCPU assertion that was missed in commit
2f0b059eeafc ("intrng: switch from MAXCPU to mp_ncpus") (which itself is
mis-titled).
Reviewed by: mmel, mhorne
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D35898
(cherry picked from commit fae8755f16ff5b9bdc32df046e0f16c0cbb48a29)
When targeting Armv8.1 we can assume FEAT_LSE is available and can use
the atomic instructions this provides without needing to check for
support first.
Reviewed by: imp, markj, emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46159
(cherry picked from commit 7a345763f96d86eee9ab578e64311bf452e58900)
Before entering the kernel exception level ensure sctlr_el2 and
sctlr_el1 are in a known state. The EOS flag needs to be set to ensure
an eret instruction is a context synchronization event.
Set spcr_el1 when entering the kernel from EL1 and use an eret
instruction to return to the caller. This ensures the CPU pstate is
consistent with the value in spcr_el1 as it is the only way to set it
directly.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45528
(cherry picked from commit 034c83fd7d85f57193850a73cc0ac957a211f725)
When entering the kernel with the E2H field set the layout of the
cnthctl_el2 register changes. Use the correct field locations to enable
access to the counter and timer registers from EL1.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45529
(cherry picked from commit 997511dffe651e1d2d708f37f2ced430a6ab3349)
To support recent extensions to the Arm architecture we may need to
store more or larger registers when sending a signal.
To support this create a list of these extra registers. Userspace that
needs to access a register in the signal handler can then walk the list
to find the correct register struct and read/write its contents.
Reviewed by: kib, markj (earlier version)
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43302
(cherry picked from commit 7e6437c08415ade75403f1ecad75167257e8ea3c)
Adjust the mair_el1 macro indentation to be consistent with the
surrounding macros.
Reviewed by: emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45524
(cherry picked from commit 86bafddd61aba115bc46bcf1d7e0afb125850b5f)
Add the pointer authentication registers to armreg.h. These will be
used to support pointer authentication in a kernel built with GCC.
Reviewed by: jhb
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45262
(cherry picked from commit 57d714a23f5ce21e389d53636b8bc6c1b45d518e)
While clang can handle numbers with a UL suffix in assembly files
gcc/gas is unable to. Switch to use the UL macro for TCR_EL1 defines as
some are used in locore.S
Reviewed by: brooks, jhb
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45261
(cherry picked from commit 29c1cf9860e531146220d9dc3596e4c79f91cfcd)
Bits [5:0] of PMBSR_MSS encodes either Buffer Status Code (BSC) or Fault
Status Code (FSC) depending on PMBSR_EC value.
Add PMBSR_MSS_{BSC,FSC} to cover this field.
Reviewed by: andrew
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45172
(cherry picked from commit 10b3eac88db689d657c4d0d0716bcbdf240ff614)
When the register is not defined in Armv8.0 i.e. added in a later
extension, like SPE added in v8.2, the alternative name format of:
S<op0>_<op1>_C<crn>_C<crm>_<op2>
should be used; otherwise, calls to {READ,WRITE}_SPECIALREG() will
fail.
Use the MRS_REG_ALT_NAME() macro for SPE changing hex to decimal as
required by the macro.
Reviewed by: andrew
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45171
(cherry picked from commit f7bdaa103eb8906fc999c7fd5e8d6af440e26e6c)
These can be used even when the compiler is too old for the register
to be included.
Reviewed by: Zachary Leaf <zachary.leaf@arm.com>
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45176
(cherry picked from commit d6d860c7ff5c4cbe9475d98000407d6f0ea84b47)
Add a macro to find which bits from far_el2 are needed to be copied
to get the full intermediate physical address (IPA).
The hpfar_el2 register only contains a 4k aligned fault address. We
need to include the lower bits from far_el2 if we need the full
faulting IPA.
(cherry picked from commit b3bbec37ba039a46f7166f6fa3571f38da4253e9)
Add a macro to get the ESR_ELx ISS value when we trap accessing a
special register.
(cherry picked from commit 09ac9cf8971a0709bb8d5a3a703cd3dbff882b6f)
No functional change.
Reviewed by: markj
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D45322
(cherry picked from commit b5e17840de65ff196b2398211b68ce96d6919f2c)
It is inherited from arm, where the global exists and is used. No
functional change.
Reviewed by: markj
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D45323
(cherry picked from commit 1d3c23676de33762fd7fc2e3d890fd14738d3ee6)
When checking if a physical address is in the DMAP region we assume
all physical addresses between DMAP_MIN_PHYSADDR and DMAP_MAX_PHYSADDR
are able to be accesses through the DMAP. It may be the case that
there is device memory in this range that shouldn't be accessed through
the DMAP mappings.
Add a check to PHYS_IN_DMAP that the translated virtual address is a
valid kernel address. To support code that already checks the address
is valid add PHYS_IN_DMAP_RANGE.
PR: 278233
Reviewed by: alc, markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D44677
(cherry picked from commit 9d40492efa467095340cf3dca5860880aa441472)
Part numbers are sourced from Linux (arch/arm64/include/asm/cputype.h).
MFC after: 2 weeks
Pull Request: https://github.com/freebsd/freebsd-src/pull/1144
(cherry picked from commit da2b732288c72b347b128e1fa9a169c7bba3beb8)
When booting a KMSAN kernel on an Ampere Altra, I've seen some boot time
hangs when the XHCI controller driver attempts to allocate memory for
32-bit DMA. The system boots fine with a GENERIC kernel; I believe that
the additional memory requirements of KMSAN push it over the edge. The
system has a bit less than 2GB of RAM below the 4GB boundary.
Allocate a new freelist to segregate memory below 4GB, as we do on
amd64, so that such memory allocation failures are less likely to occur.
Reviewed by: alc
MFC after: 1 month
Sponsored by: Klara, Inc.
Sponsored by: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D43503
(cherry picked from commit 48d5dab7bace6bae938672784ca3aa8733d72eb9)
Use pmap_early_vtophys to translate from a virtual to physical where
we were previously using the calculated delta. This means that, while
we still calculate it, we don't need to pass it to initarm or either
pmap bootstrap functions.
While here remove an unneeded printf that indirectly used it or was
related to the previous printf.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42567
(cherry picked from commit ba31362694fe465b88a025ac7c35ad7f378efc86)
(cherry picked from commit f3a83b3a6257ea0fd5aba66ff42218350228b321)
Use pmap_early_vtophys to find the physical address of the kernel base
rather than using the calculated offset as it will be removed in a
latter commit.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42566
(cherry picked from commit 5fae53589eacbf65855419922c881b3a25a68e03)
Update the HWCAP2 values to align with Linux v6.6-rc7
Reviewed by: manu, imp, emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42379
(cherry picked from commit 2ee649c1dd5c41e93f02fdca8bba9279371e4efd)
There are more than 32 arm64 HWCAP2 values. Expand the macros to
include leading zeros and mark them as unsigned long.
Reviewed by: manu, imp, emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42378
(cherry picked from commit 0f0938800989dbf84a56c370db0b9379ee730a39)
When adding support for new hardware extensions we may not want to
enable support for the FreeBSD and Linux ABIs at the same time. To
support this split the Linux ID register and hwcaps so they can be
configured separately.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42372
(cherry picked from commit e6dbc99d47ddb254d75822817592bb82b5ce4d97)
The .mcount function needs a BTI branch target. As we can't rely on
asm.h being included use the hint version of a "bti c" instruction.
This is a nop when BTI is not supported or not used.
Reviewed by: markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42230
(cherry picked from commit 0590ed0967e025e1d155da9e06509f3fe9686feb)
To support Pointer Authentication (PAC) in assembly files add a pair of
macros that sign the link register. When used before storing to the
stack it will allow hardware to detect if it has changed before using
it in the return instruction.
Reviewed by: markj, emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42226
(cherry picked from commit 2b39a6f68c3839454cd255b91978202cd995182a)
To detect when an object file is build with the Branch Target
Identification (BTI) and Pointer Authentication Code (PAC) extensions
there is an elf note the compiler will insert. It will only do so from
a high level language, e.g. C or C++.
To get the not in assembly add the GNU_PROPERTY_AARCH64_FEATURE_1_NOTE
macro that can be used to create it, and the
GNU_PROPERTY_AARCH64_FEATURE_1_VAL macro to insert the correct value
based on which combination of BTI and PAC are enabled.
Reviewed by: markj (earlier version), emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42225
(cherry picked from commit 82597d2102a02c4e9e9355717c32867d34b77ef0)
When we enable BTI iboth the first instruction in a function that could
be called indirectly, and a branch within a function need a valid
landing pad instruction.
There are three options for these instructions:
1. A breakpoint instruction
2. A pointer authentication PACIASP/PACIBSP
3. A BTI instruction
Option 1 will raise a breakpoint exception so isn't useable in either
cases. Option 2 could be used in some function entry cases, but needs
to be paired with an authentication instruction, and is normally only
used in non-leaf functions we can't use it in this case. This leaves
option 3.
There are four variants of the instruction, the C variant is used on
function entry and the J variant is for jumping within a function.
There is also a JC that works with both and one with no target that
works with neither.
Reviewed by: markj
Sponsored by: Arm Ltd
Sponsored by: The FreeBSD Foundation (earlier version)
Differential Revision: https://reviews.freebsd.org/D42078
(cherry picked from commit e340882d3e49a98aa39b13041a2bf714c30dccdf)
We could hit these when executing code marked as using BTI but jumps
to a non-branch target instruction.
Sponsored by: Arm Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39450
(cherry picked from commit 450f731b7f2351d475b43bbc6234eea28adfd2b0)
These will be used by bhyve to implement page table walking.
Sponsored by: Arm Ltd
(cherry picked from commit a35e47369f88e8c2e14b7371a1a5e1152680551f)
