Since commit 19df0c5abcb9d4e9 ("LinuxKPI: make __kmalloc() play by the
rules"), kmalloc() systematically allocates contiguous physical memory,
as it should. However, kvzalloc() was left defined in terms of
kmalloc(), which makes it allocate contiguous physical memory too. This
is a too stringent restriction, as kvzalloc() is supposed to be a simple
page-zeroing wrapper around kvmalloc().
According to Linux's documentation ("memory-allocation.rst"), kvmalloc()
first tries to allocate contiguous memory, falling back to
non-contiguous one if that fails. Thus, callers are already supposed to
deal with the possibility of non-contiguous memory being returned.
Reviewed by: bz
Fixes: 19df0c5abcb9 ("LinuxKPI: make __kmalloc() play by the rules")
MFC after: 10 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D51247
(cherry picked from commit 986edb19a49c7d7d3050c759d9b0826283492ebf)
Forgotten on commit to main/-CURRENT:
PR: 277476
Move kfree() into slab.c as an implementation and hide the private
function linux_kfree_async() entirely.
Remove a ; at the end of a define and sort some defines into place.
Remove extern from function declarations and move the closer to where
they belong.
Sort the functions into "base allocator/free" functions--these have
an implementation in slab.c and are ensuring contiguous physical
memory allocations.
Followed by inline functions using these base allocators to implement
their functionality; vmalloc/kvalloc, and misc functions.
Sponsored by: The FreeBSD Foundation
Reviewed by: dumbbell
Differential Revision: https://reviews.freebsd.org/D49572
(cherry picked from commit a3e6f97bf57c1d323487d07369aec66542f995ce)
Implement krealloc_array() using krealloc(). Implement krealloc()
doing the various size checks ourselves and use realloc() or kmalloc()
depending on old and new allocation sizes.
This way we can ensure that allocated memory stays physically contiguous.
Sponsored by: The FreeBSD Foundation
Suggested by: jhb (see D46657)
Reviewed by: jhb, markj
Differential Revision: https://reviews.freebsd.org/D49571
(cherry picked from commit 1c95d401ebe5075ebb38b57638830713a496f107)
With mallocarray() we cannot guarantee that any size larger than
PAGE_SIZE will be contiguous. Switch kmalloc_array() and
kmalloc_array_node() to use __kmalloc()/lkpi___kmalloc_node() as their
underlying implementation which now does provide that guarantee.
Likewise adjust kcalloc_node() to use kmalloc_array_node().
This means we only have two (plain + _node) underlying allocation
routines for the entire category of functions.
Also adjust kvmalloc() and kvmalloc_array() to be a "mirrored"
implementation to their non-v counterparts. These may return
non-contiguous memory so can use malloc().
Sponsored by: The FreeBSD Foundation
Reviewed by: jhb
Extra thanks to: jhb for helping sorting this out
Differential Revision: https://reviews.freebsd.org/D46657
(cherry picked from commit 1c81ebec74d8a71c770f7835d3bc6e02c13467a0)
In order to allow the allocator to change in the future move it into
the implementation file from being an inline function in the header.
While here factor out the size calculation and add a comment as-to why
this is done. We will need the size (_s) in the future to make a
decision on how to allocate.
Sponsored by: The FreeBSD Foundation
Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D45815
(cherry picked from commit 1f7df757017404011732196e65981d9325f7a89f)
Amdgpu driver does a lot of memory allocations in FPU-protected sections
of code for certain display cores, e.g. for DCN30. This does not work
on FreeBSD as its malloc function can not be run within a critical
section. Check this condition and temporally exit from FPU-protected
context to workaround issue and reduce source code patching.
Sponsored by: Serenity Cyber Security, LLC
Reviewed by: manu (previous version)
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D42822
(cherry picked from commit c0b8047bdc13040eafb162c4b7b5dba11034ff4b)
DRM drivers set some pointers to `ZERO_SIZE_PTR` directly (without
allocating anything), to treat pointers which were "initialized" (set to
`ZERO_SIZE_PTR`) with no memory allocation like really allocated
pointers. NULL isn't used because it represents a third state.
Reviewed by: emaste, manu
Approved by: emaste, manu
Differential Revision: https://reviews.freebsd.org/D39055
On Linux, the `kmalloc()` family of functions returns a special value if
the size of the allocation is zero. This macro verifies if the pointer
is NULL (the allocation failed) or the size is 0 (the allocation was not
performed AFAIU). This special value can be passed to `kfree()`.
On FreeBSD, our `malloc(9)` functions don't return a special value for
0-size allocations. Therefore we can simply compare the result against
NULL.
Reviewed by: manu
Approved by: manu
Differential Revision: https://reviews.freebsd.org/D37367
In FreeBSD, this is a wrapper on top of `realloc()`.
V2: Check if `n * size` would overflow and return `NULL` if that's the
case. Suggested by hselasky@ and emaste@.
Reviewed by: manu
Approved by: manu
Differential Revision: https://reviews.freebsd.org/D36959
Obsolete it usage but keep for a while for drm-kmod 5.4 compatibility
MFC after: 1 week
Reviewed by: hselasky, manu
Differential Revision: https://reviews.freebsd.org/D33298
as it is required by i915kms driver from Linux kernel v 5.5.
This is done with asynchronous freeing of requested memory areas from
taskqueue thread. As memory to be freed is reused to store linked list
entry, backing UMA zone item size is rounded up to pointer size.
While here, make struct linux_kmem_cache private to LKPI to reduce amount
of BSD headers included by linux/slab.h and switch RCU code to usage of
LKPI's linux_irq_work_tq taskqueue to avoid injection of current into
system-wide taskqueue_fast thread context.
Submitted by: nc (initial version for drm-kmod)
Reviewed by: manu, nc
Differential revision: https://reviews.freebsd.org/D30760
Summary:
Linux's irq_work queue was created for asynchronous execution of code from contexts where spin_lock's are not available like "hardware interrupt context". FreeBSD's fast taskqueues was created for the same purposes.
Drm-kmod 5.4 uses irq_work_queue() at least in one place to schedule execution of task/work from the critical section that triggers following INVARIANTS-induced panic:
```
panic: acquiring blockable sleep lock with spinlock or critical section held (sleep mutex) linuxkpi_short_wq @ /usr/src/sys/kern/subr_taskqueue.c:281
cpuid = 6
time = 1605048416
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe006b538c90
vpanic() at vpanic+0x182/frame 0xfffffe006b538ce0
panic() at panic+0x43/frame 0xfffffe006b538d40
witness_checkorder() at witness_checkorder+0xf3e/frame 0xfffffe006b538f00
__mtx_lock_flags() at __mtx_lock_flags+0x94/frame 0xfffffe006b538f50
taskqueue_enqueue() at taskqueue_enqueue+0x42/frame 0xfffffe006b538f70
linux_queue_work_on() at linux_queue_work_on+0xe9/frame 0xfffffe006b538fb0
irq_work_queue() at irq_work_queue+0x21/frame 0xfffffe006b538fd0
semaphore_notify() at semaphore_notify+0xb2/frame 0xfffffe006b539020
__i915_sw_fence_notify() at __i915_sw_fence_notify+0x2e/frame 0xfffffe006b539050
__i915_sw_fence_complete() at __i915_sw_fence_complete+0x63/frame 0xfffffe006b539080
i915_sw_fence_complete() at i915_sw_fence_complete+0x8e/frame 0xfffffe006b5390c0
dma_i915_sw_fence_wake() at dma_i915_sw_fence_wake+0x4f/frame 0xfffffe006b539100
dma_fence_signal_locked() at dma_fence_signal_locked+0x105/frame 0xfffffe006b539180
dma_fence_signal() at dma_fence_signal+0x72/frame 0xfffffe006b5391c0
dma_fence_is_signaled() at dma_fence_is_signaled+0x80/frame 0xfffffe006b539200
dma_resv_add_shared_fence() at dma_resv_add_shared_fence+0xb3/frame 0xfffffe006b539270
i915_vma_move_to_active() at i915_vma_move_to_active+0x18a/frame 0xfffffe006b5392b0
eb_move_to_gpu() at eb_move_to_gpu+0x3ad/frame 0xfffffe006b539320
eb_submit() at eb_submit+0x15/frame 0xfffffe006b539350
i915_gem_do_execbuffer() at i915_gem_do_execbuffer+0x7d4/frame 0xfffffe006b539570
i915_gem_execbuffer2_ioctl() at i915_gem_execbuffer2_ioctl+0x1c1/frame 0xfffffe006b539600
drm_ioctl_kernel() at drm_ioctl_kernel+0xd9/frame 0xfffffe006b539670
drm_ioctl() at drm_ioctl+0x5cd/frame 0xfffffe006b539820
linux_file_ioctl() at linux_file_ioctl+0x323/frame 0xfffffe006b539880
kern_ioctl() at kern_ioctl+0x1f4/frame 0xfffffe006b5398f0
sys_ioctl() at sys_ioctl+0x12a/frame 0xfffffe006b5399c0
amd64_syscall() at amd64_syscall+0x121/frame 0xfffffe006b539af0
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe006b539af0
--- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x800a6f09a, rsp = 0x7fffffffe588, rbp = 0x7fffffffe640 ---
KDB: enter: panic
```
Here, the dma_resv_add_shared_fence() performs a critical_enter() and following call of schedule_work() from semaphore_notify() triggers 'acquiring blockable sleep lock with spinlock or critical section held' panic.
Switching irq_work implementation to fast taskqueue fixes the panic for me.
Other report with the similar bug: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=247166
Reviewed By: hselasky
Differential Revision: https://reviews.freebsd.org/D27171
In Linux, ksize() gets the actual amount of memory allocated for a given
object. This commit adds malloc_usable_size() to FreeBSD KPI which does
the same. It also maps LinuxKPI ksize() to newly created function.
ksize() function is used by drm-kmod.
Reviewed by: hselasky, kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D26215
Put large functions into linux_slab.c instead of declaring them static
inline.
Add support for more memory allocation wrappers like kmalloc_array()
and __vmalloc().
Make sure either the M_WAITOK or the M_NOWAIT flag is set and mask
away unused memory allocation flags before calling FreeBSD's malloc()
routine.
Move kmalloc_node() definition to slab.h where it belongs.
Implement support for the SLAB_DESTROY_BY_RCU feature when creating a
kmem_cache which basically means kmem_cache memory is freed using
call_rcu().
MFC after: 1 week
Sponsored by: Mellanox Technologies
conflict with the opensolaris kernel module.
This patch solves a problem where the kernel linker will incorrectly
resolve opensolaris kmem_xxx() functions as linuxkpi ones, which leads
to a panic when these functions are used.
Submitted by: gallatin @
Sponsored by: Mellanox Technologies
MFC after: 1 week
- Move all files related to the LinuxKPI into sys/compat/linuxkpi and
its subfolders.
- Update sys/conf/files and some Makefiles to use new file locations.
- Added description of COMPAT_LINUXKPI to sys/conf/NOTES which in turn
adds the LinuxKPI to all LINT builds.
- The LinuxKPI can be added to the kernel by setting the
COMPAT_LINUXKPI option. The OFED kernel option no longer builds the
LinuxKPI into the kernel. This was done to keep the build rules for
the LinuxKPI in sys/conf/files simple.
- Extend the LinuxKPI module to include support for USB by moving the
Linux USB compat from usb.ko to linuxkpi.ko.
- Bump the FreeBSD_version.
- A universe kernel build has been done.
Reviewed by: np @ (cxgb and cxgbe related changes only)
Sponsored by: Mellanox Technologies
2015-10-29 08:28:39 +00:00
Renamed from sys/ofed/include/linux/slab.h (Browse further)
