OpenZFS release 2.2.7
Notable upstream pull request merges:
#15457022bf8637 Increase L2ARC write rate and headroom
#155271ba5b2ef8 freebsd: remove __FBSDID macro use
#157929e7af55e8 Add custom debug printing for your asserts
#15793a1ea40af8 libzfs: use zfs_strerror() in place of strerror()
#16181 -multiple zdb/ztest: improve and harmonise crash output
#16208e5e4957a5 Allow block cloning to be interrupted by a signal
#16210ba4e582a6 FreeBSD: Add const qualifier to members of struct
opensolaris_utsname
#16225a6198f34b Simplify issig()
#1633525ec9a903 zdb: fix BRT dump
#16364cf80a803d zvol: ensure device minors are properly cleaned up
#16404384b53be8 FreeBSD: Add missing memory reclamation accounting
#16492 -multiple Split "shared" userspace & kernel code into separate files
#16496f1694496a zfs_file: rename zfs_file_fallocate to zfs_file_deallocate
#16511bc0d89bfc Fix an uninitialized data access
#165292dc8529d9 Fix handling of DNS names with '-' in them for sharenfs
#1653930ea44296 zfs_log: add flex array fields to log record structs
#16546098789216 Evicting too many bytes from MFU metadata
#1655154278533a Reduce and handle EAGAIN errors on AIO label reads
#1655484ad1d536 FreeBSD: restore zfs_znode_update_vfs()
#1656521c40e6d9 FreeBSD: Sync taskq_cancel_id() returns with Linux
#1656748482bb2f Properly release key in spa_keystore_dsl_key_hold_dd()
#16584e8f4592a1 Avoid computing strlen() inside loops
#16605acc8a3186 ARC: Cache arc_c value during arc_evict()
#16650fc60e0c6e freebsd: Use compiler.h from FreeBSD's base's linuxkpi
#16667b32b35cea zdb: show bp in uberblock dump
#166841f5e1b919 Pack dmu_buf_impl_t by 16 bytes
#1668873b3e8ace Fix gcc uninitialized warning in FreeBSD zio_crypt.c
#16690727506c94 On the first vdev open ignore impossible ashift hints
#16692d83cd5307 zdb: add extra -T flag to show histograms of BRT refcounts
#1669382ab837a0 Fix gcc unused value warning in FreeBSD simd.h
#167402bba6e3c5 BRT: Don't call brt_pending_remove() on holes/embedded
#16801299da6ace Fix race in libzfs_run_process_impl
Obtained from: OpenZFS
OpenZFS commit: e269af1b3c
OpenZFS tag: zfs-2.2.7
OpenZFS upstream refactored the asm versions of sha2 to be usable on all
32-bit arm flavors, so it is not necessary to limit this to armv6 and
armv7.
Suggested by: jhb
(cherry picked from commit bc6051a060)
Similar to fb17dfa0c8, fix libicp_rescue to include asm versions of
sha2 on armv6, to unbreak the build of rescue.
Fixes: 3494f7c019
(cherry picked from commit 1a4a9a5057)
The following upstream commit:
727497ccdf module/icp/asm-arm/sha2: enable non-SIMD asm kernels on armv5/6
does indeed enable sha2 asm for earlier arm CPUs, but since libicp's
Makefile was not updated, this leads to:
ld: error: undefined reference due to --no-allow-shlib-undefined: zfs_sha256_block_armv7
Fix it by compiling sha256-armv7.S and sha512-armv7.S for armv6 too.
Fixes: 3494f7c019
(cherry picked from commit fb17dfa0c8)
This is similar to other translators and will be used in static
probes where the interface is not known.
Reviewed by: markj
Sponsored by: Netflix, Inc.
Differential Revision: https://reviews.freebsd.org/D43728
(cherry picked from commit e00fd24262)
This option can be used to specify a format to use in DTrace output.
The following formats are supported:
- json
- xml
- html
- none (default DTrace output)
This is implemented using libxo and integrated into libdtrace. Client
code only works with the following API:
- dtrace_oformat_setup(dtrace_hdl_t *) -- to be called when output is starting.
- dtrace_oformat_teardown(dtrace_hdl_t *) -- to be called when output is finished
- dtrace_oformat(dtrace_hdl_t *) -- check if oformat is enabled.
- dtrace_set_outfp(FILE *) -- sets the output file for oformat.
- Ensure that oformat is correctly checked in the drop handler and record
processing callbacks.
This commit also adds tests which check if the generated output is
valid (JSON, XML) and extends the dtrace(1) describing the structured output.
Reviewed by: markj
Discussed with: phil
MFC after: 2 months
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D41745
(cherry picked from commit 93f27766a7)
The header gives an offset in 32-bit words, and the translator is
supposed to convert that to a byte count. But, the conversion was
incorrect.
Reviewed by: tuexen, rscheff
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D43264
(cherry picked from commit c3268c23de)
OpenZFS release 2.2.1
Notable upstream pull request merges:
#143786d693e20a Large sync writes perform worse with slog
#1524378fd79eac Add zfs_prepare_disk script for disk firmware install
#15356b76724ae4 FreeBSD: Improve taskq wrapper
#15360f9a9aea12 Add mutex_enter_interruptible() for interruptible
sleeping IOCTLs
#15371e82e68400 DMU: Do not pre-read holes during write
#1539079f7de575 Remove lock from dsl_pool_need_dirty_delay()
#153976e41aca51 Trust ARC_BUF_SHARED() more
#15402eaa62d995 Properly pad struct tx_cpu to cache line
#154051cc1bf4fa Set spa_ccw_fail_time=0 when expanding a vdev
#15416edebca5df FreeBSD: taskq: Remove unused declaration
#154517aef672b7 Read prefetched buffers from L2ARC
#15452bd7a02c25 Tune zio buffer caches and their alignments
#154563ec4ea68d Unify arc_prune_async() code
#15465459c99ff2 Fix block cloning between unencrypted and encrypted
datasets
#1547876663fe37 Fix accounting error for pending sync IO ops in zpool
iostat
#1552987e9e8286 Add a tunable to disable BRT support
Obtained from: OpenZFS
OpenZFS commit: 55dd24c4cc
OpenZFS tag: zfs-2.2.1
Notable upstream pull request merges:
#15024 Add missed DMU_PROJECTUSED_OBJECT prefetch
#15029 Do not request data L1 buffers on scan prefetch
#15036 FreeBSD: catch up to __FreeBSD_version 1400093
#15039 Fix raw receive with different indirect block size
#15047 FreeBSD: Fix build on stable/13 after 1302506
#15049 Fix the ZFS checksum error histograms with larger record sizes
#15052 Reduce bloat in ereport.fs.zfs.checksum events
#15056 Avoid extra snprintf() in dsl_deadlist_merge()
#15061 Ignore pool ashift property during vdev attachment
#15063 Don't panic if setting vdev properties is unsupported for this
vdev type
#15067 spa_min_alloc should be GCD, not min
#15071 Add explicit prefetches to bpobj_iterate()
#15072 Adjust prefetch parameters
#15079 set autotrim default to 'off' everywhere
#15080 ZIL: Fix config lock deadlock
#15088 metaslab: tuneable to better control force ganging
#15096 Avoid waiting in dmu_sync_late_arrival()
#15097 BRT should return EOPNOTSUPP
#15103 Remove zl_issuer_lock from zil_suspend()
#15107 Remove fastwrite mechanism
#15113 libzfs: sendrecv: send_progress_thread: handle SIGINFO/SIGUSR1
#15122 ZIL: Second attempt to reduce scope of zl_issuer_lock
#15129 zpool_vdev_remove() should handle EALREADY error return
#15132 ZIL: Replay blocks without next block pointer
#15148 zfs_clone_range should return descriptive error codes
#15153 ZIL: Avoid dbuf_read() before dmu_sync()
#15161 Make zoned/jailed zfsprops(7) make more sense
#15172 copy_file_range: fix fallback when source create on same txg
#15180 Update outdated assertion from zio_write_compress
#15216 Relax error reporting in zpool import and zpool split
#15227 ZIL: Tune some assertions
#15228 ZIL: Revert zl_lock scope reduction
#15233 ZIL: Change ZIOs issue order
Obtained from: OpenZFS
OpenZFS commit: 32949f2560
Approved by: re (gjb)
dtrace stored its PR_RLC and PR_KLC flags in the proc_handle's flags,
where they collided with PATTACH_FORCE and PATTACH_RDONLY, respectively.
Thus, Psetflags(PR_KLC) effectively also set the PATTACH_RDONLY flag.
Since the flags are private to dtrace (at least on FreeBSD), store them in
dtrace's own dt_proc structure instead.
On FreeBSD, either PR_RLC or PR_KLC was always set, so remove code that
would handle the case where neither was set.
Reviewed by: markj
MFC after: 1 week
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D41121
They were previously being defined by cddl/lib/Makefile.inc, and as of
commit 4ae6991228 were being overridden by defaults in bsd.own.mk,
which changed SHLIBDIR to /usr/lib.
Reported by: Domagoj Stolfa <ds815@cam.ac.uk>
Fixes: 4ae6991228 ("dtrace: Add WITH_DTRACE_ASAN")
This option is a blanket for all the DTrace-related software. The option
when enabled passes in -fsanitize=address -fsanitize=undeifned, enabling
ASAN and UBSAN in the following components:
- libdtrace
- dtrace(1)
- lockstat(1)
- plockstat(1)
The option defaults to "no" and is intended as a developer aid.
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D41157
Improve consistency of the field names with tcpsinfo_t:
* Use mss instead of max_seg_size.
* Use lport and rport instead of tcp_localport and tcp_foreignport.
Use t_flags instead of flags to improve consistency with t_flags2.
Add laddr and raddr, since the addresses were missing when compared
to the output of siftr.
Reviewed by: cc
Sponsored by: Netflix, Inc.
Differential Revision: https://reviews.freebsd.org/D40834
dtrace_instr_size() is needed by the forthcoming RISC-V port of kinst,
as well as by libdtrace in D38825 for both amd64 and RISC-V.
Reviewed by: markj, mhorne
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39489
Notable upstream pull request merges:
#12194 Fix short-lived txg caused by autotrim
#13368 ZFS_IOC_COUNT_FILLED does unnecessary txg_wait_synced()
#13392 Implementation of block cloning for ZFS
#13741 SHA2 reworking and API for iterating over multiple implementations
#14282 Sync thread should avoid holding the spa config write lock
when possible
#14283 txg_sync should handle write errors in ZIL
#14359 More adaptive ARC eviction
#14469 Fix NULL pointer dereference in zio_ready()
#14479 zfs redact fails when dnodesize=auto
#14496 improve error message of zfs redact
#14500 Skip memory allocation when compressing holes
#14501 FreeBSD: don't verify recycled vnode for zfs control directory
#14502 partially revert PR 14304 (eee9362a7)
#14509 Fix per-jail zfs.mount_snapshot setting
#14514 Fix data race between zil_commit() and zil_suspend()
#14516 System-wide speculative prefetch limit
#14517 Use rw_tryupgrade() in dmu_bonus_hold_by_dnode()
#14519 Do not hold spa_config in ZIL while blocked on IO
#14523 Move dmu_buf_rele() after dsl_dataset_sync_done()
#14524 Ignore too large stack in case of dsl_deadlist_merge
#14526 Use .section .rodata instead of .rodata on FreeBSD
#14528 ICP: AES-GCM: Refactor gcm_clear_ctx()
#14529 ICP: AES-GCM: Unify gcm_init_ctx() and gmac_init_ctx()
#14532 Handle unexpected errors in zil_lwb_commit() without ASSERT()
#14544 icp: Prevent compilers from optimizing away memset()
in gcm_clear_ctx()
#14546 Revert zfeature_active() to static
#14556 Remove bad kmem_free() oversight from previous zfsdev_state_list
patch
#14563 Optimize the is_l2cacheable functions
#14565 FreeBSD: zfs_znode_alloc: lock the vnode earlier
#14566 FreeBSD: fix false assert in cache_vop_rmdir when replaying ZIL
#14567 spl: Add cmn_err_once() to log a message only on the first call
#14568 Fix incremental receive silently failing for recursive sends
#14569 Restore ASMABI and other Unify work
#14576 Fix detection of IBM Power8 machines (ISA 2.07)
#14577 Better handling for future crypto parameters
#14600 zcommon: Refactor FPU state handling in fletcher4
#14603 Fix prefetching of indirect blocks while destroying
#14633 Fixes in persistent error log
#14639 FreeBSD: Remove extra arc_reduce_target_size() call
#14641 Additional limits on hole reporting
#14649 Drop lying to the compiler in the fletcher4 code
#14652 panic loop when removing slog device
#14653 Update vdev state for spare vdev
#14655 Fix cloning into already dirty dbufs
#14678 Revert "Do not hold spa_config in ZIL while blocked on IO"
Obtained from: OpenZFS
OpenZFS commit: 431083f75b
For the TCP protocol inpcb storage specify allocation size that would
provide space to most of the data a TCP connection needs, embedding
into struct tcpcb several structures, that previously were allocated
separately.
The most import one is the inpcb itself. With embedding we can provide
strong guarantee that with a valid TCP inpcb the tcpcb is always valid
and vice versa. Also we reduce number of allocs/frees per connection.
The embedded inpcb is placed in the beginning of the struct tcpcb,
since in_pcballoc() requires that. However, later we may want to move
it around for cache line efficiency, and this can be done with a little
effort. The new intotcpcb() macro is ready for such move.
The congestion algorithm data, the TCP timers and osd(9) data are
also embedded into tcpcb, and temprorary struct tcpcb_mem goes away.
There was no extra allocation here, but we went through extra pointer
every time we accessed this data.
One interesting side effect is that now TCP data is allocated from
SMR-protected zone. Potentially this allows the TCP stacks or other
TCP related modules to utilize that for their own synchronization.
Large part of the change was done with sed script:
s/tp->ccv->/tp->t_ccv./g
s/tp->ccv/\&tp->t_ccv/g
s/tp->cc_algo/tp->t_cc/g
s/tp->t_timers->tt_/tp->tt_/g
s/CCV\(ccv, osd\)/\&CCV(ccv, t_osd)/g
Dependency side effect is that code that needs to know struct tcpcb
should also know struct inpcb, that added several <netinet/in_pcb.h>.
Differential revision: https://reviews.freebsd.org/D37127
It is useful to have zfs utilities and lib in a separate package as
it allow users to create image that can support ZFS (i.e. not with
WITHOUT_ZFS in src.conf set) without bloating the default image with
all zfs tools (for example for jails).
Differential Revision: https://reviews.freebsd.org/D36225
kinst does not instantiate its probes automatically, it only does so on
demand via an ioctl interface implemented by /dev/kinst. This change
modifies libdtrace to perform that work when the script references the
kinst provider, similar to the way pid provider probes are implemented.
Reviewed by: markj
MFC after: 3 months
Sponsored by: Google, Inc. (GSoC 2022)
Differential Revision: https://reviews.freebsd.org/D36852
The debug traces for reference counting in ZFS use the pointer of the
owning object as a "tag" for references to check that when an object
drops a reference it had actually held one. In a couple of places ZFS
drops references after freeing the owning object. In userland GCC
realizes this is a use after free. However, since only the value of
the pointer is used, and it isn't indirected, the use is harmless.
Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D36818
Notable upstream pull request merges:
#13725 Fix BLAKE3 tuneable and module loading on Linux and FreeBSD
#13756 FreeBSD: Organize sysctls
#13773 FreeBSD: add kqfilter support for zvol cdev
#13781 Importing from cachefile can trip assertion
#13794 Apply arc_shrink_shift to ARC above arc_c_min
#13798 Improve too large physical ashift handling
#13799 Revert "Avoid panic with recordsize > 128k, raw sending and
no large_blocks"
#13802 Add zfs.sync.snapshot_rename
#13831 zfs_enter rework
#13855 zfs recv hangs if max recordsize is less than received
recordsize
Obtained from: OpenZFS
OpenZFS commit: c629f0bf62
Add an option to enable/disable DTrace without disabling ZFS. New
architectures such as CHERI may support ZFS before they support DTrace
and the old model of WITHOUT_CDDL disabling both wasn't helpful.
For compatiblity, the CDDL option remains and WITHOUT_CDDL implies
WITHOUT_DTRACE. WITHOUT_DTRACE also implies WITHOUT_CTF.
As part of this change, largely convert cddl/*/Makefile to using the
more compact SUBDIR.${MK_<FOO>}+= form rather than using intermediate
variables.
Reviewed by: markj
Obtained from: CheriBSD
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D35718
This is a repeat of 13cee53cef35 ("libicp: Fix build for powerpc64le")
but for the rescue version of the library.
Fixes: 1f1e2261e3 ("zfs: merge openzfs/zfs@deb121309")
The C sources use the assembly routines for little-endian powerpc64, and
the assembly files have ppc64le in their names, but the guard here was
for big-endian powerpc64.
Fixes: 1f1e2261e3 ("zfs: merge openzfs/zfs@deb121309")
Notable upstream pull request merges:
#9078: log xattr=sa create/remove/update to ZIL
#11919: Cross-platform xattr user namespace compatibility
#13014: Report dnodes with faulty bonuslen
#13016: FreeBSD: Fix zvol_cdev_open locking
#13019: spl: Don't check FreeBSD rwlocks for double initialization
#13027: Fix clearing set-uid and set-gid bits on a file when
replying a write
#13031: Add enumerated vdev names to 'zpool iostat -v' and
'zpool list -v'
#13074: Enable encrypted raw sending to pools with greater ashift
#13076: Receive checks should allow unencrypted child datasets
#13098: Avoid dirtying the final TXGs when exporting a pool
#13172: Fix ENOSPC when unlinking multiple files from full pool
Obtained from: OpenZFS
OpenZFS commit: a86e089415
Use it instead of the existing ctf.h from OpenSolaris. This makes it
easier to use CTF in the core kernel, and to extend the CTF format to
support wider type IDs.
The imported ctf.h is modified to depend only on _types.h, and also to
provide macros which use the "parent" bit of a type ID to refer to types
in a parent CTF container.
No functional change intended.
Reviewed by: Domagoj Stolfa, emaste
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34358
Parts of zstd, used in openzfs and other places, trigger a new clang 14
-Werror warning:
```
sys/contrib/zstd/lib/decompress/huf_decompress.c:889:25: error: use of bitwise '&' with boolean operands [-Werror,-Wbitwise-instead-of-logical]
(BIT_reloadDStreamFast(&bitD1) == BIT_DStream_unfinished)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
```
While the warning is benign, it should ideally be fixed upstream and
then vendor-imported, but for now silence it selectively.
MFC after: 3 days
Notable upstream pull request merges:
#12766 Fix error propagation from lzc_send_redacted
#12805 Updated the lz4 decompressor
#12851 FreeBSD: Provide correct file generation number
#12857 Verify dRAID empty sectors
#12874 FreeBSD: Update argument types for VOP_READDIR
#12896 Reduce number of arc_prune threads
#12934 FreeBSD: Fix zvol_*_open() locking
#12947 lz4: Cherrypick fix for CVE-2021-3520
#12961 FreeBSD: Fix leaked strings in libspl mnttab
#12964 Fix handling of errors from dmu_write_uio_dbuf() on FreeBSD
#12981 Introduce a flag to skip comparing the local mac when raw sending
#12985 Avoid memory allocations in the ARC eviction thread
Obtained from: OpenZFS
OpenZFS commit: 17b2ae0b24
In 9fae47666 zfsd got a libspl dependency to avoid undefined references.
However that workaround did not help external consumers of libzfs_core.
Fix all missing dependencies lld 13 and the rtld complain about.
Reviewed by: freqlabs, markj
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D32521
Notable upstream pull request merges:
#12392 Avoid panic in case of pool errors and missing L2ARC
#12448 skip snapshot in zfs_iter_mounted()
#12516 Fix NFS and large reads on older kernels
#12533 Fail invalid incremental recursive send gracefully
#12569 FreeBSD: Really zero the zero page
#12575 Reject zfs send -RI with nonexistent fromsnap
#12602 Correct refcount_add in dmu_zfetch
#12650 zpool should call zfs_nicestrtonum() with non-NULL handle
Obtained from: OpenZFS
OpenZFS commit: ec64fdb93d
