From 9bf937a4854cf27c0d420019af781607b62e3c46 Mon Sep 17 00:00:00 2001 From: Thomas Waldmann Date: Sat, 3 Jan 2026 19:31:54 +0100 Subject: [PATCH] mount: performance improvement MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The efficiency difference between `meta.extend(bytes(N))` and `meta = meta + bytes(N)` stems from how Python manages memory and objects during these operations. - **`bytearray.extend()`**: This is an **in-place** operation. If the current memory block allocated for the `bytearray` has enough extra capacity (pre-allocated space), Python simply writes the new bytes into that space and updates the length. If it needs more space, it uses `realloc()`, which can often expand the existing memory block without moving the entire data set to a new location. - **Concatenation (`+`)**: This creates a **completely new** `bytearray` object. It allocates a new memory block large enough to hold the sum of both parts, copies the contents of `meta`, copies the contents of `bytes(N)`, and then reassigns the variable `meta` to this new object. - **`bytearray.extend()`**: In the best case (when capacity exists), it is **O(K)**, where K is the number of bytes being added. In the worst case (reallocation), it is **O(N + K)**, but Python uses an over-allocation strategy (growth factor) that amortizes this cost, making it significantly faster on average. - **Concatenation (`+`)**: It is always **O(N + K)** because it must copy the existing `N` bytes every single time. As the `bytearray` grows larger (e.g., millions of items in a backup), this leads to **O(N²)** total time complexity across multiple additions, as you are repeatedly copying an ever-growing buffer. - Concatenation briefly requires memory for **both** the old buffer and the new buffer simultaneously before the old one is garbage collected. This increases the peak memory usage of the process. - `extend()` is more memory-efficient as it minimizes the need for multiple large allocations and relies on the underlying memory manager's ability to resize buffers efficiently. In the context of `borg mount`, where `meta` can grow to be many megabytes or even gigabytes for very large repositories, using concatenation causes a noticeable slowdown as the number of archives or files increases, whereas `extend()` remains performant. --- src/borg/fuse.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/borg/fuse.py b/src/borg/fuse.py index 0d6bc13fa..fe23c4e37 100644 --- a/src/borg/fuse.py +++ b/src/borg/fuse.py @@ -177,7 +177,7 @@ class ItemCache: for key, (csize, data) in zip(archive_item_ids, self.decrypted_repository.get_many(archive_item_ids)): # Store the chunk ID in the meta-array if write_offset + 32 >= len(meta): - self.meta = meta = meta + bytes(self.GROW_META_BY) + meta.extend(bytes(self.GROW_META_BY)) meta[write_offset : write_offset + 32] = key current_id_offset = write_offset write_offset += 32 @@ -215,7 +215,7 @@ class ItemCache: msgpacked_bytes = b"" if write_offset + 9 >= len(meta): - self.meta = meta = meta + bytes(self.GROW_META_BY) + meta.extend(bytes(self.GROW_META_BY)) # item entries in the meta-array come in two different flavours, both nine bytes long. # (1) for items that span chunks: