upstream/nextcloud
1
0
Fork 0
mirror of https://github.com/nextcloud/server.git synced 2026-02-27 20:10:37 -05:00
Code Issues Projects Releases Packages Wiki Activity Actions 34
81385 commits 688 branches 1186 tags 7 GiB
Commit graph

300 commits

Author SHA1 Message Date
Louis Chmn
3cfc34ecfa fix(S3): Only append streams if non-seekable 
Later, when we rewind the stream in `writeMultiPart` during retry, both streams were rewinded, so the resulting stream was bigger than expected.

Inspired by https://github.com/aws/aws-sdk-php/blob/master/src/S3/ObjectUploader.php#L136-L146

Signed-off-by: Louis Chmn <louis@chmn.me>
 2025-10-10 12:43:19 +00:00
Robin Appelman
6842d0746d
Merge pull request #55288 from nextcloud/backport/52816/stable31 
[stable31] feat: move primary object store configuration to a single place
 2025-09-30 10:35:42 +02:00
yemkareems
d615109126 fix: small typo fix 
Signed-off-by: yemkareems <yemkareems@gmail.com>
 2025-09-29 12:24:43 +00:00
Robin Appelman
7162c3e8ed
feat: move primary object store configuration to a single place 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2025-09-24 18:39:21 +02:00
Robin Appelman
05e64418a7
fix: better object store write error propagation 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2025-08-07 13:48:43 +02:00
Robin Appelman
d1af0f34a2
fix: always do stream counting for object store upload 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2025-08-07 13:48:42 +02:00
Robin Appelman
902cb3dbb9
fix: validate written size for s3 multipart uploads 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2025-08-07 13:48:42 +02:00
Robin Appelman
e483387189
feat: more generic way of passing metadata to object storage backends for new objects 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2025-08-07 13:48:37 +02:00
Kent Delante
f40d14abac fix(s3): retry failed multipart uploads with decreased concurrency 
Signed-off-by: Kent Delante <kent.delante@proton.me>
 2025-06-20 18:24:16 +08:00
Daniel Calviño Sánchez
019b535868 fix(ObjectStore): Make S3 "connect_timeout" option configurable 
The hardcoded connection timeout of 5 seconds may not be enough in some
cases, so now it is got from the ObjectStore arguments in Nextcloud
configuration, falling back to 5 if not set.

The connection timeout is set in seconds, but decimal precision can be
used for subsecond accuracy (for example, 4.2 for 4200 milliseconds).

Signed-off-by: Daniel Calviño Sánchez <danxuliu@gmail.com>
 2025-06-17 03:30:55 +00:00
Robin Appelman
edfb7b2393 perf: set the folder size in the same query as we create it 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2025-06-02 09:56:29 +00:00
Louis Chemineau
9d26571eb2 fix(S3): Use original folder size during copy 
This prevent having copied folders with a wrongly set size of 0KB.

- Fix https://github.com/nextcloud/server/issues/51916

Signed-off-by: Louis Chemineau <louis@chmn.me>
 2025-06-02 09:56:29 +00:00
Robin Appelman
780766b63c fix: keep IObjectStoreMetaData in private namespace for backports 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2025-05-16 10:41:27 +02:00
Robin Appelman
f0cab800ab feat: add command to list objects 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2025-05-16 10:41:27 +02:00
Robin Appelman
a324dbec2e feat: add command to get object metadata 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2025-05-16 10:41:27 +02:00
Anna Larch
d393e98c9e fix(objectstorage): add retry attempts to S3 connection 
Signed-off-by: Anna Larch <anna@nextcloud.com>
 2025-04-23 18:40:23 +00:00
Louis Chemineau
27ed921ad5 fix: Transfer ownership with S3 as primary 
When using S3 as primary storage, transferring ownership with the `--move` option fail with the following error:

`SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '8-45b963397aa40d4a0063e0d85e4fe7a1' for key 'fs_storage_path_hash'`

The `--move` option moves the entire home folder from one account to another.
The error means that the move failed because the destination folder already exist in `oc_filecache`.

- With S3 as primary storage, folders only exists as entries in `oc_filecache`.
- With S3 as primary storage, `moveFromStorage(...)` only moves the cache entry, as nothing needs to be moved on disk. This cache move does not delete potentially pre-existing destination folder.
- With Local storage, `moveFromStorage(...)` calls `rename(...)` which delete pre-existing folder.

- `transfer(...)`: 687a4d9ac7/apps/files/lib/Service/OwnershipTransferService.php (L112)
- `oneTimeUserSetup(...)`: 687a4d9ac7/lib/private/Files/SetupManager.php (L261-L262)
- `mkdir(...)`: 687a4d9ac7/lib/private/Files/ObjectStore/ObjectStoreStorage.php (L91-L135)
- `moveFromStorage(...)`: 687a4d9ac7/lib/private/Files/ObjectStore/ObjectStoreStorage.php (L635-L636)

Delete pre-existing folder in `moveFromStorage(...)`

Signed-off-by: Louis Chemineau <louis@chmn.me>
 2025-04-16 10:19:17 +02:00
Benjamin Gaussorgues
d056e4dc74 perf(files): faster query to fetch incomplete directories 
Signed-off-by: Benjamin Gaussorgues <benjamin.gaussorgues@nextcloud.com>
 2025-02-13 14:24:44 +00:00
Richard Steinmetz
936ef804cd fix(s3): treat empty sse_c_key as not set 
Co-authored-by: Christian Becker <christian@dabecka.de>
Signed-off-by: Richard Steinmetz <richard@steinmetz.cloud>
 2025-01-29 17:35:52 +00:00
Ferdinand Thiessen
26c93519f2 fix: Harden files scanner for invalid null access 
Co-authored-by: Ferdinand Thiessen <opensource@fthiessen.de>
Co-authored-by: Kate <26026535+provokateurin@users.noreply.github.com>
Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
 2025-01-28 19:49:03 +00:00
Robin Appelman
5896f8c37f
chore: cleanup leftover debug statement 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2024-12-09 15:40:01 +01:00
Robin Appelman
6cf66f95ce
fix: get object size from stream where possible 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2024-12-05 15:16:22 +01:00
Robin Appelman
ca547bb913
fix: don't perform the extra buffering in s3 stream write when the stream size is known 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2024-12-05 14:07:40 +01:00
Robin Appelman
7da8951c57
fix: use php://temp instead of php://memory for multi-part upload buffer 
this should reduce potential memory issues if the limit is set very high

Signed-off-by: Robin Appelman <robin@icewind.nl>
 2024-12-05 14:07:37 +01:00
Robin Appelman
df3a9e221d
fix: throw correct exception type when we can't verify if an s3 bucket exists 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2024-11-26 16:49:55 +01:00
provokateurin
0de4843b73 refactor(Storage): Align all Storage constructors 
Signed-off-by: provokateurin <kate@provokateurin.de>
 2024-10-23 13:24:18 +02:00
provokateurin
f28e74b7a8
refactor(Storage): Make all parameter types strong types 
Signed-off-by: provokateurin <kate@provokateurin.de>
 2024-10-07 15:00:05 +02:00
Robin Appelman
3e12e1e789
fix: rework move into object store to better preserve fileids 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2024-09-27 14:46:08 +02:00
Robin Appelman
888d06dff9
fix: preserve fileid when moving from objectstore to non-objectstore 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2024-09-27 14:45:42 +02:00
Robin Appelman
da21acfb3f
fix: ensure source folder is removed from cache when moving to objectstore 
otherwise this causes confusion down the line as it's contents will be moved to the new cache

Signed-off-by: Robin Appelman <robin@icewind.nl>
 2024-09-27 14:45:42 +02:00
provokateurin
7cdccd058f
fix(Storage): Fix IStorage return types 
Signed-off-by: provokateurin <kate@provokateurin.de>
 2024-09-26 18:13:03 +02:00
Robin Appelman
47d281865e perf: reuse cache info we already have when moving to object store 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2024-09-19 19:06:06 +02:00
provokateurin
9836e9b164
chore(deps): Update nextcloud/coding-standard to v1.3.1 
Signed-off-by: provokateurin <kate@provokateurin.de>
 2024-09-19 14:21:20 +02:00
Robin Appelman
3c0c81bb94 fix: ensure parent folder exists when writing a file to object storage 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2024-09-18 14:11:27 +02:00
Robin Appelman
e0fb25f846 fix: fix object store id for test object store 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2024-09-18 13:54:55 +02:00
Robin Appelman
60f6347c97 fix: cleanup objectstore file_put_content 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2024-09-18 13:54:55 +02:00
Robin Appelman
3e94ee391f fix: create intermediate directories for objectstore moveFromStorage 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2024-09-18 13:54:55 +02:00
Robin Appelman
879fcc4253 fix: get source file before moving the cache item in objectstore moveFromStorage 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2024-09-18 13:54:55 +02:00
Stephan Orbaugh
f25acfb799
Merge pull request #47986 from cfiehe/fix_move_on_same_bucket 
perf(ObjectStoreStorage): Improve (slow) move on same object bucket
 2024-09-17 16:35:19 +02:00
provokateurin
d2d1e3284d
fix(Storage): Fix method docs inheritance 
Signed-off-by: provokateurin <kate@provokateurin.de>
 2024-09-17 10:20:38 +02:00
provokateurin
8ca6fcace7
fix(Storage): Document getOwner() can return false 
Signed-off-by: provokateurin <kate@provokateurin.de>
 2024-09-17 10:10:50 +02:00
Côme Chilliet
492e6997d8
chore: Fix psalm issues, put back private versions of getter in private Storage interface 
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
 2024-09-16 18:11:38 +02:00
Christoph Fiehe
9597072ada perf(ObjectStoreStorage): Improve (slow) move on same object bucket 
This commit fixes the issue #47856. When you upload a file into a group folder and when you use a single S3 bucket as primary storage, the final move operation hangs for a long time. In the background, Nextcloud initiates a copy-delete sequence from the bucket into the bucket, with causes a lot unnecessary overhead. Nextcloud thinks that the file must be imported to another storage and does not recognize that everything is done on the same object bucket. In that case, the import step can be completely skipped, which saves time, network bandwidth and reduces the load on the object storage.

The behavior improves a lot with https://github.com/nextcloud/server/pull/46013. However, there are still some put messages that are being sent to the object storage when you use an object storage as primary storage and upload files into a group folder.

Co-authored-by: Kate <26026535+provokateurin@users.noreply.github.com>
Signed-off-by: Christoph Fiehe <c.fiehe@eurodata.de>
 2024-09-16 15:11:09 +02:00
provokateurin
7c1c02d6a5
fix(S3ConfigTrait): Allow proxy field to take false 
Signed-off-by: provokateurin <kate@provokateurin.de>
 2024-09-15 15:41:58 +02:00
Robin Appelman
508d8e7c01
fix: make swift connect exception message more informative 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2024-09-09 17:15:01 +02:00
Julius Härtl
6d2253e692
Merge pull request #46013 from nextcloud/obj-store-move-from-storage-preserve-fileid 
fix: write object to the correct urn when moving from another storage to object store
 2024-08-26 13:47:27 +02:00
Daniel Kesselberg
af6de04e9e
style: update codestyle for coding-standard 1.2.3 
Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>
 2024-08-25 19:34:58 +02:00
Robin Appelman
bd740ac0b0
fix: write object to the correct urn when moving from another storage to object store 
Signed-off-by: Robin Appelman <robin@icewind.nl>
 2024-08-23 15:22:16 +02:00
Git'Fellow
c84e76a749 fix(s3): Don't wait indefinitely for S3 to return 
Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>

fix: lint

Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>

fix: use AwsException

Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>

fix: Throw on connection failure

Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>

fix: Wrap all in try catch block

Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>

fix: use RequestTimeout error message

Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>

log: use OCP Server class

Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>

fix: Handle connect timeout only

Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>

fix: Handle errors more generically

Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
 2024-07-05 16:24:32 +02:00
Ferdinand Thiessen
ae1f5cc56d
fix(S3): Adjust usage of guzzle promise 
`Promise\promise_for` was deprecated and is now removed and replaced with the static API (`Create::promiseFor`).

Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
 2024-06-16 15:54:44 +02:00