Commit graph

310 commits

Author SHA1 Message Date
provokateurin
8ca6fcace7
fix(Storage): Document getOwner() can return false
Signed-off-by: provokateurin <kate@provokateurin.de>
2024-09-17 10:10:50 +02:00
Côme Chilliet
492e6997d8
chore: Fix psalm issues, put back private versions of getter in private Storage interface
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2024-09-16 18:11:38 +02:00
Christoph Fiehe
9597072ada perf(ObjectStoreStorage): Improve (slow) move on same object bucket
This commit fixes the issue #47856. When you upload a file into a group folder and when you use a single S3 bucket as primary storage, the final move operation hangs for a long time. In the background, Nextcloud initiates a copy-delete sequence from the bucket into the bucket, with causes a lot unnecessary overhead. Nextcloud thinks that the file must be imported to another storage and does not recognize that everything is done on the same object bucket. In that case, the import step can be completely skipped, which saves time, network bandwidth and reduces the load on the object storage.

The behavior improves a lot with https://github.com/nextcloud/server/pull/46013. However, there are still some put messages that are being sent to the object storage when you use an object storage as primary storage and upload files into a group folder.

Co-authored-by: Kate <26026535+provokateurin@users.noreply.github.com>
Signed-off-by: Christoph Fiehe <c.fiehe@eurodata.de>
2024-09-16 15:11:09 +02:00
provokateurin
7c1c02d6a5
fix(S3ConfigTrait): Allow proxy field to take false
Signed-off-by: provokateurin <kate@provokateurin.de>
2024-09-15 15:41:58 +02:00
Robin Appelman
508d8e7c01
fix: make swift connect exception message more informative
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-09-09 17:15:01 +02:00
Julius Härtl
6d2253e692
Merge pull request #46013 from nextcloud/obj-store-move-from-storage-preserve-fileid
fix: write object to the correct urn when moving from another storage to object store
2024-08-26 13:47:27 +02:00
Daniel Kesselberg
af6de04e9e
style: update codestyle for coding-standard 1.2.3
Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>
2024-08-25 19:34:58 +02:00
Robin Appelman
bd740ac0b0
fix: write object to the correct urn when moving from another storage to object store
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-08-23 15:22:16 +02:00
Git'Fellow
c84e76a749 fix(s3): Don't wait indefinitely for S3 to return
Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>

fix: lint

Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>

fix: use AwsException

Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>

fix: Throw on connection failure

Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>

fix: Wrap all in try catch block

Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>

fix: use RequestTimeout error message

Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>

log: use OCP Server class

Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>

fix: Handle connect timeout only

Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>

fix: Handle errors more generically

Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
2024-07-05 16:24:32 +02:00
Ferdinand Thiessen
ae1f5cc56d
fix(S3): Adjust usage of guzzle promise
`Promise\promise_for` was deprecated and is now removed and replaced with the static API (`Create::promiseFor`).

Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
2024-06-16 15:54:44 +02:00
Josh
63886147ba fix(ObjectStore): handle empty S3 hostname
Fixes #45637

The support for s3-accelerate added in #44496 introduced a regression in AWS S3 environments when `hostname` is blank (which is a valid configuration w/ AWS since the hostname gets auto-generated).

Signed-off-by: Josh <josh.t.richards@gmail.com>
2024-06-12 15:43:20 -04:00
Andy Scherzinger
dae7c159f7
chore: Add SPDX header
Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
2024-05-24 13:11:22 +02:00
Robin Appelman
40d9a3fcab
feat: add option to enable s3 acceleration
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-04-26 14:40:24 +02:00
Joas Schilling
cdb40a3110
fix(openstack): Fix psalm issue with updated openstack version
Signed-off-by: Joas Schilling <coding@schilljs.com>
2024-04-24 07:01:57 +02:00
Côme Chilliet
ec5133b739 fix: Apply new coding standard to all files
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2024-04-02 14:16:21 +02:00
Ferdinand Thiessen
7880086de8
fix(S3): Adjust typing for 32bit compatibility
Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
2024-03-26 13:07:53 +01:00
Ferdinand Thiessen
32dee2f84a
feat(ObjectStore): Make S3 MultipartUpload concurrency configurable
Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
2024-03-21 16:25:49 +01:00
Ferdinand Thiessen
69c5e8ebde
fix(ObjectStore): Use common S3 trait so config and connection trait members are in sync
Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
2024-03-21 16:25:42 +01:00
Joas Schilling
951c330d39
Revert "don't perform an explicit setup when using the file cache" 2024-03-07 14:38:02 +01:00
John Molakvoæ
b7088e7071
Merge pull request #32163 from nextcloud/file-cache-gc-no-full-setup
don't perform an explicit setup when using the file cache
2024-03-07 11:09:19 +01:00
Robin Appelman
4fba4cd14c fix: fix user folder init
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-03-06 17:52:24 +01:00
Robin Appelman
c54f7bd761 fix: disable the internal s3sdk multi part copy logic
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-03-06 15:32:14 +01:00
Côme Chilliet
c0ce272e9c chore: Migrate away from OC::$server->getLogger
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2024-02-13 17:32:30 +01:00
Thomas Citharel
30798eb6c2
refactor(objectstorage): cleanup types
Signed-off-by: Thomas Citharel <tcit@tcit.fr>
2024-02-02 14:59:21 +01:00
Thomas Citharel
3be3dbdb3b
refactor(objectstorage): move $user property from ObjectStoreStorage class to HomeObjectStoreStorage
Signed-off-by: Thomas Citharel <tcit@tcit.fr>
2024-02-02 14:59:21 +01:00
Thomas Citharel
4c618c4bf1
fix(objectstorage): cleanup HomeObjectStoreStorage
Fix invalid signature for getUser() method and change occurences of OC\User\User for OCP\IUser

Signed-off-by: Thomas Citharel <tcit@tcit.fr>
2024-02-02 14:59:20 +01:00
Robin Appelman
b777304a77
also updated parent etags when a changed etag is detected during scanning
Signed-off-by: Robin Appelman <robin@icewind.nl>
2024-01-31 18:06:18 +01:00
Julius Härtl
e4054370b1
fix(s3): Add config option to disable multipart copy for certain s3 providers
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2023-12-28 15:31:38 +01:00
Robin Appelman
1043c21b35
only do a multipart s3 copy when above the regular copy limit
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-12-28 15:31:37 +01:00
Ferdinand Thiessen
5172baaf8b
fix(ObjectStore): Make copying behavior consistent with local storage
Drop file permissions on copy like we do on local storage.

Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
2023-11-17 12:54:37 +01:00
Tobias Zimmerer
cdf96fab92 fix(s3): support SSE-C headers for the MultipartCopy call
Signed-off-by: Tobias Zimmerer <3228193+ir0nhide@users.noreply.github.com>
2023-11-14 21:19:59 +01:00
Hamid Dehnavi
ea06cf2f39 Convert isset ternary to null coalescing operator
Signed-off-by: Hamid Dehnavi <hamid.dev.pro@gmail.com>
2023-09-28 17:44:19 +03:30
Côme Chilliet
dc347b39c1
Merge pull request #40501 from nextcloud/fix/fix-int-size-in-files-scanner
Store size in int|float for 32bit support
2023-09-25 16:32:32 +02:00
Robin Appelman
eab8dd3ca0 use multipart copy for s3
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-09-20 17:28:48 +02:00
Côme Chilliet
5090848b25
Store size in int|float for 32bit support
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2023-09-19 13:43:01 +02:00
Julius Härtl
88a96e4db7 fix: Only store unencrypted_size if path should be encrypted
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2023-09-06 07:25:28 +02:00
Robin Appelman
64f62f7cfd
Merge pull request #39863 from nextcloud/sharing-mask-wrapper
move share permission logic to storage wrapper
2023-09-04 14:05:14 +02:00
Thomas Citharel
3f28fc58aa fix(s3): fix handling verify_bucket_exists parameter
If 'verify_bucket_exists' is set to false in the config.php s3 configuration, it's supposed to avoid
verifying that the bucket exists. However empty(falsy) will  always return true, so this condition
would not work.

Signed-off-by: Thomas Citharel <tcit@tcit.fr>
2023-08-31 21:36:03 +02:00
Robin Appelman
a031bc4788 more share permission logic to storage wrapper
this way we only have to determine the share permissions once

Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-08-14 18:17:27 +02:00
Tobias Zimmerer
23f85b5ac2 fix(s3): add SSE-C parameters to headObject call
Signed-off-by: Tobias Zimmerer <3228193+ir0nhide@users.noreply.github.com>
2023-07-30 22:54:58 +02:00
Julius Härtl
77f249726d
Merge pull request #38934 from nextcloud/bugfix/noid/ssec-multipart 2023-07-20 11:00:08 +02:00
Robin Appelman
39a0554c09
Merge pull request #38858 from nextcloud/scanner-performance3
file scanner performance improvements
2023-06-26 17:01:18 +02:00
Julius Härtl
ea575e342f
fix(s3): Pass SSE-C parameters for multipart upload
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2023-06-21 18:32:24 +02:00
Robin Appelman
de85610385 use source cache when listing folder during recursive copy
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-06-19 16:40:15 +02:00
Robin Appelman
561536fc58
file scanner performance improvements
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-06-16 16:31:48 +02:00
Robin Appelman
ea88ec1350
add utility command for object store objects
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-06-09 18:08:18 +02:00
Robin Appelman
9f1d497a0b
Merge pull request #38261 from fsamapoor/replace_strpos_calls_in_lib_private
Refactors "strpos" calls in  lib/private to improve code readability.
2023-06-01 23:10:00 +02:00
Robin Appelman
cc0ad25c98
don't always check if we need to setup the object store root
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-05-23 17:44:40 +02:00
Faraz Samapoor
e7cc7653b8 Refactors "strpos" calls in lib/private to improve code readability.
Signed-off-by: Faraz Samapoor <fsamapoor@gmail.com>
2023-05-15 15:17:19 +03:30
Robin Appelman
2ea41dab93
repair -1 folder sizes for object store background scan
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-05-10 19:33:26 +02:00
Simon L
45a0fe490c
Merge pull request #37534 from Rsplwe/master
Adjust the value of the "max-parts" parameter of the object storage 'ListPart' interface to 1000
2023-04-18 00:16:38 +02:00
Simon L
85c9e75f33
Merge pull request #32314 from nextcloud/object-mkdir-logging
additional logging when mkdir fails for object storage
2023-04-17 23:42:08 +02:00
Robin Appelman
c6be96d04f
additional logging when mkdir fails for object storage
Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-04-05 14:52:19 +02:00
Robin Appelman
d63fa2d991
improve objectstore rmdir handling
remove folder entries as they are cleared instead of in one go afterwards

otherwise they stick around if some of the child entries can't be removed

Signed-off-by: Robin Appelman <robin@icewind.nl>
2023-04-03 18:28:29 +02:00
Rsplwe
5c6f35832c
Update lib/private/Files/ObjectStore/S3.php
Co-authored-by: Côme Chilliet <91878298+come-nc@users.noreply.github.com>
Signed-off-by: Rsplwe <i@rsplwe.com>
2023-04-03 21:59:59 +08:00
Rsplwe
d85e623916
s3: get remaining truncation part
Signed-off-by: Rsplwe <i@rsplwe.com>
2023-04-01 22:56:15 +08:00
Rsplwe
ccac296821
Adjust the value of 'maxparts' to 1000
Signed-off-by: Rsplwe <i@rsplwe.com>
2023-04-01 21:20:52 +08:00
Julius Härtl
e23aa8883e
feat(s3): Use multipart upload for chunked uploading
This allows to stream file chunks directly to S3 during upload.

Signed-off-by: Julius Härtl <jus@bitgrid.net>
2023-03-08 14:00:04 +01:00
Julius Härtl
159a0c8411
feat(s3): Add option to specify an SSE-C customer provided key
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2023-01-24 10:36:03 +01:00
Côme Chilliet
f5c361cf44
composer run cs:fix
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2023-01-20 11:45:08 +01:00
François Ménabé
cf9bdc0274 Manage s3 storage class in objectstore
Signed-off-by: François Ménabé <francois.menabe@gmail.com>
2023-01-12 14:58:27 +01:00
Maxence Lange
cc639a8877 compare cached filesize on download
Signed-off-by: Maxence Lange <maxence@artificial-owl.com>
2022-12-01 15:42:15 -01:00
Robin Appelman
a5036a973c allow disabling object store write check
Signed-off-by: Robin Appelman <robin@icewind.nl>
2022-10-26 18:45:59 +00:00
Carl Schwan
a5ea677370 Rename file1 and file2 to source and target in Storage abstraction
Signed-off-by: Carl Schwan <carl@carlschwan.eu>
2022-10-18 17:21:47 +02:00
Julius Härtl
214a50d071
Avoid allocating too much memory for the buffer
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2022-10-11 15:34:27 +02:00
Carl Schwan
3950deb42f
Merge pull request #32740 from jasperweyne/patch-2
Fix open_basedir issues by AWS SDK
2022-09-19 10:19:15 +02:00
Robin Appelman
457822c144 don't update the folder size if we know it hasn't changed
Signed-off-by: Robin Appelman <robin@icewind.nl>
2022-08-16 13:56:51 +02:00
Jasper Weyne
44f6c931e7
Merge branch 'master' into patch-2 2022-08-11 08:54:08 +02:00
Robin Appelman
61fdd91dcf
don't set null as a bundle path
Signed-off-by: Robin Appelman <robin@icewind.nl>
2022-07-27 12:58:07 +02:00
Robin Appelman
f1486890d7
only use nextcloud bundle when explicitly enabled
Signed-off-by: Robin Appelman <robin@icewind.nl>
2022-07-05 16:06:55 +02:00
Robin Appelman
bffa67c48b
also use nextcloud certificate bundle when downloading from s3
Signed-off-by: Robin Appelman <robin@icewind.nl>
2022-07-05 15:57:54 +02:00
Jasper Weyne
0633a1d9f5
Disable ~/.aws/config access by S3ConnectionTrait
This commit sets the 'use_aws_shared_config_files' option to false, in order to disable configuration loading from ~/.aws/config by the AWS SDK, specifically the S3Client. It is a continuation of #27040, as that PR only changed the behaviour of the CredentialsProvider; this change affects the ConfigurationProvider as well.

Signed-off-by: Jasper Weyne <jasperweyne@gmail.com>
2022-06-07 13:39:14 +02:00
Julius Härtl
0f0e74a78c Cleanup temporary files after finishing the write to object storage
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2022-06-02 12:57:18 +00:00
Côme Chilliet
069477e608
Migrate more classes of lib/private to LoggerInterface
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2022-04-26 16:52:52 +02:00
Carl Schwan
7817845538 Add a metadata service to store file metadata
Signed-off-by: Carl Schwan <carl@carlschwan.eu>
2022-04-13 14:06:29 +02:00
Côme Chilliet
ea23523c70
Adapt more code to migration to LoggerInterface
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2022-03-24 16:21:25 +01:00
Côme Chilliet
6be7aa112f
Migrate from ILogger to LoggerInterface in lib/private
Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2022-03-24 16:21:25 +01:00
Robin Appelman
1156214a26
don't try to get custom certs for s3 primary storage
Signed-off-by: Robin Appelman <robin@icewind.nl>
2022-03-15 15:50:08 +01:00
Robin Appelman
194a21f374
use the nextcloud certificate bundle for s3
Signed-off-by: Robin Appelman <robin@icewind.nl>
2022-03-14 18:09:48 +01:00
Louis Chemineau
bc2c02f451 Wrap S3 multipart upload exception
Signed-off-by: Louis Chemineau <louis@chmn.me>
2022-02-16 14:56:14 +01:00
Louis Chemineau
0b768c17d2 Wrap S3 multipart upload exception
Signed-off-by: Louis Chemineau <louis@chmn.me>
2022-02-15 17:35:12 +01:00
Julius Härtl
2aac757805
Make max size for single put uploads configurable
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2021-12-29 13:30:22 +01:00
Julius Härtl
e475dfe16c
Fix php-cs for S3ConnectionTrait.php
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2021-12-21 21:35:46 +01:00
Julius Härtl
1acfbd0b06
Merge pull request #27493 from cuppett/cuppett/simplify-aws-credential-provider 2021-12-21 11:58:52 +01:00
Côme Chilliet
66bdda568a
Fix Guzzle function calls
Since 1.7.0 and 2.0 these functions are turned into static methods.
See https://github.com/guzzle/psr7#upgrading-from-function-api

Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2021-11-23 09:28:57 +01:00
Stephen Cuppett
4a6a2c2b48 Simplify S3ConnectionTrait to defaultProvider plus option
When we initially added the EC2 and ECS IAM role support in #24700,
we had to use a workaround by explicitly ordering the various providers
due to an inconsistency in the AWS SDK for PHP. We submitted a PR there
to get that squared away. Now, we've consumed that version upstream
for the SDK and can update our code here to be the most concise version
as well as position ourselves to pick up new methods as those become
available and prevalent in AWS (for acquiring credentials).

See also: https://github.com/nextcloud/server/pull/24700#issuecomment-747650892
See also: https://github.com/aws/aws-sdk-php/pull/2172

Signed-off-by: Stephen Cuppett <steve@cuppett.com>
2021-11-22 07:04:48 -05:00
John Molakvoæ
6ab2feaa54
Implement multibucket shift for ObjectStore
Signed-off-by: John Molakvoæ <skjnldsv@protonmail.com>
2021-11-11 15:09:28 +01:00
Carl Schwan
df4e6bab69
Merge pull request #29115 from nextcloud/work/carl/correct-permissions-when-copying
Fix permissions when copying from ObjectStorage
2021-10-28 14:38:20 +02:00
Carl Schwan
bfa60aaf27
Fix permissions when copying from ObjectStorage
Make sure that when a user copy a file from a directory they don't have
all permissions to a directory where they have more permissions, the
permissions are correctly set to the one from the parent taget folder.

This was caused by the ObjectStoreStorage::copyFromStorage using
the jailed storage and cache entry instead of the unjailed one like other
storages (the local one).
Steps to reproduce

+ Use object storage
+ Create a groupfolder with one group having full permission and another one
  who can just read files.
+ With an user who is in the second group, copy a file from the groupfolder to
  the home folder of this user.
+ The file in the home folder of the user will be read only and can't be deleted
  even though it is in their home folder and they are the owner. In oc_filecache,
  the permissions stored for this file are 1 (READ)

Signed-off-by: Carl Schwan <carl@carlschwan.eu>
2021-10-28 13:29:50 +02:00
Robin Appelman
09ffac5e6d
s3 external storage listing rework
Signed-off-by: Robin Appelman <robin@icewind.nl>
2021-10-15 15:35:02 +02:00
Bernd Rederlechner
3866f388b1
Refactor writeObject to only use MultipartUpload when required
Signed-off-by: Bernd Rederlechner <Bernd.Rederlechner@t-systems.com>

Co-authored-by: Julius Härtl <jus@bitgrid.net>
2021-08-20 17:02:25 +02:00
Julius Härtl
3e67637a4a
Merge pull request #26463 from Worteks/fix-s3proxy 2021-07-14 23:32:05 +02:00
blizzz
62675eb5c2
Merge pull request #27586 from nextcloud/bugfi/noid/objectstore-checksum
Reset checksum when writing files to object store
2021-07-01 17:30:54 +02:00
Maxime Besson
388a458cd5 fix(proxy): use accessor
Signed-off-by: Maxime Besson <maxime.besson@worteks.com>
2021-06-30 09:25:04 +02:00
Christoph Wurst
6d5cfe0c66
Move DateTime::RFC2822 to DateTimeInterface::2822
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2021-06-23 15:30:43 +02:00
Julius Härtl
21db463bc9
Reset checksum when writing files to object store
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2021-06-21 12:04:49 +02:00
John Molakvoæ (skjnldsv)
215aef3cbd
Update php licenses
Signed-off-by: John Molakvoæ (skjnldsv) <skjnldsv@protonmail.com>
2021-06-04 22:02:41 +02:00
Morris Jobke
333665b43d
Merge pull request #27040 from nextcloud/FlorentCoppint-master
Avoid reading ~/.aws/config when using S3 provider
2021-05-25 10:19:22 +02:00
Florent
d4444f2472
Avoid reading ~/.aws/config when using S3 provider
When using S3 storage, we sometimes have open_basedir errors due to aws-sdk-php trying to read ~/.aws/config which is out of open_basedir restrictions.
Christoph Wurst already added csm=false config in #21406 but it wasn't enough, we also need to set use_arn_region=false, added in this commit.

Signed-off-by: Florent <florent@coppint.com>
2021-05-20 12:21:15 +02:00
Robin Appelman
ef6f2e68f0
explicitly close source stream on object store upload even if count wrapper isn't needed
Signed-off-by: Robin Appelman <robin@icewind.nl>
2021-04-29 17:01:19 +02:00