mattermost/server/go.mod

239 lines
11 KiB
Modula-2
Raw Permalink Normal View History

module github.com/mattermost/mattermost/server/v8
2026-05-20 12:21:14 -04:00
go 1.26.3
require (
code.sajari.com/docconv/v2 v2.0.0-pre.4
MM-68662: Add Azure Blob Storage filestore backend (#36498) * Generalize file backend error types Replace S3FileBackendAuthError and S3FileBackendNoBucketError with backend-agnostic FileBackendAuthError and FileBackendNoBucketError so non-S3 drivers can return them and the admin "Test Connection" flow keeps surfacing useful messages. The old S3-prefixed names are kept as type aliases of the generic types so external code (plugins, historical consumers) continues to compile, and so existing S3 construction sites stay untouched. The type switch in connectionTestErrorToAppError now matches the generic types, with new i18n keys (test_connection_auth.app_error and test_connection_no_bucket.app_error) whose wording does not name S3. The old S3-specific i18n keys are dropped via `make i18n-extract` since they are no longer referenced from code; the api4 test that asserted on those keys is updated, and the Cypress `MM-T996 Amazon S3 connection error messaging` spec that asserted on the old user-facing string is updated to the new wording. ------ AI assisted commit * Pull in Azure SDK and uuid dependencies Bring in github.com/Azure/azure-sdk-for-go/sdk/azcore and .../sdk/storage/azblob (with .../sdk/internal as their indirect dependency). The two are needed by the upcoming Azure Blob Storage filestore backend and its lazy-Range-backed reader. The bump of golang.org/x/{crypto,net,sys,term,text} comes transitively from azblob's minimum versions. Also promotes github.com/google/uuid from indirect to direct, since the Azure backend uses it to generate block IDs that share the same wire format the SDK itself produces in UploadStream. ------ AI assisted commit * Add azureRangeReader, a seekable Range-backed blob reader A small standalone type that satisfies the FileBackend interface's ReadCloseSeeker + the broader io.ReaderAt contract on top of Azure Blob Storage HTTP Range requests. Lands as its own commit because the upcoming Azure FileBackend driver builds on it, and the reader itself is independently useful — and independently testable against a fake downloader without standing up an Azure client. Design notes: * Read opens an HTTP Range stream lazily at the current offset and reuses it for sequential reads. Seek to a different offset closes the open stream; the next Read re-opens it. * Seek to the same offset is a no-op and does not close the open stream, so callers like zip.NewReader that probe with redundant seeks don't kick off a fresh download. * ReadAt issues a dedicated ranged DownloadStream per call and does not touch the streaming cursor — matches the io.ReaderAt contract the bulk-import worker's zip.NewReader path relies on. * Close cancels the context (which any in-flight Azure call will observe and abort), stops the deadline timer, and closes the current body if any. It is safe to call when no body was ever opened. * CancelTimeout lets long-running consumers like the import worker opt out of the per-operation deadline that would otherwise kill multi-minute downloads partway through. The implementation talks to a small blobDownloader interface rather than *blob.Client directly so the unit tests can substitute a fake downloader that records every requested Range and tracks Close calls on the bodies it hands out. ------ AI assisted commit * Add Azure Blob Storage filestore driver Implements the FileBackend interface against Azure Blob Storage in a new azurestore.go (~520 LOC). The driver is not yet selectable via NewFileBackend's switch — that wiring lands in the next commit together with the admin config surface — but the driver itself is complete and self-contained behind the FileBackendSettings struct. Filesstore.go grows three pieces of supporting infrastructure that the driver consumes: * a `driverAzure = "azureblob"` constant alongside the existing driverS3 and driverLocal, * an Azure-specific block on FileBackendSettings (storage account, access key, container, path prefix, endpoint, SSL flag, request timeout), * a CheckMandatoryAzureFields validator that mirrors CheckMandatoryS3Fields. Behavioural notes that warrant calling out: * Reader returns the previously-added azureRangeReader, so reads stream lazily over HTTP Range and ReadAt is available for the bulk-import worker's zip.NewReader path. The deadline timer is armed before the initial GetProperties call so the HEAD itself is bounded. * WriteFile and AppendFile both go through StageBlock + CommitBlockList via a shared stageBlocks helper, never the SDK's UploadStream. UploadStream's small-payload fast path falls back to single-shot PutBlob, which leaves the resulting blob with no committed block list; a subsequent AppendFile that calls CommitBlockList on that blob would then clobber its content. Routing every write through the block-list mechanism keeps AppendFile correct regardless of payload size. * AppendFile stages the new chunk as one or more blocks and commits the existing committed block list plus the newly staged IDs. The new bytes go up exactly once — no re-download, no re-concatenate, no re-upload of the prior contents. * WriteFileContext does not wrap the caller-supplied context with its own timeout — that timeout is applied in WriteFile only, matching the S3 driver, so long-running TryWriteFileContext callers (like message-export bulk writes) opt out of the per-operation timeout the way the abstraction documents. Authentication is shared-key only for this drop; Microsoft Entra ID / managed identity is deferred to a follow-up. The endpoint is configurable so the same code targets the production Azure host (vhost style — {account}.blob.core.windows.net) or Azurite / Azure Government / sovereign clouds (path style — host[:port]/{account}). ------ AI assisted commit * Wire Azure backend into config, validation, and driver selection This commit registers the previously-added AzureFileBackend driver with the rest of the system. Until now the driver was usable only via direct construction; after this commit, `DriverName: "azureblob"` in config.json is a fully-supported deployment configuration. Five integration sites are touched: * `newFileBackend` in filesstore.go now dispatches `driverAzure` to NewAzureFileBackend, alongside the existing s3 and local cases. NewFileBackendSettingsFromConfig (and its export counterpart) gain an Azure branch that maps the model.FileSettings fields onto the Azure-specific FileBackendSettings fields. * `model.FileSettings` grows the user-facing Azure config schema: storage account, access key, container, path prefix, endpoint, SSL flag, request timeout, plus matching Export* fields for the dedicated export store. SetDefaults populates them so deployments that never opted into Azure don't carry nil pointers. `isValid` accepts the new ImageDriverAzure constant. * `Config.Sanitize()` masks AzureAccessKey and ExportAzureAccessKey the same way it masks AmazonS3SecretAccessKey, so the shared key never reaches an API consumer in plain text. * `desanitize()` restores the masked keys on a config write so a PATCH that doesn't touch the key doesn't clobber it with the FakeSetting placeholder. * `configSensitivePaths` covers both Azure key paths so audit diffs don't include them either. * `ConfigToFileBackendSettings` in the `mattermost db` CLI helper gets the Azure branch its production counterpart already has — without it, `mattermost db migrate` / `db downgrade` would fail on Azure-configured deployments with "missing azure storage account setting". Finally, the shared FileBackendTestSuite is now wired against Azurite via TestAzureFileBackendTestSuite, which skips when CI_AZURITE_HOST is unreachable. The test-infra wiring (the docker service, the env vars, the start_dependencies entry) landed in a previous PR; this commit is what makes the suite actually exercise the Azure driver end to end. ------ AI assisted commit * Validate Azure timeout and path prefix in Config.IsValid Parity with the S3-side checks that already cover AmazonS3RequestTimeoutMilliseconds and AmazonS3PathPrefix. Without these, a zero/negative AzureRequestTimeoutMilliseconds passes validation and later creates immediately-expired request contexts, and leading/trailing whitespace in AzurePathPrefix produces blob keys that don't match what the admin configured. Same checks added for the Export* counterparts. The file_driver.app_error translation is updated to mention the new 'azureblob' option alongside 'local' and 'amazons3'. ------ AI assisted commit * Stream zip entries from the Azure backend writeZipEntry was calling ReadFile, which loads the entire blob into memory before writing it to the archive. For large blobs or deep directories this spikes RSS or OOMs the goroutine. Switch to Reader (the streaming azureRangeReader) and io.Copy into the zip entry so memory stays bounded regardless of blob size. ------ AI assisted commit * Use a backend-agnostic fallback for FileBackendNoBucketError The fallback Error() message was "no such bucket", which leaks S3 terminology when an Azure caller returns the type with no wrapped Err. Use "no such bucket or container" so logs and external error handling stay neutral across backends. ------ AI assisted commit * Defend Azure path prefix against directory traversal Reject ".." in AzurePathPrefix and ExportAzurePathPrefix at config validation time, since path.Join collapses traversal segments and a prefix like "../other-tenant" would otherwise escape the configured isolation boundary. Harden the prefix helper as a second line of defense: if the joined path no longer sits inside pathPrefix, fall back to joining the prefix with the base name of the caller-supplied path. That preserves the prefix invariant for plugin and import paths that the upload code does not sanitize uniformly. ------ AI assisted commit * Honor SkipVerify when constructing the Azure client FileBackendSettings.SkipVerify is plumbed through from the System Console the same way it is for S3, so admins toggling the flag for self-signed endpoints (Azurite, sovereign clouds) get the behavior they expect without having to drop SSL entirely and send the shared key in clear text. ------ AI assisted commit * Warn when the Azure request timeout falls back to its default Config.IsValid already rejects non-positive AzureRequestTimeoutMilliseconds for any path that goes through config validation, so this warn only fires for direct callers that bypass validation (tests, helpers). Logging the substitution turns a silent coercion into something an operator can correlate against unexpected request behavior. ------ AI assisted commit * Cap Azure request timeout at 10 minutes Reject AzureRequestTimeoutMilliseconds values above the ceiling so an operator (or someone who has admin access) cannot effectively disable timeouts by setting the value to math.MaxInt64. A hung Azure call then holds a goroutine open until the OS gives up. Applies the same bound to ExportAzureRequestTimeoutMilliseconds. S3 has the same gap; treating it is out of scope here but worth a follow-up. ------ AI assisted commit * Refuse AppendFile on blobs without a committed block list A blob written by another tool (Azure portal, azcopy, a migration script, a plugin using Put Blob) has its content in the blob but an empty committed-block list. Committing a new block list against such a blob silently replaces the existing content with only the appended bytes. Check the blob's properties before staging when the committed-block list is empty, and refuse with a clear error if the blob has content. Same hazard for an admin pointing the backend at an existing container with pre-existing files. Adds an integration test against Azurite to lock the behavior in. ------ AI assisted commit * Surface truncated reads from azureRangeReader Read closed the body cleanly and returned io.EOF even when the remote stream terminated before the blob's content length. Callers (and any retry layer above) then accepted a partial blob as complete. ReadAt unconditionally rewrote io.ErrUnexpectedEOF to io.EOF, which made truncated downloads indistinguishable from clean reads. That is exactly what zip.NewReader consumes for archive readers, so the bulk-import worker would silently import partial archives. Read now closes the body, nils it, and returns io.ErrUnexpectedEOF when EOF arrives before offset reaches size. ReadAt only collapses ErrUnexpectedEOF to EOF when the full count was delivered and the stream was consumed to the end of the blob. Otherwise the truncation propagates with context. Both code paths are exercised by new fakeDownloader-backed tests. ------ AI assisted commit * Move container provisioning out of Azure TestConnection Auto-creating the container inside TestConnection meant a typo in the System Console (mattermosst instead of mattermost) silently provisioned an unwanted container in the admin's Azure subscription, with no audit log and no warning. They'd discover it later when uploads landed somewhere unexpected. TestConnection now returns FileBackendNoBucketError when the container is missing, mirroring the S3 contract. A new MakeContainer method mirrors S3FileBackend.MakeBucket, and Server.Start dispatches via two capability interfaces (bucketMaker / containerMaker) instead of a hard S3 type assertion — so the NoBucket error is no longer silently swallowed for backends Server.Start has not been taught about. ------ AI assisted commit * Carry file backend auth detail through to AppError The Test Connection button collapsed every typed backend failure into the same generic i18n message. Operators trying to debug bad credentials or a missing bucket only saw "Unable to authenticate against the file storage backend" with no SDK code to grep for in their logs. Use errors.As so the typed checks survive future wrapping, and pass the underlying error string through the NewAppError details argument. The AppError serializer surfaces that detail to the admin console alongside the translated message, so a bad S3 InvalidAccessKeyId or an Azure AuthenticationFailed shows up in the toast without an i18n schema change. ------ AI assisted commit * Remove non-ascii characters from comments ------ AI assisted commit * Make linter happy ------ AI assisted commit * Harden Azure prefix boundary check strings.HasPrefix on the joined path is a string-level check, not a path-level one, so a configured prefix of "mattermost" accepts a joined result of "mattermost-evil/...". A crafted caller path like "../mattermost-evil/secrets" would collapse via path.Join to that exact sibling and slip through the boundary check, escaping the configured prefix scope. Require the joined path to be the cleaned prefix itself or to start with the prefix followed by a path separator. The fallback path.Join uses the same cleaned prefix for consistency. ------ AI assisted commit * Provision Azurite container in standalone test setup The shared FileBackendTestSuite's SetupTest already handles a missing container by detecting FileBackendNoBucketError from TestConnection and calling MakeContainer, but TestAzureFileBackendAppendRefusesNonBlockBlob bypasses SetupTest and calls TestConnection directly. On a fresh Azurite instance the test would fail before exercising the append-refusal logic. Extract a newAzuriteBackend(t) helper alongside azuriteSettings(t) that builds the backend and ensures the container exists, mirroring the suite's setup. Use errors.As for forward compatibility with future wrapping. ------ AI assisted commit * Fix grammar in email-settings i18n string "Email settings has unset values." -> "Email settings have unset values." ------ AI assisted commit * Make Azure MakeContainer idempotent Treat a ContainerAlreadyExists response as success so that two nodes racing through TestConnection plus MakeContainer at boot both converge instead of having the loser fail. Mirrors how the S3 backend handles the equivalent BucketAlreadyOwnedByYou case. ------ AI assisted commit * Narrow AzureEndpoint comment to path-style only The setting only builds path-style URLs, so it cannot reach sovereign clouds like Azure Government or Azure China, which require vhost-style endpoints. Update the comment to reflect what the code actually does and document that sovereign-cloud support is out of scope. ------ AI assisted commit
2026-05-14 12:59:18 -04:00
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.21.1
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.4
github.com/Masterminds/semver/v3 v3.5.0
github.com/avct/uasurfer v0.0.0-20250915105040-a942f6fb6edc
github.com/aws/aws-sdk-go-v2 v1.41.7
github.com/aws/aws-sdk-go-v2/config v1.32.17
github.com/aws/aws-sdk-go-v2/credentials v1.19.16
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.23
github.com/aws/aws-sdk-go-v2/service/marketplacemetering v1.36.5
github.com/bep/imagemeta v0.12.0
github.com/blang/semver/v4 v4.0.0
Add board channel types (BO/BP) for Integrated Boards (#35887) * Add board channel types (BO/BP) with POST /boards API Introduces board channel types as a new channel variant that reuses the Channels table but is fully isolated from all /channels endpoints. Model: - Add ChannelTypeOpenBoard ("BO") and ChannelTypePrivateBoard ("BP") - Add IsBoard(), IsOpenBoard(), IsPrivateBoard() helpers - Add board-specific websocket events (board_created/updated/deleted/restored) Store: - SaveBoardChannel: atomic channel + view creation in a single transaction - Save() rejects board types (forces use of SaveBoardChannel) - Exclude boards from all channel listing/search queries (GetTeamChannels, GetAll, GetChannels, GetChannelsByUser, GetDeleted, autocomplete, search) API: - POST /boards: create board channel (feature-flagged behind IntegratedBoards) - All /channels write endpoints reject board types with 400 - All /channels read endpoints reject or exclude board types - Open boards get same public-read semantics as open channels Tests: - 15 rejection tests covering every /channels write + read endpoint - 9 exclusion tests covering every listing/search endpoint - 8 store tests for SaveBoardChannel + Save rejection - 4 board creation API tests (create, private, flag off, sidebar exclusion) - 3 authorization tests for board permission semantics Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Update generated files: i18n, go.mod, migrations list Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add i18n translations for board channel error strings Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix board guard ordering in getChannelMembers and getChannelStats Move the board rejection check after the permission check so that nonexistent channel IDs still return 403 (not 404) matching the original behavior expected by TestGetChannelMembers. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Filter boards at store level instead of API guards Store.Get() now excludes board types via WHERE clause, making boards invisible to all /channels endpoints. Added GetBoardChannel() for /boards endpoints. Removed redundant API-level rejectBoardChannel guards from 10 handlers that already call GetChannel(). Kept explicit guards only on 3 handlers that don't fetch the channel. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix empty i18n translation for app.channel.save_member.app_error Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add board system properties, kanban column config, and audit logging Migration: - Register "boards" property group with system-wide Assignee (user) and Status (select: Todo/In Progress/Complete) fields, both protected - Idempotent migration following content flagging pattern Board creation: - Look up boards fields by name, set board:linked_properties on channel - Build kanban view props with group_by mapping status options to columns - Add typed KanbanProps/KanbanColumn/KanbanGroupBy structs with ToProps()/KanbanPropsFromProps() for round-tripping - Add audit record logging for POST /boards - Add early team_id validation in API handler - Error on missing status options instead of silent empty columns Tests: - Migration test: field creation + idempotent re-run - Board creation test: verify kanban props + linked_properties - Fix updateChannelMemberRoles test to use valid role string Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add boards migration mock to testlib store setup The boards property migration calls System().GetByName() which needs a matching mock expectation, same pattern as content_flagging_setup_done. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add kanban view props validation and tests Validate kanban View.Props in IsValid(): group_by required with valid field_id, 1-100 columns, each column needs id, name, and at least one option_id. Update all test helpers to produce valid kanban props. 11 dedicated validation tests + round-trip test for KanbanProps. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Validate board display name is not empty Add early DisplayName validation in CreateBoardChannel with a clear error. Add tests for empty and whitespace-only display names. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Exclude boards from GetMany and getChannelsMemberCount Add board type exclusion to Store.GetMany() and use filtered channel IDs in getChannelsMemberCount handler so board channels don't leak into member count results. Add test covering the endpoint. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix review issues: drop search indexing for boards, use request context - Remove search layer indexing of board channels so they stay invisible to Elasticsearch/Bleve-powered search and autocomplete - Replace context.Background() with rctx.Context() for proper cancellation and tracing in CreateBoardChannel Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix gofmt alignment in websocket_message.go after merge Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Regenerate server i18n after merge * Restore translation for permission_policy.app_error * Filter board channels in name lookups, autocomplete, and indexing The store-layer board exclusion filter was missing from getByName, getByNames, GetDeletedByName, the global Autocomplete, and GetChannelsBatchForIndexing — leaving boards reachable via name lookups, the no-team-filter search path, and admin reindex jobs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Reject boards in id-batch lookups, unread, and member-mutation endpoints - GetChannelsByIds, GetChannelsWithTeamDataByIds, and GetChannelUnread now exclude BO/BP at the store layer so boards can't slip through if callers stop filtering first. - updateChannelMemberNotifyProps, updateChannelMemberAutotranslation, and viewChannel now reject board IDs explicitly via the existing rejectBoardChannelByID helper, matching the other write endpoints. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Gate boards properties setup on the IntegratedBoards feature flag doSetupBoardsProperties registered the boards property group and fields at every server boot regardless of the IntegratedBoards feature flag. Skip the migration when the flag is disabled so the property metadata only appears once boards are actually enabled. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Use App accessors for boards property lookups CreateBoardChannel reached into a.Srv().PropertyService() directly instead of going through the App-level GetPropertyGroup and GetPropertyFieldByName methods that already wrap the service. Switch to the standard App accessors so the calls match the rest of the codebase. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Log full channel input on createBoard audit record createBoard only captured team_id and type on the audit record, so failed creations lost most of the request payload. Use AddEventParameterAuditableToAuditRec with the full channel struct, matching createChannel. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Use allow-list of message channel types in store filters Inverted every sq.NotEq{[BO, BP]} filter into sq.Eq{messageChannelTypes} (or teamMessageChannelTypes for queries that also exclude direct channels) so that any future non-message channel type — wikis, etc. — is excluded by default rather than requiring every existing call site to be updated. Also rewrote GetChannelUnread on top of the squirrel builder so the same allow-list slice can be reused. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * go.mod: promote prometheus/common to direct after merge Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Use model.NewPointer for boards property permission field Drops the local permNone variable in doSetupBoardsProperties and uses model.NewPointer(model.PermissionLevelNone) inline, matching the surrounding ContentFlagging/ManagedCategory code. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Extract saveViewT to share Views insert between Save and SaveBoardChannel ViewStore.Save and SaveBoardChannel both built the same INSERT INTO Views statement, so a future column addition would need updates in two places. Extract the insert (plus PreSave/IsValid) into a private saveViewT method that accepts any sqlxExecutor — the regular master handle for ViewStore.Save, and the channel transaction for SaveBoardChannel. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add IsMessageChannel helper on model.Channel Mirrors IsBoard for the positive case: returns true for Open, Private, Direct, and Group channel types. Lets future filtering code be expressed against the allow-list rather than enumerating board types, so newly introduced non-message channel types are excluded by default. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Move board input validation into Channel.IsValidBoard CreateBoardChannel inlined four guards for type, team, and display name. Move the type/team_id/display_name checks into a new Channel.IsValidBoard method so the rules live with the model and return the AppError directly. The TrimSpace on DisplayName stays at the call site to match how CreateChannel sanitizes before validating. Drops the now-unused app.channel.create_board_channel.{invalid_type, no_team,no_display_name} translations and adds matching model.channel.is_valid_board.* keys. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Extract buildBoardKanbanView from CreateBoardChannel The kanban view construction (read status options, build columns, serialize props, assemble *model.View) only depends on the status property field and the creator id. Pulling it into its own helper shrinks CreateBoardChannel and makes the column-building logic testable in isolation. Adds board_test.go with coverage for the empty-options error path, the standard happy path, and the option-skipping branches. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Test Channel.IsValidBoard Cover the four reject cases (wrong type, missing team_id, empty display name) plus the open and private board accept cases. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * gofmt board_test.go Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Document POST /api/v4/boards in OpenAPI spec * Add valid kanban props to api4 makeTestViewForAPI helper * Assert kanban.ToProps error in makeTestViewForAPI helper The helper used to swallow the error from kanban.ToProps. Take *testing.T and require.NoError so a serialization failure surfaces immediately at the call site instead of producing a malformed view. * Run boards properties setup unconditionally The feature-flag gate added in 6298e15e86 made the test suite impossible: test infra applies FeatureFlags overrides only after app.NewServer returns, but doAppMigrations runs during NewServer, so the flag was always false at migration time. The migration short-circuited, the boards property group was never registered, and every CreateBoardChannel test 500'd with "boards property group not found." Drop the gate. The migration is idempotent (keyed in System) and benign — matches doSetupContentFlaggingProperties and doSetupManagedCategoryProperties. IntegratedBoards still gates route registration (api4/board.go) and the CreateBoardChannel runtime entry (app/board.go), so the property group sits unused until the feature is enabled. * Add Client4.CreateBoard and use it in tests Adds boardsRoute() and CreateBoard(ctx, channel) on Client4 mirroring CreateChannel/CreateView. Refactors api4 board tests off the raw DoAPIPost("/boards", ...) calls and the SaveBoardChannel store inserts that predated the API; both now exercise the public client method, drop the makeTestBoardView helper and the manual SaveMember follow-ups, and route through setupBoardTest so IntegratedBoards is enabled where needed. * make modules-tidy --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-13 19:25:08 -04:00
github.com/boxes-ltd/imaging v1.7.5
github.com/cespare/xxhash/v2 v2.3.0
github.com/dgryski/dgoogauth v0.0.0-20190221195224-5a805980a5f3
github.com/dyatlov/go-opengraph/opengraph v0.0.0-20220524092352-606d7b1e5f8a
github.com/elastic/go-elasticsearch/v8 v8.19.6
github.com/fatih/color v1.19.0
github.com/getsentry/sentry-go v0.46.2
github.com/goccy/go-yaml v1.19.2
github.com/golang-jwt/jwt/v5 v5.3.1
github.com/golang-migrate/migrate/v4 v4.19.1
github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0
github.com/golang/mock v1.6.0
MM-68662: Add Azure Blob Storage filestore backend (#36498) * Generalize file backend error types Replace S3FileBackendAuthError and S3FileBackendNoBucketError with backend-agnostic FileBackendAuthError and FileBackendNoBucketError so non-S3 drivers can return them and the admin "Test Connection" flow keeps surfacing useful messages. The old S3-prefixed names are kept as type aliases of the generic types so external code (plugins, historical consumers) continues to compile, and so existing S3 construction sites stay untouched. The type switch in connectionTestErrorToAppError now matches the generic types, with new i18n keys (test_connection_auth.app_error and test_connection_no_bucket.app_error) whose wording does not name S3. The old S3-specific i18n keys are dropped via `make i18n-extract` since they are no longer referenced from code; the api4 test that asserted on those keys is updated, and the Cypress `MM-T996 Amazon S3 connection error messaging` spec that asserted on the old user-facing string is updated to the new wording. ------ AI assisted commit * Pull in Azure SDK and uuid dependencies Bring in github.com/Azure/azure-sdk-for-go/sdk/azcore and .../sdk/storage/azblob (with .../sdk/internal as their indirect dependency). The two are needed by the upcoming Azure Blob Storage filestore backend and its lazy-Range-backed reader. The bump of golang.org/x/{crypto,net,sys,term,text} comes transitively from azblob's minimum versions. Also promotes github.com/google/uuid from indirect to direct, since the Azure backend uses it to generate block IDs that share the same wire format the SDK itself produces in UploadStream. ------ AI assisted commit * Add azureRangeReader, a seekable Range-backed blob reader A small standalone type that satisfies the FileBackend interface's ReadCloseSeeker + the broader io.ReaderAt contract on top of Azure Blob Storage HTTP Range requests. Lands as its own commit because the upcoming Azure FileBackend driver builds on it, and the reader itself is independently useful — and independently testable against a fake downloader without standing up an Azure client. Design notes: * Read opens an HTTP Range stream lazily at the current offset and reuses it for sequential reads. Seek to a different offset closes the open stream; the next Read re-opens it. * Seek to the same offset is a no-op and does not close the open stream, so callers like zip.NewReader that probe with redundant seeks don't kick off a fresh download. * ReadAt issues a dedicated ranged DownloadStream per call and does not touch the streaming cursor — matches the io.ReaderAt contract the bulk-import worker's zip.NewReader path relies on. * Close cancels the context (which any in-flight Azure call will observe and abort), stops the deadline timer, and closes the current body if any. It is safe to call when no body was ever opened. * CancelTimeout lets long-running consumers like the import worker opt out of the per-operation deadline that would otherwise kill multi-minute downloads partway through. The implementation talks to a small blobDownloader interface rather than *blob.Client directly so the unit tests can substitute a fake downloader that records every requested Range and tracks Close calls on the bodies it hands out. ------ AI assisted commit * Add Azure Blob Storage filestore driver Implements the FileBackend interface against Azure Blob Storage in a new azurestore.go (~520 LOC). The driver is not yet selectable via NewFileBackend's switch — that wiring lands in the next commit together with the admin config surface — but the driver itself is complete and self-contained behind the FileBackendSettings struct. Filesstore.go grows three pieces of supporting infrastructure that the driver consumes: * a `driverAzure = "azureblob"` constant alongside the existing driverS3 and driverLocal, * an Azure-specific block on FileBackendSettings (storage account, access key, container, path prefix, endpoint, SSL flag, request timeout), * a CheckMandatoryAzureFields validator that mirrors CheckMandatoryS3Fields. Behavioural notes that warrant calling out: * Reader returns the previously-added azureRangeReader, so reads stream lazily over HTTP Range and ReadAt is available for the bulk-import worker's zip.NewReader path. The deadline timer is armed before the initial GetProperties call so the HEAD itself is bounded. * WriteFile and AppendFile both go through StageBlock + CommitBlockList via a shared stageBlocks helper, never the SDK's UploadStream. UploadStream's small-payload fast path falls back to single-shot PutBlob, which leaves the resulting blob with no committed block list; a subsequent AppendFile that calls CommitBlockList on that blob would then clobber its content. Routing every write through the block-list mechanism keeps AppendFile correct regardless of payload size. * AppendFile stages the new chunk as one or more blocks and commits the existing committed block list plus the newly staged IDs. The new bytes go up exactly once — no re-download, no re-concatenate, no re-upload of the prior contents. * WriteFileContext does not wrap the caller-supplied context with its own timeout — that timeout is applied in WriteFile only, matching the S3 driver, so long-running TryWriteFileContext callers (like message-export bulk writes) opt out of the per-operation timeout the way the abstraction documents. Authentication is shared-key only for this drop; Microsoft Entra ID / managed identity is deferred to a follow-up. The endpoint is configurable so the same code targets the production Azure host (vhost style — {account}.blob.core.windows.net) or Azurite / Azure Government / sovereign clouds (path style — host[:port]/{account}). ------ AI assisted commit * Wire Azure backend into config, validation, and driver selection This commit registers the previously-added AzureFileBackend driver with the rest of the system. Until now the driver was usable only via direct construction; after this commit, `DriverName: "azureblob"` in config.json is a fully-supported deployment configuration. Five integration sites are touched: * `newFileBackend` in filesstore.go now dispatches `driverAzure` to NewAzureFileBackend, alongside the existing s3 and local cases. NewFileBackendSettingsFromConfig (and its export counterpart) gain an Azure branch that maps the model.FileSettings fields onto the Azure-specific FileBackendSettings fields. * `model.FileSettings` grows the user-facing Azure config schema: storage account, access key, container, path prefix, endpoint, SSL flag, request timeout, plus matching Export* fields for the dedicated export store. SetDefaults populates them so deployments that never opted into Azure don't carry nil pointers. `isValid` accepts the new ImageDriverAzure constant. * `Config.Sanitize()` masks AzureAccessKey and ExportAzureAccessKey the same way it masks AmazonS3SecretAccessKey, so the shared key never reaches an API consumer in plain text. * `desanitize()` restores the masked keys on a config write so a PATCH that doesn't touch the key doesn't clobber it with the FakeSetting placeholder. * `configSensitivePaths` covers both Azure key paths so audit diffs don't include them either. * `ConfigToFileBackendSettings` in the `mattermost db` CLI helper gets the Azure branch its production counterpart already has — without it, `mattermost db migrate` / `db downgrade` would fail on Azure-configured deployments with "missing azure storage account setting". Finally, the shared FileBackendTestSuite is now wired against Azurite via TestAzureFileBackendTestSuite, which skips when CI_AZURITE_HOST is unreachable. The test-infra wiring (the docker service, the env vars, the start_dependencies entry) landed in a previous PR; this commit is what makes the suite actually exercise the Azure driver end to end. ------ AI assisted commit * Validate Azure timeout and path prefix in Config.IsValid Parity with the S3-side checks that already cover AmazonS3RequestTimeoutMilliseconds and AmazonS3PathPrefix. Without these, a zero/negative AzureRequestTimeoutMilliseconds passes validation and later creates immediately-expired request contexts, and leading/trailing whitespace in AzurePathPrefix produces blob keys that don't match what the admin configured. Same checks added for the Export* counterparts. The file_driver.app_error translation is updated to mention the new 'azureblob' option alongside 'local' and 'amazons3'. ------ AI assisted commit * Stream zip entries from the Azure backend writeZipEntry was calling ReadFile, which loads the entire blob into memory before writing it to the archive. For large blobs or deep directories this spikes RSS or OOMs the goroutine. Switch to Reader (the streaming azureRangeReader) and io.Copy into the zip entry so memory stays bounded regardless of blob size. ------ AI assisted commit * Use a backend-agnostic fallback for FileBackendNoBucketError The fallback Error() message was "no such bucket", which leaks S3 terminology when an Azure caller returns the type with no wrapped Err. Use "no such bucket or container" so logs and external error handling stay neutral across backends. ------ AI assisted commit * Defend Azure path prefix against directory traversal Reject ".." in AzurePathPrefix and ExportAzurePathPrefix at config validation time, since path.Join collapses traversal segments and a prefix like "../other-tenant" would otherwise escape the configured isolation boundary. Harden the prefix helper as a second line of defense: if the joined path no longer sits inside pathPrefix, fall back to joining the prefix with the base name of the caller-supplied path. That preserves the prefix invariant for plugin and import paths that the upload code does not sanitize uniformly. ------ AI assisted commit * Honor SkipVerify when constructing the Azure client FileBackendSettings.SkipVerify is plumbed through from the System Console the same way it is for S3, so admins toggling the flag for self-signed endpoints (Azurite, sovereign clouds) get the behavior they expect without having to drop SSL entirely and send the shared key in clear text. ------ AI assisted commit * Warn when the Azure request timeout falls back to its default Config.IsValid already rejects non-positive AzureRequestTimeoutMilliseconds for any path that goes through config validation, so this warn only fires for direct callers that bypass validation (tests, helpers). Logging the substitution turns a silent coercion into something an operator can correlate against unexpected request behavior. ------ AI assisted commit * Cap Azure request timeout at 10 minutes Reject AzureRequestTimeoutMilliseconds values above the ceiling so an operator (or someone who has admin access) cannot effectively disable timeouts by setting the value to math.MaxInt64. A hung Azure call then holds a goroutine open until the OS gives up. Applies the same bound to ExportAzureRequestTimeoutMilliseconds. S3 has the same gap; treating it is out of scope here but worth a follow-up. ------ AI assisted commit * Refuse AppendFile on blobs without a committed block list A blob written by another tool (Azure portal, azcopy, a migration script, a plugin using Put Blob) has its content in the blob but an empty committed-block list. Committing a new block list against such a blob silently replaces the existing content with only the appended bytes. Check the blob's properties before staging when the committed-block list is empty, and refuse with a clear error if the blob has content. Same hazard for an admin pointing the backend at an existing container with pre-existing files. Adds an integration test against Azurite to lock the behavior in. ------ AI assisted commit * Surface truncated reads from azureRangeReader Read closed the body cleanly and returned io.EOF even when the remote stream terminated before the blob's content length. Callers (and any retry layer above) then accepted a partial blob as complete. ReadAt unconditionally rewrote io.ErrUnexpectedEOF to io.EOF, which made truncated downloads indistinguishable from clean reads. That is exactly what zip.NewReader consumes for archive readers, so the bulk-import worker would silently import partial archives. Read now closes the body, nils it, and returns io.ErrUnexpectedEOF when EOF arrives before offset reaches size. ReadAt only collapses ErrUnexpectedEOF to EOF when the full count was delivered and the stream was consumed to the end of the blob. Otherwise the truncation propagates with context. Both code paths are exercised by new fakeDownloader-backed tests. ------ AI assisted commit * Move container provisioning out of Azure TestConnection Auto-creating the container inside TestConnection meant a typo in the System Console (mattermosst instead of mattermost) silently provisioned an unwanted container in the admin's Azure subscription, with no audit log and no warning. They'd discover it later when uploads landed somewhere unexpected. TestConnection now returns FileBackendNoBucketError when the container is missing, mirroring the S3 contract. A new MakeContainer method mirrors S3FileBackend.MakeBucket, and Server.Start dispatches via two capability interfaces (bucketMaker / containerMaker) instead of a hard S3 type assertion — so the NoBucket error is no longer silently swallowed for backends Server.Start has not been taught about. ------ AI assisted commit * Carry file backend auth detail through to AppError The Test Connection button collapsed every typed backend failure into the same generic i18n message. Operators trying to debug bad credentials or a missing bucket only saw "Unable to authenticate against the file storage backend" with no SDK code to grep for in their logs. Use errors.As so the typed checks survive future wrapping, and pass the underlying error string through the NewAppError details argument. The AppError serializer surfaces that detail to the admin console alongside the translated message, so a bad S3 InvalidAccessKeyId or an Azure AuthenticationFailed shows up in the toast without an i18n schema change. ------ AI assisted commit * Remove non-ascii characters from comments ------ AI assisted commit * Make linter happy ------ AI assisted commit * Harden Azure prefix boundary check strings.HasPrefix on the joined path is a string-level check, not a path-level one, so a configured prefix of "mattermost" accepts a joined result of "mattermost-evil/...". A crafted caller path like "../mattermost-evil/secrets" would collapse via path.Join to that exact sibling and slip through the boundary check, escaping the configured prefix scope. Require the joined path to be the cleaned prefix itself or to start with the prefix followed by a path separator. The fallback path.Join uses the same cleaned prefix for consistency. ------ AI assisted commit * Provision Azurite container in standalone test setup The shared FileBackendTestSuite's SetupTest already handles a missing container by detecting FileBackendNoBucketError from TestConnection and calling MakeContainer, but TestAzureFileBackendAppendRefusesNonBlockBlob bypasses SetupTest and calls TestConnection directly. On a fresh Azurite instance the test would fail before exercising the append-refusal logic. Extract a newAzuriteBackend(t) helper alongside azuriteSettings(t) that builds the backend and ensures the container exists, mirroring the suite's setup. Use errors.As for forward compatibility with future wrapping. ------ AI assisted commit * Fix grammar in email-settings i18n string "Email settings has unset values." -> "Email settings have unset values." ------ AI assisted commit * Make Azure MakeContainer idempotent Treat a ContainerAlreadyExists response as success so that two nodes racing through TestConnection plus MakeContainer at boot both converge instead of having the loser fail. Mirrors how the S3 backend handles the equivalent BucketAlreadyOwnedByYou case. ------ AI assisted commit * Narrow AzureEndpoint comment to path-style only The setting only builds path-style URLs, so it cannot reach sovereign clouds like Azure Government or Azure China, which require vhost-style endpoints. Update the comment to reflect what the code actually does and document that sovereign-cloud support is out of scope. ------ AI assisted commit
2026-05-14 12:59:18 -04:00
github.com/google/uuid v1.6.0
github.com/gorilla/handlers v1.5.2
github.com/gorilla/mux v1.8.1
github.com/gorilla/schema v1.4.1
github.com/gorilla/websocket v1.5.3
MM-19606- Rework Prepackaged Plugins (#13449) * MM-19609 - Add new prepackage configuration settings (#13062) * Add signatures to the prepackaged plugins (#13138) * MM-19612 - Support querying local plugin marketplace when upst… (#13250) * MM-19612 - Support querying local plugin marketplace when upstream unavailable or disabled * Update translations file * Fixed comment * Updated to check EnableRemoteMarketplace setting and LocalOnly to get marketplace plugins * Fixed unit tests * Tests cleanup code * Removed unused error message * Updated tests * MM-19614- Updated Marketplace Service error id (#13388) * [MM-19610] Consume prepackaged plugins (#13005) * consume prepackaged plugins into memory * missing i18n * remove spurious .gitignore changes * return on failure to install prepackged plugins * cleanup * s/plugins/availablePlugins * whitespace * don't return extractDir when not needed * s/plug/plugin * error on icon, cleanup * update armored version of testplugin signature * honour AutomaticPrepackagedPlugins * document getPrepackagedPlugin * MM-19613 - Include prepackaged plugins in marketplace results (#13433) * Added prepackaged plugins to marketplace results * PR Feedback * PR Feedback * Update error where definition * Removing unnecessary var declaration * Updated comments * MM-21263 - Use EnableRemoteMarketplace in marketplace install… (#13438) * MM-21263 - Use EnableRemoteMarketplace in marketplace install endpoint * Call updateConfig before calling NewServer in TestHelper * Added translations * PR feedback * Translations * Feedback * s/helpers.go/download.go * Converging env.PrepackagedPlugins * Initial PR feedback * Ordered imports properly * Updated DownloadURL to return slice of bytes * Fixed method typo * Fixed logging * Added read lock for prepackaged plugins list * PR Feedback * Added condition to only install prepackaged plugin if it was previously enabled * Linting * Updated to check plugin state in config * Closing filereader * Only add local label if remote marketplace is enabled * Updated local tag description * Fixed tests Co-authored-by: Ali Farooq <ali.farooq0@pm.me> Co-authored-by: Shota Gvinepadze <wineson@gmail.com> Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com> Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2020-01-15 13:38:55 -05:00
github.com/h2non/go-is-svg v0.0.0-20160927212452-35e8c4b0612c
github.com/hako/durafmt v0.0.0-20210608085754-5c1018a4e16b
github.com/hashicorp/go-multierror v1.1.1
github.com/hashicorp/memberlist v0.5.4
github.com/icrowley/fake v0.0.0-20240710202011-f797eb4a99c0
github.com/isacikgoz/prompt v0.1.0
github.com/jmoiron/sqlx v1.4.0
github.com/klauspost/compress v1.18.6
github.com/ledongthuc/pdf v0.0.0-20250511090121-5959a4027728
github.com/lib/pq v1.12.3
github.com/mattermost/go-i18n v1.11.1-0.20211013152124-5c415071e404
github.com/mattermost/gosaml2 v0.10.0
github.com/mattermost/ldap v0.0.0-20231116144001-0f480c025956
2025-04-28 10:20:42 -04:00
github.com/mattermost/logr/v2 v2.0.22
github.com/mattermost/mattermost-plugin-ai v1.14.0
github.com/mattermost/mattermost/server/public v0.4.0
github.com/mattermost/morph v1.1.0
github.com/mattermost/rsc v0.0.0-20160330161541-bbaefb05eaa0
github.com/mattermost/squirrel v0.5.0
github.com/mholt/archives v0.1.5
github.com/microcosm-cc/bluemonday v1.0.27
github.com/minio/minio-go/v7 v7.1.0
github.com/opensearch-project/opensearch-go/v4 v4.6.0
2020-02-03 11:19:38 -05:00
github.com/pkg/errors v0.9.1
github.com/prometheus/client_golang v1.23.2
github.com/prometheus/client_model v0.6.2
[MM-68655] Surface RPC errors from plugin hooks (#36414) * add WithRPCErr hooks (server-facing/internal only) * zero _returns on RPC failure in WithRPCErr companions Aligns the WithRPCErr template with the HooksRPCErr godoc contract: when g.client.Call returns a transport error, gob may have partially decoded the reply. Reassign _returns to a zero value before destructuring so callers always receive zeroed outputs alongside a non-nil transport error. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * rename HooksRPCErr to HooksWithRPCErr for naming consistency Every related symbol uses the WithRPCErr suffix (MessageHasBeenPostedWithRPCErr, RunMultiPluginHookWithRPCErr, RunMultiHookWithRPCErr, etc.). Aligning the interface name removes the only outlier and makes the convention uniform. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * rename rpcErrImpl to hooksWithRPCErrImpl Mirrors the existing hooksImpl/Hooks naming pattern on hooksTimerLayer. * add supervisor.HooksWithRPCErr() and drop runtime type assertion The old path did rp.supervisor.Hooks().(HooksWithRPCErr) and handled the "doesn't implement" branch — but that branch was structurally unreachable (the compile-time `_ HooksWithRPCErr = (*hooksTimerLayer)(nil)` assertion guards it). Change supervisor.hooks from `Hooks` to the concrete `*hooksTimerLayer` (which implements both interfaces, enforced at field assignment), add a parallel HooksWithRPCErr() accessor, and call it directly. Hooks() keeps its public Hooks-interface signature via implicit conversion at return. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * drop "implemented by" clause from HooksWithRPCErr godoc Both hooksRPCClient and hooksTimerLayer satisfy the interface, and naming implementations in interface godocs adds rot — the contract is what readers need, not the list of wrappers. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 10:41:11 -04:00
github.com/prometheus/common v0.67.5
github.com/redis/rueidis v1.0.75
github.com/reflog/dateconstraints v0.2.1
github.com/rs/cors v1.11.1
github.com/sirupsen/logrus v1.9.4
github.com/spf13/cobra v1.10.2
github.com/spf13/viper v1.21.0
github.com/splitio/go-client/v6 v6.10.0
github.com/stretchr/testify v1.11.1
github.com/throttled/throttled/v2 v2.15.0
github.com/tinylib/msgp v1.6.4
github.com/tylerb/graceful v1.2.15
github.com/vmihailenco/msgpack/v5 v5.4.1
github.com/wiggin77/merror v1.0.5
github.com/xtgo/uuid v0.0.0-20140804021211-a0b114877d4c
github.com/yuin/goldmark v1.8.2
golang.org/x/crypto v0.51.0
golang.org/x/image v0.40.0
golang.org/x/net v0.54.0
golang.org/x/sync v0.20.0
golang.org/x/sys v0.44.0
golang.org/x/term v0.43.0
golang.org/x/text v0.37.0
gopkg.in/mail.v2 v2.3.1
)
Optimize gzip compression of responses (#15416) It looks like maintainers aren't very active on the repo and https://github.com/nytimes/gziphandler/pull/107 has come to a standstill. I don't want to wait forever for this to go in. Let's use a replace directive to point to the fork. When the PR gets merged, we can just bump the dependency and remove the replace directive. Re: the PR, copying the text from the PR description This gives 40-50% improvements in CPU with very minor increase in memory, just as a drop-in replacement. I think that is a very reasonable tradeoff. The library is mature and safe to be used in production. ``` name old time/op new time/op delta GzipHandler_S2k-8 74.9µs ± 2% 34.4µs ± 2% -54.07% (p=0.000 n=10+9) GzipHandler_S20k-8 379µs ± 1% 226µs ± 3% -40.42% (p=0.000 n=9+10) GzipHandler_S100k-8 1.95ms ± 2% 1.15ms ± 1% -41.27% (p=0.000 n=9+9) GzipHandler_P2k-8 24.3µs ±25% 10.7µs ±25% -55.80% (p=0.000 n=10+10) GzipHandler_P20k-8 132µs ± 2% 75µs ± 1% -42.95% (p=0.000 n=9+10) GzipHandler_P100k-8 658µs ± 2% 371µs ± 3% -43.68% (p=0.000 n=9+10) name old alloc/op new alloc/op delta GzipHandler_S2k-8 7.71kB ± 5% 9.13kB ± 7% +18.33% (p=0.000 n=10+10) GzipHandler_S20k-8 65.1kB ± 3% 70.3kB ± 3% +8.05% (p=0.000 n=10+10) GzipHandler_S100k-8 348kB ± 4% 382kB ± 2% +9.85% (p=0.000 n=10+10) GzipHandler_P2k-8 7.60kB ± 1% 7.93kB ± 2% +4.33% (p=0.000 n=10+10) GzipHandler_P20k-8 64.4kB ± 1% 66.3kB ± 2% +2.92% (p=0.000 n=10+10) GzipHandler_P100k-8 304kB ± 1% 309kB ± 1% +1.67% (p=0.000 n=10+9) name old allocs/op new allocs/op delta GzipHandler_S2k-8 21.0 ± 0% 21.0 ± 0% ~ (all equal) GzipHandler_S20k-8 24.0 ± 0% 24.0 ± 0% ~ (all equal) GzipHandler_S100k-8 27.0 ± 0% 27.0 ± 0% ~ (all equal) GzipHandler_P2k-8 21.0 ± 0% 21.0 ± 0% ~ (all equal) GzipHandler_P20k-8 24.0 ± 0% 24.0 ± 0% ~ (all equal) GzipHandler_P100k-8 26.0 ± 0% 26.0 ± 0% ~ (all equal) ``` https://mattermost.atlassian.net/browse/MM-28491
2020-09-09 23:32:20 -04:00
require (
filippo.io/edwards25519 v1.2.0 // indirect
MM-68662: Add Azure Blob Storage filestore backend (#36498) * Generalize file backend error types Replace S3FileBackendAuthError and S3FileBackendNoBucketError with backend-agnostic FileBackendAuthError and FileBackendNoBucketError so non-S3 drivers can return them and the admin "Test Connection" flow keeps surfacing useful messages. The old S3-prefixed names are kept as type aliases of the generic types so external code (plugins, historical consumers) continues to compile, and so existing S3 construction sites stay untouched. The type switch in connectionTestErrorToAppError now matches the generic types, with new i18n keys (test_connection_auth.app_error and test_connection_no_bucket.app_error) whose wording does not name S3. The old S3-specific i18n keys are dropped via `make i18n-extract` since they are no longer referenced from code; the api4 test that asserted on those keys is updated, and the Cypress `MM-T996 Amazon S3 connection error messaging` spec that asserted on the old user-facing string is updated to the new wording. ------ AI assisted commit * Pull in Azure SDK and uuid dependencies Bring in github.com/Azure/azure-sdk-for-go/sdk/azcore and .../sdk/storage/azblob (with .../sdk/internal as their indirect dependency). The two are needed by the upcoming Azure Blob Storage filestore backend and its lazy-Range-backed reader. The bump of golang.org/x/{crypto,net,sys,term,text} comes transitively from azblob's minimum versions. Also promotes github.com/google/uuid from indirect to direct, since the Azure backend uses it to generate block IDs that share the same wire format the SDK itself produces in UploadStream. ------ AI assisted commit * Add azureRangeReader, a seekable Range-backed blob reader A small standalone type that satisfies the FileBackend interface's ReadCloseSeeker + the broader io.ReaderAt contract on top of Azure Blob Storage HTTP Range requests. Lands as its own commit because the upcoming Azure FileBackend driver builds on it, and the reader itself is independently useful — and independently testable against a fake downloader without standing up an Azure client. Design notes: * Read opens an HTTP Range stream lazily at the current offset and reuses it for sequential reads. Seek to a different offset closes the open stream; the next Read re-opens it. * Seek to the same offset is a no-op and does not close the open stream, so callers like zip.NewReader that probe with redundant seeks don't kick off a fresh download. * ReadAt issues a dedicated ranged DownloadStream per call and does not touch the streaming cursor — matches the io.ReaderAt contract the bulk-import worker's zip.NewReader path relies on. * Close cancels the context (which any in-flight Azure call will observe and abort), stops the deadline timer, and closes the current body if any. It is safe to call when no body was ever opened. * CancelTimeout lets long-running consumers like the import worker opt out of the per-operation deadline that would otherwise kill multi-minute downloads partway through. The implementation talks to a small blobDownloader interface rather than *blob.Client directly so the unit tests can substitute a fake downloader that records every requested Range and tracks Close calls on the bodies it hands out. ------ AI assisted commit * Add Azure Blob Storage filestore driver Implements the FileBackend interface against Azure Blob Storage in a new azurestore.go (~520 LOC). The driver is not yet selectable via NewFileBackend's switch — that wiring lands in the next commit together with the admin config surface — but the driver itself is complete and self-contained behind the FileBackendSettings struct. Filesstore.go grows three pieces of supporting infrastructure that the driver consumes: * a `driverAzure = "azureblob"` constant alongside the existing driverS3 and driverLocal, * an Azure-specific block on FileBackendSettings (storage account, access key, container, path prefix, endpoint, SSL flag, request timeout), * a CheckMandatoryAzureFields validator that mirrors CheckMandatoryS3Fields. Behavioural notes that warrant calling out: * Reader returns the previously-added azureRangeReader, so reads stream lazily over HTTP Range and ReadAt is available for the bulk-import worker's zip.NewReader path. The deadline timer is armed before the initial GetProperties call so the HEAD itself is bounded. * WriteFile and AppendFile both go through StageBlock + CommitBlockList via a shared stageBlocks helper, never the SDK's UploadStream. UploadStream's small-payload fast path falls back to single-shot PutBlob, which leaves the resulting blob with no committed block list; a subsequent AppendFile that calls CommitBlockList on that blob would then clobber its content. Routing every write through the block-list mechanism keeps AppendFile correct regardless of payload size. * AppendFile stages the new chunk as one or more blocks and commits the existing committed block list plus the newly staged IDs. The new bytes go up exactly once — no re-download, no re-concatenate, no re-upload of the prior contents. * WriteFileContext does not wrap the caller-supplied context with its own timeout — that timeout is applied in WriteFile only, matching the S3 driver, so long-running TryWriteFileContext callers (like message-export bulk writes) opt out of the per-operation timeout the way the abstraction documents. Authentication is shared-key only for this drop; Microsoft Entra ID / managed identity is deferred to a follow-up. The endpoint is configurable so the same code targets the production Azure host (vhost style — {account}.blob.core.windows.net) or Azurite / Azure Government / sovereign clouds (path style — host[:port]/{account}). ------ AI assisted commit * Wire Azure backend into config, validation, and driver selection This commit registers the previously-added AzureFileBackend driver with the rest of the system. Until now the driver was usable only via direct construction; after this commit, `DriverName: "azureblob"` in config.json is a fully-supported deployment configuration. Five integration sites are touched: * `newFileBackend` in filesstore.go now dispatches `driverAzure` to NewAzureFileBackend, alongside the existing s3 and local cases. NewFileBackendSettingsFromConfig (and its export counterpart) gain an Azure branch that maps the model.FileSettings fields onto the Azure-specific FileBackendSettings fields. * `model.FileSettings` grows the user-facing Azure config schema: storage account, access key, container, path prefix, endpoint, SSL flag, request timeout, plus matching Export* fields for the dedicated export store. SetDefaults populates them so deployments that never opted into Azure don't carry nil pointers. `isValid` accepts the new ImageDriverAzure constant. * `Config.Sanitize()` masks AzureAccessKey and ExportAzureAccessKey the same way it masks AmazonS3SecretAccessKey, so the shared key never reaches an API consumer in plain text. * `desanitize()` restores the masked keys on a config write so a PATCH that doesn't touch the key doesn't clobber it with the FakeSetting placeholder. * `configSensitivePaths` covers both Azure key paths so audit diffs don't include them either. * `ConfigToFileBackendSettings` in the `mattermost db` CLI helper gets the Azure branch its production counterpart already has — without it, `mattermost db migrate` / `db downgrade` would fail on Azure-configured deployments with "missing azure storage account setting". Finally, the shared FileBackendTestSuite is now wired against Azurite via TestAzureFileBackendTestSuite, which skips when CI_AZURITE_HOST is unreachable. The test-infra wiring (the docker service, the env vars, the start_dependencies entry) landed in a previous PR; this commit is what makes the suite actually exercise the Azure driver end to end. ------ AI assisted commit * Validate Azure timeout and path prefix in Config.IsValid Parity with the S3-side checks that already cover AmazonS3RequestTimeoutMilliseconds and AmazonS3PathPrefix. Without these, a zero/negative AzureRequestTimeoutMilliseconds passes validation and later creates immediately-expired request contexts, and leading/trailing whitespace in AzurePathPrefix produces blob keys that don't match what the admin configured. Same checks added for the Export* counterparts. The file_driver.app_error translation is updated to mention the new 'azureblob' option alongside 'local' and 'amazons3'. ------ AI assisted commit * Stream zip entries from the Azure backend writeZipEntry was calling ReadFile, which loads the entire blob into memory before writing it to the archive. For large blobs or deep directories this spikes RSS or OOMs the goroutine. Switch to Reader (the streaming azureRangeReader) and io.Copy into the zip entry so memory stays bounded regardless of blob size. ------ AI assisted commit * Use a backend-agnostic fallback for FileBackendNoBucketError The fallback Error() message was "no such bucket", which leaks S3 terminology when an Azure caller returns the type with no wrapped Err. Use "no such bucket or container" so logs and external error handling stay neutral across backends. ------ AI assisted commit * Defend Azure path prefix against directory traversal Reject ".." in AzurePathPrefix and ExportAzurePathPrefix at config validation time, since path.Join collapses traversal segments and a prefix like "../other-tenant" would otherwise escape the configured isolation boundary. Harden the prefix helper as a second line of defense: if the joined path no longer sits inside pathPrefix, fall back to joining the prefix with the base name of the caller-supplied path. That preserves the prefix invariant for plugin and import paths that the upload code does not sanitize uniformly. ------ AI assisted commit * Honor SkipVerify when constructing the Azure client FileBackendSettings.SkipVerify is plumbed through from the System Console the same way it is for S3, so admins toggling the flag for self-signed endpoints (Azurite, sovereign clouds) get the behavior they expect without having to drop SSL entirely and send the shared key in clear text. ------ AI assisted commit * Warn when the Azure request timeout falls back to its default Config.IsValid already rejects non-positive AzureRequestTimeoutMilliseconds for any path that goes through config validation, so this warn only fires for direct callers that bypass validation (tests, helpers). Logging the substitution turns a silent coercion into something an operator can correlate against unexpected request behavior. ------ AI assisted commit * Cap Azure request timeout at 10 minutes Reject AzureRequestTimeoutMilliseconds values above the ceiling so an operator (or someone who has admin access) cannot effectively disable timeouts by setting the value to math.MaxInt64. A hung Azure call then holds a goroutine open until the OS gives up. Applies the same bound to ExportAzureRequestTimeoutMilliseconds. S3 has the same gap; treating it is out of scope here but worth a follow-up. ------ AI assisted commit * Refuse AppendFile on blobs without a committed block list A blob written by another tool (Azure portal, azcopy, a migration script, a plugin using Put Blob) has its content in the blob but an empty committed-block list. Committing a new block list against such a blob silently replaces the existing content with only the appended bytes. Check the blob's properties before staging when the committed-block list is empty, and refuse with a clear error if the blob has content. Same hazard for an admin pointing the backend at an existing container with pre-existing files. Adds an integration test against Azurite to lock the behavior in. ------ AI assisted commit * Surface truncated reads from azureRangeReader Read closed the body cleanly and returned io.EOF even when the remote stream terminated before the blob's content length. Callers (and any retry layer above) then accepted a partial blob as complete. ReadAt unconditionally rewrote io.ErrUnexpectedEOF to io.EOF, which made truncated downloads indistinguishable from clean reads. That is exactly what zip.NewReader consumes for archive readers, so the bulk-import worker would silently import partial archives. Read now closes the body, nils it, and returns io.ErrUnexpectedEOF when EOF arrives before offset reaches size. ReadAt only collapses ErrUnexpectedEOF to EOF when the full count was delivered and the stream was consumed to the end of the blob. Otherwise the truncation propagates with context. Both code paths are exercised by new fakeDownloader-backed tests. ------ AI assisted commit * Move container provisioning out of Azure TestConnection Auto-creating the container inside TestConnection meant a typo in the System Console (mattermosst instead of mattermost) silently provisioned an unwanted container in the admin's Azure subscription, with no audit log and no warning. They'd discover it later when uploads landed somewhere unexpected. TestConnection now returns FileBackendNoBucketError when the container is missing, mirroring the S3 contract. A new MakeContainer method mirrors S3FileBackend.MakeBucket, and Server.Start dispatches via two capability interfaces (bucketMaker / containerMaker) instead of a hard S3 type assertion — so the NoBucket error is no longer silently swallowed for backends Server.Start has not been taught about. ------ AI assisted commit * Carry file backend auth detail through to AppError The Test Connection button collapsed every typed backend failure into the same generic i18n message. Operators trying to debug bad credentials or a missing bucket only saw "Unable to authenticate against the file storage backend" with no SDK code to grep for in their logs. Use errors.As so the typed checks survive future wrapping, and pass the underlying error string through the NewAppError details argument. The AppError serializer surfaces that detail to the admin console alongside the translated message, so a bad S3 InvalidAccessKeyId or an Azure AuthenticationFailed shows up in the toast without an i18n schema change. ------ AI assisted commit * Remove non-ascii characters from comments ------ AI assisted commit * Make linter happy ------ AI assisted commit * Harden Azure prefix boundary check strings.HasPrefix on the joined path is a string-level check, not a path-level one, so a configured prefix of "mattermost" accepts a joined result of "mattermost-evil/...". A crafted caller path like "../mattermost-evil/secrets" would collapse via path.Join to that exact sibling and slip through the boundary check, escaping the configured prefix scope. Require the joined path to be the cleaned prefix itself or to start with the prefix followed by a path separator. The fallback path.Join uses the same cleaned prefix for consistency. ------ AI assisted commit * Provision Azurite container in standalone test setup The shared FileBackendTestSuite's SetupTest already handles a missing container by detecting FileBackendNoBucketError from TestConnection and calling MakeContainer, but TestAzureFileBackendAppendRefusesNonBlockBlob bypasses SetupTest and calls TestConnection directly. On a fresh Azurite instance the test would fail before exercising the append-refusal logic. Extract a newAzuriteBackend(t) helper alongside azuriteSettings(t) that builds the backend and ensures the container exists, mirroring the suite's setup. Use errors.As for forward compatibility with future wrapping. ------ AI assisted commit * Fix grammar in email-settings i18n string "Email settings has unset values." -> "Email settings have unset values." ------ AI assisted commit * Make Azure MakeContainer idempotent Treat a ContainerAlreadyExists response as success so that two nodes racing through TestConnection plus MakeContainer at boot both converge instead of having the loser fail. Mirrors how the S3 backend handles the equivalent BucketAlreadyOwnedByYou case. ------ AI assisted commit * Narrow AzureEndpoint comment to path-style only The setting only builds path-style URLs, so it cannot reach sovereign clouds like Azure Government or Azure China, which require vhost-style endpoints. Update the comment to reflect what the code actually does and document that sovereign-cloud support is out of scope. ------ AI assisted commit
2026-05-14 12:59:18 -04:00
github.com/Azure/azure-sdk-for-go/sdk/internal v1.12.0 // indirect
github.com/JalfResi/justext v0.0.0-20221106200834-be571e3e3052 // indirect
github.com/PuerkitoBio/goquery v1.12.0 // indirect
github.com/STARRY-S/zip v0.2.3 // indirect
github.com/advancedlogic/GoOse v0.0.0-20231203033844-ae6b36caf275 // indirect
github.com/andybalholm/brotli v1.2.1 // indirect
github.com/andybalholm/cascadia v1.3.3 // indirect
github.com/araddon/dateparse v0.0.0-20210429162001-6b43995a97de // indirect
github.com/armon/go-metrics v0.4.1 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.23 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.23 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.24 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.9 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.23 // indirect
github.com/aws/aws-sdk-go-v2/service/signin v1.0.11 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.30.17 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.21 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.42.1 // indirect
github.com/aws/smithy-go v1.25.1 // indirect
github.com/aymerick/douceur v0.2.0 // indirect
github.com/beevik/etree v1.6.0 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/bits-and-blooms/bitset v1.24.4 // indirect
github.com/bits-and-blooms/bloom/v3 v3.7.1 // indirect
github.com/bodgit/plumbing v1.3.0 // indirect
github.com/bodgit/sevenzip v1.6.2 // indirect
github.com/bodgit/windows v1.0.1 // indirect
github.com/clipperhouse/displaywidth v0.11.0 // indirect
github.com/clipperhouse/uax29/v2 v2.7.0 // indirect
github.com/corpix/uarand v0.2.0 // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.7 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/dsnet/compress v0.0.2-0.20230904184137-39efe44ab707 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/elastic/elastic-transport-go/v8 v8.11.0 // indirect
github.com/fatih/set v0.2.1 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/francoispqt/gojay v1.2.13 // indirect
github.com/fsnotify/fsnotify v1.10.1 // indirect
github.com/gigawattio/window v0.0.0-20180317192513-0f5467e35573 // indirect
github.com/go-asn1-ber/asn1-ber v1.5.7 // indirect
github.com/go-ini/ini v1.67.0 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-resty/resty/v2 v2.17.2 // indirect
github.com/go-sql-driver/mysql v1.10.0 // indirect
github.com/go-viper/mapstructure/v2 v2.5.0 // indirect
github.com/goccy/go-json v0.10.6 // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/gomodule/redigo v2.0.0+incompatible // indirect
github.com/google/btree v1.1.3 // indirect
github.com/google/jsonschema-go v0.4.3 // indirect
github.com/gorilla/css v1.0.1 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-hclog v1.6.3 // indirect
github.com/hashicorp/go-immutable-radix v1.3.1 // indirect
github.com/hashicorp/go-metrics v0.5.4 // indirect
github.com/hashicorp/go-msgpack/v2 v2.1.5 // indirect
github.com/hashicorp/go-plugin v1.8.0 // indirect
github.com/hashicorp/go-sockaddr v1.0.7 // indirect
github.com/hashicorp/golang-lru v1.0.2 // indirect
github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
github.com/hashicorp/yamux v0.1.2 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/isacikgoz/fuzzy v0.2.0 // indirect
github.com/jaytaylor/html2text v0.0.0-20260303211410-1a4bdc82ecec // indirect
github.com/jonboulle/clockwork v0.5.0 // indirect
github.com/klauspost/cpuid/v2 v2.3.0 // indirect
github.com/klauspost/crc32 v1.3.0 // indirect
github.com/klauspost/pgzip v1.2.6 // indirect
github.com/lann/builder v0.0.0-20180802200727-47ae307949d0 // indirect
github.com/lann/ps v0.0.0-20150810152359-62de8c46ede0 // indirect
github.com/levigross/exp-html v0.0.0-20120902181939-8df60c69a8f5 // indirect
github.com/mattermost/xml-roundtrip-validator v0.1.0 // indirect
github.com/mattn/go-colorable v0.1.14 // indirect
github.com/mattn/go-isatty v0.0.22 // indirect
github.com/mattn/go-runewidth v0.0.23 // indirect
github.com/mattn/go-sqlite3 v2.0.3+incompatible // indirect
github.com/miekg/dns v1.1.72 // indirect
github.com/mikelolasagasti/xz v1.0.1 // indirect
github.com/minio/crc64nvme v1.1.1 // indirect
github.com/minio/md5-simd v1.1.2 // indirect
github.com/minio/minlz v1.1.1 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/ncruces/go-strftime v1.0.0 // indirect
github.com/nwaples/rardecode/v2 v2.2.2 // indirect
github.com/oklog/run v1.2.0 // indirect
github.com/olekukonko/cat v0.0.0-20250911104152-50322a0618f6 // indirect
github.com/olekukonko/errors v1.3.0 // indirect
github.com/olekukonko/ll v0.1.8 // indirect
github.com/olekukonko/tablewriter v1.1.4 // indirect
github.com/otiai10/gosseract/v2 v2.4.1 // indirect
github.com/pborman/uuid v1.2.1 // indirect
2022-05-10 03:58:09 -04:00
github.com/pelletier/go-toml v1.9.5 // indirect
github.com/pelletier/go-toml/v2 v2.3.1 // indirect
github.com/philhofer/fwd v1.2.0 // indirect
github.com/pierrec/lz4/v4 v4.1.26 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/prometheus/procfs v0.20.1 // indirect
github.com/redis/go-redis/v9 v9.19.0 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
github.com/richardlehane/mscfb v1.0.6 // indirect
github.com/richardlehane/msoleps v1.0.6 // indirect
github.com/rs/xid v1.6.0 // indirect
github.com/russellhaering/goxmldsig v1.6.0 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/sagikazarmark/locafero v0.12.0 // indirect
github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529 // indirect
github.com/sorairolake/lzip-go v0.3.8 // indirect
github.com/spf13/afero v1.15.0 // indirect
github.com/spf13/cast v1.10.0 // indirect
github.com/spf13/pflag v1.0.10 // indirect
github.com/splitio/go-split-commons/v9 v9.1.0 // indirect
github.com/splitio/go-toolkit/v5 v5.4.1 // indirect
github.com/ssor/bom v0.0.0-20170718123548-6386211fdfcf // indirect
github.com/stretchr/objx v0.5.3 // indirect
github.com/subosito/gotenv v1.6.0 // indirect
github.com/ulikunitz/xz v0.5.15 // indirect
github.com/vmihailenco/tagparser/v2 v2.0.0 // indirect
github.com/wiggin77/srslog v1.0.1 // indirect
github.com/zeebo/xxh3 v1.1.0 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/otel v1.43.0 // indirect
go.opentelemetry.io/otel/metric v1.43.0 // indirect
go.opentelemetry.io/otel/trace v1.43.0 // indirect
go.uber.org/atomic v1.11.0 // indirect
go.yaml.in/yaml/v2 v2.4.4 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
go4.org v0.0.0-20260112195520-a5071408f32f // indirect
golang.org/x/exp v0.0.0-20260508232706-74f9aab9d74a // indirect
golang.org/x/mod v0.36.0 // indirect
golang.org/x/tools v0.45.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260511170946-3700d4141b60 // indirect
google.golang.org/grpc v1.81.0 // indirect
google.golang.org/protobuf v1.36.11 // indirect
gopkg.in/alexcesaro/quotedprintable.v3 v3.0.0-20150716171945-2caba252f4dc // indirect
gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect
2025-01-13 14:23:09 -05:00
gopkg.in/yaml.v2 v2.4.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
modernc.org/libc v1.72.3 // indirect
modernc.org/mathutil v1.7.1 // indirect
modernc.org/memory v1.11.0 // indirect
modernc.org/sqlite v1.50.1 // indirect
)
// See MM-66167, MM-68222 for more details.
replace github.com/vmihailenco/msgpack/v5 => github.com/mattermost/msgpack/v5 v5.0.0-20260408165622-cadfad56a815
// See MM-63434 for more details.
replace github.com/ledongthuc/pdf => github.com/jgheithcock/pdf v0.0.0-20260404175814-28cd6530c1fe