77 commits

Author	SHA1	Message	Date
Christopher Speller	9b01e406f4	Move password hashers from server/v8 to server/public to fix module layering violation (#35805)a ... * Move password hashers from server/v8 to server/public to fix layering violation * Revert "Move password hashers from server/v8 to server/public to fix layering violation" This reverts commit 8cad5b8dc9. * invert dependency between hashers and model * make modules-tidy --------- Co-authored-by: Jesse Hallam <jesse@mattermost.com> Co-authored-by: Mattermost Build <build@mattermost.com>	2026-04-01 15:20:12 +00:00
Jesse Hallam	f7f2d944e8	upgrade golangci-lint (#35845)	2026-03-30 13:41:32 -03:00
Miguel de la Cruz	48f2fd0873	Merge the Integrated Boards MVP feature branch (#35796) ... * Add CreatedBy and UpdatedBy to the properties fields and values (#34485) * Add CreatedBy and UpdatedBy to the properties fields and values * Fix types --------- Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es> * Adds ObjectType to the property fields table (#34908) Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es> * Update ObjectType migration setting an empty value and marking the column as not null (#34915) Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es> * Adds uniqueness mechanisms to the property fields (#35058) * Adds uniqueness mechanisms to the property fields After adding ObjectType, this commit ensures that both the PSAv1 and PSAv2 schemas are supported, and enforces property uniqueness through both database indexes and a logical check when creating new property fields. * Adds uniqueness check to property updates Updates are covered on this commit and we refactor as well the SQL code to use the squirrel builder and work better with the conditional addition of the `existingID` piece of the query. * Add translations to error messages * Fixing retrylayer mocks * Remove retrylayer duplication * Address review comments * Fix comment to avoid linter issues * Address PR comments * Update server/channels/db/migrations/postgres/000157_add_object_type_to_property_fields.down.sql Co-authored-by: Alejandro García Montoro <alejandro.garciamontoro@gmail.com> * Update server/channels/db/migrations/postgres/000157_add_object_type_to_property_fields.up.sql Co-authored-by: Alejandro García Montoro <alejandro.garciamontoro@gmail.com> * Update server/channels/db/migrations/postgres/000157_add_object_type_to_property_fields.up.sql Co-authored-by: Alejandro García Montoro <alejandro.garciamontoro@gmail.com> * Update field validation to check only for valid target types * Update migrations to avoid concurrent index creation within a transaction * Update migrations to make all index ops concurrent * Update tests to use valid PSAv2 property fields * Adds a helper for valid PSAv2 TargetTypes --------- Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es> Co-authored-by: Alejandro García Montoro <alejandro.garciamontoro@gmail.com> * Fix property tests (#35388) Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es> * Adds Integrated Boards feature flag (#35378) Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es> * Adds Integrated Boards MVP API changes (#34822) This PR includes the necessary changes for channels and posts endpoints and adds a set of generic endpoints to retrieve and manage property fields and values following the new Property System approach. Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es> Co-authored-by: Mattermost Build <build@mattermost.com> * Property System Architecture permissions for v2 (#35113) * Adds uniqueness mechanisms to the property fields After adding ObjectType, this commit ensures that both the PSAv1 and PSAv2 schemas are supported, and enforces property uniqueness through both database indexes and a logical check when creating new property fields. * Adds uniqueness check to property updates Updates are covered on this commit and we refactor as well the SQL code to use the squirrel builder and work better with the conditional addition of the `existingID` piece of the query. * Add translations to error messages * Add the permissions to the migrations, model and update the store calls * Adds the property field and property group app layer * Adds authorization helpers for property fields and values * Make sure that users cannot lock themselves out of property fields * Migrate permissions from a JSON column to three normalized columns * Remove the audit comment * Use target level constants in authorization * Log authorization membership failures * Rename admin to sysadmin * Fix i18n sorting --------- Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es> * Add Views store and app layer (#35361) * Add Views store and app layer for Integrated Boards Implements the View entity (model, SQL store, service, app) as described in the Integrated Boards tech spec. Views are channel-scoped board configurations with typed props (board, kanban subviews) and soft-delete. - public/model: View, ViewBoardProps, Subview, ViewPatch types with PreSave/PreUpdate/IsValid/Patch/Clone/Auditable - Migration 158: Views table with jsonb Props column and indexes - SqlViewStore: CRUD with nil-safe Props marshaling (AppendBinaryFlag) - ViewService: CreateView seeds default kanban subview and links the boards property field; caches boardPropertyFieldID at startup - App layer: CreateView/GetView/GetViewsForChannel/UpdateView/DeleteView with channel-membership permission checks and WebSocket events (view_created, view_updated, view_deleted) - doSetupBoardsPropertyField: registers the Boards property group and board field in NewServer() before ViewService construction - GetFieldByName now returns store.ErrNotFound instead of raw sql.ErrNoRows * Move permission checks out of App layer for views - Remove HasPermissionToChannel calls from all App view methods - Drop userID params from GetView, GetViewsForChannel, UpdateView, DeleteView - Fix doSetupBoardsPropertyField to include required TargetType for PSAv2 field * Make View service generic and enforce board validation in model - Remove board-specific auto-setup from service and server startup - Enforce that board views require Props, at least one subview, and at least one linked property in IsValid() - Move default subview seeding out of app layer; callers must provide valid props - Call PreSave on subviews during PreUpdate to assign IDs to new subviews - Update all tests to reflect the new validation requirements * Restore migrations files to match base branch * Distinguish ErrNotFound from other errors in view store Get * Use CONCURRENTLY and nontransactional for index operations in views migration * Split views index creation into separate nontransactional migrations * Update migrations.list * Update i18n translations for views * Fix makeView helper to include required Props for board view validation * Rename ctx parameter from c to rctx in OAuthProvider mock * Remove views service layer, call store directly from app * Return 500 for unexpected DB errors in GetView, 404 only for not-found * Harden View model: deep-copy Props, validate linked property IDs - Add ViewBoardProps.Clone() to deep-copy LinkedProperties and Subviews - Use it in View.Clone() and View.Patch() to prevent shared-slice aliasing - Iterate over LinkedProperties in View.IsValid() and reject invalid IDs with a dedicated i18n key - Register ViewStore in storetest AssertExpectations so mock expectations are enforced - Add tests covering all new behaviours * Restore autotranslation worker_stopped i18n translation * Fix view store test IDs and improve error handling in app layer - Use model.NewId() for linked property IDs in testUpdateView to fix validation failure (IsValid rejects non-UUID strings) - Fix import grouping in app/view.go (stdlib imports in one block) - Return 404 instead of 500 when Update/Delete store calls return ErrNotFound (e.g. concurrent deletion TOCTOU race) * Add View store mock to retrylayer test genStore helper The View store was added to the store interface but the genStore() helper in retrylayer_test.go was not updated, causing TestRetry to panic. Also removes the duplicate Recap mock registration. * Refactor view deletion and websocket event handling; update SQL store methods to use query builder * revert property field store * Remove useless migrations * Add cursor-based pagination to View store GetForChannel - Add ViewQueryCursor and ViewQueryOpts types with validation - Return (views, cursor, error) for caller-driven pagination - PerPage clamping: <=0 defaults to 20, >200 clamps to 200 - Support IncludeDeleted filter - Add comprehensive store tests for pagination, cursor edge cases, PerPage clamping, and invalid input rejection - Add app layer test for empty channelID → 400 - Update interface, retrylayer, timerlayer, and mock signatures Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Refactor test loops in ViewStore tests for improved readability * change pagination to limit/offset * Add upper-bound limits on View Subviews and LinkedProperties Defense-in-depth validation: cap Subviews at 50 and LinkedProperties at 500 to prevent abuse below the 300KB payload limit. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * MM-67388, MM-66528, MM-67750: Add View REST API endpoints, websocket events, and sort order (#35442) * Add Views store and app layer for Integrated Boards Implements the View entity (model, SQL store, service, app) as described in the Integrated Boards tech spec. Views are channel-scoped board configurations with typed props (board, kanban subviews) and soft-delete. - public/model: View, ViewBoardProps, Subview, ViewPatch types with PreSave/PreUpdate/IsValid/Patch/Clone/Auditable - Migration 158: Views table with jsonb Props column and indexes - SqlViewStore: CRUD with nil-safe Props marshaling (AppendBinaryFlag) - ViewService: CreateView seeds default kanban subview and links the boards property field; caches boardPropertyFieldID at startup - App layer: CreateView/GetView/GetViewsForChannel/UpdateView/DeleteView with channel-membership permission checks and WebSocket events (view_created, view_updated, view_deleted) - doSetupBoardsPropertyField: registers the Boards property group and board field in NewServer() before ViewService construction - GetFieldByName now returns store.ErrNotFound instead of raw sql.ErrNoRows * Move permission checks out of App layer for views - Remove HasPermissionToChannel calls from all App view methods - Drop userID params from GetView, GetViewsForChannel, UpdateView, DeleteView - Fix doSetupBoardsPropertyField to include required TargetType for PSAv2 field * Make View service generic and enforce board validation in model - Remove board-specific auto-setup from service and server startup - Enforce that board views require Props, at least one subview, and at least one linked property in IsValid() - Move default subview seeding out of app layer; callers must provide valid props - Call PreSave on subviews during PreUpdate to assign IDs to new subviews - Update all tests to reflect the new validation requirements * Restore migrations files to match base branch * Distinguish ErrNotFound from other errors in view store Get * Use CONCURRENTLY and nontransactional for index operations in views migration * Split views index creation into separate nontransactional migrations * Update migrations.list * Update i18n translations for views * Fix makeView helper to include required Props for board view validation * Rename ctx parameter from c to rctx in OAuthProvider mock * Remove views service layer, call store directly from app * Return 500 for unexpected DB errors in GetView, 404 only for not-found * Harden View model: deep-copy Props, validate linked property IDs - Add ViewBoardProps.Clone() to deep-copy LinkedProperties and Subviews - Use it in View.Clone() and View.Patch() to prevent shared-slice aliasing - Iterate over LinkedProperties in View.IsValid() and reject invalid IDs with a dedicated i18n key - Register ViewStore in storetest AssertExpectations so mock expectations are enforced - Add tests covering all new behaviours * Restore autotranslation worker_stopped i18n translation * Fix view store test IDs and improve error handling in app layer - Use model.NewId() for linked property IDs in testUpdateView to fix validation failure (IsValid rejects non-UUID strings) - Fix import grouping in app/view.go (stdlib imports in one block) - Return 404 instead of 500 when Update/Delete store calls return ErrNotFound (e.g. concurrent deletion TOCTOU race) * Add View store mock to retrylayer test genStore helper The View store was added to the store interface but the genStore() helper in retrylayer_test.go was not updated, causing TestRetry to panic. Also removes the duplicate Recap mock registration. * Refactor view deletion and websocket event handling; update SQL store methods to use query builder * revert property field store * Add View API endpoints with OpenAPI spec, client methods, and i18n Implement REST API for channel views (board-type) behind the IntegratedBoards feature flag. Adds CRUD endpoints under /api/v4/channels/{channel_id}/views with permission checks matching the channel bookmark pattern. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Remove useless migrations * Add cursor-based pagination to View store GetForChannel - Add ViewQueryCursor and ViewQueryOpts types with validation - Return (views, cursor, error) for caller-driven pagination - PerPage clamping: <=0 defaults to 20, >200 clamps to 200 - Support IncludeDeleted filter - Add comprehensive store tests for pagination, cursor edge cases, PerPage clamping, and invalid input rejection - Add app layer test for empty channelID → 400 - Update interface, retrylayer, timerlayer, and mock signatures Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add cursor-based pagination to View API for channel views * Enhance cursor handling in getViewsForChannel and update tests for pagination * Refactor test loops in ViewStore tests for improved readability * Refactor loop in TestGetViewsForChannel for improved readability * change pagination to limit/offset * switch to limit/offset pagination * Add upper-bound limits on View Subviews and LinkedProperties Defense-in-depth validation: cap Subviews at 50 and LinkedProperties at 500 to prevent abuse below the 300KB payload limit. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add view sort order API endpoint Add POST /api/v4/channels/{channel_id}/views/{view_id}/sort_order endpoint following the channel bookmarks reorder pattern. Includes store, app, and API layers with full test coverage at each layer. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add connectionId to view WebSocket events and sort_order API spec Thread connectionId from request header through all view handlers (create, update, delete, sort_order) to WebSocket events, matching the channel bookmarks pattern. Add sort_order endpoint to OpenAPI spec. Update minimum server version to 11.6. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Remove duplicate View/ViewPatch definitions from definitions.yaml The merge from integrated-boards-mvp introduced duplicate View and ViewPatch schema definitions that were already defined earlier in the file with more detail (including ViewBoardProps ref and enums). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Update minimum server version to 11.6 in views API spec Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add missing translations for view sort order error messages Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Merge integrated-boards-mvp into ibmvp_api-views; remove spec files Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix flaky TestViewStore timestamp test on CI Add sleep before UpdateSortOrder to ensure timestamps differ, preventing same-millisecond comparisons on fast CI machines. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * remove duplicate views.yaml imclude * Use c.boolString() for include_deleted query param in GetViewsForChannel Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix views.yaml sort order schema: use integer type and require body Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Refactor view sort order tests to use named IDs instead of array indices Extract idA/idB/idC from views slice and add BEFORE/AFTER comments to make stateful subtest ordering easier to follow. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Return 404 instead of 403 for view operations on deleted channels Deleted channels should appear non-existent to callers rather than revealing their existence via a 403. Detailed error text explains the context for debugging. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * add missing channel deleteat checks * Use c.Params.Page instead of manual page query param parsing in getViewsForChannel c.Params already validates and defaults page/per_page, so the manual parsing was redundant. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add support for total count in views retrieval * Add tests for handling deleted views in GetViewsForChannel and GetView * Short-circuit negative newIndex in UpdateSortOrder before opening transaction Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add per-channel limit on views to bound UpdateSortOrder cost Without a cap, unbounded view creation makes sort-order updates increasingly expensive (CASE WHEN per view, row locks). Adds MaxViewsPerChannel=50 constant and enforces it in the app layer before saving. Includes API and app layer tests. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Remove include_deleted support from views API Soft-deleted views are structural metadata with low risk, but no other similar endpoint (e.g. channel bookmarks) exposes deleted records without an admin gate. Rather than adding an admin-only permission check for consistency, remove the feature entirely since there is no current use case. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Update view permissions to require `create_post` instead of channel management permissions * Remove obsolete view management error messages for direct and group messages --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * feat(migrations): add user tracking and object type to property fields - Introduced user tracking columns (CreatedBy, UpdatedBy) to PropertyFields and PropertyValues. - Added ObjectType column to PropertyFields with associated unique indexes for legacy and typed properties. - Created new migration scripts for adding and dropping these features, including necessary indexes for data integrity. - Established views for managing property fields with new attributes. This update enhances the schema to support better tracking and categorization of property fields. * Add Property System Architecture v2 API endpoints (#35583) * Adds uniqueness mechanisms to the property fields After adding ObjectType, this commit ensures that both the PSAv1 and PSAv2 schemas are supported, and enforces property uniqueness through both database indexes and a logical check when creating new property fields. * Adds uniqueness check to property updates Updates are covered on this commit and we refactor as well the SQL code to use the squirrel builder and work better with the conditional addition of the `existingID` piece of the query. * Add translations to error messages * Add the permissions to the migrations, model and update the store calls * Adds the property field and property group app layer * Adds authorization helpers for property fields and values * Make sure that users cannot lock themselves out of property fields * Migrate permissions from a JSON column to three normalized columns * Remove the audit comment * Use target level constants in authorization * Log authorization membership failures * Rename admin to sysadmin * Adds the Property System Architecture v2 API endpoints * Adds permission checks to the create field endpoint * Add target access checks to value endpoints * Add default branches for object_type and target_type and extra guards for cursor client4 methods * Fix vet API mismatch * Fix error checks * Fix linter * Add merge semantics for property patch logic and API endpoint * Fix i18n * Fix duplicated patch elements and early return on bad cursor * Update docs to use enums * Fix i18n sorting * Update app layer to return model.AppError * Adds a limit to the number of property values that can be patched in the same request * Require target_type filter when searching property fields * Add objectType validation as part of field.IsValid() * Fix linter * Fix test with bad objecttpye * Fix test grouping --------- Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es> * MM-67968: Flatten view model — remove icon, subviews, typed board props (#35726) * feat(views): flatten view model by removing icon, subview, and board props Simplifies the View data model as part of MM-67968: removes Icon, Subview, and ViewBoardProps types; renames ViewTypeBoard to ViewTypeKanban; replaces typed Props with StringInterface (map[string]any); adds migration 000167 to drop the Icon column from the Views table. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * feat(api): update views OpenAPI spec to reflect flattened model Removes ViewBoardProps, Subview, and icon from the View and ViewPatch schemas. Changes type enum from board to kanban. Replaces typed props with a free-form StringInterface object. Aligns with MM-67968. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * refactor(views): simplify store by dropping dbView and marshalViewProps StringInterface already implements driver.Valuer and sql.Scanner, so the manual JSON marshal/unmarshal and the dbView intermediate struct were redundant. model.View now scans directly from the database. Also removes the dead ViewMaxLinkedProperties constant and wraps the Commit() error in UpdateSortOrder. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * fix(api): allow arbitrary JSON in view props OpenAPI schema The props field was restricted to string values via additionalProperties: { type: string }, conflicting with the Go model's StringInterface (map[string]any). Changed to additionalProperties: true in View, ViewPatch, and inline POST schemas. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * Adds basic implementation of the generic redux store for PSAv2 (#35512) * Adds basic implementation of the generic redux store for PSAv2 * Add created_by and updated_by to the test fixtures * Make target_id, target_type and object_type mandatory * Wrap getPropertyFieldsByIds and getPropertyValuesForTargetByFieldIds with createSelector * Address PR comments --------- Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es> * Adds websocket messages for the PSAv2 API events (#35696) * Adds uniqueness mechanisms to the property fields After adding ObjectType, this commit ensures that both the PSAv1 and PSAv2 schemas are supported, and enforces property uniqueness through both database indexes and a logical check when creating new property fields. * Adds uniqueness check to property updates Updates are covered on this commit and we refactor as well the SQL code to use the squirrel builder and work better with the conditional addition of the `existingID` piece of the query. * Add translations to error messages * Add the permissions to the migrations, model and update the store calls * Adds the property field and property group app layer * Adds authorization helpers for property fields and values * Make sure that users cannot lock themselves out of property fields * Migrate permissions from a JSON column to three normalized columns * Remove the audit comment * Use target level constants in authorization * Log authorization membership failures * Rename admin to sysadmin * Adds the Property System Architecture v2 API endpoints * Adds permission checks to the create field endpoint * Add target access checks to value endpoints * Add default branches for object_type and target_type and extra guards for cursor client4 methods * Fix vet API mismatch * Fix error checks * Fix linter * Add merge semantics for property patch logic and API endpoint * Fix i18n * Fix duplicated patch elements and early return on bad cursor * Update docs to use enums * Fix i18n sorting * Update app layer to return model.AppError * Adds a limit to the number of property values that can be patched in the same request * Adds websocket messages for the PSAv2 API events * Add IsPSAv2 helper to the property field for clarity * Add guard against nil returns on field deletion * Add docs to the websocket endpoints --------- Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es> * migrations: consolidate views migrations and reorder after master - Merged 000165 (create Views) with 000167 (drop Icon) since Icon was never needed - Renumbered branch migrations 159-166 → 160-167 so master's 000159 (deduplicate_policy_names) runs first - Regenerated migrations.list Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add API endpoint to retrieve posts for a specific view (#35604) Automatic Merge * Apply fixes after merge * Return a more specific error from getting multiple fields * Prevent getting broadcast params on field deletion if not needed * Remove duplicated migration code * Update property conflict code to always use master * Adds nil guard when iterating on property fields * Check that permission level is valid before getting rejected by the database * Validate correctness on TargetID for PSAv2 fields * Avoid PSAv1 using permissions or protected * Fix test data after validation change * Fix flaky search test * Adds more posts for filter use cases to properly test exclusions --------- Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es> Co-authored-by: Alejandro García Montoro <alejandro.garciamontoro@gmail.com> Co-authored-by: Julien Tant <julien@craftyx.fr> Co-authored-by: Mattermost Build <build@mattermost.com> Co-authored-by: Julien Tant <785518+JulienTant@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-27 10:36:35 +01:00
Doug Lauder	3a2cc5e242	ES search: include footer and author name in indexed fields. (#35603) ... * Include footer and author name in indexed fields.	2026-03-16 15:35:02 -04:00
Ben Schumacher	b6e5264731	[MM-67739] Rename SlackAttachment to MessageAttachment across the codebase (#35445) ... Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Mattermost Build <build@mattermost.com>	2026-03-10 16:37:21 +01:00
Christopher Poile	7e0af2de57	MM-67540 - Align Elasticsearch backfill throttle to 10k requests/sec (#35539) ... The OpenSearch implementation used 10,000 requests/sec but Elasticsearch was set to 1,000. Align both to 10,000.	2026-03-09 15:01:58 -04:00
Christopher Poile	2ada8d7659	MM-67540 - Allow searching public channel messages without channel membership (#35298) ... * UpdateByQuery methods for channel_type; rewrite reindexChannelPosts log pre-fetch error in channel Update and upgrade reindexChannelPosts to error level * add backfill orchestration, config listener, webapp toggle changes fix misleading backfill complete log when SaveOrUpdate fails * add integration & unit tests for public channel search and backfill * fix nil pointer dereference on UpdateByQuery response and log partial failures * add tests for compliance mode override and P channel post leakage * update system console snapshots * add instructions to error message * improve compliance-mode test * getAllChannels doesn't filter by O/S, need to do ourselves * in search, load channel info for channels we're not a member of * blank commit -- something is wrong with github, maybe this will help * improve channelType passing; error msg; simplify settings naming * debug logging for timing backfill and channel change; TO BE REVERTED * fix getMissingChannelsFromFiles in search as well * blank commit	2026-03-09 14:07:44 -04:00
Alejandro García Montoro	79ee7d9e16	MM-67686: Elasticsearch indexing job progress estimation (#35433) ... * Improve job progress estimation with no data For the Elasticsearch indexing job, we compute the job progress executing analytics queries to get the total number of posts, channels, users and files in the database. Until now, if that call failed, we instead used an estimate. That estimate is a hardcoded number that is in no way related to the server data. If that estimate was smaller than the number of already processed entities, the job progress would show up as larger than 100%. This commit changes that behaviour by caching the result of the analytics query: 1. If the analytics query succeeds, we store that value in the job data. 2. If the analytics query fails, we pick a value for the total as follows: - Use the value previously stored in the job data if available. - If not, use the hardcoded estimate. - If the hardcoded estimate is smaller than the current count of processed entities, use that count instead. * Add defensive code against division by zero --------- Co-authored-by: Mattermost Build <build@mattermost.com>	2026-03-09 17:04:07 +00:00
Ben Schumacher	99d119cf85	[MM-66846] Fix missing ES version and plugins in diagnostics (#35475) ... Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-05 23:55:32 +01:00
Doug Lauder	5d6e6733d5	MM-45293 Index all attachment content fields in Elasticsearch (#35454) ... * MM-45293 Index all Slack attachment content fields in Elasticsearch Previously only the "text" field from Slack attachments was indexed. This adds title, pretext, fallback, and field titles/values, making posts from integrations (JIRA, GitHub, CI bots) fully searchable.	2026-03-05 09:49:20 -05:00
Ben Schumacher	60fbce7d03	[MM-67671] Add CJK Post search support for PostgreSQL (#35260) ... Add LIKE-based CJK (Chinese, Japanese, Korean) search support for PostgreSQL, gated behind a `CJKSearch` feature flag. PostgreSQL's built-in full-text search (`to_tsvector`/`to_tsquery`) does not support CJK languages because it relies on whitespace-based tokenization, which doesn't work for logographic/syllabic scripts that don't use spaces between words. This PR adds a `LIKE`-based fallback for search terms containing CJK characters. **How it works:** When the `CJKSearch` feature flag is enabled and a search term contains CJK characters (Han, Hiragana, Katakana, or Hangul), the query builder generates `LIKE '%term%'` clauses instead of `to_tsvector @@ to_tsquery` expressions. Case-sensitive `LIKE` is used rather than `ILIKE` since CJK scripts have no letter casing, which also allows potential use of standard B-tree indexes.	2026-02-25 10:18:51 -05:00
Ben Cooke	932086e29c	separate websocket event for translations metrics (#35296)	2026-02-17 23:29:01 -05:00
M-ZubairAhmed	cd8b22af99	[MM-65979] Add Prometheus metrics for plugin webapp performance (#35075)	2026-02-13 18:07:54 +05:30
Carlos Garcia	feca30d85f	support for Elastic(Open)search CJK analysis plugins (#34784) ... * support for Elastic(Open)search CJK analysis plugins * addresses PR review comments * addresses PR comments * moves CJK based tests to own file and adds some more Dockerfile for open and elasticsearch are changed to install required plugins for testing or running locally. * properly sort error messages * fix style issues * removes trailing space	2026-02-12 10:05:23 +01:00
Ben Cooke	4195b8bc5c	Metrics for Autotranslations (#34900)	2026-01-29 05:46:45 -05:00
Carlos Garcia	3388093c00	updates opensearch library dependency and adds tests for caused_by error reason returned (#34826)	2026-01-27 16:01:22 +01:00
Ben Cooke	a1c85007e1	Autotranslations MVP (#34696) ... --------- Co-authored-by: Elias Nahum <nahumhbl@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Nick Misasi <nick.misasi@mattermost.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Matthew Birtch <mattbirtch@gmail.com> Co-authored-by: Mattermost Build <build@mattermost.com>	2026-01-26 17:05:34 -05:00
Ibrahim Serdar Acikgoz	084006c0ea	[MM-61758] Burn on read feature (#34703) ... * Add read receipt store for burn on read message types * update mocks * fix invalidation target * have consistent case on index creation * Add temporary posts table * add mock * add transaction support * reflect review comments * wip: Add reveal endpoint * user check error id instead * wip: Add ws events and cleanup for burn on read posts * add burn endpoint for explicitly burning messages * add translations * Added logic to associate files of BoR post with the post * Added test * fixes * disable pinning posts and review comments * MM-66594 - Burn on read UI integration (#34647) * MM-66244 - add BoR visual components to message editor * MM-66246 - BoR visual indicator for sender and receiver * MM-66607 - bor - add timer countdown and autodeletion * add the system console max time to live config * use the max expire at and create global scheduler to register bor messages * use seconds for BoR config values in BE * implement the read by text shown in the tooltip logic * unestack the posts from same receiver and BoR and fix styling * avoid opening reply RHS * remove unused dispatchers * persis the BoR label in the drafts * move expiration value to metadata * adjust unit tests to metadata insted of props * code clean up and some performance improvements; add period grace for deletion too * adjust migration serie number * hide bor messages when config is off * performance improvements on post component and code clean up * keep bor existing post functionality if config is disabled * Add read receipt store for burn on read message types * Add temporary posts table * add transaction support * reflect review comments * wip: Add reveal endpoint * user check error id instead * wip: Add ws events and cleanup for burn on read posts * avoid reacting to unrevealed bor messages * adjust migration number * Add read receipt store for burn on read message types * have consistent case on index creation * Add temporary posts table * add mock * add transaction support * reflect review comments * wip: Add reveal endpoint * user check error id instead * wip: Add ws events and cleanup for burn on read posts * add burn endpoint for explicitly burning messages * adjust post reveal and type with backend changes * use real config values, adjust icon usage and style * adjust the delete from from sender and receiver * improve self deleting logic by placing in badge, use burn endpoint * adjust websocket events handling for the read by sender label information * adjust styling for concealed and error state * update burn-on-read post event handling for improved recipient tracking and multi-device sync * replace burn_on_read with type in database migrations and model * remove burn_on_read metadata from PostMetadata and related structures * Added logic to associate files of BoR post with the post * Added test * adjust migration name and fix linter * Add read receipt store for burn on read message types * update mocks * have consistent case on index creation * Add temporary posts table * add mock * add transaction support * reflect review comments * wip: Add reveal endpoint * user check error id instead * wip: Add ws events and cleanup for burn on read posts * add burn endpoint for explicitly burning messages * Added logic to associate files of BoR post with the post * Added test * disable pinning posts and review comments * show attachment on bor reveal * remove unused translation * Enhance burn-on-read post handling and refine previous post ID retrieval logic * adjust the returning chunk to work with bor messages * read temp post from master db * read from master * show the copy link button to the sender * revert unnecessary check * restore correct json tag * remove unused error handling and clarify burn-on-read comment * improve type safety and use proper selectors * eliminate code duplication in deletion handler * optimize performance and add documentation * delete bor message for sender once all receivers reveal it * add burn on read to scheduled posts * add feature enable check * use master to avoid all read recipients race condition --------- Co-authored-by: Mattermost Build <build@mattermost.com> Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com> Co-authored-by: Harshil Sharma <harshilsharma63@gmail.com> * squash migrations into single file * add configuration for the scheduler * don't run messagehasbeenposted hook * remove parallel tests on burn on read * add clean up for closing opened modals from previous tests * simplify delete menu item rendering * add cleanup step to close open modals after each test to prevent pollution * streamline delete button visibility logic for Burn on Read posts * improve reliability of closing post menu and modals by using body ESC key --------- Co-authored-by: Harshil Sharma <harshilsharma63@gmail.com> Co-authored-by: Pablo Vélez <pablovv2012@gmail.com> Co-authored-by: Mattermost Build <build@mattermost.com>	2025-12-11 07:59:50 +01:00
Elias Nahum	4589005a54	feat: Add Microsoft Intune MAM authentication support (#34577) ... * Add Entra ID token authentication and Intune MAM config exposure * Add Intune MAM toggle to Mobile Security admin console * Add IntuneSettings with the AuthService to use and its own TenantID andClientID for the Entra App registration Include Admin console changes switch from /oauth/entra to /oauth/intune endpoint * openAPI documentation --------- Co-authored-by: Mattermost Build <build@mattermost.com> Co-authored-by: yasser khan <attitude3cena.yf@gmail.com>	2025-12-10 08:31:53 +02:00
Eva Sarafianou	0d181ca215	Push Proxy Authentication (#34211) ... * Initial Implementation of Push Proxy Authentication * Include Config Listener for Leader plus delete startup function as job scheduler runs on initialization * Remove push proxy auth from local imports * Add push proxy auth to external imports * Add push proxy auth error messages * Update error codes * Fix enterprise dep definition * make i18n-extract * Mock System store Get * m * m * m * m * Update serverID header * Add install type env var to docker * Update Push Proxy config with new options Global, US, Germany and Japan. Previous configurations will keep working * use model.SafeDereference * Delete token when new push proxy URL is empty * ServerID header only if auth token is available --------- Co-authored-by: Daniel Schalla <daniel@mattermost.com> Co-authored-by: Nick Misasi <nick.misasi@mattermost.com> Co-authored-by: Mattermost Build <build@mattermost.com>	2025-11-12 20:16:44 +02:00
Ben Schumacher	b2df9be70b	Fix errcheck linter errors in helpers (#31578)	2025-11-12 13:00:51 +01:00
Ben Schumacher	892a7c9c69	Use golangci-lints's build-in modernize linter (#34341)	2025-11-04 12:09:11 +01:00
Jesse Hallam	a020d7ab97	Revert "Allow degraded opensearch startup (#34228)" (#34232) ... This reverts commit a1072142e1.	2025-10-27 19:51:47 +01:00
Jesse Hallam	a1072142e1	Allow degraded opensearch startup (#34228) ... In the event template creation fails for opensearch, allow the search engine to still startup normally, accepting a potentially degraded search state as a consequence. Relates-to: https://mattermost.atlassian.net/browse/MM-65177	2025-10-21 16:33:12 +00:00
Alejandro García Montoro	0b7b968087	Update dependencies (#34071) ... * Update dependencies * Undo hack needed by go-elasticsearch v8.18.0 See https://github.com/mattermost/mattermost/pull/31021 for more information. * Another breaking change in a minor version :) * Remove old exclude rules * Unify ES/OS tests when getting no documents * Clarify weird assert on ES/OS common tests * Bump dependencies now that archives v0.1.5 is out	2025-10-14 13:00:39 +02:00
Harshil Sharma	3265054ad5	Migrate content flagging settings to database (#33989) ... * lint fix * CI * added new migration mocks * Used setup for tests * some comment * Removed unnecesseery nil check * Form validation * WIP tests * WIP tests * WIP tests * fix: mock content flagging config selector with correct reasons format Co-authored-by: aider (anthropic/claude-sonnet-4-20250514) <aider@aider.chat> * fix: add mock for getContentFlaggingConfig in flag post modal test Co-authored-by: aider (anthropic/claude-sonnet-4-20250514) <aider@aider.chat> * Updated error code order in API docs * removed empty files * Added tests * lint fixes * minor tweak * lint fix * type fix * fixed test * nit * test enhancements * API WIP * API WIP * creating values * creating content flagging channel and properties * Able to save properties * Added another property field * WIP * WIP * Added validations * Added data validations and hidden post if confifgured to * lint fixes * Added API spec * Added some tests * Added tests for getContentReviewBot * test: add comprehensive tests for getContentReviewChannels function * Added more app layer tests * Added TestCanFlagPost * test: Add comprehensive tests for FlagPost function * Added all app layer tests * Removed a file that was reamoved downstream * test: add content flagging test file * test: add comprehensive tests for FlagContentRequest.IsValid method * Added model tests * test: add comprehensive tests for SqlPropertyValueStore.CreateMany * test: add comprehensive tests for flagPost() API function * Added API tests * linter fix * WIP * sent post flagging confirmation message * fixed i18n nissues * fixed i18n nissues * CI * WIP * WIP * Added API call * test: add test for Client4.flagPost API call in FlagPostModal * fix: remove userEvent.setup() from flag post modal test * test: wrap submit button click in act for proper state updates * Updated tests * lint fix * Updated test * fix: reset contentFlaggingGroupId for test isolation in content flagging tests * removed cached group ID * removed debug log * CI * Updated to allow special characters in comments * Handled empty comment * Created getContentFlaggingFields API * created getPostPropertyValues API * Used finally * WIP * Created useContentFlaggingFields hook * WIP * WIP * Added option to retain data for reviewers * Displayed deleted post's preview * DIsplayed all properties * Adding field name i18n * WIP - managing i18n able texts * Finished displaying all fields * Manual cleanup * lint fixes * team role filter logic fix * Fixed tests * created new API to fetch flagged posts * lint fix * Added new client methods * test: add comprehensive tests for content flagging APIs * Added new API tests * fixed openapi spec * Fixed DataSpillageReport tests * Fixed PostMarkdown test * Fixed PostPreviewPropertyRenderer test * Added metadata to card renderer * test fixes * Added no comment placeholder * Added view detail button * Created RemoveFlaggedMessageConfirmationModal modal * Added key and remove flag request modal * IMplemented delete flagged post * Handled edge cases of deleting flagged post * keep message * UI integration * Added WS event for post report update and handled deleted files of flagged post * Added error handling in keep/remove forms * i18n fixes * Fixed test * Updated OpenAPI specs * fixed types * fixed types * refactoring * refactor: improve test mocking for data spillage report component * test mock updates * Fixed tests * Updated reducer * not resetting mocks * Added migrations for content flagging tables * Created new structure * review fixes * Used correct ot name * WIP * review fixes * review fixes * Added new property translations * CI * CI * CI * Improved test * fixed test * CI * New UI component * WIP * Updated settings APIs * cached DB data * used cached reviewer data * Updated tests * Lint fixes * test: add tests for saveContentFlaggingSettings and getContentFlaggingSettings APIs * test fix * test: add tests for SaveContentFlaggingConfig and GetContentFlaggingConfigReviewerIDs * Updated tests * test: add content flagging test for local cache layer * test: add comprehensive tests for content flagging store cache * Updated tests * lint fix * Updated mobile text * Added content flagging SQL store mocks * Added API specs for new APIs * fixed tests * feat: add TestContentFlaggingStore function for content flagging store testing * feat: add comprehensive tests for content flagging store * Added SQL store tests * test: add content flagging test for local cache layer * test: add tests for content flagging store caching * Added cache layer tests * Updated tests * Fixed * Handled JSON error * fixes * fixes * Fixed retry layer test * fixerdf i18n * Fixed test * CI * building index concurrently * CI * fixed a test * CI * cleanup * Integrate flag post api (#33798) * WIP * WIP * Added API call * test: add test for Client4.flagPost API call in FlagPostModal * fix: remove userEvent.setup() from flag post modal test * test: wrap submit button click in act for proper state updates * Updated tests * lint fix * CI * Updated to allow special characters in comments * Handled empty comment * Used finally * CI * Fixed test * Spillage card integration (#33832) * Created getContentFlaggingFields API * created getPostPropertyValues API * WIP * Created useContentFlaggingFields hook * WIP * WIP * Added option to retain data for reviewers * Displayed deleted post's preview * DIsplayed all properties * Adding field name i18n * WIP - managing i18n able texts * Finished displaying all fields * Manual cleanup * lint fixes * team role filter logic fix * Fixed tests * created new API to fetch flagged posts * lint fix * Added new client methods * test: add comprehensive tests for content flagging APIs * Added new API tests * fixed openapi spec * Fixed DataSpillageReport tests * Fixed PostMarkdown test * Fixed PostPreviewPropertyRenderer test * Added metadata to card renderer * test fixes * Added no comment placeholder * Fixed test * refactor: improve test mocking for data spillage report component * test mock updates * Updated reducer * not resetting mocks * WIP * review fixes * CI * Fixed * fixes * Content flagging actions implementation (#33852) * Added view detail button * Created RemoveFlaggedMessageConfirmationModal modal * Added key and remove flag request modal * IMplemented delete flagged post * Handled edge cases of deleting flagged post * keep message * UI integration * Added WS event for post report update and handled deleted files of flagged post * Added error handling in keep/remove forms * i18n fixes * Updated OpenAPI specs * fixed types * fixed types * refactoring * Fixed tests * review fixes * Added new property translations * Improved test * fixed test * CI * fixes * CI * fixed a test * CI * Review fixes --------- Co-authored-by: aider (anthropic/claude-sonnet-4-20250514) <aider@aider.chat>	2025-10-13 12:24:01 +05:30
Jesse Hallam	de72b41232	relocate message export (#33805)	2025-08-27 23:22:05 +03:00
Alejandro García Montoro	553f99612e	MM-60441: Re-index public channels when a user joins a team (#33400) ... * Index all public channels when a user joins a team * Precompute team members for indexChannelsForTeam * Refactor RequestContextWithMaster to store package This way, we can import it from both the sqlstore and the searchlayer packages. The alternative for this is duplicating the code in those two packages, but that will *not* work: The context package expects custom types for the keys stored in it, so that different packages never clash with each other when trying to register a new key. See the docs for the WithValue function: https://pkg.go.dev/context#WithValue If we try to duplicate the storeContextKey type in both the sqlstore and searchlayer packages, although they *look* the same, they are not, and HasMaster will fail to get the value of the storeContextKey(useMaster) key if it's from the other package. * Use master in call to GetTeamMembersForChannel In GetTeamMembersForChannel, use the DB from the newly passed context, which will be the receiving context everywhere except in the call done from indexChannelsForTeam, to avoid the read after write issue when saving a team member. * Fix GetPublicChannelsForTeam paging We were using the page and perPage arguments as is in the call to GetPublicChannelsForTeam, but that function expects and offset and a limit as understood by SQL. Although perPage and limit are interchangeable, offset is not equal to page, but to page * perPage. * Add a synchronous bulk indexer for Opensearch * Implement Opensearch's SyncBulkIndexChannels * Add a synchronous bulk indexer for Elasticsearch * Implement Elasticsearch's SynkBulkIndexChannels * Test SyncBulkIndexChannels * make mocks * Bulk index channels on indexChannelsForTeam * Handle error from SyncBulkIndexChannels * Fix style * Revert indexChannelWithTeamMembers refactor * Remove defensive code on sync bulk processor * Revert "Add a synchronous bulk indexer for Opensearch" This reverts commit bfe4671d96. * Revert "Add a synchronous bulk indexer for Elasticsearch" This reverts commit 6643ae3f30. * Refactor bulk indexers with a common interface * Test all the different implementations Assisted by Claude * Remove debug statements * Refactor common code into _stop * Rename getUserIDsFor{,Private}Channel * Wrap error * Make perPage a const * Fix typos * Call GetTeamsForUser only if needed * Differentiate errors for sync/async processors --------- Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com> Co-authored-by: Mattermost Build <build@mattermost.com>	2025-08-25 19:28:19 +02:00
Agniva De Sarker	9dd8c056e7	MM-63368: Remove MySQL (#33458) ... https://mattermost.atlassian.net/browse/MM-63368 ```release-note Remove MySQL support from the codebase entirely. ```	2025-07-22 20:40:55 +05:30
Ben Schumacher	9add320011	[MM-64654] Migrate to modern Go features (#31820)	2025-07-18 12:54:51 +02:00
Christopher Poile	9b1e03a6b8	[MM-63557] mmctl: Add compliance export create cmd (#30594) ... * Refactor job retrieval to support multiple statuses & multiple types - Updated job retrieval functions to handle multiple job statuses. - Renamed `GetJobsByTypeAndStatus` to `GetJobsByTypesAndStatuses` for consistency across the codebase. - Adjusted related function signatures and implementations in the job store and retry layer to accommodate the new method. - Updated tests to reflect changes in job retrieval logic and ensure proper functionality. * Add compliance export create command and tests - Introduced `ComplianceExportCreateCmd` to facilitate the creation of compliance export jobs with options for date, start, and end timestamps. - Added unit tests for the new command, covering various scenarios including valid and invalid inputs. - Updated documentation to include usage examples and options for the new command. - Enhanced existing tests to ensure proper functionality of compliance export job handling. * update docs * update tests for new logic * Refactor message export job tests to use DefaultPreviousJobPageSize - Updated all test cases in worker_test.go to replace hardcoded page size of 100 with DefaultPreviousJobPageSize for consistency. - Adjusted the worker.go file to define DefaultPreviousJobPageSize and use it in job retrieval logic. - Ensured that the changes maintain the functionality of job data initialization and retrieval tests. * PR comments * PR comments, simplifications, clarifications, formatting * prefer hypen over underscore in command names * merge conflict * update mmctl docs	2025-06-24 21:38:30 +00:00
David Krauser	761584c040	[MM-64244] Add websocket disconnect reason metric (#31032) ... We've recently spent some effort improving websocket reconnection logic. With this commit, I've augmented the websocket reconnect metric to include a disconnect reason. This will help us measure the impact of these changes in production.	2025-05-30 08:15:20 -04:00
Agniva De Sarker	79bf1c34db	MM-63217: Bump dependencies (#31021) ... * MM-63217: Bump dependencies Skip-Enterprise-PR: true https://mattermost.atlassian.net/browse/MM-63217 ```release-note NONE ``` * fix go.mod ```release-note NONE ``` * fix test ```release-note NONE ```	2025-05-29 13:08:00 +05:30
Ibrahim Serdar Acikgoz	a344b3225b	[MM-61756] Attribute Based Access Control - Phase 1 (#30785) ... Attribute Based Access Control - Base * MM-63662 * MM-63919 * MM-63954 * MM-63955 * MM-63425 * MM-63426 * MM-63458 * MM-63459 * MM-63603 * MM-63845 * MM-64146 * MM-64199 * MM-64201 * MM-64233 * MM-64247 * MM-64268 --------- Co-authored-by: Harshil Sharma <harshilsharma63@gmail.com> Co-authored-by: Pablo Andrés Vélez Vidal <pablovv2012@gmail.com> Co-authored-by: abhijit-singh <abhijitsingh0702@gmail.com> Co-authored-by: Harrison Healey <harrisonmhealey@gmail.com>	2025-05-15 11:33:08 +02:00
Ibrahim Serdar Acikgoz	d452f4f043	[MM-63661] add access control metrics (#30680)	2025-05-11 22:11:42 +02:00
Agniva De Sarker	271d928e87	MM-63467: Refactor constLabels to a common initializer (#30779) ... This is useful to avoid mistakes while adding new metrics. https://mattermost.atlassian.net/browse/MM-63467 ```release-note NONE ``` Co-authored-by: Mattermost Build <build@mattermost.com>	2025-04-23 21:59:51 +05:30
Agniva De Sarker	09488558a0	MM-63298: [AI assisted] Elasticsearch add a global search prefix (#30417) ... This PR adds functionality to search by a global search prefix. This allows Mattermost to be used across multiple data centers with multiple Elasticsearch instances synchronized using cross-cluster replication. While here, we also add tests cases to cover for some missing search interface methods. For now, no system console setting is exposed. Because IndexPrefix is also not exposed. It can be added later if a need arises. https://mattermost.atlassian.net/browse/MM-63298 ```release-note A new config setting ElasticsearchSettings.GlobalSearchPrefix is added which can be used to search across multiple indices having a common prefix. This is useful in a scenario with multiple Elasticsearch instances, where multiple instances are writing to different indices with different prefixes using the ElasticsearchSettings.IndexPrefix setting. ``` Co-authored-by: Mattermost Build <build@mattermost.com>	2025-04-04 11:42:12 +05:30
Ben Schumacher	9b5d8d52bf	[MM-62427] Add message attachments validation (#30180) ... * Add message attachments validation * Add props validation * Validate slack attachment fields * Update tests and library usage * Improve interactive dialog error for length checks * Allow predefined colors for slack attachments * Fix TestPostAction * Use const for data source * Add tests * Cleanup unused props * Add happy path tests * lint fixes * Add validation for PostActionOptions	2025-03-20 12:53:50 +01:00
Christopher Poile	c049748b88	[MM-63314] Fix ClaimJob in HA environments (#30383) ... * ClaimJob now returns newly claimed job * internal code affected by change * test changes required * two branches: for mysql, use transaction; for postgres, use returning * two branches: for mysql, use transaction; for postgres, use returning * use same millis value for LastActivityAt and StartAt * blank commit --------- Co-authored-by: Mattermost Build <build@mattermost.com>	2025-03-14 10:24:26 -04:00
Agniva De Sarker	1a58f923e0	[aider assisted] MM-61888: Add ClientSideUserIds field to MetricsSettings (#30127) ... We add a new config setting to allow the admin to set a fixed list of userIDs to track for all client side webapp metrics. This gives the admin to get a deeper look at how the application is behaving for a single user. A new section in the system console is also added for the user to edit this setting from the UI. https://mattermost.atlassian.net/browse/MM-61888 ```release-note A new config setting MetricsSettings.ClientSideUserIds is added where you can set the user ids you want to track for client side webapp metrics. ``` * fix lint errors ```release-note NONE ``` * fixing tests ```release-note NONE ```	2025-02-13 21:10:34 +05:30
Agniva De Sarker	d34dcc2dda	MM-62901: Apply constLabels on all metrics (#30125) ... When the client side metrics were added constLabels got missed out. Without this, we cannot do the high level grouping by rings. https://mattermost.atlassian.net/browse/MM-62901 ```release-note NONE ```	2025-02-06 21:45:57 +05:30
Harshil Sharma	35e776d805	Removed Elasticsearch and Opensearch channel index schedma check (#30102)	2025-02-06 11:50:37 +05:30
Christopher Poile	8b65771a31	[MM-62897] Compliance export backwards compatibility (#30120) ... * add backwards compatibility for <10.5 * test: backwards compatibility for <10.5 * spelling	2025-02-05 21:20:28 +00:00
Agniva De Sarker	afbd9d64c3	[AI assisted]: MM-62295: Search and index archived channels as well. (#29796) ... We add a new field delete_at in the channels template. This field is then searched in the SearchChannels function. Also added tests to verify that archived channels are searched properly, and also indexed correctly. https://mattermost.atlassian.net/browse/MM-62295 ```release-note - Now archived channels are searchable with ES/OS if TeamSettings.ExperimentalViewArchivedChannels is enabled. - If there are old channels which were archived before a bulk index was run, users would need to purge indexes, and do bulk index again. Because those old archived channels are removed from the index when a bulk index is run. ``` Co-authored-by: Mattermost Build <build@mattermost.com>	2025-01-28 19:44:55 +05:30
Christopher Poile	737bed311c	MM-60115, MM-62436, MM-62493: Compliance export downloads from local and s3; e2e tests (#29806) ... * add ZipReader method to filestore and file/s3 backends to merge to merge * add WriteStreamResponse as an alternative to WriteFileResponse * add generated layers * enable download link; download job endpoint now streams export dir zips * fix MM-62493 * re-enable e2e tests--we have download links, folks * Add tests for ZipReader in filestore and s3store * remove unnecessary error return on ZipReader * little cleanup * improve tests; some refactoring: s.Nil(err) -> s.NoError(err) * blank commit * backwards compatability for pre-10.5 job downloads * compress file response; better errors; better comments; PR comments * update generated app layers * improve/widen tests; improve comments; simplify localstore ZipReader * regenerate layers * follow GoDoc conventions * update generated layers * remove unnecessary comment * in jobs/job-id/download, clean exportDir before sending to ZipReader * better comments; add an error return on ZipReader * improve file permissions * adjust tests for new error returns * linting * i18n	2025-01-27 09:28:07 +05:30
Rahim Rahman	540408545a	feat(MM-61865): Add mobile client content load network metrics (#29601) ... * MM-61865: Add mobile client content load network metrics * added new common label * renaming from MobileClientContentLoad to MobileClientNetworkRequests * content_load_group => network_request_group * refactor more NetworkRequest-* changes * replace contentLoadGroup to networkRequestGroup * new metrics elapsedTime * Refactor urlCount to totalRequests * add averageSpeed metric * replace contentLoadGroup with networkRequestGroup * use h.Labels vs commonLabels for network_request_group * add agent * add effective latency metrics * add total parallel requests & total sequential * mocks generated by mockery * did a bit of cleanup and sorting * formatting * updated the AcceptedNetworkRequestGroups * cleanup and sorting	2025-01-16 11:03:41 -07:00
Christopher Poile	aba4434dab	MM-59966 - Compliance Export overhaul - feature branch (#29789) ... * [MM-59089] Add a compliance export constant (#27919) * add a useful constant * i18n * another constant * another i18n * [MM-60422] Add GetChannelsWithActivityDuring (#28301) * modify GetUsersInChannelDuring to accept a slice of channelIds * add GetChannelsWithActivityDuring * add compliance export progress message; remove unused custom status * linting * tests running too fast * add batch size config settings * add store tests * linting * empty commit * i18n changes * fix i18n ordering * MM-60570 - Server-side changes consolidating the export CLI with server/ent code (#28640) * add an i18n field; add the CLI's export directory * int64 -> int * Add UntilUpdateAt for MessageExport and AnalyticsPostCount to merge * remove now-unused i18n strings * add TranslationsPreInitFromBuffer to allow CLI to use i18n * use GetBuilder to simplify; rename TranslationsPreInitFromFileBytes * [MM-59089] Improve compliance export timings (#1733 - Enterprise repo) * MM-60422 - Performance and logic fixes for Compliance Exports (#1757 - Enterprise repo) * MM-60570 - Enterprise-side changes consolidating the export CLI with server/ent code (#1769 - Enterprise repo) * merge conflicts; missed file from ent branch * MM-61038 - Add an option to sqlstore.New (#28702) remove useless comment add test add an option to sqlstore.New * MM-60976: Remove RunExport command from Mattermost binary (#28805) * remove RunExport command from mattermost binary * remove the code it was calling * fix i18n * remove test (was only testing license, not functionality) * empty commit * fix flaky GetChannelsWithActivityDuring test * MM-60063: Dedicated Export Filestore fix, redo of #1772 (enterprise) (#28803) * redo filestore fix #1772 (enterprise repo) on top of MM-59966 feature * add new e2e tests for export filestore * golint * ok, note to self: shadowing bad, actually (when there's a defer) * empty commit * MM-61137 - Message export: Support 7.8.11 era dbs (#28824) * support 7.8.11 era dbs by wrapping the store using only what we need * fix flaky GetChannelsWithActivityDuring test * add a comment * only need to define the MEFileInfoStore (the one that'll be overridden) * blank commit * MM-60974 - Message Export: Add performance metrics (#28836) * support 7.8.11 era dbs by wrapping the store using only what we need * fix flaky GetChannelsWithActivityDuring test * add a comment * only need to define the MEFileInfoStore (the one that'll be overridden) * performance metrics * cleanup unneeded named returns * blank commit * MM-60975 - Message export: Add startTime and endTime to export folder name (#28840) * support 7.8.11 era dbs by wrapping the store using only what we need * fix flaky GetChannelsWithActivityDuring test * add a comment * only need to define the MEFileInfoStore (the one that'll be overridden) * performance metrics * output startTime and endTime in export folder * empty commit * merge conflict * MM-60978 - Message export: Improve xml fields; fix delete semantics (#28873) * support 7.8.11 era dbs by wrapping the store using only what we need * fix flaky GetChannelsWithActivityDuring test * add a comment * only need to define the MEFileInfoStore (the one that'll be overridden) * performance metrics * output startTime and endTime in export folder * empty commit * add xml fields, omit when empty, tests * fix delete semantics; test (and test for update semantics) * clarify comments * simplify edited post detection, now there's no edge case. * add some spacing to help fast running tests * merge conflicts/updates needed for new deleted post semantics * linting; fixing tests from upstream merge * use SafeDereference * linting * stronger typing; better wrapped errors; better formatting * blank commit * goimports formatting * fix merge mistake * minor fixes due to changes in master * MM-61755 - Simplifying and Support reporting to the db from the CLI (#29281) * finally clean up JobData struct and stringMap; prep for CLI using db * and now simplify using StringMapToJobDataWithZeroValues * remove unused fn * create JobDataExported; clean up errors * MM-60176 - Message Export: Global relay cleanup (#29168) * move global relay logic into global_relay_export * blank commit * blank commit * improve errors * MM-60693 - Refactor CSV to use same codepath as Actiance (#29191) * move global relay logic into global_relay_export * blank commit * refactor (and simplify) ExportParams into shared * blank commit * remove unused fn * csv now uses pre-calculated joins/leaves like actiance * improve errors * remove nil post check; remove ignoredPosts metric * remove unneeded copy * MM-61696 - Refactor GlobalRelay to use same codepath as Actiance (#29225) * move global relay logic into global_relay_export * blank commit * refactor (and simplify) ExportParams into shared * blank commit * remove unused fn * csv now uses pre-calculated joins/leaves like actiance * remove newly unneeded function and its test. goodbye. * refactor GetPostAttachments for csv + global relay to share * refactor global_relay_export and fix tests (no changes to output) * improve errors * remove nil post check; remove ignoredPosts metric * remove unneeded copy * remove unneeded nil check * PR comments * MM-61715 - Generalize e2e to all export types 🤖 (#29369) * move global relay logic into global_relay_export * blank commit * refactor (and simplify) ExportParams into shared * blank commit * remove unused fn * csv now uses pre-calculated joins/leaves like actiance * remove newly unneeded function and its test. goodbye. * refactor GetPostAttachments for csv + global relay to share * refactor global_relay_export and fix tests (no changes to output) * improve errors * remove nil post check; remove ignoredPosts metric * remove unneeded copy * remove unneeded nil check * PR comments * refactor isDeletedMsg for all export types * fix start and endtime, nasty csv createAt bug; bring closer to Actiance * align unit tests with new logic (e.g. starttime / endtime) * refactor a TimestampConvert fn for code + tests * bug: pass templates to global relay (hurray for e2e tests, otherwise...) * add global relay zip to allowed list (only for tests) * test helpers * new templates for e2e tests * e2e tests... phew. * linting * merge conflicts * unexport PostToRow; add test helper marker * cleanup, shortening, thanks to PR comments * MM-61972 - Generalize export data path - Actiance (#29399) * extract and generalize the export data generation functions * finish moving test (bc of previous extraction) * lift a function from common -> shared (to break an import cycle) * actiance now takes general export data, processes it into actiance data * bring tests in line with correct sorting rules (upadateAt, messageId) * fixups, PR comments * turn strings.Repeat into a more descriptive const amended: one letter fix; bad rebase * MM-62009 - e2e clock heisenbug (#29434) * consolidate assertions; output debuggable diffs (keeping for future) * refactor test output generator to generators file * waitUntilZeroPosts + pass through until to job = fix all clock issues * simplify messages to model.NewId(); remove unneeded waitUntilZeroPosts * model.NewId() -> storetest.NewTestID() * MM-61980 - Generalize export data path - CSV (#29482) * simple refactoring * increase sleep times for (very) rare test failures * add extra information to the generic export for CSV * adj Actiance to handle new generic export (no difference in its output) * no longer need mergePosts (yay), move getJoinLeavePosts for everyone * adjust tests for new csv semantics (detailed in summary) * and need to add the new exported data to the export_data_tests * rearrange csv writing to happen after data export (more logical) * linting * remove debug statements * figured out what was wrong with global relay e2e test 3; solid now * PR comments * MM-61718 - Generalize export data path - Global Relay (#29508) * move global relay over to using the generalized export data * performance pass -- not much can be done * Update server/enterprise/message_export/global_relay_export/global_relay_export.go Co-authored-by: Claudio Costa <cstcld91@gmail.com> --------- Co-authored-by: Claudio Costa <cstcld91@gmail.com> * MM-62058 - Align CSV with Actiance (#29551) * refactoring actiance files and var names for clarity * bug found in exported attachments (we used to miss some start/ends) * changes needed for actiance due to new generic exports * bringing CSV up to actiance standards * fixing global relay b/c of new semantics (adding a note on an edge case) * aligning e2e tests, adding comments to clarify what is expected/tested * necessary changes; 1 more test for added functionality (ignoreDeleted) * comment style * MM-62059 - Align Global Relay with Actiance/CSV; many fixes (#29665) * core logic changes to general export_data and the specific export paths * unit tests and e2e tests, covering all new edge cases and all logic * linting * better var naming, const value, and cleaning up functions calls * MM-62436 - Temporarily skip cypress tests that require download link (#29772) --------- Co-authored-by: Claudio Costa <cstcld91@gmail.com>	2025-01-10 16:56:02 -05:00
Agniva De Sarker	647c9b6808	[MM-62293]: Remove Elasticsearch from translation strings (#29728) ... We cannot simply use Elasticsearch any more because we support Opensearch as well. Therefore changing Elasticsearch to just say Search. Note: There are still some cases where Elasticsearch is used. That is intentional and necessary. ```release-note NONE ```	2025-01-08 11:11:59 +05:30
Agniva De Sarker	42ff2a2639	[Aider assisted]: Templatize rest of the elasticsearch translation strings (#29582) ... After https://github.com/mattermost/mattermost/pull/29209, there were some more strings which were left out. Sending another PR to fix all of them. The config validation strings are made generic for simplicity, because they are called before the search backend is started, and even though you could check for the config setting, it forces the code order. Therefore, changed the words to just say "Search" for simplicity. I tried to use Aider initially for a little while, but ended up fighting a lot with the search/replace and having to repeat myself over and over again to get it to do the refactoring. Eventually gave up and did rest of the changes myself. ```release-note NONE ```	2024-12-13 19:25:08 +05:30
Agniva De Sarker	59b8532a89	metrics/histogram: Fixing incorrect label_name (#29492) ... We were incorrectly overwriting the same key - label_name. ```release-note NONE ```	2024-12-06 19:34:46 +05:30