* Replace hardcoded test passwords with model.NewTestPassword()
Add model.NewTestPassword() utility that generates 14+ character
passwords meeting complexity requirements for FIPS compliance. Replace
all short hardcoded test passwords across the test suite with calls to
this function.
* Enforce FIPS compliance for passwords and HMAC keys
FIPS OpenSSL requires HMAC keys to be at least 14 bytes. PBKDF2 uses
the password as the HMAC key internally, so short passwords cause
PKCS5_PBKDF2_HMAC to fail.
- Add FIPSEnabled and PasswordFIPSMinimumLength build-tag constants
- Raise the password minimum length floor to 14 when compiled with
requirefips, applied in SetDefaults only when unset and validated
independently in IsValid
- Return ErrMismatchedHashAndPassword for too-short passwords in
PBKDF2 CompareHashAndPassword rather than a cryptic OpenSSL error
- Validate atmos/camo HMAC key length under FIPS and lengthen test
keys accordingly
- Adjust password validation tests to use PasswordFIPSMinimumLength
so they work under both FIPS and non-FIPS builds
* CI: shard FIPS test suite and extract merge template
Run FIPS tests on PRs that touch go.mod or have 'fips' in the branch
name. Shard FIPS tests across 4 runners matching the normal Postgres
suite. Extract the test result merge logic into a reusable workflow
template to deduplicate the normal and FIPS merge jobs.
* more
* Fix email test helper to respect FIPS minimum password length
* Fix test helpers to respect FIPS minimum password length
* Remove unnecessary "disable strict password requirements" blocks from test helpers
* Fix CodeRabbit review comments on PR #35905
- Add server-test-merge-template.yml to server-ci.yml pull_request.paths
so changes to the reusable merge workflow trigger Server CI validation
- Skip merge-postgres-fips-test-results job when test-postgres-normal-fips
was skipped, preventing failures due to missing artifacts
- Set guest.Password on returned guest in CreateGuestAndClient helper
to keep contract consistent with CreateUserWithClient
- Use shared LowercaseLetters/UppercaseLetters/NUMBERS/PasswordFIPSMinimumLength
constants in NewTestPassword() to avoid drift if FIPS floor changes
https://claude.ai/code/session_01HmE9QkZM3cAoXn2J7XrK2f
* Rename FIPS test artifact to match server-ci-report pattern
The server-ci-report job searches for artifacts matching "*-test-logs",
so rename from postgres-server-test-logs-fips to
postgres-server-fips-test-logs to be included in the report.
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Add CreatedBy and UpdatedBy to the properties fields and values (#34485)
* Add CreatedBy and UpdatedBy to the properties fields and values
* Fix types
---------
Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
* Adds ObjectType to the property fields table (#34908)
Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
* Update ObjectType migration setting an empty value and marking the column as not null (#34915)
Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
* Adds uniqueness mechanisms to the property fields (#35058)
* Adds uniqueness mechanisms to the property fields
After adding ObjectType, this commit ensures that both the PSAv1 and
PSAv2 schemas are supported, and enforces property uniqueness through
both database indexes and a logical check when creating new property
fields.
* Adds uniqueness check to property updates
Updates are covered on this commit and we refactor as well the SQL
code to use the squirrel builder and work better with the conditional
addition of the `existingID` piece of the query.
* Add translations to error messages
* Fixing retrylayer mocks
* Remove retrylayer duplication
* Address review comments
* Fix comment to avoid linter issues
* Address PR comments
* Update server/channels/db/migrations/postgres/000157_add_object_type_to_property_fields.down.sql
Co-authored-by: Alejandro García Montoro <alejandro.garciamontoro@gmail.com>
* Update server/channels/db/migrations/postgres/000157_add_object_type_to_property_fields.up.sql
Co-authored-by: Alejandro García Montoro <alejandro.garciamontoro@gmail.com>
* Update server/channels/db/migrations/postgres/000157_add_object_type_to_property_fields.up.sql
Co-authored-by: Alejandro García Montoro <alejandro.garciamontoro@gmail.com>
* Update field validation to check only for valid target types
* Update migrations to avoid concurrent index creation within a transaction
* Update migrations to make all index ops concurrent
* Update tests to use valid PSAv2 property fields
* Adds a helper for valid PSAv2 TargetTypes
---------
Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
Co-authored-by: Alejandro García Montoro <alejandro.garciamontoro@gmail.com>
* Fix property tests (#35388)
Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
* Adds Integrated Boards feature flag (#35378)
Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
* Adds Integrated Boards MVP API changes (#34822)
This PR includes the necessary changes for channels and posts
endpoints and adds a set of generic endpoints to retrieve and manage
property fields and values following the new Property System approach.
Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
Co-authored-by: Mattermost Build <build@mattermost.com>
* Property System Architecture permissions for v2 (#35113)
* Adds uniqueness mechanisms to the property fields
After adding ObjectType, this commit ensures that both the PSAv1 and
PSAv2 schemas are supported, and enforces property uniqueness through
both database indexes and a logical check when creating new property
fields.
* Adds uniqueness check to property updates
Updates are covered on this commit and we refactor as well the SQL
code to use the squirrel builder and work better with the conditional
addition of the `existingID` piece of the query.
* Add translations to error messages
* Add the permissions to the migrations, model and update the store calls
* Adds the property field and property group app layer
* Adds authorization helpers for property fields and values
* Make sure that users cannot lock themselves out of property fields
* Migrate permissions from a JSON column to three normalized columns
* Remove the audit comment
* Use target level constants in authorization
* Log authorization membership failures
* Rename admin to sysadmin
* Fix i18n sorting
---------
Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
* Add Views store and app layer (#35361)
* Add Views store and app layer for Integrated Boards
Implements the View entity (model, SQL store, service, app) as described
in the Integrated Boards tech spec. Views are channel-scoped board
configurations with typed props (board, kanban subviews) and soft-delete.
- public/model: View, ViewBoardProps, Subview, ViewPatch types with
PreSave/PreUpdate/IsValid/Patch/Clone/Auditable
- Migration 158: Views table with jsonb Props column and indexes
- SqlViewStore: CRUD with nil-safe Props marshaling (AppendBinaryFlag)
- ViewService: CreateView seeds default kanban subview and links the
boards property field; caches boardPropertyFieldID at startup
- App layer: CreateView/GetView/GetViewsForChannel/UpdateView/DeleteView
with channel-membership permission checks and WebSocket events
(view_created, view_updated, view_deleted)
- doSetupBoardsPropertyField: registers the Boards property group and
board field in NewServer() before ViewService construction
- GetFieldByName now returns store.ErrNotFound instead of raw sql.ErrNoRows
* Move permission checks out of App layer for views
- Remove HasPermissionToChannel calls from all App view methods
- Drop userID params from GetView, GetViewsForChannel, UpdateView, DeleteView
- Fix doSetupBoardsPropertyField to include required TargetType for PSAv2 field
* Make View service generic and enforce board validation in model
- Remove board-specific auto-setup from service and server startup
- Enforce that board views require Props, at least one subview, and at least one linked property in IsValid()
- Move default subview seeding out of app layer; callers must provide valid props
- Call PreSave on subviews during PreUpdate to assign IDs to new subviews
- Update all tests to reflect the new validation requirements
* Restore migrations files to match base branch
* Distinguish ErrNotFound from other errors in view store Get
* Use CONCURRENTLY and nontransactional for index operations in views migration
* Split views index creation into separate nontransactional migrations
* Update migrations.list
* Update i18n translations for views
* Fix makeView helper to include required Props for board view validation
* Rename ctx parameter from c to rctx in OAuthProvider mock
* Remove views service layer, call store directly from app
* Return 500 for unexpected DB errors in GetView, 404 only for not-found
* Harden View model: deep-copy Props, validate linked property IDs
- Add ViewBoardProps.Clone() to deep-copy LinkedProperties and Subviews
- Use it in View.Clone() and View.Patch() to prevent shared-slice aliasing
- Iterate over LinkedProperties in View.IsValid() and reject invalid IDs
with a dedicated i18n key
- Register ViewStore in storetest AssertExpectations so mock expectations
are enforced
- Add tests covering all new behaviours
* Restore autotranslation worker_stopped i18n translation
* Fix view store test IDs and improve error handling in app layer
- Use model.NewId() for linked property IDs in testUpdateView to fix
validation failure (IsValid rejects non-UUID strings)
- Fix import grouping in app/view.go (stdlib imports in one block)
- Return 404 instead of 500 when Update/Delete store calls return
ErrNotFound (e.g. concurrent deletion TOCTOU race)
* Add View store mock to retrylayer test genStore helper
The View store was added to the store interface but the genStore()
helper in retrylayer_test.go was not updated, causing TestRetry to panic.
Also removes the duplicate Recap mock registration.
* Refactor view deletion and websocket event handling; update SQL store methods to use query builder
* revert property field store
* Remove useless migrations
* Add cursor-based pagination to View store GetForChannel
- Add ViewQueryCursor and ViewQueryOpts types with validation
- Return (views, cursor, error) for caller-driven pagination
- PerPage clamping: <=0 defaults to 20, >200 clamps to 200
- Support IncludeDeleted filter
- Add comprehensive store tests for pagination, cursor edge cases,
PerPage clamping, and invalid input rejection
- Add app layer test for empty channelID → 400
- Update interface, retrylayer, timerlayer, and mock signatures
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Refactor test loops in ViewStore tests for improved readability
* change pagination to limit/offset
* Add upper-bound limits on View Subviews and LinkedProperties
Defense-in-depth validation: cap Subviews at 50 and LinkedProperties
at 500 to prevent abuse below the 300KB payload limit.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* MM-67388, MM-66528, MM-67750: Add View REST API endpoints, websocket events, and sort order (#35442)
* Add Views store and app layer for Integrated Boards
Implements the View entity (model, SQL store, service, app) as described
in the Integrated Boards tech spec. Views are channel-scoped board
configurations with typed props (board, kanban subviews) and soft-delete.
- public/model: View, ViewBoardProps, Subview, ViewPatch types with
PreSave/PreUpdate/IsValid/Patch/Clone/Auditable
- Migration 158: Views table with jsonb Props column and indexes
- SqlViewStore: CRUD with nil-safe Props marshaling (AppendBinaryFlag)
- ViewService: CreateView seeds default kanban subview and links the
boards property field; caches boardPropertyFieldID at startup
- App layer: CreateView/GetView/GetViewsForChannel/UpdateView/DeleteView
with channel-membership permission checks and WebSocket events
(view_created, view_updated, view_deleted)
- doSetupBoardsPropertyField: registers the Boards property group and
board field in NewServer() before ViewService construction
- GetFieldByName now returns store.ErrNotFound instead of raw sql.ErrNoRows
* Move permission checks out of App layer for views
- Remove HasPermissionToChannel calls from all App view methods
- Drop userID params from GetView, GetViewsForChannel, UpdateView, DeleteView
- Fix doSetupBoardsPropertyField to include required TargetType for PSAv2 field
* Make View service generic and enforce board validation in model
- Remove board-specific auto-setup from service and server startup
- Enforce that board views require Props, at least one subview, and at least one linked property in IsValid()
- Move default subview seeding out of app layer; callers must provide valid props
- Call PreSave on subviews during PreUpdate to assign IDs to new subviews
- Update all tests to reflect the new validation requirements
* Restore migrations files to match base branch
* Distinguish ErrNotFound from other errors in view store Get
* Use CONCURRENTLY and nontransactional for index operations in views migration
* Split views index creation into separate nontransactional migrations
* Update migrations.list
* Update i18n translations for views
* Fix makeView helper to include required Props for board view validation
* Rename ctx parameter from c to rctx in OAuthProvider mock
* Remove views service layer, call store directly from app
* Return 500 for unexpected DB errors in GetView, 404 only for not-found
* Harden View model: deep-copy Props, validate linked property IDs
- Add ViewBoardProps.Clone() to deep-copy LinkedProperties and Subviews
- Use it in View.Clone() and View.Patch() to prevent shared-slice aliasing
- Iterate over LinkedProperties in View.IsValid() and reject invalid IDs
with a dedicated i18n key
- Register ViewStore in storetest AssertExpectations so mock expectations
are enforced
- Add tests covering all new behaviours
* Restore autotranslation worker_stopped i18n translation
* Fix view store test IDs and improve error handling in app layer
- Use model.NewId() for linked property IDs in testUpdateView to fix
validation failure (IsValid rejects non-UUID strings)
- Fix import grouping in app/view.go (stdlib imports in one block)
- Return 404 instead of 500 when Update/Delete store calls return
ErrNotFound (e.g. concurrent deletion TOCTOU race)
* Add View store mock to retrylayer test genStore helper
The View store was added to the store interface but the genStore()
helper in retrylayer_test.go was not updated, causing TestRetry to panic.
Also removes the duplicate Recap mock registration.
* Refactor view deletion and websocket event handling; update SQL store methods to use query builder
* revert property field store
* Add View API endpoints with OpenAPI spec, client methods, and i18n
Implement REST API for channel views (board-type) behind the
IntegratedBoards feature flag. Adds CRUD endpoints under
/api/v4/channels/{channel_id}/views with permission checks
matching the channel bookmark pattern.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Remove useless migrations
* Add cursor-based pagination to View store GetForChannel
- Add ViewQueryCursor and ViewQueryOpts types with validation
- Return (views, cursor, error) for caller-driven pagination
- PerPage clamping: <=0 defaults to 20, >200 clamps to 200
- Support IncludeDeleted filter
- Add comprehensive store tests for pagination, cursor edge cases,
PerPage clamping, and invalid input rejection
- Add app layer test for empty channelID → 400
- Update interface, retrylayer, timerlayer, and mock signatures
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add cursor-based pagination to View API for channel views
* Enhance cursor handling in getViewsForChannel and update tests for pagination
* Refactor test loops in ViewStore tests for improved readability
* Refactor loop in TestGetViewsForChannel for improved readability
* change pagination to limit/offset
* switch to limit/offset pagination
* Add upper-bound limits on View Subviews and LinkedProperties
Defense-in-depth validation: cap Subviews at 50 and LinkedProperties
at 500 to prevent abuse below the 300KB payload limit.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add view sort order API endpoint
Add POST /api/v4/channels/{channel_id}/views/{view_id}/sort_order
endpoint following the channel bookmarks reorder pattern. Includes
store, app, and API layers with full test coverage at each layer.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add connectionId to view WebSocket events and sort_order API spec
Thread connectionId from request header through all view handlers
(create, update, delete, sort_order) to WebSocket events, matching
the channel bookmarks pattern. Add sort_order endpoint to OpenAPI
spec. Update minimum server version to 11.6.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Remove duplicate View/ViewPatch definitions from definitions.yaml
The merge from integrated-boards-mvp introduced duplicate View and
ViewPatch schema definitions that were already defined earlier in
the file with more detail (including ViewBoardProps ref and enums).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Update minimum server version to 11.6 in views API spec
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add missing translations for view sort order error messages
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Merge integrated-boards-mvp into ibmvp_api-views; remove spec files
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix flaky TestViewStore timestamp test on CI
Add sleep before UpdateSortOrder to ensure timestamps differ,
preventing same-millisecond comparisons on fast CI machines.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* remove duplicate views.yaml imclude
* Use c.boolString() for include_deleted query param in GetViewsForChannel
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix views.yaml sort order schema: use integer type and require body
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Refactor view sort order tests to use named IDs instead of array indices
Extract idA/idB/idC from views slice and add BEFORE/AFTER comments
to make stateful subtest ordering easier to follow.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Return 404 instead of 403 for view operations on deleted channels
Deleted channels should appear non-existent to callers rather than
revealing their existence via a 403. Detailed error text explains
the context for debugging.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* add missing channel deleteat checks
* Use c.Params.Page instead of manual page query param parsing in getViewsForChannel
c.Params already validates and defaults page/per_page, so the manual
parsing was redundant.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add support for total count in views retrieval
* Add tests for handling deleted views in GetViewsForChannel and GetView
* Short-circuit negative newIndex in UpdateSortOrder before opening transaction
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add per-channel limit on views to bound UpdateSortOrder cost
Without a cap, unbounded view creation makes sort-order updates
increasingly expensive (CASE WHEN per view, row locks). Adds
MaxViewsPerChannel=50 constant and enforces it in the app layer
before saving. Includes API and app layer tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Remove include_deleted support from views API
Soft-deleted views are structural metadata with low risk, but no other
similar endpoint (e.g. channel bookmarks) exposes deleted records without
an admin gate. Rather than adding an admin-only permission check for
consistency, remove the feature entirely since there is no current use case.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Update view permissions to require `create_post` instead of channel management permissions
* Remove obsolete view management error messages for direct and group messages
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat(migrations): add user tracking and object type to property fields
- Introduced user tracking columns (CreatedBy, UpdatedBy) to PropertyFields and PropertyValues.
- Added ObjectType column to PropertyFields with associated unique indexes for legacy and typed properties.
- Created new migration scripts for adding and dropping these features, including necessary indexes for data integrity.
- Established views for managing property fields with new attributes.
This update enhances the schema to support better tracking and categorization of property fields.
* Add Property System Architecture v2 API endpoints (#35583)
* Adds uniqueness mechanisms to the property fields
After adding ObjectType, this commit ensures that both the PSAv1 and
PSAv2 schemas are supported, and enforces property uniqueness through
both database indexes and a logical check when creating new property
fields.
* Adds uniqueness check to property updates
Updates are covered on this commit and we refactor as well the SQL
code to use the squirrel builder and work better with the conditional
addition of the `existingID` piece of the query.
* Add translations to error messages
* Add the permissions to the migrations, model and update the store calls
* Adds the property field and property group app layer
* Adds authorization helpers for property fields and values
* Make sure that users cannot lock themselves out of property fields
* Migrate permissions from a JSON column to three normalized columns
* Remove the audit comment
* Use target level constants in authorization
* Log authorization membership failures
* Rename admin to sysadmin
* Adds the Property System Architecture v2 API endpoints
* Adds permission checks to the create field endpoint
* Add target access checks to value endpoints
* Add default branches for object_type and target_type and extra guards for cursor client4 methods
* Fix vet API mismatch
* Fix error checks
* Fix linter
* Add merge semantics for property patch logic and API endpoint
* Fix i18n
* Fix duplicated patch elements and early return on bad cursor
* Update docs to use enums
* Fix i18n sorting
* Update app layer to return model.AppError
* Adds a limit to the number of property values that can be patched in the same request
* Require target_type filter when searching property fields
* Add objectType validation as part of field.IsValid()
* Fix linter
* Fix test with bad objecttpye
* Fix test grouping
---------
Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
* MM-67968: Flatten view model — remove icon, subviews, typed board props (#35726)
* feat(views): flatten view model by removing icon, subview, and board props
Simplifies the View data model as part of MM-67968: removes Icon, Subview,
and ViewBoardProps types; renames ViewTypeBoard to ViewTypeKanban; replaces
typed Props with StringInterface (map[string]any); adds migration 000167
to drop the Icon column from the Views table.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* feat(api): update views OpenAPI spec to reflect flattened model
Removes ViewBoardProps, Subview, and icon from the View and ViewPatch
schemas. Changes type enum from board to kanban. Replaces typed props
with a free-form StringInterface object. Aligns with MM-67968.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* refactor(views): simplify store by dropping dbView and marshalViewProps
StringInterface already implements driver.Valuer and sql.Scanner, so the
manual JSON marshal/unmarshal and the dbView intermediate struct were
redundant. model.View now scans directly from the database. Also removes
the dead ViewMaxLinkedProperties constant and wraps the Commit() error in
UpdateSortOrder.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(api): allow arbitrary JSON in view props OpenAPI schema
The props field was restricted to string values via
additionalProperties: { type: string }, conflicting with the Go model's
StringInterface (map[string]any). Changed to additionalProperties: true
in View, ViewPatch, and inline POST schemas.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Adds basic implementation of the generic redux store for PSAv2 (#35512)
* Adds basic implementation of the generic redux store for PSAv2
* Add created_by and updated_by to the test fixtures
* Make target_id, target_type and object_type mandatory
* Wrap getPropertyFieldsByIds and getPropertyValuesForTargetByFieldIds with createSelector
* Address PR comments
---------
Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
* Adds websocket messages for the PSAv2 API events (#35696)
* Adds uniqueness mechanisms to the property fields
After adding ObjectType, this commit ensures that both the PSAv1 and
PSAv2 schemas are supported, and enforces property uniqueness through
both database indexes and a logical check when creating new property
fields.
* Adds uniqueness check to property updates
Updates are covered on this commit and we refactor as well the SQL
code to use the squirrel builder and work better with the conditional
addition of the `existingID` piece of the query.
* Add translations to error messages
* Add the permissions to the migrations, model and update the store calls
* Adds the property field and property group app layer
* Adds authorization helpers for property fields and values
* Make sure that users cannot lock themselves out of property fields
* Migrate permissions from a JSON column to three normalized columns
* Remove the audit comment
* Use target level constants in authorization
* Log authorization membership failures
* Rename admin to sysadmin
* Adds the Property System Architecture v2 API endpoints
* Adds permission checks to the create field endpoint
* Add target access checks to value endpoints
* Add default branches for object_type and target_type and extra guards for cursor client4 methods
* Fix vet API mismatch
* Fix error checks
* Fix linter
* Add merge semantics for property patch logic and API endpoint
* Fix i18n
* Fix duplicated patch elements and early return on bad cursor
* Update docs to use enums
* Fix i18n sorting
* Update app layer to return model.AppError
* Adds a limit to the number of property values that can be patched in the same request
* Adds websocket messages for the PSAv2 API events
* Add IsPSAv2 helper to the property field for clarity
* Add guard against nil returns on field deletion
* Add docs to the websocket endpoints
---------
Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
* migrations: consolidate views migrations and reorder after master
- Merged 000165 (create Views) with 000167 (drop Icon) since Icon was never needed
- Renumbered branch migrations 159-166 → 160-167 so master's 000159 (deduplicate_policy_names) runs first
- Regenerated migrations.list
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Add API endpoint to retrieve posts for a specific view (#35604)
Automatic Merge
* Apply fixes after merge
* Return a more specific error from getting multiple fields
* Prevent getting broadcast params on field deletion if not needed
* Remove duplicated migration code
* Update property conflict code to always use master
* Adds nil guard when iterating on property fields
* Check that permission level is valid before getting rejected by the database
* Validate correctness on TargetID for PSAv2 fields
* Avoid PSAv1 using permissions or protected
* Fix test data after validation change
* Fix flaky search test
* Adds more posts for filter use cases to properly test exclusions
---------
Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
Co-authored-by: Alejandro García Montoro <alejandro.garciamontoro@gmail.com>
Co-authored-by: Julien Tant <julien@craftyx.fr>
Co-authored-by: Mattermost Build <build@mattermost.com>
Co-authored-by: Julien Tant <785518+JulienTant@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Add LIKE-based CJK (Chinese, Japanese, Korean) search support for PostgreSQL, gated behind a `CJKSearch` feature flag.
PostgreSQL's built-in full-text search (`to_tsvector`/`to_tsquery`) does not support CJK languages because it relies on whitespace-based tokenization, which doesn't work for logographic/syllabic scripts that don't use spaces between words. This PR adds a `LIKE`-based fallback for search terms containing CJK characters.
**How it works:** When the `CJKSearch` feature flag is enabled and a search term contains CJK characters (Han, Hiragana, Katakana, or Hangul), the query builder generates `LIKE '%term%'` clauses instead of `to_tsvector @@ to_tsquery` expressions. Case-sensitive `LIKE` is used rather than `ILIKE` since CJK scripts have no letter casing, which also allows potential use of standard B-tree indexes.
* support for Elastic(Open)search CJK analysis plugins
* addresses PR review comments
* addresses PR comments
* moves CJK based tests to own file and adds some more
Dockerfile for open and elasticsearch are changed to install required
plugins for testing or running locally.
* properly sort error messages
* fix style issues
* removes trailing space
This commit reverts PR #30214, which addressed bug MM-60790 but caused a
performance regression tracked by MM-66782.
This revert has two implications:
1. The performance issue is solved.
2. The original bug is re-introduced.
Re-introducing the original bug seems not to be ideal, but I argue that
the original PR did not actually fix the bug:
- Before that PR, looking for a quoted string would return additional
results: the UX was slightly confusing, because when the user looked
for the word "stateful", the results would contain matches like
"states" (see MM-60790).
- After that PR, looking for a quoted string can timeout, so that the
list of results becomes empty. The UX here may be less confusing,
since the user simply doesn't find what they're looking for, and they
may assume that string is not present in any post, but it's completely
wrong: the result list is empty because the SQL query timed out and
thus the endpoint returned 0 results.
The solution to the original issue should be addressed via
Elasticsearch, which should provide a more refined and precise search
results.
For more information on the investigation on this issue and the
motivation behind the revert, see
https://mattermost.atlassian.net/wiki/x/IYAk_w
Co-authored-by: Mattermost Build <build@mattermost.com>
* Remove legacy quoteColumnName() utility
Since Mattermost only supports PostgreSQL, the quoteColumnName() helper
that was designed to handle database-specific column quoting is no longer
needed. The function was a no-op that simply returned the column name
unchanged.
Remove the function from utils.go and update status_store.go to use
the "Manual" column name directly.
* Remove legacy driver checks from store.go
Since Mattermost only supports PostgreSQL, remove conditional checks
for different database drivers:
- Simplify specialSearchChars() to always return PostgreSQL-compatible chars
- Remove driver check from computeBinaryParam()
- Remove driver check from computeDefaultTextSearchConfig()
- Simplify GetDbVersion() to use PostgreSQL syntax directly
- Remove switch statement from ensureMinimumDBVersion()
- Remove unused driver parameter from versionString()
* Remove MySQL alternatives for batch delete operations
Since Mattermost only supports PostgreSQL, remove the MySQL-specific
DELETE...LIMIT syntax and keep only the PostgreSQL array-based approach:
- reaction_store.go: Use PostgreSQL array syntax for PermanentDeleteBatch
- file_info_store.go: Use PostgreSQL array syntax for PermanentDeleteBatch
- preference_store.go: Use PostgreSQL tuple IN subquery for DeleteInvalidVisibleDmsGms
* Remove MySQL alternatives for UPDATE...FROM syntax
Since Mattermost only supports PostgreSQL, remove the MySQL-specific
UPDATE syntax that joins tables differently:
- thread_store.go: Use PostgreSQL UPDATE...FROM syntax in
MarkAllAsReadByChannels and MarkAllAsReadByTeam
- post_store.go: Use PostgreSQL UPDATE...FROM syntax in deleteThreadFiles
* Remove MySQL alternatives for JSON and subquery operations
Since Mattermost only supports PostgreSQL, remove the MySQL-specific
JSON and subquery syntax:
- thread_store.go: Use PostgreSQL JSONB operators for updating participants
- access_control_policy_store.go: Use PostgreSQL JSONB @> operator for
querying JSON imports
- session_store.go: Use PostgreSQL subquery syntax for Cleanup
- job_store.go: Use PostgreSQL subquery syntax for Cleanup
* Remove MySQL alternatives for CTE queries
Since Mattermost only supports PostgreSQL, simplify code that
uses CTEs (Common Table Expressions):
- channel_store.go: Remove MySQL CASE-based fallback in
UpdateLastViewedAt and use PostgreSQL CTE exclusively
- draft_store.go: Remove driver checks in DeleteEmptyDraftsByCreateAtAndUserId,
DeleteOrphanDraftsByCreateAtAndUserId, and determineMaxDraftSize
* Remove driver checks in migrate.go and schema_dump.go
Simplify migration code to use PostgreSQL driver directly since
PostgreSQL is the only supported database.
* Remove driver checks in sqlx_wrapper.go
Always apply lowercase named parameter transformation since PostgreSQL
is the only supported database.
* Remove driver checks in user_store.go
Simplify user store functions to use PostgreSQL-only code paths:
- Remove isPostgreSQL parameter from helper functions
- Use LEFT JOIN pattern instead of subqueries for bot filtering
- Always use case-insensitive LIKE with lower() for search
- Remove MySQL-specific role filtering alternatives
* Remove driver checks in post_store.go
Simplify post_store.go to use PostgreSQL-only code paths:
- Inline getParentsPostsPostgreSQL into getParentsPosts
- Use PostgreSQL TO_CHAR/TO_TIMESTAMP for date formatting in analytics
- Use PostgreSQL array syntax for batch deletes
- Simplify determineMaxPostSize to always use information_schema
- Use PostgreSQL jsonb subtraction for thread participants
- Always execute RefreshPostStats (PostgreSQL materialized views)
- Use materialized views for AnalyticsPostCountsByDay
- Simplify AnalyticsPostCountByTeam to always use countByTeam
* Remove driver checks in channel_store.go
Simplify channel_store.go to use PostgreSQL-only code paths:
- Always use sq.Dollar.ReplacePlaceholders for UNION queries
- Use PostgreSQL LEFT JOIN for retention policy exclusion
- Use PostgreSQL jsonb @> operator for access control policy imports
- Simplify buildLIKEClause to always use LOWER() for case-insensitive search
- Simplify buildFulltextClauseX to always use PostgreSQL to_tsvector/to_tsquery
- Simplify searchGroupChannelsQuery to use ARRAY_TO_STRING/ARRAY_AGG
* Remove driver checks in file_info_store.go
Simplify file_info_store.go to use PostgreSQL-only code paths:
- Always use PostgreSQL to_tsvector/to_tsquery for file search
- Use file_stats materialized view for CountAll()
- Use file_stats materialized view for GetStorageUsage() when not including deleted
- Always execute RefreshFileStats() for materialized view refresh
* Remove driver checks in attributes_store.go
Simplify attributes_store.go to use PostgreSQL-only code paths:
- Always execute RefreshAttributes() for materialized view refresh
- Remove isPostgreSQL parameter from generateSearchQueryForExpression
- Always use PostgreSQL LOWER() LIKE LOWER() syntax for case-insensitive search
* Remove driver checks in retention_policy_store.go
Simplify retention_policy_store.go to use PostgreSQL-only code paths:
- Remove isPostgres parameter from scanRetentionIdsForDeletion
- Always use pq.Array for scanning retention IDs
- Always use pq.Array for inserting retention IDs
- Remove unused json import
* Remove driver checks in property stores
Simplify property_field_store.go and property_value_store.go to use
PostgreSQL-only code paths:
- Always use PostgreSQL type casts (::text, ::jsonb, ::bigint, etc.)
- Remove isPostgres variable and conditionals
* Remove driver checks in channel_member_history_store.go
Simplify PermanentDeleteBatch to use PostgreSQL-only code path:
- Always use ctid-based subquery for DELETE with LIMIT
* Remove remaining driver checks in user_store.go
Simplify user_store.go to use PostgreSQL-only code paths:
- Use LEFT JOIN for bot exclusion in AnalyticsActiveCountForPeriod
- Use LEFT JOIN for bot exclusion in IsEmpty
* Simplify fulltext search by consolidating buildFulltextClause functions
Remove convertMySQLFullTextColumnsToPostgres and consolidate
buildFulltextClause and buildFulltextClauseX into a single function
that takes variadic column arguments and returns sq.Sqlizer.
* Simplify SQL stores leveraging PostgreSQL-only support
- Simplify UpdateMembersRole in channel_store.go and team_store.go
to use UPDATE...RETURNING instead of SELECT + UPDATE
- Simplify GetPostReminders in post_store.go to use DELETE...RETURNING
- Simplify DeleteOrphanedRows queries by removing MySQL workarounds
for subquery locking issues
- Simplify UpdateUserLastSyncAt to use UPDATE...FROM...RETURNING
instead of fetching user first then updating
- Remove MySQL index hint workarounds in ORDER BY clauses
- Update outdated comments referencing MySQL
- Consolidate buildFulltextClause and remove convertMySQLFullTextColumnsToPostgres
* Remove MySQL-specific test artifacts
- Delete unused MySQLStopWords variable and stop_word.go file
- Remove redundant testSearchEmailAddressesWithQuotes test
(already covered by testSearchEmailAddresses)
- Update comment that referenced MySQL query planning
* Remove MySQL references from server code outside sqlstore
- Update config example and DSN parsing docs to reflect PostgreSQL-only support
- Remove mysql:// scheme check from IsDatabaseDSN
- Simplify SanitizeDataSource to only handle PostgreSQL
- Remove outdated MySQL comments from model and plugin code
* Remove MySQL references from test files
- Update test DSNs to use PostgreSQL format
- Remove dead mysql-replica flag and replicaFlag variable
- Simplify tests that had MySQL/PostgreSQL branches
* Update docs and test config to use PostgreSQL
- Update mmctl config set example to use postgres driver
- Update test-config.json to use PostgreSQL DSN format
* Remove MySQL migration scripts, test data, and docker image
Delete MySQL-related files that are no longer needed:
- ESR upgrade scripts (esr.*.mysql.*.sql)
- MySQL schema dumps (mattermost-mysql-*.sql)
- MySQL replication test scripts (replica-*.sh, mysql-migration-test.sh)
- MySQL test warmup data (mysql_migration_warmup.sql)
- MySQL docker image reference from mirror-docker-images.json
* Remove MySQL references from webapp
- Simplify minimumHashtagLength description to remove MySQL-specific configuration note
- Remove unused HIDE_MYSQL_STATS_NOTIFICATION preference constant
- Update en.json i18n source file
* clean up e2e-tests
* rm server/tests/template.load
* Use teamMemberSliceColumns() in UpdateMembersRole RETURNING clause
Refactor to use the existing helper function instead of hardcoding
the column names, ensuring consistency if the columns are updated.
* u.id -> u.Id
* address code review feedback
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
This PR adds functionality to search by a global search prefix.
This allows Mattermost to be used across multiple data centers
with multiple Elasticsearch instances synchronized using
cross-cluster replication.
While here, we also add tests cases to cover for some missing
search interface methods.
For now, no system console setting is exposed. Because IndexPrefix
is also not exposed. It can be added later if a need arises.
https://mattermost.atlassian.net/browse/MM-63298
```release-note
A new config setting ElasticsearchSettings.GlobalSearchPrefix is added
which can be used to search across multiple indices having a common prefix.
This is useful in a scenario with multiple Elasticsearch instances, where
multiple instances are writing to different indices with different prefixes
using the ElasticsearchSettings.IndexPrefix setting.
```
Co-authored-by: Mattermost Build <build@mattermost.com>
When a search term is a fully quoted string, we want to avoid
stemming if it is supported by the search backend.
For Postgres, it does provide a feature by which if we use the
"simple" search config, then no stemming is performed and an exact
match with the word is done without having to resort
to LIKE queries.
Unfortunately, for ES/OS this is not an option because
the message field is a text field, which means ES/OS will analyze it,
stem it and store it in its root form. Therefore, no exact match
can be possible with ES/OS.
The only solution here is to have yet another keyword field
for message which will store it in its raw form. But this
will effectively double the disk storage for post indices
and not a good design choice.
Ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-term-query.html#avoid-term-query-text-fieldshttps://mattermost.atlassian.net/browse/MM-60790
```release-note
NONE
```
* re-arrange the tests to run only on DB
```release-note
NONE
```
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* poc - wip
* add search files across teams
* eslint
* fix existing tests
* fix webapp style
* fix test
* add api doc
* change initial state in test
* add tests on API
* add tests on file info layer
* fix file search tags
* add rhs reducer test
* reset team selected when the RHS is suppressed
* change css to reflect UI
* fix style
* fix doc wording
* make getSearchTeam return currentTeamId when value is not set
* await is unnecessary
* revert boolean check and add test
* add comment to getSearchTeam to let dev knows it defaults to currentTeam
* remove redundant team check
* simplfy test
* fix style check
---------
Co-authored-by: Caleb Roseland <caleb@calebroseland.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
* Fixed the issue for DB layer, ES to go
* Handled channel bookmarks
* Handled Bleve
* Lint fix
* Added channel bookmark test
* Skip bleve test
* Used common function
* SKipping ES as indexing logic in unavailable in test
* added rctx to reuired files
* added more file changes after issues on startup
* added context to draft_store
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* Refactor: Use generic to change func StringInSlice to Contains
* Refactor: Use generic to change func stringNotInSlice to Contains
Make utils.go (dir server/public/utils) and func Contains
* Refactor: Move func Contains from channels/utils to public/utils
Move func Contains from channels/utils to public/utils
Fix import declarations line
* Docs: Add a description of the Contains function
* Test: add TestContains
Add a test code for a Contain function
https://mattermost.atlassian.net/browse/MM-52532
- Replace golint with revive
- Add makezero linter
- Fix all the required linter failures
Some issues in enterprise and public modules
are yet to be fixed. We send this to expediate things.
It was a good decision in hindsight to keep the public module as 0.x
because this would have been a breaking change again.
https://mattermost.atlassian.net/browse/MM-53032
```release-note
Changed the Go module path from github.com/mattermost/mattermost-server/server/v8 to github.com/mattermost/mattermost/server/v8.
For the public facing module, it's path is also changed from github.com/mattermost/mattermost-server/server/public to github.com/mattermost/mattermost/server/public
```
https://mattermost.atlassian.net/browse/MM-52079
```release-note
We upgrade the module version to 8.0. The new module path is github.com/mattermost-server/server/v8.
```
Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
* MM-45956: Optimize FileInfo stats query
We Denormalize Post.ChannelId on FileInfo.ChannelId
```release-note
The file info stats query is now optimized by denormalizing the channelID column into the table itself. This will speed up the query to get the file count for a channel on clicking the RHS.
Migration times:
On a MySQL 8.0.31 DB with
1405 rows in FileInfo and 11M posts, it took around 0.3s
On a Postgres 12.14 DB with
1731 rows in FileInfo and 11M posts, it took around 0.27s
```
https://mattermost.atlassian.net/browse/MM-45956
