* Adds a mechanism to delete CPA values for a given user
This requires improving the Property Value service to enable delete
all values for a given target, so a new method was created that allows
to delete filtering by targetType and targetID (required) and
optionally for a specific groupID in case the caller wants to affect
all values for a target (useful in case you remove a post for example
and want to delete all values pointing to that post regardless of the
feature they belong to) or only those that belong to a specific
feature.
* Fix property value tests
* Fix after merge and update method name
* Fix linter
---------
Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
Co-authored-by: Mattermost Build <build@mattermost.com>
* Adds direct participants to the channel invite
The channel invite now contains the sanitized users that are local to
the node that is sending the invite. In the event that the receiving
server doesn't have those users in its local database, it can create
them from the invite and correctly generate the DM or GM with them as
members.
* Use IsRemote instead of directly checking user attributes
---------
Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
Co-authored-by: Mattermost Build <build@mattermost.com>
* Move the sanitization and validation of CPA values to the model
* Fix CI
* Use proper IDs instead of strings
---------
Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
Co-authored-by: Mattermost Build <build@mattermost.com>
Fixes https://github.com/mattermost/mattermost/issues/29102
- Fix error handling in fileutils.FindDir() call by properly checking the returned boolean value
- Add proper error handling for SetPhase2PermissionsMigrationStatus calls
- Remove channels/app/support_packet_test.go from the errcheck exception list in .golangci.yml
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-authored-by: Claude <noreply@anthropic.com>
* allow opening angle bracket before links
* cut off link before an angle bracket(both open & close)
* add more test cases for ParseURLAutolink func
* add more test cases for ParseWWWAutolink func
* add more test cases for TrimTrailingCharactersFromLink func
* add more test cases for GetFirstLinkAndImages func
* add test cases for isAllowedBeforeWWWLink func
* allow closing angle bracket and opening paranthesis before links
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
This PR adds functionality to search by a global search prefix.
This allows Mattermost to be used across multiple data centers
with multiple Elasticsearch instances synchronized using
cross-cluster replication.
While here, we also add tests cases to cover for some missing
search interface methods.
For now, no system console setting is exposed. Because IndexPrefix
is also not exposed. It can be added later if a need arises.
https://mattermost.atlassian.net/browse/MM-63298
```release-note
A new config setting ElasticsearchSettings.GlobalSearchPrefix is added
which can be used to search across multiple indices having a common prefix.
This is useful in a scenario with multiple Elasticsearch instances, where
multiple instances are writing to different indices with different prefixes
using the ElasticsearchSettings.IndexPrefix setting.
```
Co-authored-by: Mattermost Build <build@mattermost.com>
* MM-62157: Avoid SELECT * in user_store.go
- Replace raw SQL query "SELECT * FROM Users" with query builder in Update method
- Replace raw SQL query "SELECT * from Users" with query builder in ClearAllCustomRoleAssignments
- Change Select("*") to Select("data.*") in GetUserReport method
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* MM-62157: Avoid more instances of SELECT * in user_store.go
- Replace raw SQL query "Users.*, cm.ChannelId" with usersQuery in getUsersInGroupChannels
- Replace "Users.*" with getUsersColumns() in GetUserReport method
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Adds a groupID filter to the property service methods
This allows the property service caller to directly ensure that a
given call is only going to affect a field or value that belongs to a
given group, instead of (for example) retrieving a property value
before deleting it by id to ensure that the value belongs to a
specific property group. The groupID filter is optional and has no
effect if called with the empty string value.
The changes also remove references to input sanitization on trimming
the whitespace for the CPA field names and validate at the API level
the input for the field patch endpoint.
* Fix linter
---------
Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
Co-authored-by: Mattermost Build <build@mattermost.com>
* MM-62162: Replace SELECT * in user_access_token_store.go
- Replaced all SELECT * queries with explicit column selection
- Used query builder instead of raw SQL strings for all queries
- Added reusable userAccessTokensSelectQuery in the store constructor
- Added comprehensive test for pagination and IsActive flag
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix code formatting in user_access_token_store test file
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* MM-62162: Update UserAccessToken Search method to use explicit column names
- Fixed "ambiguous column" errors by explicitly naming columns with table qualifiers
- Added test for the Search functionality
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* MM-62162: Update UserAccessToken Search method to use explicit column names
- Fixed "ambiguous column" errors by explicitly naming columns with table qualifiers
- Maintained exact semantics of original query's LIKE filters
- Refactored to use query builder instead of raw SQL
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* MM-62162: Update UserAccessToken Search method to use reusable query
- Modified userAccessTokensSelectQuery to use fully qualified column names
- Updated Search method to use the existing query builder
- Fixed "ambiguous column" errors by using full column qualifiers
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* MM-62162: Remove unnecessary comment in user_access_token_store.go
Removed redundant comment in Search method explaining the use of qualified column names, as the code is self-documenting.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* MM-62162: Use t.Cleanup() for token cleanup in userAccessTokenPagination test
* MM-62162: Handle error in Cleanup function
* linting
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* MM-62154: Avoid SELECT * in webhook_store.go
- Defined column list functions for both IncomingWebhooks and OutgoingWebhooks tables
- Converted all uses of SELECT * to use the column list functions
- Used the QueryBuilder pattern for SQL queries for better consistency and readability
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* MM-62154: Improve webhook store implementation using query builders
- Refactored webhook store to follow session_store.go pattern
- Added select query builders in the constructor
- Used pre-built queries throughout the store implementation for better maintainability and performance
- Removed redundant incomingWebhookColumns() and outgoingWebhookColumns() functions
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* MM-62154: Fix whitespace style issues in webhook_store.go
- Ran gofmt to remove trailing whitespace
- Ensures consistent code formatting
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* MM-62154: Remove unnecessary int(0) casting
- Simplified DeleteAt=0 condition by removing redundant int casting
- Maintains same behavior while making code cleaner
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* MM-62160: Avoid SELECT * in user_terms_of_service.go
- Added userTermsOfServiceSelectQuery in the constructor
- Replaced raw SQL query with query builder pattern
- Used explicit column selection instead of SELECT *
- Made the implementation more resilient to schema changes
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* MM-62160: Also avoid SELECT * in terms_of_service_store.go
- Added termsOfServiceSelectQuery field to store struct
- Replaced SELECT * with explicit column selection
- Updated GetLatest and Get methods to use the query builder pattern
- Made the implementation more resilient to schema changes
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* MM-62155: Avoid SELECT * in team_store.go
- Added explicit column lists for all SELECT queries
- Created teamSelectQuery for reused queries
- Replaced raw SQL queries with query builder pattern
* Refactor getTeamMembersWithSchemeSelectQuery to use teamMembersQuery builder
Instead of duplicating TeamMembers columns, use the existing builder to avoid redundancy.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Remove redundant comments from getTeamMembersWithSchemeSelectQuery
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Add config settings for additional security features on mobile
* Add system console settings for mobile security
* Update svg and link
* Fix strings
* Add test for the discovery feature
* Fix tests
* Add permission migrations
* Add relevant e2e tests
* Fix key alignment
* fix tests
* Fix lint
* Mock new migration
* Fix playwright prettier
* Add new section to delegated permissions
* Update snapshots
* Fix flakyness in playwright test
---------
Co-authored-by: Elias Nahum <nahumhbl@gmail.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
We used < which meant we missed sending
reminders created for that timestamp, because
the job runs on that exact time.
Using <= to fix that.
https://mattermost.atlassian.net/browse/MM-63545
```release-note
NONE
```
* test: Add unit tests for custom profile attributes select options
* feat: Add custom profile attributes model with validation and constants
* refactor: Trim spaces from name and color in custom profile attribute select option constructor
* gofmt
* refactor: Fix typo in custom profile attributes select option function name
* feat: Add IsValid method to validate CustomProfileAttributesSelectOptions
* refactor: Replace map[string]bool with map[string]struct{} for key existence check
* refactor: Rename NewCustomProfileAttributeSelectOption to NewCustomProfileAttributesSelectOption
* feat: Add validation to prevent empty custom profile attribute options
* refactor: Add validation and creation methods for custom profile attributes
* feat: Add index number to validation error messages in custom profile attributes
* fix tests
* add default visibility
* feat: Add comprehensive test cases for custom profile attributes field validation
* fix: Update custom profile attributes map keys to use capitalized names
* feat: Add support for lowercase and title case keys in custom profile attributes map
* test: Add comprehensive test for NewCustomProfileAttributesSelectOptionFromMap
* feat: Add validation for custom profile attributes fields
* refactor: Update CustomProfileAttributesSelectOption constructor to prioritize ID parameter
* test: Add test cases for preserving IDs in custom profile attributes
* feat: Enhance ID validation and trimming in custom profile attributes
* don't do validation in constructor
* test: Add test case for preserving option IDs when patching select field
* improve test
* i18n
* refactor: Modify CustomProfileAttributesSelectOption to use lowercase JSON keys
* fix casing in custom profilte attributes test
* refactor: Use consistent "ValidateCPAField" in error messages for custom profile attributes
* use custom types rather than string
* lint
* fix api test
* refactor: Make color field optional in custom profile attributes
* style
* generic options
* removed unused i18n
* test: Add tests for NewCPAFieldFromPropertyField and CPAFieldToPropertyField
* test: Add test case for property field with empty attributes
* refactor: Cleanup whitespace and remove empty Attrs in custom profile attributes test
* test: Add test case for CPA field with empty attributes
* refactor: Improve custom profile attributes field handling and validation
* refactor: Move validateCustomProfileAttributesField to Validate method on CPAField struct
* use CPAField
* code style
* add validation and tests
* tests
* i18n
* err->appErr
* fix TestDeleteCPAField test
* i18n
* Add SAML and LDAP attr
* rename CustomProfileAttributes in method to CPA
* rename CPASortOrder method
* rearrange consts
* use Len test method
* sanitize and validate
* manage error the same way property field and value do
* fix: Update test error ID for custom profile attributes validation
* test: Update error ID expectations in custom profile attributes tests
* refactor: Convert CPAAttrs.SortOrder from string to int
* json uses float64
* feat: Add length validation for custom profile attribute option name and color
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
The file already has the ChannelID field. Therefore, we avoid querying
the database again. This sometimes causes errors in customer environments
where there is noticeable replication lag, causing files not to be
indexed entirely.
https://mattermost.atlassian.net/browse/MM-62900
```release-note
NONE
```
Co-authored-by: Mattermost Build <build@mattermost.com>
Previously, we would set it to nil pointer which would eventually
cause a panic when ReplicaLagAbs/ReplicaLagTime would get called.
https://mattermost.atlassian.net/browse/MM-62964
```release-note
NONE
```
Co-authored-by: Mattermost Build <build@mattermost.com>
* MM-62159: Avoid SELECT * in channel_store_categories.go
- Added sidebarCategorySelectQuery field to SqlChannelStore struct
- Replaced SELECT * with explicit column selection in GetSidebarCategory and getSidebarCategoriesT functions
- Updated raw SQL query in addChannelToFavoritesCategoryT to use explicit column selection
- Made the implementation more resilient to schema changes
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* use sc alias, simplify
* MM-62159: Fix ambiguous ID column in sidebar category queries
- Modified sidebarCategorySelectQuery initialization to explicitly use "sc" table alias for all columns
- Prevents "Column 'Id' in field list is ambiguous" error when joining with other tables
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* MM-62159: Consistently use table aliases in sidebar categories queries
- Added 'sc' table alias to all sidebar category queries
- Ensures consistency and avoids ambiguous column errors in future joins
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* MM-62159: Replace 'sc' alias with full 'SidebarCategories' table name
- Replaced all instances of the 'sc' alias with the full table name 'SidebarCategories'
- Updated the SidebarCategories query builder to use the full table name
- Removed commented-out debug printf statement
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
When a search term is a fully quoted string, we want to avoid
stemming if it is supported by the search backend.
For Postgres, it does provide a feature by which if we use the
"simple" search config, then no stemming is performed and an exact
match with the word is done without having to resort
to LIKE queries.
Unfortunately, for ES/OS this is not an option because
the message field is a text field, which means ES/OS will analyze it,
stem it and store it in its root form. Therefore, no exact match
can be possible with ES/OS.
The only solution here is to have yet another keyword field
for message which will store it in its raw form. But this
will effectively double the disk storage for post indices
and not a good design choice.
Ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-term-query.html#avoid-term-query-text-fieldshttps://mattermost.atlassian.net/browse/MM-60790
```release-note
NONE
```
* re-arrange the tests to run only on DB
```release-note
NONE
```
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* Update Go version to v1.23.7
* Bump golangci-lint to a version supporting Go 1.23
* Fix golangci-lint warnings
Several rules from gosimple, revive and staticcheck linters were
failing:
- Redefinition of built-in identifiers (max, min, new, recover...)
- Use of printf-like functions with simple strings
- Check for nil slices, when len already takes it into account
* Trigger Build
* Trigger Build
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* ClaimJob now returns newly claimed job
* internal code affected by change
* test changes required
* two branches: for mysql, use transaction; for postgres, use returning
* two branches: for mysql, use transaction; for postgres, use returning
* use same millis value for LastActivityAt and StartAt
* blank commit
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* MM-62161: Replace SELECT * in session_store.go
- Replaced all SELECT * queries with explicit column selection
- Used query builder instead of raw SQL strings where possible
- Added reusable sessionSelectQuery in the store constructor
- Added comprehensive test for GetSessionsWithActiveDeviceIds
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix variable shadowing issues in session_store.go
Resolved variable shadowing by reassigning to the existing error variables instead of declaring new ones in scoped blocks.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix code formatting in session_store test file
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Updates the property service and store method signatures
Getters can now receive a `groupID` that narrows down the query if
present, so it's not necessary to check for the group ID on the
returning values from the outside layers.
The Search methods now receive the `groupID` and the `targetID`
explicitly as parameters, incentivizing the use of the indexes that
the underlying tables have on the searches.
* Fix tests
---------
Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
Co-authored-by: Mattermost Build <build@mattermost.com>
If the deadQueue was full, we would be incorrectly sending
the wrong index for dqPtr. Fixed that and added a test case.
https://mattermost.atlassian.net/browse/MM-63304
```release-note
NONE
```
* MM-63205 Revert emoji store methods to return empty arrays instead of nil
* Update tests for getEmojiList
* Add case to TestGetEmojisByNames
* Update tests for searchEmoji and autocompleteEmoji
* [MM-62687] Patch permission check to avoid modifying the system admin
* Check for manage system first
* PR feedback
* Add another test
* Lint
* Fix test
* Bump Go version to 1.23.6
* Update CodeQL Github action as well
* Use server's Go version for CodeQL action
Co-authored-by: Antonis Stamatiou <stamatiou.antonis@gmail.com>
* Empty commit to trigger CI
* Bump golangci-lint to a version supporting Go 1.23
* Fix golangci-lint warnings
Several rules from gosimple, revive and staticcheck linters were
failing:
- Redefinition of built-in identifiers (max, min, new, recover...)
- Use of printf-like functions with simple strings
- Check for nil slices, when len already takes it into account
---------
Co-authored-by: Antonis Stamatiou <stamatiou.antonis@gmail.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
* MM-61947 Run DND expiry job more often and round expiry time to match interval
* Move comment to make it godoc-compatible
* Change truncateDNDEndTime to work with seconds
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* Adde MySQL and Postgres migrations
* Replaced select * with column names
* removed all * from channel SQL store
* cleanup
* Fixed a duplicate column
* cleanup
* Added migrations and store support
* WIP
* used channelname slice in a missed place
* Handled patch
* Added app level tests
* Added API layer tests
* Added API layer tests
* WIP
* converted to query builder
* cleanupo
* added not null and default constraints
* Fixed test
* fixed file name
* review fixes
* review fixes
* updated migration file
* fixed text
* Review fixes
* Adde MySQL and Postgres migrations
* Replaced select * with column names
* removed all * from channel SQL store
* cleanup
* Fixed a duplicate column
* cleanup
* used channelname slice in a missed place
* WIP
* converted to query builder
* cleanupo
* Review fixes
When Channel iteration mode is enabled, we need to ensure that
channel scoped events do not fall through to the connIndex.All()
condition. This is possible because there are multiple hubs
in a given system. So a single event will flow through all of them
and in some hubs, a channel scoped event might not have any connections.
In that case, we need to stop processing further.
https://mattermost.atlassian.net/browse/MM-62960
```release-note
NONE
```
When we added iteration by channelID, this was a known tradeoff during that.
However, it has been observed that the regular connection removal function
creates considerable blocking of the processing loop, leading to high
CPU usage and API latencies.
To fix this, we add a reverse mapping of channelIDs to connections
and their positions in the slice. This helps us to remove the connection
from the slice without iteration.
Unfortunately, this still needs to iterate through all channelIDs
during invalidation of the channel member cache. However, the user
cache invalidation is not a regular activity. So it should be an acceptable
tradeoff to make.
https://mattermost.atlassian.net/browse/MM-62960
```release-note
A new config knob ServiceSettings.EnableWebHubChannelIteration which allows a user to control the performance of websocket broadcasting. By default, this setting is turned off. If it is turned on, it improves the websocket broadcasting performance at the expense of poor performance when users join/leave a channel. It is not recommended to turn it on unless you have atleast 200,000 concurrent users actively using MM.
```
Co-authored-by: Mattermost Build <build@mattermost.com>
We add a new config setting to allow the admin to set a fixed
list of userIDs to track for all client side webapp metrics.
This gives the admin to get a deeper look at how the application
is behaving for a single user.
A new section in the system console is also added for the user
to edit this setting from the UI.
https://mattermost.atlassian.net/browse/MM-61888
```release-note
A new config setting MetricsSettings.ClientSideUserIds is added
where you can set the user ids you want to track for client side webapp
metrics.
```
* fix lint errors
```release-note
NONE
```
* fixing tests
```release-note
NONE
```
* refactor: Replace pagination with cursor-based pagination for custom profile attributes
* remove pagination loop on property value retrieval for CPA
* add migrations to optimize pagination on property fields and values
* adapt test to remove pagination check
* update migrations list
* postgres: drop index concurrently
* concurrent index manipulation must be done outside of a Tx
* fix: Correct SQL index drop syntax from "OM" to "ON" in migration files
* test: Add CountForGroup test cases for property field store
* refactor: Add CountForGroup method to PropertyFieldStore interface and implementations
* Fix style and i18n
* feat: Add optional deleted property field filtering to CountForGroup method
* refactor: Update CountForGroup to support optional deleted property fields
* test: Add comprehensive tests for CountForGroup with includeDeleted parameter
* adapt test + gen layers
* rename property service method and set the includeDelete to false
* refactor: Remove redundant constant and use CustomProfileAttributesFieldLimit directly
* fix tests
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* Adds websocket messages to Custom Profile Attributes
The app layer now fires a websocket event as part of the operations
over Custom Profile Attribute fields and values. It updates as well
the Patch method for CPA values so all the changes are commited as
part of the same transaction.
To be able to do this last operation, the change adds methods to
upsert CPA values in both the store and the property service.
* Fix i18n strings
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
* Updates the property field and value update methods to use a single query for multiple entities
* Update server/channels/store/sqlstore/property_field_store.go
Co-authored-by: Julien Tant <785518+JulienTant@users.noreply.github.com>
---------
Co-authored-by: Miguel de la Cruz (aider) <miguel@ctrlz.es>
Co-authored-by: Julien Tant <785518+JulienTant@users.noreply.github.com>
We were redirecting any logging happening via the Go standard logger
to our own mlog instance. The issue is that all those logs were happening
at LvlStdLog level which is higher than LvlDebug.
LvlStdLog is at 10, and LvlDebug is at 5. So unless a customer
sets the log level specifically to "stdlog" or sets up advanced
logging to specifically log for that level, no standard logs will
ever get logged.
We fix this by logging it at warn level. The reason for warn
is that mostly external libraries use this facility to log
out-of-band errors that could not be returned using standard
error returns.
I saw another plugin error which was logged similarly. This
was never surfaced before due to this bug.
https://mattermost.atlassian.net/browse/MM-62926
```release-note
NONE
```
* Revert "Revert "[MM-62142] Avoid SELECT * in status_store.go (#29610)" (#29985)"
This reverts commit d345e92136.
* add tests for StatusGet and StatusGetByIds
* handle NULL columns in the Status Store
* simplify status store
* more builder simplifications and tests
* expose GetQueryPlaceholder
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* add SelectBuilderCtx; change the few calls where it's appropriate to use
* tests (wip)
* tests improved
* use SelectCtx's own WithTimeout
* improve tests based on PR feedback
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* refactor: Move property value sanitization to model layer
* feat: Add value sanitization for custom profile attributes
* refactor: Update custom profile attributes to use json.RawMessage
* refactor: Update patchCustomProfileAttribute to handle json.RawMessage directly
* refactor: Refactor custom profile attributes handler with improved validation
* refactor: Rename `patchCustomProfileAttribute` to `patchCPAValues`
* refactor: Replace ReturnJSON with json.NewEncoder and add error logging
* feat: Add encoding/json import to property_value.go
* refactor: Update property value tests to use json.RawMessage
* fix: Convert string value to json.RawMessage in property value test
* fix: Convert string literals to json.RawMessage in property value tests
* fix: Add missing encoding/json import in custom_profile_attributes.go
* fix: Preserve JSON RawMessage type in listCPAValues function
* fix: Update custom profile attributes test to use json.RawMessage
* feat: Add json import to custom_profile_attributes_test.go
* refactor: Update ListCPAValues and PatchCPAValues to use json.RawMessage
* refactor: Rename `actualValue` to `updatedValue` in custom profile attributes test
* refactor: Improve user permission and audit logging for custom profile attributes patch
* refactor: Optimize CPA field lookup by using ListCPAFields() and map
* fix: Correct user ID reference in custom profile attributes patch endpoint
* refactor: Change patchCPAValues to use map[string]json.RawMessage for results
* refactor: format and fix tests
* test: Add comprehensive unit tests for sanitizePropertyValue function
* test: Add test case for invalid property value type
* feat: Use `model.NewId()` to generate valid IDs in custom profile attributes tests
* refactor: Replace hardcoded IDs with dynamic variables in custom profile attributes test
* refactor: restore variable name
* refactor: drop undesired changes
* chore: refresh app layers
* feat: Update API definition to support string or string array values for custom profile attributes
* test: Add test cases for multiselect custom profile attribute values
* test: Add tests for multiselect custom profile attribute values
* test: Isolate array value test in separate t.Run
* test: Add test case for multiselect array values in custom profile attributes
* refactor: Move array value test from TestCreateCPAField to TestPatchCPAValue
* test: Update custom profile attributes test assertions
* test: add test case for handling array values in GetCPAValue
* test: Add array value tests for property value store
* refactor(store): no need to convert to json the rawmessage
* chore: lint
* i18n
* use model to interface with sqlx
* fix: Allow empty strings for text, date, and select profile attributes
* refactor: Filter out empty strings in multiselect and multiuser fields
* refactor: Update multiuser field sanitization to validate and error on invalid IDs
* refactor: Simplify sanitizePropertyValue function with reduced code duplication
* fix: Allow empty user ID in custom profile attribute sanitization
* refactor: Convert comment-based subtests to nested t.Run in TestSanitizePropertyValue
* refactor: Convert comment-based subtests to nested t.Run tests in TestSanitizePropertyValue
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* do not error on exportFile error
* add tests for local and s3 storage exporting with missing file
* linting
* fix attachment path validation in mmctl
* Prompt Team Admin before joining private channel
* fix and implement prompt when team admin joins via channel link
* update premission check and unit tests
* remove comment
* clean up some code
* update check to fix E2E tests
* Audit calls to searchPosts API
* Update server/public/model/post_search_results.go
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
* Update server/public/model/post_search_results.go
* Update server/public/model/post_search_results.go
---------
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
We add a new field delete_at in the channels template.
This field is then searched in the SearchChannels function.
Also added tests to verify that archived channels are searched
properly, and also indexed correctly.
https://mattermost.atlassian.net/browse/MM-62295
```release-note
- Now archived channels are searchable with ES/OS if TeamSettings.ExperimentalViewArchivedChannels is enabled.
- If there are old channels which were archived before a bulk index was run, users would need to purge indexes, and do bulk index again. Because those old archived channels are removed from the index when a bulk index is run.
```
Co-authored-by: Mattermost Build <build@mattermost.com>
* Step 1: Migrate any SELECT * queries written as strings into builder queries.
* Step 2: Migrate any instances of SELECT * into explicitly enumerating the needed columns
* revert `GetByName` to read from master as opposed to master.
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* Fix flakyness in channel bookmark tests
* Use existing util to initialize client
* Update server/channels/api4/apitestlib.go
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
* Remove redundant close
---------
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
* Adds Custom Profile Attributes feature license and API endpoints check
* Fix linter
* Remove the specific license feature and fallback to checking for license presence
* Add translation
* Update the checks and tests to enable endpoints only for Enterprise licenses
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* fix for deleting non-bulk export files and directories
* add job worker setup and running to helper_test.go
* add infrastructure for e2e testing of workers; test ExportDelete
* linting
* use retention days in test instead of magic number
* SELECT * to query builder
* execBuilder, manual column name and one more select * remaining instance
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
There are several steps that a server runs through inside (*Server).Start
after ch.initPlugins() till the signal handler is reached which handles
the server shutdown procedure.
The issue arises when the server is shutdown after ch.initPlugins() completes
but before (*Server).Start finishes. In that case, the plugins are all started
but they won't be shut down cleanly.
To fix this edge-case, we set up an intermediate signal handler, which
attaches itself as soon as ch.initPlugins is finished, allowing us to run
the cleanup code in case the shutdown happens before (*Server).Start finishes.
And when we do reach the main signal handler, we don't need this intermediate
handler any more. So we just reset the handlers and use the main signal handler
which takes care of shutting down the whole server.
Note: This is still not 100% bug-proof because ch.initPlugins() will initialize
_all_ plugins, and the shutdown can happen just after one plugin is initialized.
To handle that case will require the need to set up signal handlers after every
plugin init which feels like overkill to me.
A sample flow diagram to visualize better:
Edge-case
server.Start()
|
ch.initPlugins()
|
<ctrl-c>
|
execute signal handler, os.Exit(1)
Happy-path
server.Start()
|
ch.initPlugins()
|
server.Start() finished
|
reset old signal handler
|
setup main signal handler
|
server runs on as usual until shutdown
https://mattermost.atlassian.net/browse/MM-49353
```release-note
NONE
```
* migrate Token Store GetByToken query to SQLBuilder
* explicitly enumerate columns, factor out query into constructor
* fix syntax error
* fix error handling
* spread args... instead of passing the array as a single param
* fix MySQL by casing columns correctly
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com>
We do a cluster request to get the active and dead queues
from other nodes in the cluster to sync any missing
information.
We check the dead queue in the other nodes to see
if there's been any message loss or not. Accordingly,
we send just the active queue or both active and dead queues.
There's still an edge case that is left out where
a client could have potentially connected and reconnected
to multiple nodes leaving multiple active queues
in multiple nodes. We don't handle this scenario
because then potentially we need to create
a slice of sendQueueSize * number_of_nodes. And then
this can happen again, leading to an infinite increase
in sendQueueSize.
We leave this edge-case to Redis, acknowledging
a limitation in our architecture.
In this PR, when there's no message loss, we just
take the active queue from the last node it connected
to.
And if there's message loss where the client's
seqNum is within the last node's dead queue, we also
handle that.
But if there's severe message loss where the client's
seqNum falls within the dead queue of another node, then
we just send the data from that node to reconstruct the
data as much as possible. It could be possible to set
a new connection ID in this case, but this involves
more data transfer always from all nodes and recomputing
the state in the requestor node.
https://mattermost.atlassian.net/browse/MM-61904
```release-note
NONE
```
Co-authored-by: Mattermost Build <build@mattermost.com>
* MM-61865: Add mobile client content load network metrics
* added new common label
* renaming from MobileClientContentLoad to MobileClientNetworkRequests
* content_load_group => network_request_group
* refactor more NetworkRequest-* changes
* replace contentLoadGroup to networkRequestGroup
* new metrics elapsedTime
* Refactor urlCount to totalRequests
* add averageSpeed metric
* replace contentLoadGroup with networkRequestGroup
* use h.Labels vs commonLabels for network_request_group
* add agent
* add effective latency metrics
* add total parallel requests & total sequential
* mocks generated by mockery
* did a bit of cleanup and sorting
* formatting
* updated the AcceptedNetworkRequestGroups
* cleanup and sorting
Within the Users and Groups store, converge on just naming the `Users`
table instead of sometimes aliasing it and sometimes not. This makes an
informal, but unambiguous interface when sharing logic between stores,
and simplifies the next step of refactoring away from `Users.*` queries.
Relates-to: MM-62158
* Adds the main Property System Architecture components
This change adds the necessary migrations for the Property Groups,
Fields and Values tables to be created, the store layer and a Property
Service that can be used from the app layer.
* Adds Custom Profile Attributes endpoints and app layer
* implement get and patch cpa values
* run i18n-extract
* Update property field type to use user instead of person
* Update PropertyFields to allow for unique nondeleted fields and remove redundant indexes
* Update PropertyValues to allow for unique nondeleted fields and remove redundant indexes
* Use StringMap instead of the map[string]any on property fields
* Add i18n strings
* Revert "Use StringMap instead of the map[string]any on property fields"
This reverts commit e2735ab0f8.
* Cast JSON binary data to string and add todo note for StringMap use
* Add mocks to the retrylayer tests
* Cast JSON binary data to string in property value store
* Check for binary parameter instead of casting to string for JSON data
* Fix bad merge
* Check property field type is one of the allowed ones
* Avoid reusing err variable to be explicit about the returned value
* Merge Property System Migrations into one file
* Adds NOT NULL to timestamps at the DB level
* Update stores to use tableSelectQuery instead of a slice var
* Update PropertyField model translations to be more explicit and avoid repetition
* Update PropertyValue model translations to be more explicit and avoid repetition
* Use ExecBuilder instead of ToSql&Exec
* Update property field errors to add context
* Ensure PerPage is greater than zero
* Update store errors to give more context
* Use ExecBuilder in the property stores where possible
* Add an on conflict suffix to the group register to avoid race conditions
* Remove user profile API documentation changes
* Update patchCPAValues endpoint and docs to return the updated information
* Merge two similar error conditions
* Use a route function for ListCPAValues
* Remove badly used translation string
* Remove unused get in register group method
* Adds input sanitization and validation to the CPA API endpoints
* Takes login outside of one test case to make it clear it affects multiple t.Runs
* Fix wrap error and return code when property field has been deleted
* Fix receiver name
* Adds comment to move the CPA group ID to the db cache
* Set the PerPage of CPA fields to the fields limit
* Update server/channels/app/custom_profile_attributes_test.go
Co-authored-by: Alejandro García Montoro <alejandro.garciamontoro@gmail.com>
* Standardize group ID access
* Avoid polluting the state between tests
* Use specific errors for the retrieval of CPA group
---------
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
Co-authored-by: Alejandro García Montoro <alejandro.garciamontoro@gmail.com>
* Updated patch/update post API to allow file modification (#29447)
* WIP
* WIP
* Atatched new files ton post
* WIP: deleting removed files
* Deleted removed files and invalidated file metadata cache
* removed file ignore logif from update post API
* Added TestFindExclusives
* Added tests for DeleteForPostByIds
* Added app layer tests
* Added tests
* Added API level tests
* test enhancements
* Fixed a test
* Edit history include file metadata (#29505)
* Send file metadata in edit history metadata
* Added app tests
* Added store tests
* Added tests for populateEditHistoryFileMetadata{
* Added cache to avoid repetitigve DB calls for edits with only message changes
* Added API tests
* i18m fix
* removed commented code
* Improved test helper
* Show attachments in edit history RHS (#29519)
* Send file metadata in edit history metadata
* Added app tests
* Added store tests
* Added tests for populateEditHistoryFileMetadata{
* Added cache to avoid repetitigve DB calls for edits with only message changes
* Added API tests
* i18m fix
* WIUP: displa files in edit
* removed commented code
* Displayed file in edit history
* Handled file icon
* Fixed closing history component on clicking on file
* Simplified selector
* Simplified selector
* Improved test helper
* Disabled action menu on edit history file
* Added tests
* Improved selector
* Updated snapshot
* review Fixes
* restructured componnets
* Updated test
* Updated test
* Restore post api (#29643)
* Restore post version API WIP
* Undelete files WIP
* Added store tests
* Created post restore API
* Updated updatepost safeUpdate signature
* review fixex and improvements
* Fixed an app test
* Added API laer tests
* Added API tests and OpenAPI specs
* Fixed a typo
* Allow editing files when editing posts (#29709)
* WIP - basic view files when editing post
* Cleanup
* bg color
* Added text editor tests for files
* WIP
* WIP
* removed debug log
* Allowed admin to add and remove files on someone else's post
* Handled drafts and scheduled posts
* linter fixes
* Updated snapshot
* server test fix
* CI
* Added doc
* Restore post api integration (#29719)
* WIP - basic view files when editing post
* Cleanup
* bg color
* Added text editor tests for files
* WIP
* WIP
* removed debug log
* Allowed admin to add and remove files on someone else's post
* Handled drafts and scheduled posts
* linter fixes
* Updated snapshot
* server test fix
* Used new API to restore post
* handled edut limit and undo
* lint fix
* added comments
* Fixed edit post item tests
* Fixed buttons
* Aded snapshots
* fix test
* Updated snapshot
* Minor fixes
* fixed snapshot
* Edit file dnd area (#29763)
* dnd wip
* DND continued
* Supported multiple unbind dragster funcs
* lint fixes
* Got center channel file drop working when editing a post
* file dnd working with center channel and rhs
* file dnd working with center channel and rhs
* removed unneeded stopPropogation calls
* cleanup
* DND overlay fix
* Lint fix
* Advanced text editor test updates for file upload overlay
* fixed use upload hook tests
* Updated some more snapshots
* minor cleanup
* Updated i18n
* removed need of array for dragster unbind events
* lint fixes
* edit history cursor
* Fixed bugu causing faliure to delete empty posts (#29778)
* Files in restore confirmation (#29781)
* Added files to restore post confirmation dialog
* Fixed post restore toast colors
* Fixed restore bug
* Fixed restore confirmation toast tests
* a11y improvement and modal width fix
* Edit attachment misc fixes (#29808)
* Removed single image actions in restore post confirmation dialog
* Fixed file drop overlay size and position
* Made edit indiator accessible
* Lint fix
* Added bunch of more tests
* ANother test migrated from enzyme to react testing library
* More test enhancements
* More test enhancements
* More test enhancements
* lint fixes
* Fixed a test
* Added missing snapshots
* Test fixes
* Adds the main Property System Architecture components
This change adds the necessary migrations for the Property Groups,
Fields and Values tables to be created, the store layer and a Property
Service that can be used from the app layer.
* Update property field type to use user instead of person
* Update PropertyFields to allow for unique nondeleted fields and remove redundant indexes
* Update PropertyValues to allow for unique nondeleted fields and remove redundant indexes
* Use StringMap instead of the map[string]any on property fields
* Add i18n strings
* Revert "Use StringMap instead of the map[string]any on property fields"
This reverts commit e2735ab0f8.
* Cast JSON binary data to string and add todo note for StringMap use
* Add mocks to the retrylayer tests
* Cast JSON binary data to string in property value store
* Check for binary parameter instead of casting to string for JSON data
* Check property field type is one of the allowed ones
* Avoid reusing err variable to be explicit about the returned value
* Merge Property System Migrations into one file
* Adds NOT NULL to timestamps at the DB level
* Update stores to use tableSelectQuery instead of a slice var
* Update PropertyField model translations to be more explicit and avoid repetition
* Update PropertyValue model translations to be more explicit and avoid repetition
* Use ExecBuilder instead of ToSql&Exec
* Update property field errors to add context
* Ensure PerPage is greater than zero
* Update store errors to give more context
* Use ExecBuilder in the property stores where possible
* Add an on conflict suffix to the group register to avoid race conditions
* Remove badly used translation string
* Remove unused get in register group method
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* [MM-59089] Add a compliance export constant (#27919)
* add a useful constant
* i18n
* another constant
* another i18n
* [MM-60422] Add GetChannelsWithActivityDuring (#28301)
* modify GetUsersInChannelDuring to accept a slice of channelIds
* add GetChannelsWithActivityDuring
* add compliance export progress message; remove unused custom status
* linting
* tests running too fast
* add batch size config settings
* add store tests
* linting
* empty commit
* i18n changes
* fix i18n ordering
* MM-60570 - Server-side changes consolidating the export CLI with server/ent code (#28640)
* add an i18n field; add the CLI's export directory
* int64 -> int
* Add UntilUpdateAt for MessageExport and AnalyticsPostCount
to merge
* remove now-unused i18n strings
* add TranslationsPreInitFromBuffer to allow CLI to use i18n
* use GetBuilder to simplify; rename TranslationsPreInitFromFileBytes
* [MM-59089] Improve compliance export timings (#1733 - Enterprise repo)
* MM-60422 - Performance and logic fixes for Compliance Exports (#1757 - Enterprise repo)
* MM-60570 - Enterprise-side changes consolidating the export CLI with server/ent code (#1769 - Enterprise repo)
* merge conflicts; missed file from ent branch
* MM-61038 - Add an option to sqlstore.New (#28702)
remove useless comment
add test
add an option to sqlstore.New
* MM-60976: Remove RunExport command from Mattermost binary (#28805)
* remove RunExport command from mattermost binary
* remove the code it was calling
* fix i18n
* remove test (was only testing license, not functionality)
* empty commit
* fix flaky GetChannelsWithActivityDuring test
* MM-60063: Dedicated Export Filestore fix, redo of #1772 (enterprise) (#28803)
* redo filestore fix#1772 (enterprise repo) on top of MM-59966 feature
* add new e2e tests for export filestore
* golint
* ok, note to self: shadowing bad, actually (when there's a defer)
* empty commit
* MM-61137 - Message export: Support 7.8.11 era dbs (#28824)
* support 7.8.11 era dbs by wrapping the store using only what we need
* fix flaky GetChannelsWithActivityDuring test
* add a comment
* only need to define the MEFileInfoStore (the one that'll be overridden)
* blank commit
* MM-60974 - Message Export: Add performance metrics (#28836)
* support 7.8.11 era dbs by wrapping the store using only what we need
* fix flaky GetChannelsWithActivityDuring test
* add a comment
* only need to define the MEFileInfoStore (the one that'll be overridden)
* performance metrics
* cleanup unneeded named returns
* blank commit
* MM-60975 - Message export: Add startTime and endTime to export folder name (#28840)
* support 7.8.11 era dbs by wrapping the store using only what we need
* fix flaky GetChannelsWithActivityDuring test
* add a comment
* only need to define the MEFileInfoStore (the one that'll be overridden)
* performance metrics
* output startTime and endTime in export folder
* empty commit
* merge conflict
* MM-60978 - Message export: Improve xml fields; fix delete semantics (#28873)
* support 7.8.11 era dbs by wrapping the store using only what we need
* fix flaky GetChannelsWithActivityDuring test
* add a comment
* only need to define the MEFileInfoStore (the one that'll be overridden)
* performance metrics
* output startTime and endTime in export folder
* empty commit
* add xml fields, omit when empty, tests
* fix delete semantics; test (and test for update semantics)
* clarify comments
* simplify edited post detection, now there's no edge case.
* add some spacing to help fast running tests
* merge conflicts/updates needed for new deleted post semantics
* linting; fixing tests from upstream merge
* use SafeDereference
* linting
* stronger typing; better wrapped errors; better formatting
* blank commit
* goimports formatting
* fix merge mistake
* minor fixes due to changes in master
* MM-61755 - Simplifying and Support reporting to the db from the CLI (#29281)
* finally clean up JobData struct and stringMap; prep for CLI using db
* and now simplify using StringMapToJobDataWithZeroValues
* remove unused fn
* create JobDataExported; clean up errors
* MM-60176 - Message Export: Global relay cleanup (#29168)
* move global relay logic into global_relay_export
* blank commit
* blank commit
* improve errors
* MM-60693 - Refactor CSV to use same codepath as Actiance (#29191)
* move global relay logic into global_relay_export
* blank commit
* refactor (and simplify) ExportParams into shared
* blank commit
* remove unused fn
* csv now uses pre-calculated joins/leaves like actiance
* improve errors
* remove nil post check; remove ignoredPosts metric
* remove unneeded copy
* MM-61696 - Refactor GlobalRelay to use same codepath as Actiance (#29225)
* move global relay logic into global_relay_export
* blank commit
* refactor (and simplify) ExportParams into shared
* blank commit
* remove unused fn
* csv now uses pre-calculated joins/leaves like actiance
* remove newly unneeded function and its test. goodbye.
* refactor GetPostAttachments for csv + global relay to share
* refactor global_relay_export and fix tests (no changes to output)
* improve errors
* remove nil post check; remove ignoredPosts metric
* remove unneeded copy
* remove unneeded nil check
* PR comments
* MM-61715 - Generalize e2e to all export types 🤖 (#29369)
* move global relay logic into global_relay_export
* blank commit
* refactor (and simplify) ExportParams into shared
* blank commit
* remove unused fn
* csv now uses pre-calculated joins/leaves like actiance
* remove newly unneeded function and its test. goodbye.
* refactor GetPostAttachments for csv + global relay to share
* refactor global_relay_export and fix tests (no changes to output)
* improve errors
* remove nil post check; remove ignoredPosts metric
* remove unneeded copy
* remove unneeded nil check
* PR comments
* refactor isDeletedMsg for all export types
* fix start and endtime, nasty csv createAt bug; bring closer to Actiance
* align unit tests with new logic (e.g. starttime / endtime)
* refactor a TimestampConvert fn for code + tests
* bug: pass templates to global relay (hurray for e2e tests, otherwise...)
* add global relay zip to allowed list (only for tests)
* test helpers
* new templates for e2e tests
* e2e tests... phew.
* linting
* merge conflicts
* unexport PostToRow; add test helper marker
* cleanup, shortening, thanks to PR comments
* MM-61972 - Generalize export data path - Actiance (#29399)
* extract and generalize the export data generation functions
* finish moving test (bc of previous extraction)
* lift a function from common -> shared (to break an import cycle)
* actiance now takes general export data, processes it into actiance data
* bring tests in line with correct sorting rules (upadateAt, messageId)
* fixups, PR comments
* turn strings.Repeat into a more descriptive const
amended: one letter fix; bad rebase
* MM-62009 - e2e clock heisenbug (#29434)
* consolidate assertions; output debuggable diffs (keeping for future)
* refactor test output generator to generators file
* waitUntilZeroPosts + pass through until to job = fix all clock issues
* simplify messages to model.NewId(); remove unneeded waitUntilZeroPosts
* model.NewId() -> storetest.NewTestID()
* MM-61980 - Generalize export data path - CSV (#29482)
* simple refactoring
* increase sleep times for (very) rare test failures
* add extra information to the generic export for CSV
* adj Actiance to handle new generic export (no difference in its output)
* no longer need mergePosts (yay), move getJoinLeavePosts for everyone
* adjust tests for new csv semantics (detailed in summary)
* and need to add the new exported data to the export_data_tests
* rearrange csv writing to happen after data export (more logical)
* linting
* remove debug statements
* figured out what was wrong with global relay e2e test 3; solid now
* PR comments
* MM-61718 - Generalize export data path - Global Relay (#29508)
* move global relay over to using the generalized export data
* performance pass -- not much can be done
* Update server/enterprise/message_export/global_relay_export/global_relay_export.go
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
---------
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* MM-62058 - Align CSV with Actiance (#29551)
* refactoring actiance files and var names for clarity
* bug found in exported attachments (we used to miss some start/ends)
* changes needed for actiance due to new generic exports
* bringing CSV up to actiance standards
* fixing global relay b/c of new semantics (adding a note on an edge case)
* aligning e2e tests, adding comments to clarify what is expected/tested
* necessary changes; 1 more test for added functionality (ignoreDeleted)
* comment style
* MM-62059 - Align Global Relay with Actiance/CSV; many fixes (#29665)
* core logic changes to general export_data and the specific export paths
* unit tests and e2e tests, covering all new edge cases and all logic
* linting
* better var naming, const value, and cleaning up functions calls
* MM-62436 - Temporarily skip cypress tests that require download link (#29772)
---------
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
The default role permissions weren't being saved
to a separate variable which would have not restored
them properly. This was a problem in various other
tests, potentially causing side-effects in other
flaky tests as well.
We fix that, as well as clean up the for loop
a bit and mention how many messages were recieved
in the failure message so that it becomes clear
whether no messages were recieved, or atleast some
were received.
https://mattermost.atlassian.net/browse/MM-62079
```release-note
NONE
```
* MM-62378: Initialize status and session cache before loading license
ps.LoadLicense could end up calling InvalidateAllCaches. Therefore,
we need to intialize those caches before calling LoadLicense
to prevent a panic from happening.
While here, we also remove some unused code.
https://mattermost.atlassian.net/browse/MM-62378
```release-note
NONE
```
* fix lint errors
```release-note
NONE
```
* fix test
```release-note
NONE
```
* revert the api4 portion of 57372bd06b
* test: Add system admin channel access tests
* skip a.HasPermissionToTeam for channels without a team
* fix TestPostGetInfo
* Add DeleteSidebarCategory method and channel category API tests
* Add test to verify channels move to default categories when custom category is deleted
* rename channel category tests and remove duplicate context
* TestNotifications fix
---------
Co-authored-by: Fume <contact@fumedev.com>
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
This test failed because TestWebHubCloseConnOnDBFail failed
and therefore the ChannelMembers table failed to be
renamed back.
We can see that in the logs:
```
teams","request_id":"sdxjxsd1wibd9mb7kc3ixj6g1w","ip_addr":"127.0.0.1","user_id":"afw9qqjh7pnddqakraxnjf1oph","method":"POST","user_id":"afw9qqjh7pnddqakraxnjf1oph","team_id":"bocubwmn4jyrmc1p9uxrc4pwfc","error":"JoinDefaultChannels: Unable to save direct channel., channel_members_save: Error 1146 (42S02): Table 'dbrt3jzcosj3nb8g6mgwmqyss9rh.ChannelMembers' doesn't exist"}
{"timestamp":"2024-11-10 13:52:34.260 Z","level":"error","msg":"Error while invalidating channel member cache","caller":"platform/web_hub.go:524","user_id":"afw9qqjh7pnddqakraxnjf1oph","error":"failed to find ChannelMembers, TeamScheme and ChannelScheme data: Error 1146 (42S02): Table 'dbrt3jzcosj3nb8g6mgwmqyss9rh.ChannelMembers' doesn't exist"}
{"timestamp":"2024-11-10 13:52:34.260 Z","level":"error","msg":"Error while invalidating channel member cache","caller":"platform/web_hub.go:524","user_id":"afw9qqjh7pnddqakraxnjf1oph","error":"failed to find ChannelMembers, TeamScheme and ChannelScheme data: Error 1146 (42S02): Table 'dbrt3jzcosj3nb8g6mgwmqyss9rh.ChannelMembers' doesn't exist"}
```
There is nothing wrong with this test. And while
we are here, we slightly improve the for-select loop
to exit faster.
https://mattermost.atlassian.net/browse/MM-61699
```release-note
NONE
```
Finally I figured out why the log message for
/api/v4/websocket does not appear. It is because
the log gets generated only when the request returns,
and for websockets, the request doesn't return
until the client closes. And because we were closing
the client in a defer clause, the flushing of the logger
would happen before closing the client, therefore
leading to a race condition of the log not appearing
from time to time.
https://mattermost.atlassian.net/browse/MM-61698
```release-note
NONE
```
https://github.com/mattermost/mattermost/pull/29214 fixed the
race condition in web_hub initialization.
The failure in the test is strange because there's no log
for the websocket request. It's as if th.CreateWebSocketClientWithClient()
never got called at all.
I am re-arranging the lines so that even if it fails,
the table rename happens anyways. But overall, I don't see
any reason for the failure.
https://mattermost.atlassian.net/browse/MM-61698
```release-note
NONE
```
While investigating https://mattermost.atlassian.net/browse/MM-61529,
I noticed that we weren't adding the internal error which made
it impossible to understand what was really happening.
```release-note
NONE
```
I was planning to kick off a campaign for this. But aider can do this just fine.
Starting off with a single file for now. But will slowly send more PRs with larger
commit diffs.
```release-note
NONE
```
Co-authored-by: Mattermost Build <build@mattermost.com>
Drop the legacy `X` suffix from `GetMasterX` and `GetReplicaX`. The
presence of the suffix suggests there's a `non-X` version: but in fact
we migrated these away a long time ago, so remove the cognitive
overhead.
As an aside, this additionally helps avoid trip up LLMs that interpret
this as "something to fix".
We were incorrectly not broadcasting status cache updates
inspite of that being an LRU cache.
We were also not doing it for profilesInChannel cache.
Now we fix it by properly checking the invalidationEvent
which is something local to the cache itself rather
than the cache provider.
https://mattermost.atlassian.net/browse/MM-62077
```release-note
NONE
```
This introduces a new entry in the `Main Menu -> About` modal with the hostname of the currently connected websocket. This will be used to aid debugging issues in clustered environments by showing which node in the cluster is servicing requests for a particular websocket.
This information is only visible in self-managed instances. It will not be visible on cloud instances.
* [MM-61477] Fixed exif rotates images width and height is not correctly calculated
* [MM-61477] Apply rotation ony to png or jepg
* [MM-61477] Added test cases to verify correct rotation
* fix: fixed wrong format
* uff
* fix: exif rotation is only possible on jpeg images
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
We create a custom histogram metric that logs the userID
when the observed value is greater or equal to the last bucket value.
This allows us to start tracking the slowest users of a system
while at the same time not polluting the Prometheus metrics
by storing a userID for every observation.
https://mattermost.atlassian.net/browse/MM-61887
```release-note
NONE
```
Page load is one of the metrics that we track and present
to MLT. However, in its current form, it is not very
actionable because it also contains the network latency.
We split the whole metric into these parts:
startTime
|
responseStart = TTFB
|
responseEnd = TTLB
|
domInteractive = Start of processing phase
|
loadEventEnd = Load complete
This gives us better visibility into exactly
which phase in the load process is slow.
I have experimented with other metrics like
- domContentLoadedEventStart
- domContentLoadedEventEnd
- domComplete
and observed that they do not have sufficient
gaps in the timespan to have any relevance.
Additionally, I have moved TTFB from being a
web vitals metric to being tracked from the performance
metrics to remain consistent with the other navigation
metrics measured.
Lastly, I took this chance to improve some of the
validation errors that we threw to include more
context into the input that was passed and why
does it fail.
This also meant that I had to change the tests
to check for error strings rather than direct
errors which is a bad thing, but I don't think
it's worth the effort trying to have named error
variables for all of them.
https://mattermost.atlassian.net/browse/MM-61886
```release-note
NONE
```
* poc - wip
* add search files across teams
* eslint
* fix existing tests
* fix webapp style
* fix test
* add api doc
* change initial state in test
* add tests on API
* add tests on file info layer
* fix file search tags
* add rhs reducer test
* reset team selected when the RHS is suppressed
* change css to reflect UI
* fix style
* fix doc wording
* make getSearchTeam return currentTeamId when value is not set
* await is unnecessary
* revert boolean check and add test
* add comment to getSearchTeam to let dev knows it defaults to currentTeam
* remove redundant team check
* simplfy test
* fix style check
---------
Co-authored-by: Caleb Roseland <caleb@calebroseland.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
* add webhook id to logs
* Move hook_id context to api level from app level.
* pass errCtx to other references of same app-error-id
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* ensure user has invite team permission in order to change setting
* add tests and handle UI
* lint fixes
* revert changes to invite section input
* update tests
* revert bad merge
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* Removed test_update_user_auth_plugin from golangci
The test_update_user_auth_plugin has been removed from the .golangci.yml configuration file. This change simplifies the linting process by excluding unnecessary files.
* Refactored error handling in user auth tests
The error handling for the 'expectUserAuth' function within the user authentication plugin tests has been refactored. Previously, errors were not being explicitly checked after each call to this function. Now, an error check is performed after each invocation and if an error is found, it's immediately returned. This change improves the robustness of our test suite by ensuring that potential issues are caught and handled appropriately during testing.
* Refactor error handling in UpdateUserAuth
The error handling in the UpdateUserAuth function has been refactored. The variable 'err' was replaced with 'appErr' to better reflect its purpose and improve code readability.
* Updated user authentication in plugin API
The user authentication method in the plugin API has been updated. Previously, it was fetching the user based on a static configuration value. Now, it fetches the user dynamically using the provided userID. This change makes the function more flexible and adaptable to different use cases.
* Updated golangci configuration
Removed a test file from the exclusion list in the golangci.yml configuration. This will allow linting checks to be performed on this previously excluded file, improving code quality and consistency.
* Updated golangci configuration
Removed a test file from the exclusion list in the golangci configuration. This will ensure that our linting tools also cover this previously excluded test file, improving overall code quality checks.
---------
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
We don't clear the Redis cache for every test.
This can cause issues because the cache is shared
across test. We fix that with this PR.
```release-note
NONE
```
There were couple of errors in the test:
1. UpdatedScheduledPost will automatically
set the ProcessedAt to now internally inside
toUpdateMap. So setting the value from outside
has no effect.
2. The bug was that if it took more than a
milisecond to capture the time, and then
do the internal call, then the Get call
will have a higher value and therefore fail.
Since UpdatedScheduledPost doesn't return
an updated post, so there is no need
to compare the timestamps at all.
```release-note
NONE
```
* Stop recording unknown platforms as "Windows" in Sessions table
* Add more tests to user_agent_test.go
* MM-61012 Add special handling to OS detection for mobile apps
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
Essentially applying the fix for https://github.com/mattermost/mattermost/pull/29214
to the web package. It seems like the app.StoreOverrideWithCache
function was never used in this package.
Just a small highlight of how much boilerplate is repeated
in multiple packages
```release-note
NONE
```
Co-authored-by: Mattermost Build <build@mattermost.com>
And re-order the server initialization (AGAIN).
For the nth time, we found several bugs in the initialization
process.
1. filestore.NewExportFileBackend and filestore.NewFileBackend
depended on license, but the license wasn't even loaded until
later!
2. The `ps.sqlStore.UpdateLicense` call also didn't work
because the license wouldn't get loaded. It only accidentally
worked because of `ps.AddLicenseListener` which would update
the license later on. We remove that.
Ideally, we would have loaded the license first and then
checked for redis client, but it's very difficult to do that.
Reasons are explained in the code comment.
So we just wait until the license is loaded, and simply throw an
error later.
https://mattermost.atlassian.net/browse/MM-61229
```release-note
NONE
```
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* MM-61731-enhance-failed-sent-system-bot-message
* adjust test and adjust i18n messages
* set back correct value
* improve naming and get channels logic
* hide channel name if the channel is private
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
Since we have the auto-retry mechanism for replicas,
there is not much benefit in trying 5 times before deciding
to move on.
In the earlier model, we would fail the server startup,
so it made sense to check as many times as possible.
Also reducing the sleep interval so that we can improve
the boot up time in case of a bad replica.
```release-note
NONE
```
* fix: Add job name to the publish report step
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
Co-authored-by: Antonis Stamatiou <stamatiou.antonis@gmail.com>
* MM-60506 - notify user for failed scheduled messages
* add unit tests for handleFailedScheduledMessages and send system-bot message
* fix vet issues
* adjust test for vet report
* make sure to send the message to every user there was a failed message
---------
Co-authored-by: Harshil Sharma <harshil.sharma@mattermost.com>
* Added test for user beloonging to channel but not team
* Added test for read onmly channel
* Added test for fetching scheduled posts for team you don;'t belong to
* Added more tests
* test enhancements
* CI
The Store variable was written to _after_ the server
started which was causing the race.
We simply move it to before we start the server.
While we are here, we fix yet another race condition
which was unrelated, but doing it in one sweep. This
was related to the user props access. When a user is updated,
there was 3 ws events that get sent out, 2 were deep-copied
whereas 1 was not. This led to race condition in postgres
binary-param mode where we were trying to set the user props.
https://mattermost.atlassian.net/browse/MM-61700
* Add test notification tool
* Add frontend styles
* Remove option from admin view
* Refactor create post and add translations
* Fix several CI errors
* Fix API and frontend snapshots
* Refactor trailing and leading icon on buttons
* Add different button states
* i18n-extract
* Fix wrong text
* Add tests
* Fix wrong string
* Fix test
* feat: E2E send test notifications (#28371)
* Refactor send desktop notification
* Address rest of the feedback
* Fix tests
* Add correct link
* Fix test
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
Co-authored-by: yasserfaraazkhan <attitude3cena.yf@gmail.com>
Tests at very high scale indicates that the iteration
of all connections during websocket broadcast starts
to become a bottleneck.
To optimize this, we move the channelMember cache from
inside web_conn.go to the hubConnectionIndex.
This involves adding a new map keyed by the channelID
and containing all webConns where the user is a member
of that channel. Subsequently, a new method needed to
be added to invalidate the cache which previously
used to happen in web_conn.
And as a last step, we remove the cache from web_conn
to reduce SQL queries to the DB.
https://mattermost.atlassian.net/browse/MM-61130
```release-note
NONE
```
* shared channel event tracking
* Update notification.go
Co-authored-by: Miguel de la Cruz <miguel@mcrx.me>
---------
Co-authored-by: Miguel de la Cruz <miguel@mcrx.me>
* Create scheduled post api (#27920)
* Added migration files for Postgres
* Added migrations for MySQL
* Added store method
* Added API and store tests
* Renamed migration after syncing with master
* Added app layer tests
* API is ready
* API is ready
* API is ready
* Renamed migration after syncing with master
* Updated migration list
* Fixed retry layer tests
* Allowed posts with empty messages
* Review fixes
* Reverted an incorrect change
* Renamed migration and fixed ID assignment
* CI
* Send post button changes (#28019)
* added Split button
* WIP
* Added core menu options
* WIP
* WIP
* WIP
* Handled displaying error in creating scheduled post
* lint fixes
* webapp i18n fix
* Review fixes
* Fixed a webapp test
* A few more fixes
* Removed a duplicate comment
* Scheduled post job (#28088)
* Added the job function
* Added query for fetching scheduled posts for pricessing
* WIP
* WIP
* WIP
* WIP
* WIP
* WIP
* Reafactoring of scheduled post job
* Lint fixes
* Updated i18n files
* FInishing touches
* Added tests for GetScheduledPosts
* Added tests for PermanentlyDeleteScheduledPosts
* Updated all layer
* Some changes as discussed with team
* Added tests for UpdatedScheduledPost
* Code review refactoring
* Added job test
* MM-60120 - Custom time selection (#28120)
* Added a common date time picker modal and used it for post reminder
* Added a common date time picker modal and used it for post reminderggp
* Added modal for custom schedule time and fixed TZ issue
* WIP
* Removed event from useSubmit hook
* Removed event from useSubmit hook
* Added timezone handling
* fixed type error
* Updated i18n strings
* Minor cleanup
* updated snapshots
* review fixes
* Handled event
* Supported for having a DM thread open in RHS while in a regular channel
* Review fixes
* MM-60136 - Scheduled messages tab (#28133)
* WIP
* WIP
* Created Tabs and Tab wrapper with added styling
* Added API to get scheduled posts
* WIP
* Displated scheduled post count
* i18n fix
* Added tests
* Handled asetting active tab absed on URL:
* Reverted unintended change
* Added API to client ad OpenAPI specs
* Renamed file
* Adding fileinfo to schedule posts
* Partial review fixes
* Made get scheduled post API return posts by teamID
* review fixes
* Moved scheduled post redux code to MM-redux package
* Usedd selector factory
* WIP:
* WIP:
* Lint fix
* Fixed an incorrect openapi spec file
* Removed redundent permission check
* Clreaed scheduled post data on logout
* Removed unused i18n string:
* lint fix
* Render scheduled posts (#28208)
* WIP
* WIP
* Created Tabs and Tab wrapper with added styling
* Added API to get scheduled posts
* WIP
* Displated scheduled post count
* i18n fix
* Added tests
* Handled asetting active tab absed on URL:
* Reverted unintended change
* Added API to client ad OpenAPI specs
* Renamed file
* Created common component for draft list item
* WIP
* WIP
* Adding fileinfo to schedule posts
* Basic rendering
* Added count badge to tabs
* WIP
* Made the Drafts LHS iteam appear if no drafts exist but scheduled posts do
* Fixed icon size
* Partial review fixes
* Made get scheduled post API return posts by teamID
* Handled initial vs team switch load
* Displayed scheduled date in panel header
* Added error message and error indiocator
* WIP
* review fixes
* WIP Adding error reason tag
* Added error codes
* Moved scheduled post redux code to MM-redux package
* Usedd selector factory
* WIP:
* WIP:
* Lint fix
* Fixed an incorrect openapi spec file
* Removed redundent permission check
* Clreaed scheduled post data on logout
* Removed unused i18n string:
* lint fix
* Opened rescheduling modal
* Updated graphic for empty state of schduled post list
* Added delete scheduled post option and modal
* Badge and timezone fix
* WIP:
* Added send now confirmation modal
* lint
* Webapp i18n fix
* Fixed webapp test
* Fixed a bug where DM/GM scheduled posts weren't immideatly showing up in UI
* Minor fixes
* WIP
* Review fixes
* Review fixes
* Optimisations
* Fixed reducer name
* Moment optimizatin
* Updated route check
* MM-60144 - added API to update a scheduled post (#28248)
* WIP
* Added api and ap layer for update scheduled post ̛̦̄
* Added API to OpenAI specs, Go client and TS client
* removed permissio check
* Added tests
* Fixed tests
* Added PreUpdate method on scheduled post model
* MM-60131 - Reschedule post integration (#28281)
* Handled rescheduling post in webapp
* Added error handling
* MM-60146 - Delete scheduled post api (#28265)
* WIP
* Added api and ap layer for update scheduled post ̛̦̄
* Added API to OpenAI specs, Go client and TS client
* removed permissio check
* Added tests
* Fixed tests
* Added PreUpdate method on scheduled post model
* Added delete scheduled post API
* Added API to Go client and OpenAPI specs
* Added API to TS client
* Added tests
* CI
* Rmeoved two incorrect code comments
* MM-60653 - Integrated delete scheduled post API (#28296)
* Integrated delete scheduled apost API
* Lint fix
* Review fixes
* Excluded draft checks from scheduled posts (#28370)
* Excluded draft checks from scheduled posts
* Added a removed todo
* MM-60125 - Scheduled post channel indicator (#28320)
* Integrated delete scheduled apost API
* Lint fix
* Added state for storing scheduled posts by channel ID
* Refactored redux store to store scheudled posts by ID, thens tore IDs everywhere
* Refactored redux store to store scheudled posts by ID, thens tore IDs everywhere
* WIP
* Added scheduled post indiocator
* Handled single and multiple scheudled posts
* Review fixes
* Fixed styling and handled center channel, RHS and threads view
* Lint fix
* i18n fix
* Fixed a cycling dependency
* Lint fix
* Added some more comments
* Updated styling
* Review fixes
* Added common component for remote user time and scheduled post indicator
* Updated scheduled post count
* Minor change
* Moved CSS code around
* Fixed a bug where files in scheduled post didn't show up until refresh (#28359)
---------
Co-authored-by: Daniel Espino García <larkox@gmail.com>
* Scheduled post config (#28485)
* Added config
* Added config on server and webapp side
* Added config check in server and webapp
* Added license check
* Added license check
* Added placeholder help text
* Added license check to job
* Fixed job test
* Review fixes
* Updated English text
* Review fixes
* MM-60118 - Added index on ScheduledPosts table (#28579)
* Added index
* Updated indexes
* Scheduled posts misc fixes (#28625)
* Added detailed logging for scheduled post job
* Limited scheduled posts processing to 24 hours
* Marked old scheduled posts as unable to send
* Added t5ests
* converted some logs to trace level
* Fixed a bug causing error message to show up on deleting a scheduled post in a deleted thread (#28630)
* Fixed scheduled posts link in RHS (#28659)
* Fixed scheduled posts link in RHS
* Review fixes
* Fix permission name in scheduled posts by team (#28580)
* Fix permission name
* fix wording
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* FIxed width of generic modal header to fix browser channel modal (#28639)
* Only consider error-free scheduled posts for indicator in channel and RHS (#28683)
* Show only errro free scheudled posts in post box indicator
* Fixed a bug to handle no scheduled posts
* Fixed draft and scheudled post UI in mobile view (#28680)
* MM-60873 and MM-60872 - Fixed a bug with updating scheduled posts (#28656)
* Fixed a bug with updating scheduled posts
* Better selectors
* MOved shceuled post message length validation to app layer
* MM-60732 - Scheduled posts channel link now takes you to the first scheduled post in channel/thread in list (#28768)
* Ordered scheudle dposts by schgeudled at nad create at
* Ordered in client
* Added scroll to target
* Removed classname prop
* Fixed tests
* Added doc
* Import fix
* MM-60961 - Fixed a bug where API used incoming create at date for scheduled post (#28703)
* Fixed a bug where API used incoming create at date for scheduled post
* Stopped sending created at value for scheduled post
* MM-60785 - Fixed a bug where scheduled posts of channel we are no longer member of didn't show up (#28637)
* Fixed a bug where scheduled posts of channel we are no longer member of didn't show up
* Added a comment
* CI
* Used data loader to optimise laoding missing channels
* Minor refactoring
* MM-60963 - Added common checks for post and scheduled posts (#28713)
* Added commen checks for post and scheuled posts
* Sanitised scheduled posts
* Fixed tests
* Splitted post checks into app and context functions
* Added checks on scheduiled posts job as well:
* i18n fix
* Fixed a test
* Renamed a func
* removed duplicate check
* Scheduled posts UI fixes (#28828)
* Fixed send button and time picker borders
* Fixed center alignment of time picker
* Removed on for today and tomorrow
* Lint fix
* Date time modal hover state fix
* Badge fix
* Fixed a mnerge issue
* Scheduled Post send now and add schedule on draft (#28851)
* Added send now option on scheduled posts
* Minor refactoring
* WIP
* WIP
* WIP
* Lint fix
* i18n fix
* Snapshot update
* Review fixes
* Scheduled post inline editing (#28893)
* Added send now option on scheduled posts
* Minor refactoring
* WIP
* WIP
* WIP
* Lint fix
* i18n fix
* Snapshot update
* Displayed editing component in scheduled post
* Added handling for updating scheduled post
* Handle events
* Fixed escape key issue in scheudled post editing
* Fixes
* Displayed error message for editing error
* Don't show mention warning
* Handled dev mode (#28918)
* MInor fixes
* client fix
* Fixes
* CI
* Removed dev mode behaviour temperorily (#29008)
---------
Co-authored-by: Daniel Espino García <larkox@gmail.com>
Co-authored-by: Eva Sarafianou <eva.sarafianou@gmail.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
This test actually tested that there is no race
_if_ session pooling is there. But there is no session pooling
any more so this test is irrelevant.
https://mattermost.atlassian.net/browse/MM-61225
```release-note
NONE
```
This originated from https://github.com/mattermost/mattermost/issues/15249.
However, the original idea was discarded https://github.com/mattermost/mattermost/issues/15249#issuecomment-709713065
as being too complicated to implement. Then I had another
idea to implement it just for session objects.
My thinking was that since every single request allocates a new
session struct, it would be good to use a sync.Pool for that.
However, 4 years later, now we know that the primary bottleneck
in app performance comes from websocket event marshalling.
Therefore, while it would be good to do this, it is difficult
to do it correctly (as shown by the numerous racy tests).
Hence, reverting this.
```release-note
NONE
```
* add tests
* pass userID to the function instead of the user object.
* remove concurrent login simulation
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* Forbid users from creating DMs and GMs with remote users
This change prevents the application from creating new DMs and GMs
when remote users are involved, and prevents as well new posts from
being created into preexisting DMs and GMs with remote users created
in previous versions.
* Adds i18n
* Skip preexisting DM tests and create a new one for the new behavior
* Excludes remote channels from channel search
This is done through a new body parameter in the SearchAllChannels
endpoint that allows to search for local only channels, which are
either channels that are shared but marked as homed locally, or
channels that are not shared at all.
* fix lint
* Fix tests
---------
Co-authored-by: Caleb Roseland <caleb@calebroseland.com>
Support for RemoteClusters without a default team id is in place for
old servers that created those connections before v10.1. This change
forbids the creation of new RemoteClusters without providing this
field, and will be removed when manual invites are implemented.
Co-authored-by: Caleb Roseland <caleb@calebroseland.com>
* additional validation for channelbookmark
* add fixes for webapp
* only set permissions correct for type
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
We remove the remaining special casing for **model.User
and add unit tests to lock in the behavior.
Additional load tests were done locally to confirm
there are no hidden code paths left out.
https://mattermost.atlassian.net/browse/MM-59947
```release-note
NONE
```
- Improve some code to use ToSQL
- Fix a bug in SqlReactionStore.GetForPostSince
where OrderBy wasn't added properly. Added a test.
- Improve a method to return a slice rather than
pass a pointer to slice.
```release-note
NONE
```
* MM-57516 - restrict activation/deactivation over ldap users
* Add unit tests
* refactor test to unify repeated actions
* add disable actions in user details too
* migrate test to use react-testing-library
* add new ldap user test and fix other existing tests
* restrict ldap users status management via api
* use correct server status and update tests
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
After creating a remote cluster, we were using its token field to
generate the remote invite. As we're going to return that remote
cluster in the response, we sanitize the entity, emptying the token
field before generating the invite, and causing it not to contain the
token itself.
This changes store the token in an independent variable to avoid
mutating it as part of the sanitization.
Co-authored-by: Mattermost Build <build@mattermost.com>
* Fixed errcheck issues in server/channels/api4/ip_filtering.go
* Added separate go routine to SendIPFiltersChangedEmail and removed redundant return
* Added import for mlog
* Changed from returning error to logging it
---------
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
* added unit tests for UpdateUserCustomStatus and RemoveUserCustomStatus functions
* stylistic improvements in status endpoint tests
* linting fix
---------
Co-authored-by: Fume <contact@fumedev.com>
* MM-60222_apply filter to export csv
* get the report exporting with the filters ready
* add unit tests
* cover one more file with some tests
* style the confirm modal note
* add translations
* remove unnecessary print line
* disable export button if there is no data to export
* fix linter issues
* fix linter errors
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* Split up handling of permalinks and other links in getLinkMetadata
* MM-60351 Use oEmbed for YouTube links
* Explicitly request json from the oEmbed provider
* Fix linter
* Fix type of CacheAge field
* Address feedback
Setting the flag to false never worked, and it was never
caught because there wasn't an instance when this method
was called with allowFromCache=false.
https://mattermost.atlassian.net/browse/MM-60606
```release-note
NONE
```
We sprinkle a bit of generic magic to refactor a lot
of duplicate code.
To avoid exposing unnecessary code, I duplicated the function
twice. But let me know if you have strong opinions about this.
https://mattermost.atlassian.net/browse/MM-60171
```release-note
NONE
```
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* Add descriptions to test cases for getNeededAtMentionedUsernames
* MM-59854 Load users who are at-mentioned in message attachment fields
* MM-59854 Add server support for at-mentioning users in message attachment fields
* Migrate support/external_commands from JS to TS
* MM-59584 Ensure that at-mentions never show in other message attachment fields
* MM-59584 Add E2E tests for how we load users based on at mentions
* Use new E2E test helpers in other places
* Update snapshots
* Add metrics for mobile versions snapshots
* Add notifications disabled and fix lint
* Address feedback
* Verify all references to JobTypeActiveUsers
* Fix typos
* Improve platform values
* Add test and MySQL support
On user login, we were aggressively clearing the channel member
cache when it wasn't necessary. There is no channel membership
info that is changing on a user login.
Additionally, we fix some more issues with pointer passing
to the cache. It's a known problem that our value passing
style isn't consistent when we fetch items from cache.
Sometimes we pass pointer-to-pointer, sometimes it's just a
pointer. This is also the reason for https://github.com/mattermost/mattermost/pull/27830.
We fix 2 such cases where we passed pointer-to-pointer
but didn't handle the special case inside the cache.
This time, we actually fix it in the app layer instead
of the cache layer and also add a test for good measure.
https://mattermost.atlassian.net/browse/MM-60480
```release-note
NONE
```
Co-authored-by: Mattermost Build <build@mattermost.com>
* ignore bot accounts when counting deactivated users
* moved query to squirrel
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
* set is null right
* Run a different query depending on driver due to performance reasons.
---------
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
While using Redis, there is no need to invalidate
and then update the new count from the DB when
we know the exact number it is going to be incremented
or decremented by.
In that case, we can directly use Redis primitives
to update the cache and prevent yet another DB query.
To achieve this, we modify the LRU cache get/set
paths slightly to not marshal into byte slices
for *int64 values. This is needed for Redis to operate
the INCR/DECR commands.
https://mattermost.atlassian.net/browse/MM-59933
```release-note
NONE
```
Co-authored-by: Mattermost Build <build@mattermost.com>
- Update library version.
- Added MaxFlush delay to help reduce CPU usage.
- Fall back to LRU cache for the caches which use SCAN.
- Added mattermost-redis and running for all api layer
tests in Postgres.
https://mattermost.atlassian.net/browse/MM-59934
```release-note
NONE
```
We were using a mock cache provider which mixed up LRU caches
with each other. This led to incorrect unmarshalling method calls.
We fix this by using the real cache provider.
https://mattermost.atlassian.net/browse/MM-60413
```release-note
NONE
```
* app/config.go: add a nil check for a required field
* Apply suggestions from code review
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
---------
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
* Changes the event to send when a channel is shared or unshared
Before we were sending the CHANNEL_CONVERTED websockets event, which
is intended to notify of a public channel being converted into a
private one, so the interface was showing the channel as private until
the client was refreshed.
Now a full channel update event is sent when the channel is shared or
unshared, so the interface gets the new information correctly.
* Fix message type on command tests
* Allows invites to be sent to offline remotes
Invites sent to remotes marked as offline will be stored as pending,
and when the remote comes back online, it will process the invites as
part of the synchronization process.
* Update condition name for excluding confirmed invites
* Adds logical deletes to shared channel remotes and remote clusters
Instead of physically deleting the shared channel remote and remote
clusters records when a channel is unshared, a remote uninvited or a
remote cluster is deleted, now those have a logical `DeleteAt` field
that is set.
This allows us to safely restore shared channels between two remote
clusters (as of now resetting the cursor without backfilling their
contents) and to know which connections were established in the past
and now are severed.
* Delete the index in remoteclusters before adding the new column
* Fix bad error check
* Ignore at and tilde for username and channel searches.
* allow using tilde in search suggestions
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* Ignore performance counts if notifications are blocked by the device
* Change the endpoint to allow more information
* Add tests and API description
* Remove wrong test
* Address feedback
* Only update the cache when there is no error
* Follow same casing as other props
* use one single endpoint
* Fix tests
* Fix i18n
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
PR: https://github.com/mattermost/mattermost/pull/27204 fixed the issue
where `invalidateWebConnSessionCacheForUser` was not broadcasted across
the cluster. However, we became too conservative and started to invalidate
the session cache as well.
But the session cache only contains the session details for the user
and the team membership. There is no channel membership related info.
Therefore, we avoid that and improve performance as well.
https://mattermost.atlassian.net/browse/MM-60478
```release-note
NONE
```
Co-authored-by: Mattermost Build <build@mattermost.com>
* Adds status sync to shared channels
To allow for status to be synced, this changes add a new type of
shared channel internal task. This task has its contents pre-fetched
and stored in the `existingMsg` property, and it is keyed with a user
ID besides a channel ID, so it doesn't conflict with channel-driven
synchronization tasks.
All status synchronizations are triggered from the app layer, so there
is no need of watching for new WebSocket events. Although right now
we're only syncing one user status per message, the changes account
for a list of statuses in case we want to batch them in the future.
The feature is gated by a configuration property and can be disabled
independently of the rest of Shared Channels if it's necessary. It is
backwards compatible as well, and should cause no problems with
servers running older Mattermost versions.
* Adds status sync error management and retry
* Adds DisableSharedChannelsStatusSync to the telemetry report
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* Adds ServerFederationSettings configuration block
The Server Federation related configuration properties were located in
the `ExperimentalSettings` config block. Now that the feature is going
to get out of beta, and foreseeing more configuration properties being
added to it, we're moving them to a block of its own.
If the properties were set in the old location, their values should be
carried over to the new ones. The old properties are left in place and
marked as deprecated not to cause issues with the model.
* Update System Console to include Server Federation section
* Remove system console section
* Move the configuration properties to ConnectedWorkspaces
* Add ConnectedWorkspacesSettings to the telemetry
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* Adds Shared Channel management API endpoints
New endpoints for the following routes are added:
- Get Shared Channel Remotes by Remote Cluster at `GET
/api/v4/remotecluster/{remote_id}/sharedchannelremotes`
- Invite Remote Cluster to Channel at `POST
/api/v4/remotecluster/{remote_id}/channels/invite`
- Uninvite Remote Cluster to Channel at `POST
/api/v4/remotecluster/{remote_id}/channels/uninvite`
These endpoints are planned to be used from the system console, and
gated through the `manage_secure_connections` permission.
* Adds i18n messages for API errors
* Fix pagination flaky test
* Fix linter
* Adds the posibility of filtering shared channel remotes by home
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* removed an incorretc return that caused missing API timing metrics for 4xx and 5xx errors
* Added tests and recorded metric for URL length limit as well
* Status code fix
We were invalidating the userProfileByIds cache unnecessarily
when it wasn't really needed. It's just that the InvalidateCacheForUser
method wiped off all user related caches, when we just needed to wipe
user-channel related caches.
To fix this, we split the user cache invalidation method into two
and just call the InvalidateChannelCacheForUser method from app/channel.go.
https://mattermost.atlassian.net/browse/MM-59827
```release-note
NONE
```
- We introduce 2 new APIs:
1. Scan: this allows incremental iteration
without blocking the Redis server and is the
recommended way to iterate over keys. With this,
we have entirely removed the need for Keys.
2. RemoveMulti: this allows deletion of multiple
keys in a single operation which optimizes
network round trips.
- While here, we make a small improvement to
GetStatusFromCache, where we remove the shallow
copy which wasn't necessary because we always
serialize the data from the cache.
- We do not use Redis for session cache because of
frequent requests to iterate the entire cache which leads
to a lot of `SCAN` calls.
- Avoid broadcasting status update messages for Redis case.
- Setting cache expiry for status cache
- Removing .Set method altogether to prevent
any chances of setting an item with no expiry.
https://mattermost.atlassian.net/browse/MM-59932
```release-note
NONE
```
* Adds default team to the remote cluster entity
A new DefaultTeamId field is added to the RemoteCluster entity and its
endpoints, and used when receiving channel invites to choose in which
team to create a new channel.
This will be later extended with the ability for the system admin to
manually accept invites, choosing which team to create the channel on
each. This use case will be triggered when the DefaultTeamId field is
empty, which now simply chooses the first team it finds in the
database as a fallback.
* Fix migrations list
* Fixes channelinvite test case
* Fix i18n
* Fix migration list
* Allows creating new remote clusters without providing a password
If the endpoint receives a request with no password, it will generate
one internally and return it in the response, so the frotend can show
it to the user.
* Use a random string instead of a UUID for the generated password
* Update function name to avoid CString reference and adds assertion
* Update server/channels/utils/textgeneration.go
Co-authored-by: Eva Sarafianou <eva.sarafianou@gmail.com>
* Extends the charset
---------
Co-authored-by: Eva Sarafianou <eva.sarafianou@gmail.com>
* [MM-59911] Force individual users to receive the `user_added` and `group_added` events instead of relying on the channel membership
* Fix redundant JSON serializing
* [MM-58492] Remove subsequent check for team permissions on open channels
* Removed extra check for open channel for webhooks as well
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* Cleanup post embed for WebSocket payload
* Removing post embeds in create post API
* Fixed a nil check:
* Removed a now-irrelavent test
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* Refactored channel saving logic
The code for saving channels has been refactored handling different database drivers. The query string is now dynamically generated based on the driver in use. Additionally, error handling has been improved to account for cases where no rows are affected by the insert operation. Because a conflict should be detected due to the complex flow under the call of the function.
* Refactor variable name and error handling in channel store
The variable 'q' has been renamed to 'insert' for better readability. Error handling after executing the insert statement has been improved by checking for errors immediately after execution, rather than waiting until rows affected are checked. This provides a more immediate response to potential issues during the insert operation.
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
Overall, there is no bug here. But opportunities for
improvement.
- Used the maps.Clone function.
- Handled the error while writing to the network.
https://mattermost.atlassian.net/browse/MM-59283
```release-note
NONE
```
* MM-59875: default apps enabled false
We stopped prepackaging the apps framework and stopped supporting it in
Mattermost Cloud a year ago. As part of v10, we're formally deprecating
it, while making allowances for customers who still need to transition
by not making this a breaking change immediately. To help prevent new
adoption, let's ensure the feature flag is disabled by default.
Fixes: https://mattermost.atlassian.net/browse/MM-59875
* fix tests
* [MM-58492][MM-58523] Fixed some access control bugs around archived channels by replacing the permission check with HasPermissionToReadChannel
* Fix lint, add ChannelId to uploads
* Fix MMCTL tests and remove unnecessary check for the error message that doesn't work anyways
* Include channel map for getting flagged posts
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* Fixed the issue for DB layer, ES to go
* Handled channel bookmarks
* Handled Bleve
* Lint fix
* Added channel bookmark test
* Skip bleve test
* Used common function
* SKipping ES as indexing logic in unavailable in test
* make /ancillary a post
* remove get from client, fix tests
* remove GET for retrieving ancillary permissions
* Update permissions.yaml
* Update permissions.yaml
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* Added a bunch of debugging logs for Elasticsearch index check
* Update server/channels/app/elasticsearch.go
---------
Co-authored-by: Harrison Healey <harrisonmhealey@gmail.com>
* add more fields to sanitizeInput on User
* add test for user sanoitizeInput
* add more fields
* remove line, lint fix
* additional fields and sanitize update
* Update user_test.go
* remove fields that are unnecessary to check
* add check to test
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* Updated required role for user report page GET API
* Updated permission in error message
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* Opened modal from system console
* WIP
* WIP
* WIP
* Handled saving user
* Successfully updated user based settings
* WIP
* WIP
* All settings are updating well
* Fixed modal style
* Added admin mode indicators in modal
* Added confirmation dialog
* Lint fixes
* Added license check
* Added permission check
* Fixed i18n file order
* type fix
* Updated snapshots
* Handled performance debugging setting
* Some styling tweaks
* Fixed text alighnment
* Updated license required from professional to enterprise
* Handled long user names
* review fixes
* Added manage setting option in user list page context menu
* Added loader
* Minor reordering
* Removed confirm modal
* Updated snapshots for removed modal
* Added some tests
* Lint fix
* Used new selector in user detail page
* Used new selector in user list page
* Updated tests
* Fixed an incorrect default test
* Improve mocking of imported resources in unit tests
We have Webpack configured so that, when code imports an image or other resource, the code gets the URL of that image. Jest now matches that behaviour which is needed because React Testing Library would previously throw an error.
* Polyfill ResizeObserver in all unit tests
* Ensure haveIChannelPermission always returns a boolean value
The previous code could sometimes return undefined. While that should behave the same in practice, it can cause React to print prop type warnings
* MM-58535 Add region label to LCP metrics
* MM-58535 Upgrade web-vitals and add INP attribution
* Change new labels to use snake_case
* Remove replaceGlobalStore option from renderWithContext
I was going to add this in case any tests failed with this option set to false, but after running those tests, that's not the case. I'm going to remove this as an option since it seems more likely than not that anyone using RTL would prefer to have this on.
* remove manage team permissions from sysconsole_write_user_management_chanels and sysconsole_write_user_management_groups
* update migrations, add unit tests
* run migrations-extract
* make updating ancillary permissions a post
* update file names
* add new api to doc, update body to just be []string
* revert moving ancillary permissions to post
* fix queries after final testing
* Update channel.go
* Update channel.go
* Update 000124_remove_manage_team_permission.up.sql
* Update 000124_remove_manage_team_permission.up.sql
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* on login add UserId and SessionId to audit's Actor field to match logout
* lint
* simplify to add only userId and sessionId
* AddToEventActor -> AddUser/SessionToEventActor
* fill in missing session data when logging the audit record
* why did it bump that? reverting.
* make modules-tidy
* trigger build
* add more context to the comment
* Add platform information for push notification metrics
* Address feedback
* Add the client platform returned by the devices to the normalize function
* Add "no platform" platform label to distinguish from unknown
* Adds Remote Cluster related API endpoints
New endpoints for the following routes are added:
- Get Remote Clusters at `GET /api/v4/remotecluster`
- Create Remote Cluster at `POST /api/v4/remotecluster`
- Accept Remote Cluster invite at `POST
/api/v4/remotecluster/accept_invite`
- Generate Remote Cluster invite at `POST
/api/v4/remotecluster/{remote_id}/generate_invite`
- Get Remote Cluster at `GET /api/v4/remotecluster/{remote_id}`
- Patch Remote Cluster at `PATCH /api/v4/remotecluster/{remote_id}`
- Delete Remote Cluster at `DELETE /api/v4/remotecluster/{remote_id}`
These endpoints are planned to be used from the system console, and
gated through the `manage_secure_connections` permission.
* Update server/channels/api4/remote_cluster_test.go
Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
* Fix AppError names
---------
Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
Co-authored-by: Mattermost Build <build@mattermost.com>
