If the store fails to initialize (e.g. run a migration), it would `log.Fatal` and then `os.Exit`. Unfortunately, this trips up `TestMain`, which happily keeps running tests, now guaranteed to fail.
Avoid this by instead returning an error from the store initialization, handling appropriately at the layer above.
After some analysis of vacuuming statistics of our large
Postgres customers, it was observed that the default vacuuming
thresholds for some tables are not optimal enough for our workload.
We need more frequent vacuuming for some tables. The analysis
showed that the top 4 tables that needed rectification
were posts, preferences, threadmemberships and fileinfo.
So we reduce the threshold by half for those tables, triggering
autovacuum and autoanalyze doubly often.
```release-note
NONE
```
We missed this out last time. It's possible in an HA
scenario for a second pod to start later while the other
job is in-progress. In that case, it would schedule
two jobs.
https://mattermost.atlassian.net/browse/MM-53747
```release-note
NONE
```
Bifrost now encodes all image paths. Due to this
one-way translation, we need to encode all the older
image paths as well.
After this is done, we can remove the double-lookup.
https://mattermost.atlassian.net/browse/MM-53747
```release-note
NONE
```
Co-authored-by: Mattermost Build <build@mattermost.com>
* Revert "POC: Cross-team recent search (#20027)"
This reverts commit aa59c28b04, preserving
a few code tidyings unrelated to the original PR.
* Revert "Add feature flag for command palette (#20011)"
This reverts commit c78c5ce3f3.
* add base for calling the endpoint
* add endpoint and handler
* update store and layers
* call the endpoint
* align types
* update app layers
* generate mocks
* complete handler
* finish store query
* add todos
* add ui for member count
* add selector
* add a todo
* add cache layer
* optimize calls in FE
* handle invalidation of the cache
* fix go style
* fix test
* use existing channel layer count
* fix import error
* delete unnecessary code
* write tests for channel cache layer
* fix testname
* fix mocks
* fix cache layer test
* fix a test
* really fix the test
* write more tests for server
* address PR comments
* remove comment
* rename more_channels to browse_channels
* fix style
* update snapshot
* add translations
* Revert "add translations"
This reverts commit 56476a5dab.
* add only related translations
* address PR review points
* add test
* fix test
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
There was already a cache present for emoji names. But we weren't using
it for the GetMultipleByName method. Now we implement that method
to look up the cache for every emoji name passed.
Secondly, a bigger problem was that we were making the DB call for system
emojis as well. Since system emojis aren't stored in the DB, it would
fall through the cache layer and always make a redundant DB call. In the
profiles, this should up as taking 16% of the total time to serve
a getPostsForChannel API endpoint.
We fix this by filtering the emojis to only custom emojis
before making the call.
https://mattermost.atlassian.net/browse/MM-53669
```release-note
NONE
```
* [MM-53192] Patch full name leak in Insights team_members API
* Update server/channels/app/team.go
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
---------
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
Sometimes a broad distinction of just a channelID or a userID
is not enough to efficiently send a websocket event to users.
In several cases, depending on the user and channel, we might
need to modify the message. Therefore, we introduce the
concept of a channel hook that will get executed if the scope
is set to a channel. This hook can be populated at the app layer
to perform any application specific logic to the event.
Care must be taken to avoid race conditions as the passed event
is not deep copied. It is left to the user to treat it carefully.
For this issue, the main problem was that since we don't know
which users have permissions to which channels, we had to go through
_all_ members of a channel to figure that out. This was redundant
since a large portion of those users might not even be connected
at that time.
We solve this with the channel hook where we push this check
to be performed later while actually sending the event. This
reduces the computation to be done only for _connected_ users
rather than _all_ users of a channel.
The next iteration of this should be to use websocket subscriptions
to monitor exactly which users are on that channel to even
trim down that list. That is a larger initiative to be taken later.
Tested locally with a channel of 50 users. Here are rough results:
```
With PR:
patchPost 97ms
createPost 90ms
Master:
patchPost 306ms
createPost - 298ms
```
https://mattermost.atlassian.net/browse/MM-42810
```release-note
Improve performance while sending messages
with permalinks to channels with large number of users.
```
* add teams to allowed frame-ancestors
* fix unit tests
* set SameSite attribute for session cookie
* further restrict ancestors
* skip landing page if in iframe
* Only set cookie SameSite=None if embedded in iframe
* don't set MMEMBED cookie on landing page (check only)
* fully parse MMEMBED cookie
* add comment
* more comments
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* WIP
* Add rate limiting for desktop token API
* Missing mocks
* Style fixes
* Update snapshots
* Maybe use an actual redirect link :P
* Refactoring for tests
* Add tests for server
* Fix lint issue
* Fix tests
* Fix lint
* Add front-end screen component
* Component logic
* Style changes
* Quick style fix
* Lint fixes
* Initial PR feedback
* Enable logging into the browser as well when completing the login process
* Refactor to push more logic to the other component
* Remove unnecessary helper code
* Fix i18n
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* update for guest demotion and sysadmin edits
* update unit test
* update unit test
* use existing disabled flag
* remove commented line
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* temp commit
* update test to allow bot creation
* add bot check to updateUser and deleteUser
* add more unit tests
* lint fixes
* lint fix
* update based on doc
* add more unit tests
* lint fixes
* fix unit tests
* fix unit tests
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
For MySQL, a query for type LEFT JOIN .. IS NULL
leads to a nested antijoin which leads to poor performance.
We fix this by rewriting the query to avoid the antijoin.
See the JIRA epic for more context behind this.
We also make another improvement to remove the DISTINCT
clause. It didn't serve any purpose since userids would
already be unique.
https://mattermost.atlassian.net/browse/MM-53406
```release-note
NONE
```
In a single createPost flow, there were
3 separate calls to App.IsCRTEnabled. This
showed up very slightly in the CPU profiles.
Not a big deal, but good to get it out of
the way.
```release-note
NONE
```
* Remove global draft feature flag
* More removal - in progress
* Removed the rest in webapp
* Removed the rest in webapp
* Fix tests
* Update feature_flags.go
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
The UpdateProductNotices method runs in a separate goroutine
which modifies a global structure called noticesCache.
This creates race condition when one test finishes
but the goroutine hasn't finished running.
To fix this, we simply prevent the spawning of the goroutine
by disabling the feature in the config.
Ideally, we'd not have global state in the first place.
But that's a separate matter.
https://mattermost.atlassian.net/browse/MM-52646
```release-note
NONE
```
Remove changes related to the unshipped threads everywhere feature, including commits b8da473da7 and 9f9e19e05d.
Since a version of Playbooks shipped calling this experimental API, keep a `nil` implementation to avoid breaking compatibility. We remove the hooks altogether, but keep the numbering again to avoid breaking compatbility.
Fixes: https://mattermost.atlassian.net/browse/MM-53358
Having cache layer as part of the store call
Prometheus metrics can be somewhat misleading
because cache-backed methods can have very high
volume and yet not cause any load on the DB at all.
However, if the admin is not aware which calls
are cache-backed and which are not, then
it can be hard to understand why a store call
of high volume isn't causing any stress on the DB at all.
Therefore, for simplicity we reorder the layers
such that cache layer is handled outside the timer layer.
We already have cache related metrics like hit/miss
which can be used to monitor cache usage.
https://mattermost.atlassian.net/browse/MM-53058
```release-note
NONE
```
Co-authored-by: Mattermost Build <build@mattermost.com>
* Remove build references
* Remove playbooks webapp and server, and add the prepackaged plugin
* Remove translations
* Add ProductSettings to the playwright type
* Restore playbooks as a prepackaged plugin for cypress e2e tests
https://mattermost.atlassian.net/browse/MM-52532
- Replace golint with revive
- Add makezero linter
- Fix all the required linter failures
Some issues in enterprise and public modules
are yet to be fixed. We send this to expediate things.
It was a good decision in hindsight to keep the public module as 0.x
because this would have been a breaking change again.
https://mattermost.atlassian.net/browse/MM-53032
```release-note
Changed the Go module path from github.com/mattermost/mattermost-server/server/v8 to github.com/mattermost/mattermost/server/v8.
For the public facing module, it's path is also changed from github.com/mattermost/mattermost-server/server/public to github.com/mattermost/mattermost/server/public
```
* disable coverage
This reduces runtime of the server test suite from ~30m to ~10m, and as far as I can see: we discarded the coverage output anyway.
* allow morph 60s to migrate when running tests
* fix fileutils.TestFindFile on MacOS
* introduce model.ExternalServiceEnvironment
* pick license public key from external service env
* pick Stripe public key from external service env
* pick Rudder key from external service env
* configure Sentry DSN from external service env
* always log external_service_environment, Unsetenv
* clear faked BuildEnv, improve logging
* strip out unset GOTAGS
* fix Sentry tests
* simplify to just ServiceEnvironment
* relocate ServiceEnvironment in client config
* initialize CWS URLs based on service environment
* unset rudder key for boards dev
* harden service environment to avoid accidental production
* fix TestSentry again
* fix DEFAULT -> ENTERPRISE
* s/dev/test when naming playbooks rudder key
* simplify boards rudder key switch
* use uniform rudderKey variable names
* retain compatibility with existing pipeline
* reduce to just production/test
* unit test with valid test license
* simplify Playbooks telemetry initialization
* restore dev service environment
* emit ServiceEnvironment when running e2e tests
* Migrate all method in model/client4.go to accept a context.Context
* Fix th.*Client
* Fix remaining issues
* Empty commit to triger CI
* Fix test
* Add cancellation test
* Test that returned error is context.Canceled
* Fix bad merge
* Update mmctl code
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* create config setting to enable/disable playbooks product
* fix to config name
* fix typo
* revert changes to package-lock.json
* update name of test
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* mark thread files as deleted
* add missing check
* improve query
* Stopped rendering post preview if post is deleted
* Fixed lint error
* Fixed test
* updated types
* Removed deleted post from other post's embed data
* Added tests
* Apply suggestions from code review
Co-authored-by: Daniel Espino García <larkox@gmail.com>
* lint fix
---------
Co-authored-by: Konstantinos Pittas <konstantinos.pittas@mattermost.com>
Co-authored-by: Harshil Sharma <harshilsharma63@gmail.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
Co-authored-by: Harshil Sharma <18575143+harshilsharma63@users.noreply.github.com>
Co-authored-by: Daniel Espino García <larkox@gmail.com>
* Avoid calling the user count query in future if we get a count > 0
* re-adding mock session to avoid adding the old mitigation in future
* adjustments based on feedback
* MM-46410: adds urgency on mention counts
We have introduced priority for posts in
https://github.com/mattermost/mattermost-webapp/pull/10951.
We do need to color the mention badges in the webapp with a prominent
color when a mention is posted in an urgent message.
A thread has urgent mentions if the root post is marked as urgent, and
the replies contain mentions to the user viewing the thread.
This PR adds two columns, urgentmentioncount, and isurgent, in
channelmembers, and threads tables respectively.
Furthermore when asking for team/thread mention counts, we also return
urgent mention counts for the user.
* Adds PostAcknowledgements table and apis
* job init and fetch mentions
* add-migrations
* delete-expired
* send-notifications
* Fetches post priority in batches
* stop-notifications
* stop-notification-on-reply
* MM-47750: Adds PostAcknowledgements table and apis
- Adds post acknowledgement api/app/store methods to be able to save and
delete post acknowledgements by users.
- Adds wesbsocket events for acknowledgement created/deleted
- Returns post acknowledgements in the post's metadata
* add-license-check
* add-pagination
* delete on channel and team
* validate guests
* add configs
* move create priority post check from app to api
* Add desktop notifications
* check status
* use config in job
* add IsUrgent check
* Add last-sent-at
* validate max recipients
* Update lastSentAt
* Validate min. recipient
* send email notification only once
* remove email notifications
* use latest time from config to run job
* Add notifications counter
* publish events to mentioned users only
* pickup license updates in scheduler
* don't allow post owner to stop notifications
* follow normal notifications behaviour
* Validates persistent notifications interval
* move logic of handling valid and expired posts into sql
* Adds persistent notifications in the webapp
---------
Co-authored-by: koox00 <3829551+koox00@users.noreply.github.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
I mistakenly assumed that those packages are naturally imported.
And also atleast one test would have broken.
None of them are true. :)
So we just import it again at a better place
This reverts commit 30a053314b.
```release-note
NONE
```
* platform/web_conn: fix a possible data race issue
* reflect review comments
---------
Co-authored-by: Ibrahim Serdar Acikgoz <ibrahim@ibrahims-mac.local>
Co-authored-by: Mattermost Build <build@mattermost.com>
* Regenerate default profile picture if username has changed
- Only actions is profile picture has not been changed
- Adjusts ResetLastPictureUpdate store function to store
-curTime instead of 0
- This is to support updating the default picture while still
retaining the ability to discern a default image from a set one.
- Changes SetDefaultProfileImage to leverage UpdateDefaultProfileImage
- Test updates around updating user default profile pictures
* App interface updates
* Only display picture update date if non-negative
- Ensures we don't display negative timestamps (default images)
- Change ported for mono-repo changes
* Remove duplicate test assertion
---------
Co-authored-by: Nathan Geist <ngeist@spiria.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
We strengthen some checks, forcing them to be changed
only via config.
https://mattermost.atlassian.net/browse/MM-37691
```release-note
NONE
```
Co-authored-by: Mattermost Build <build@mattermost.com>
* [MM-47492] - Don't count system messages as replies to a thread
* Add test assertion
* revert store changes`
---------
Co-authored-by: Nevyana Angelova <nevyangelova@Nevyanas-MacBook-Pro.local>
Co-authored-by: Mattermost Build <build@mattermost.com>
* WIP
* Fetched ES index for type checking
* WIP
* Moved migration key to model
* CI
* CI
* Fixed existing tests
* Updated einterface mocks
* CI
* WIP
* Reverted test makefile changes
* Fixed error level
* CI
* Not attempting to launch job if not enterprise ready
* CI
* Fixing job trigger condition
* CI
* Updated eemocks
* CI
* MM-50960 - store system organization name
* restore the preparing workspace plugins and invite screens
* add back the page lines for the design
* add lines back and organize styles
* set back documentation to monorepo style and disable board as a product
* fix organization link and style skip button
* create team on organization name screen continue button click
* make sure there are not already created team and if so just update team name
* update the team display name if team has already been created
* cover error scenarios during team creation
* add pr feedback and add a couple of unit tests
* fix translation server error; make sure only update display name if it has changed in the form
* temp advances
* rewrite unit tests using react-testing library; fix unit tests
* fix translations
* make sure the launching workspace finish in cloud installations
* remove redundant validation
* fix unit tests
* remove unintended config value left after merge conflict
* MM-50550: Filter out threads from "left" channels v2
Currently leaving a channel doesn't affect the thread memberships of
that user/channel combination.
This PR aims to filter out all threads from those channels for the user.
Adds a DeleteAt column in the ThreadMemberships table, and filter out
all thread memberships that are "deleted".
Each time a user leaves a channel all thread memberships are going to be
marked as deleted, and when a user joins a channel again all those
existing thread memberships will be re-instantiated.
Adds a migration to mark all existing thread memberships as deleted
depending on whether there exists a channel membership for that
channel/user.
* Added migration files into list
* Fixes tests
* Fixes case where DeleteAt would be null
* Guard thread API endpoints with appropriate perms
* Deletes ThreadMembership rows upon leaving channel
* Minor style changes
* Use NoTranslation error
* Refactors tests
* Adds API tests to assert permissions on Team
* Adds tests, and fixes migrations
* Fixes test description
* Fix test
* Removes check on DM/GMs
* Change the MySQL query in the migration
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
We monitor the health of DB replicas, and on a fatal error,
take them out of the pool.
On a separate goroutine, we keep pinging the unhealthy replicas,
and on getting a good response back, we add them back to the pool.
https://mattermost.atlassian.net/browse/MM-50427
```release-note
Mattermost is now resilient against DB replica outages and will
dynamically choose a replica if it's alive.
Also added a config parameter ReplicaMonitorIntervalSeconds
whose default value is 5. This controls how frequently unhealthy
replicas will be monitored for liveness check.
```
Co-authored-by: Mattermost Build <build@mattermost.com>
* Enable products for channels tests
* increase unit test timeout; check IsConfigReadOnly
* make app-layers
* Avoid loading boards tempaltes between tests to improve speed
* Fix delete query to be compatible with both databases
* Avoid preserving the templates for boards store tests
* Run all tests in one command
* Revert "Run all tests in one command"
This reverts commit 0330f7cd8f.
* concurrent pkg group tests in CI
* Revert "Revert "Run all tests in one command""
This reverts commit 73892fec77.
* Revert "concurrent pkg group tests in CI"
This reverts commit 550fb6cdd4.
* try testing 3 subsets of packages concurrently to improve time taken
* Revert "try testing 3 subsets of packages concurrently to improve time taken"
This reverts commit 97475f3c4e.
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
Co-authored-by: wiggin77 <wiggin77@warpmail.net>
https://mattermost.atlassian.net/browse/MM-52079
```release-note
We upgrade the module version to 8.0. The new module path is github.com/mattermost-server/server/v8.
```
Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
This will be shown in the system console to let users know
which search backend is active.
Implemented via adding an extra param in the ping response.
https://mattermost.atlassian.net/browse/MM-49984
```release-note
The database section in the system console now has an additional
read-only section which shows the active search backend in use.
This can be helpful to confirm which is the currently active
search engine when there are multiple of them configured.
```
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
We would return "nil, nil" if a plugin was on the blocklist.
This would cause a nil dereference panic while trying to add
struct fields from manifest to the audit logs.
To avoid it, we return an error explicitly and ignore the
error ids to unnecessarily have it log an error.
https://mattermost.atlassian.net/browse/MM-51786
```release-note
NONE
```
After the monorepo integration, the boards product
was started before we started the hub.
This led to a data race where the GetHubForUserId was actually
called before the hub was initialized.
To fix this we initialize the platform first, and then the products.
https://mattermost.atlassian.net/browse/MM-52009
```release-note
NONE
```
Mockery started to throw errors like "Unexpected package creation during export data loading".
This was consistently reproducible locally and on upgrade, it has gone away.
```release-note
NONE
```
* bind to localhost:0 for tests
Explicitly bind to `localhost:0` instead of just `:0` to avoid binding to all available interfaces and in turn avoid triggering firewall warnings on MacOS.
* fix Playbooks to use dynamic port
* TestMetrics: handle ipv4 localhost too
* TestMetrics: linting
* MM-49564: Upsert in the Store vs App layer
Refactor drafts so that Upserting a draft would happen in the DB and not
in the app layer.
* Fixes mocks
* Fixes tests
* Fixes translations
* Fixes tests
* Update tests
* Fixes tests
* Addresses review comments
- renames Save => Upsert
- removes Sleep from tests
* Fixes flaky test
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* remove Boards feature flag from server, makefile, and docker
* Update server/boards/server/boards_service_util.go
Co-authored-by: Miguel de la Cruz <miguel@mcrx.me>
---------
Co-authored-by: Miguel de la Cruz <miguel@mcrx.me>
* Add new trial form to enrich trial requests with more customer info
* Update snapshots
* One more addition
* Fixes from PR feedback
* Fix types, i18n, update e2e tests
* Fix linter
* Fix i18n?
* Fix blank translation
* update snapshot
* Update snapshot properly
* Remove inapplicable test
* Fix business email validation only happening once
* UX Feedback
* Fix linter
* Fix linter again, not working locally
* FIX LINTER
* Move isvalid check until after some fields are set
* Fix for overlapping modals
* Fix linter
* UX feedback
* UX Feedback
* Fix typo in error modal
* [MM-51551] Add new Trial Form to Playbooks trial requests (#22650)
* Playbooks start trial entrypoints opens new trial form modal
* Fix style
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* [MM-51347] Trial form error modal for embargoed and air gapped entities (#22656)
* Playbooks start trial entrypoints opens new trial form modal
* Add support for air gapped environments when making trial requests
* Add specific handling for embargoed entities
* undo some code
* Fix linter
* Fix types
* Fix style
* Updates because TE has to upgrade to E0 before it can activate a trial
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
We log warnings whenever our websocket buffer sizes exceed
certain thresholds. The problem with that is, when this happens,
the logs are completely spammed with these lines
making it annoying for the customer.
To improve the situation, we use a timer that only gets reset
every minute.
https://mattermost.atlassian.net/browse/MM-51700
```release-note
NONE
```
* Revert "fix store issue take two"
This reverts commit 59f943c2c7.
* Revert "fix store override issue"
This reverts commit 29c346757a.
* Revert "Fix TestPushNotificationRace"
This reverts commit 6d62dddf86.
* Revert "fix default DSN for CI"
This reverts commit e0e69cdbb0.
* Revert "disable playbooks from more unit tests"
This reverts commit a1e97a9e96.
* Revert "disable playbooks for more tests"
This reverts commit 4d2dc74f05.
* Revert "disable playbooks for TestSAMLSettings"
This reverts commit 35c1a4312d.
* Revert "disable playbooks for more unit tests"
This reverts commit c049631a14.
* Revert "disable playbooks for mocked enterprise tests"
This reverts commit 829317fddb.
* Partially revert "disable playbooks for channel/apps mocked tests"
This reverts commit 52b4a0a6cf.
* Revert "fix TestUnitUpdateConfig"
This reverts commit 8f134f2a8a.
* Revert "add plugin mock to TestUnitUpdateConfig"
This reverts commit 3ec5419092.
* Revert "disable Boards for more test helpers"
This reverts commit 5d4d0d02d9.
* Revert "disable boards at correct place in test helpers"
This reverts commit 0c9e175f79.
* Partially revert "disable boards for slash cmd tests"
This reverts commit fad8d9de93.
* Revert "disable Boards for channels web tests"
This reverts commit 15540fdfc0.
* Revert "Adds a teardown function to playbook server tests to disable and reenable boards"
This reverts commit 9a46e3d0f4.
* Revert "Test disable boards through feature flag"
This reverts commit 787044add8.
* TestUnitUpdateConfig: restore callback check
* Revert "Revert "fix default DSN for CI""
This reverts commit 01b879d55a.
Spawn a goroutine from the server layer to have better
control for compliance tests.
https://mattermost.atlassian.net/browse/MM-51504
```release-note
NONE
```
Co-authored-by: Mattermost Build <build@mattermost.com>
* MM-45956: Optimize FileInfo stats query
We Denormalize Post.ChannelId on FileInfo.ChannelId
```release-note
The file info stats query is now optimized by denormalizing the channelID column into the table itself. This will speed up the query to get the file count for a channel on clicking the RHS.
Migration times:
On a MySQL 8.0.31 DB with
1405 rows in FileInfo and 11M posts, it took around 0.3s
On a Postgres 12.14 DB with
1731 rows in FileInfo and 11M posts, it took around 0.27s
```
https://mattermost.atlassian.net/browse/MM-45956
There is a standard IDNA2008 which defines how unicode characters
in domain names can be compared using UTS (Unicode Technical Standard) 46.
We use that to convert any links in messages to ASCII and
then compare with the domain list in config.
https://mattermost.atlassian.net/browse/MM-51486
```release-note
NONE
```
