2020-11-10 00:13:45 -05:00
|
|
|
// Copyright (c) 2015-present Mattermost, Inc. All Rights Reserved.
|
|
|
|
|
// See LICENSE.txt for license information.
|
|
|
|
|
|
|
|
|
|
package sqlstore
|
|
|
|
|
|
2021-01-07 12:12:43 -05:00
|
|
|
import (
|
|
|
|
|
"context"
|
2023-09-11 11:07:29 -04:00
|
|
|
|
|
|
|
|
"github.com/mattermost/mattermost/server/public/shared/request"
|
MM-60441: Re-index public channels when a user joins a team (#33400)
* Index all public channels when a user joins a team
* Precompute team members for indexChannelsForTeam
* Refactor RequestContextWithMaster to store package
This way, we can import it from both the sqlstore and the searchlayer
packages. The alternative for this is duplicating the code in those two
packages, but that will *not* work:
The context package expects custom types for the keys stored in it, so
that different packages never clash with each other when trying to
register a new key. See the docs for the WithValue function:
https://pkg.go.dev/context#WithValue
If we try to duplicate the storeContextKey type in both the sqlstore and
searchlayer packages, although they *look* the same, they are not, and
HasMaster will fail to get the value of the storeContextKey(useMaster)
key if it's from the other package.
* Use master in call to GetTeamMembersForChannel
In GetTeamMembersForChannel, use the DB from the newly passed context,
which will be the receiving context everywhere except in the call done
from indexChannelsForTeam, to avoid the read after write issue when
saving a team member.
* Fix GetPublicChannelsForTeam paging
We were using the page and perPage arguments as is in the call to
GetPublicChannelsForTeam, but that function expects and offset and a
limit as understood by SQL. Although perPage and limit are
interchangeable, offset is not equal to page, but to page * perPage.
* Add a synchronous bulk indexer for Opensearch
* Implement Opensearch's SyncBulkIndexChannels
* Add a synchronous bulk indexer for Elasticsearch
* Implement Elasticsearch's SynkBulkIndexChannels
* Test SyncBulkIndexChannels
* make mocks
* Bulk index channels on indexChannelsForTeam
* Handle error from SyncBulkIndexChannels
* Fix style
* Revert indexChannelWithTeamMembers refactor
* Remove defensive code on sync bulk processor
* Revert "Add a synchronous bulk indexer for Opensearch"
This reverts commit bfe4671d96bffa9ca27ed3c655fc5527b72bbafb.
* Revert "Add a synchronous bulk indexer for Elasticsearch"
This reverts commit 6643ae3f30c461544d0861aec7dee1f24e507c37.
* Refactor bulk indexers with a common interface
* Test all the different implementations
Assisted by Claude
* Remove debug statements
* Refactor common code into _stop
* Rename getUserIDsFor{,Private}Channel
* Wrap error
* Make perPage a const
* Fix typos
* Call GetTeamsForUser only if needed
* Differentiate errors for sync/async processors
---------
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
2025-08-25 13:28:19 -04:00
|
|
|
"github.com/mattermost/mattermost/server/v8/channels/store"
|
2020-11-10 00:13:45 -05:00
|
|
|
)
|
|
|
|
|
|
MM-30882: Fix read-after-write issue for demoting user (#16911)
* MM-30882: Fix read-after-write issue for demoting user
In (*App).DemoteUserToGuest, we would demote a user, and then immediately
read it back to do future operations from the user. This reading back
of the user had the effect of sticking the old value into the cache
after which it would never be updated.
There was another issue along with this, which was when the invalidation
message would broadcast across the cluster, it would hit the cache invalidation
problem where an unrelated store call would miss the cache because
it was invalidated, and then again read from replica and stick the old value.
To fix all these, we return the new value directly from the store method
to avoid having the app to read it again.
And we add a map in the localcache layer which tracks invalidations made,
and then switch to use master if it's true.
The core change is fairly limited, but due to changing the store method signatures,
a lot of code needed to be updated to pass "context.Background". Therefore the PR
just "appears" to be big, but the main changes are limited to app/user.go,
sqlstore/user_store.go and user_layer.go
https://mattermost.atlassian.net/browse/MM-30882
```release-note
Fix an issue where demoting a user to guest would not take effect in
an environment with read replicas.
```
* Fix concurrent map access
* Fixing mistakes
* fix tests
2021-02-12 08:34:05 -05:00
|
|
|
// WithMaster adds the context value that master DB should be selected for this request.
|
2023-09-11 11:07:29 -04:00
|
|
|
//
|
|
|
|
|
// Deprecated: This method is deprecated and there's ongoing change to use `request.CTX` across
|
|
|
|
|
// instead of `context.Context`. Please use `RequestContextWithMaster` instead.
|
MM-30882: Fix read-after-write issue for demoting user (#16911)
* MM-30882: Fix read-after-write issue for demoting user
In (*App).DemoteUserToGuest, we would demote a user, and then immediately
read it back to do future operations from the user. This reading back
of the user had the effect of sticking the old value into the cache
after which it would never be updated.
There was another issue along with this, which was when the invalidation
message would broadcast across the cluster, it would hit the cache invalidation
problem where an unrelated store call would miss the cache because
it was invalidated, and then again read from replica and stick the old value.
To fix all these, we return the new value directly from the store method
to avoid having the app to read it again.
And we add a map in the localcache layer which tracks invalidations made,
and then switch to use master if it's true.
The core change is fairly limited, but due to changing the store method signatures,
a lot of code needed to be updated to pass "context.Background". Therefore the PR
just "appears" to be big, but the main changes are limited to app/user.go,
sqlstore/user_store.go and user_layer.go
https://mattermost.atlassian.net/browse/MM-30882
```release-note
Fix an issue where demoting a user to guest would not take effect in
an environment with read replicas.
```
* Fix concurrent map access
* Fixing mistakes
* fix tests
2021-02-12 08:34:05 -05:00
|
|
|
func WithMaster(ctx context.Context) context.Context {
|
MM-60441: Re-index public channels when a user joins a team (#33400)
* Index all public channels when a user joins a team
* Precompute team members for indexChannelsForTeam
* Refactor RequestContextWithMaster to store package
This way, we can import it from both the sqlstore and the searchlayer
packages. The alternative for this is duplicating the code in those two
packages, but that will *not* work:
The context package expects custom types for the keys stored in it, so
that different packages never clash with each other when trying to
register a new key. See the docs for the WithValue function:
https://pkg.go.dev/context#WithValue
If we try to duplicate the storeContextKey type in both the sqlstore and
searchlayer packages, although they *look* the same, they are not, and
HasMaster will fail to get the value of the storeContextKey(useMaster)
key if it's from the other package.
* Use master in call to GetTeamMembersForChannel
In GetTeamMembersForChannel, use the DB from the newly passed context,
which will be the receiving context everywhere except in the call done
from indexChannelsForTeam, to avoid the read after write issue when
saving a team member.
* Fix GetPublicChannelsForTeam paging
We were using the page and perPage arguments as is in the call to
GetPublicChannelsForTeam, but that function expects and offset and a
limit as understood by SQL. Although perPage and limit are
interchangeable, offset is not equal to page, but to page * perPage.
* Add a synchronous bulk indexer for Opensearch
* Implement Opensearch's SyncBulkIndexChannels
* Add a synchronous bulk indexer for Elasticsearch
* Implement Elasticsearch's SynkBulkIndexChannels
* Test SyncBulkIndexChannels
* make mocks
* Bulk index channels on indexChannelsForTeam
* Handle error from SyncBulkIndexChannels
* Fix style
* Revert indexChannelWithTeamMembers refactor
* Remove defensive code on sync bulk processor
* Revert "Add a synchronous bulk indexer for Opensearch"
This reverts commit bfe4671d96bffa9ca27ed3c655fc5527b72bbafb.
* Revert "Add a synchronous bulk indexer for Elasticsearch"
This reverts commit 6643ae3f30c461544d0861aec7dee1f24e507c37.
* Refactor bulk indexers with a common interface
* Test all the different implementations
Assisted by Claude
* Remove debug statements
* Refactor common code into _stop
* Rename getUserIDsFor{,Private}Channel
* Wrap error
* Make perPage a const
* Fix typos
* Call GetTeamsForUser only if needed
* Differentiate errors for sync/async processors
---------
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
2025-08-25 13:28:19 -04:00
|
|
|
return store.WithMaster(ctx)
|
2020-11-10 00:13:45 -05:00
|
|
|
}
|
|
|
|
|
|
2023-09-11 11:07:29 -04:00
|
|
|
// RequestContextWithMaster adds the context value that master DB should be selected for this request.
|
2025-09-10 09:11:32 -04:00
|
|
|
func RequestContextWithMaster(rctx request.CTX) request.CTX {
|
|
|
|
|
return store.RequestContextWithMaster(rctx)
|
2023-09-11 11:07:29 -04:00
|
|
|
}
|
|
|
|
|
|
2023-10-11 07:08:55 -04:00
|
|
|
// HasMaster is a helper function to check whether master DB should be selected or not.
|
|
|
|
|
func HasMaster(ctx context.Context) bool {
|
MM-60441: Re-index public channels when a user joins a team (#33400)
* Index all public channels when a user joins a team
* Precompute team members for indexChannelsForTeam
* Refactor RequestContextWithMaster to store package
This way, we can import it from both the sqlstore and the searchlayer
packages. The alternative for this is duplicating the code in those two
packages, but that will *not* work:
The context package expects custom types for the keys stored in it, so
that different packages never clash with each other when trying to
register a new key. See the docs for the WithValue function:
https://pkg.go.dev/context#WithValue
If we try to duplicate the storeContextKey type in both the sqlstore and
searchlayer packages, although they *look* the same, they are not, and
HasMaster will fail to get the value of the storeContextKey(useMaster)
key if it's from the other package.
* Use master in call to GetTeamMembersForChannel
In GetTeamMembersForChannel, use the DB from the newly passed context,
which will be the receiving context everywhere except in the call done
from indexChannelsForTeam, to avoid the read after write issue when
saving a team member.
* Fix GetPublicChannelsForTeam paging
We were using the page and perPage arguments as is in the call to
GetPublicChannelsForTeam, but that function expects and offset and a
limit as understood by SQL. Although perPage and limit are
interchangeable, offset is not equal to page, but to page * perPage.
* Add a synchronous bulk indexer for Opensearch
* Implement Opensearch's SyncBulkIndexChannels
* Add a synchronous bulk indexer for Elasticsearch
* Implement Elasticsearch's SynkBulkIndexChannels
* Test SyncBulkIndexChannels
* make mocks
* Bulk index channels on indexChannelsForTeam
* Handle error from SyncBulkIndexChannels
* Fix style
* Revert indexChannelWithTeamMembers refactor
* Remove defensive code on sync bulk processor
* Revert "Add a synchronous bulk indexer for Opensearch"
This reverts commit bfe4671d96bffa9ca27ed3c655fc5527b72bbafb.
* Revert "Add a synchronous bulk indexer for Elasticsearch"
This reverts commit 6643ae3f30c461544d0861aec7dee1f24e507c37.
* Refactor bulk indexers with a common interface
* Test all the different implementations
Assisted by Claude
* Remove debug statements
* Refactor common code into _stop
* Rename getUserIDsFor{,Private}Channel
* Wrap error
* Make perPage a const
* Fix typos
* Call GetTeamsForUser only if needed
* Differentiate errors for sync/async processors
---------
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
2025-08-25 13:28:19 -04:00
|
|
|
return store.HasMaster(ctx)
|
2020-11-10 00:13:45 -05:00
|
|
|
}
|
2021-02-16 03:30:03 -05:00
|
|
|
|
2021-09-14 04:16:42 -04:00
|
|
|
// DBXFromContext is a helper utility that returns the sqlx DB handle from a given context.
|
|
|
|
|
func (ss *SqlStore) DBXFromContext(ctx context.Context) *sqlxDBWrapper {
|
2023-10-11 07:08:55 -04:00
|
|
|
if HasMaster(ctx) {
|
2024-12-10 08:57:19 -05:00
|
|
|
return ss.GetMaster()
|
2021-09-14 04:16:42 -04:00
|
|
|
}
|
2024-12-10 08:57:19 -05:00
|
|
|
return ss.GetReplica()
|
2021-09-14 04:16:42 -04:00
|
|
|
}
|