upstream/mattermost
1
0
Fork 0
mirror of https://github.com/mattermost/mattermost.git synced 2026-05-28 04:35:04 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

MM-64209: Optimize completePopulatingCategoryChannelsT for MySQL (#30963)

Browse source 
For our MySQL customers, we have discovered that the query is not
able to choose the right plan by itself without adequate hints.

This is only for MySQL as we have confirmed from multiple customers
that Postgres takes the right index idx_sidebarcategories_userid_teamid
for the sidebarCategories table. And if it doesn't, then a VACUUM ANALYZE
fixes it.

But for MySQL, we have to do two things:
- Pass an index hint to let it use idx_sidebarcategories_userid_teamid.
- Pass an optimizer hint to materialize the sub-query. This is used
to materialize the doesNotHaveSidebarChannel sub-query into a temporary
table, letting MySQL reuse the contents of the table for further processing
in the parent sections of the query.

I have confirmed both locally and in the customer environment
that it gives a clear benefit.

*LOCAL*

OLD:
```
| -> Nested loop antijoin  (cost=2889.85 rows=19767) (actual time=3.355..38.033 rows=15 loops=1)
    -> Nested loop inner join  (cost=66.65 rows=110) (actual time=0.410..1.689 rows=220 loops=1)
        -> Filter: ((Channels.DeleteAt = 0) and (Channels.`Type` in ('O','P')))  (cost=25.25 rows=110) (actual time=0.394..0.886 rows=220 loops=1)
            -> Index lookup on Channels using idx_channels_team_id_display_name (TeamId='team01'), with index condition: (Channels.Id is not null)  (cost=25.25 rows=220) (actual time=0.389..0.793 rows=220 loops=1)
        -> Single-row covering index lookup on ChannelMembers using PRIMARY (ChannelId=Channels.Id, UserId='user000')  (cost=0.28 rows=1) (actual time=0.003..0.003 rows=1 loops=220)
    -> Nested loop inner join  (cost=4967.50 rows=180) (actual time=0.165..0.165 rows=1 loops=220)
        -> Covering index lookup on SidebarChannels using PRIMARY (ChannelId=Channels.Id)  (cost=7.86 rows=180) (actual time=0.055..0.062 rows=13 loops=220)
        -> Filter: ((SidebarCategories.TeamId = 'team01') and (SidebarCategories.UserId = 'user000'))  (cost=44.93 rows=1) (actual time=0.008..0.008 rows=0 loops=2881)
            -> Single-row index lookup on SidebarCategories using PRIMARY (Id=SidebarChannels.CategoryId)  (cost=44.93 rows=1) (actual time=0.006..0.006 rows=1 loops=2881)
 |
```

NEW:
```
 | -> Nested loop antijoin  (cost=5879.73 rows=58021) (actual time=1.544..3.135 rows=15 loops=1)
    -> Nested loop inner join  (cost=66.65 rows=110) (actual time=0.421..1.778 rows=220 loops=1)
        -> Filter: ((Channels.DeleteAt = 0) and (Channels.`Type` in ('O','P')))  (cost=25.25 rows=110) (actual time=0.405..0.945 rows=220 loops=1)
            -> Index lookup on Channels using idx_channels_team_id_display_name (TeamId='team01'), with index condition: (Channels.Id is not null)  (cost=25.25 rows=220) (actual time=0.400..0.859 rows=220 loops=1)
        -> Single-row covering index lookup on ChannelMembers using PRIMARY (ChannelId=Channels.Id, UserId='user000')  (cost=0.28 rows=1) (actual time=0.003..0.004 rows=1 loops=220)
    -> Single-row index lookup on <subquery2> using <auto_distinct_key> (ChannelId=Channels.Id)  (cost=130.37..130.37 rows=1) (actual time=0.006..0.006 rows=1 loops=220)
        -> Materialize with deduplication  (cost=130.35..130.35 rows=527) (actual time=1.118..1.118 rows=205 loops=1)
            -> Filter: (SidebarChannels.ChannelId is not null)  (cost=77.61 rows=527) (actual time=0.059..0.851 rows=523 loops=1)
                -> Nested loop inner join  (cost=77.61 rows=527) (actual time=0.058..0.786 rows=523 loops=1)
                    -> Covering index lookup on SidebarCategories using idx_sidebarcategories_userid_teamid (UserId='user000', TeamId='team01')  (cost=2.81 rows=15) (actual time=0.025..0.031 rows=15 loops=1)
                    -> Covering index lookup on SidebarChannels using idx_sidebarchannels_categoryid (CategoryId=SidebarCategories.Id)  (cost=1.70 rows=35) (actual time=0.032..0.046 rows=35 loops=15)
```

Performance improvement from 38ms to 3ms.

*CUSTOMER ENV* (with sensitive data wiped off)

OLD:
```
| -> Sort: channels.DisplayName  (actual time=512..512 rows=5 loops=1)
    -> Stream results  (cost=3.28 rows=1.44) (actual time=223..512 rows=5 loops=1)
        -> Nested loop antijoin  (cost=3.28 rows=1.44) (actual time=223..512 rows=5 loops=1)
            -> Nested loop inner join  (cost=3.02 rows=0.3) (actual time=0.025..0.0878 rows=5 loops=1)
                -> Covering index lookup on ChannelMembers using idx_channelmembers_user_id_channel_id_last_viewed_at (UserId='')  (cost=0.916 rows=6) (actual time=0.0146..0.023 rows=6 loops=1)
                -> Filter: ((channels.DeleteAt = 0) and (channels.TeamId = '') and (channels.`Type` in ('O','P')))  (cost=0.251 rows=0.05) (actual time=0.00999..0.0102 rows=0.833 loops=6)
                    -> Single-row index lookup on Channels using PRIMARY (Id=channelmembers.ChannelId)  (cost=0.251 rows=1) (actual time=0.00778..0.00785 rows=1 loops=6)
            -> Nested loop inner join  (cost=2.85 rows=4.81) (actual time=102..102 rows=0 loops=5)
                -> Covering index lookup on SidebarChannels using PRIMARY (ChannelId=channelmembers.ChannelId)  (cost=2.01 rows=4.81) (actual time=0.0125..13.8 rows=24134 loops=5)
                -> Filter: ((sidebarcategories.TeamId = '') and (sidebarcategories.UserId = ''))  (cost=1.54 rows=1) (actual time=0.00359..0.00359 rows=0 loops=120671)
                    -> Single-row index lookup on SidebarCategories using PRIMARY (Id=sidebarchannels.CategoryId)  (cost=1.54 rows=1) (actual time=0.00316..0.00319 rows=1 loops=120671)
```

NEW:
```
Here is the output

| -> Sort: channels.DisplayName  (actual time=0.12..0.12 rows=5 loops=1)
    -> Stream results  (cost=3.45 rows=4.01) (actual time=0.0797..0.11 rows=5 loops=1)
        -> Nested loop antijoin  (cost=3.45 rows=4.01) (actual time=0.0769..0.106 rows=5 loops=1)
            -> Nested loop inner join  (cost=3.02 rows=0.3) (actual time=0.0291..0.0555 rows=5 loops=1)
                -> Covering index lookup on ChannelMembers using idx_channelmembers_user_id_channel_id_last_viewed_at (UserId='')  (cost=0.916 rows=6) (actual time=0.0145..0.0162 rows=6 loops=1)
                -> Filter: ((channels.DeleteAt = 0) and (channels.TeamId = '') and (channels.`Type` in ('O','P')))  (cost=0.251 rows=0.05) (actual time=0.00611..0.00619 rows=0.833 loops=6)
                    -> Single-row index lookup on Channels using PRIMARY (Id=channelmembers.ChannelId)  (cost=0.251 rows=1) (actual time=0.0053..0.00534 rows=1 loops=6)
            -> Single-row index lookup on <subquery2> using <auto_distinct_key> (ChannelId=channelmembers.ChannelId)  (cost=7.01..7.01 rows=1) (actual time=0.00956..0.00956 rows=0 loops=5)
                -> Materialize with deduplication  (cost=7..7 rows=13.4) (actual time=0.0451..0.0451 rows=0 loops=1)
                    -> Filter: (sidebarchannels.ChannelId is not null)  (cost=5.66 rows=13.4) (actual time=0.0441..0.0441 rows=0 loops=1)
                        -> Nested loop inner join  (cost=5.66 rows=13.4) (actual time=0.0439..0.0439 rows=0 loops=1)
                            -> Covering index lookup on SidebarCategories using idx_sidebarcategories_userid_teamid (UserId='', TeamId='')  (cost=0.592 rows=3) (actual time=0.0105..0.0134 rows=3 loops=1)
                            -> Covering index lookup on SidebarChannels using idx_sidebarchannels_categoryid (CategoryId=sidebarcategories.Id)  (cost=1.39 rows=4.46) (actual time=0.00999..0.00999 rows=0 loops=3)
```

Performance improvement from 512ms to 0.12ms.

https://mattermost.atlassian.net/browse/MM-64209

```release-note
NONE
```
This commit is contained in:
Agniva De Sarker 2025-05-09 09:48:07 +05:30 committed by GitHub
parent 3be58f5f34
commit 0ebd3e8085
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 33 additions and 11 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

44
server/channels/store/sqlstore/channel_store_categories.go
View file

@ -442,6 +442,8 @@ func (s SqlChannelStore) completePopulatingCategoryChannelsT(db dbSelecter, cate
return category, nil
}
isMySQL := s.DriverName() == model.DatabaseDriverMysql
var channelTypeFilter sq.Sqlizer
if category.Type == model.SidebarCategoryDirectMessages {
// any DM/GM channels that aren't in any category should be returned as part of the Direct Messages category
@ -457,20 +459,40 @@ func (s SqlChannelStore) completePopulatingCategoryChannelsT(db dbSelecter, cate
}
// A subquery that is true if the channel does not have a SidebarChannel entry for the current user on the current team
doesNotHaveSidebarChannel := sq.Select("1").
Prefix("NOT EXISTS (").
From("SidebarChannels").
Join("SidebarCategories on SidebarChannels.CategoryId=SidebarCategories.Id").
Where(sq.And{
sq.Expr("SidebarChannels.ChannelId = ChannelMembers.ChannelId"),
sq.Eq{"SidebarCategories.UserId": category.UserId},
sq.Eq{"SidebarCategories.TeamId": category.TeamId},
}).
Suffix(")")
var doesNotHaveSidebarChannel sq.SelectBuilder
if isMySQL {
doesNotHaveSidebarChannel = sq.Select("/*+ QB_NAME(subq1) */ 1").
Prefix("NOT EXISTS (").
From("SidebarChannels").
// we have to use an index hint for MySQL.
Join("SidebarCategories USE INDEX(idx_sidebarcategories_userid_teamid) on SidebarChannels.CategoryId=SidebarCategories.Id").
Suffix(")")
} else {
doesNotHaveSidebarChannel = sq.Select("1").
Prefix("NOT EXISTS (").
From("SidebarChannels").
Join("SidebarCategories on SidebarChannels.CategoryId=SidebarCategories.Id").
Suffix(")")
}
doesNotHaveSidebarChannel = doesNotHaveSidebarChannel.Where(sq.And{
sq.Expr("SidebarChannels.ChannelId = ChannelMembers.ChannelId"),
sq.Eq{"SidebarCategories.UserId": category.UserId},
sq.Eq{"SidebarCategories.TeamId": category.TeamId},
})
channels := []string{}
var col string
if isMySQL {
// This is a materialization hint for MySQL to materialize
// the doesNotHaveSidebarChannel sub-query
// Without this hint, MySQL is unable to come up with this plan by itself.
col = "/*+ SEMIJOIN(@subq1 MATERIALIZATION) */ Id"
} else {
col = "Id"
}
sql, args, err := s.getQueryBuilder().
Select("Id").
Select(col).
From("ChannelMembers").
LeftJoin("Channels ON Channels.Id=ChannelMembers.ChannelId").
Where(sq.And{