test-server-race wasn't using the same set of steps
that the test-server step did. Therefore one test was failing.
Refactored it such that scripts/test.sh can be used to run
normal and race tests as well
```release-note
NONE
```
* MM-31094: Adds tooling to develop and test using a MySQL instance with replication lag. Adds some lazy lookups to fallback to master if results are not found.
* MM-31094: Removes mysql-read-replica from default docker services.
* MM-31094: Switches (store..SessionStore).Get and (store.TeamStore).GetMember to using context.Context.
* MM-31094: Updates (store.UsersStore).Get to use context.
* MM-31094: Updates (store.PostStore).Get to use context.
* MM-31094: Removes feature flag and config setting.
* MM-31094: Rolls back some master reads.
* MM-31094: Rolls a non-cache read.
* MM-31094: Removes feature flag from the store.
* MM-31094: Removes unused constant and struct field.
* MM-31094: Removes some old feature flag references.
* MM-31094: Fixes some tests.
* MM-31094: App layers fix.
* MM-31094: Fixes mocks.
* MM-31094: Don't reparse flag.
* MM-31094: No reparse.
* MM-31094: Removed unused FeatureFlags field.
* MM-31094: Removes unnecessary feature flags variable declarations.
* MM-31094: Fixes copy-paste error.
* MM-31094: Fixes logical error.
* MM-30194: Removes test method from store.
* Revert "MM-30194: Removes test method from store."
This reverts commit d5a6e8529b.
* MM-31094: Conforming to make's strange syntax.
* MM-31094: Configures helper for read replica with option.
* MM-31094: Adds some missing ctx's.
* MM-31094: WIP
* MM-31094: Updates test names.
* MM-31094: WIP
* MM-31094: Removes unnecessary master reads.
* MM-31094: ID case changes out of scope.
* MM-31094: Removes unused context.
* MM-31094: Switches to a helper. Removes some var naming changes. Fixes a merge error.
* MM-31094: Removes SQLITE db driver ref.
* MM-31094: Layer generate fix.
* MM-31094: Removes unnecessary changes.
* MM-31094: Moves test method.
* MM-31094: Re-add previous fix.
* MM-31094: Removes make command for dev.
* MM-31094: Fix for login.
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Create basic make commands for configuring golang-migrate
* Showcase full flow with new migrations
* Apply PR suggestions
* Migrate over team members
* Update mocks
* Fix specs
* Move columns that added after table creation onto separate stmts
* Put back gorp table definitions
* Fix issues with golang-migrate that not tracks underlying db driver
* Help prompt after new migration and consistent checksum for bindata
* Put gorp mapping back
* Apply PR suggestiong
* Close migrations after they run
* Add migration file to bindata check
* Updates needed
* Reset store_test
* Add copyright
* Apply PR suggestions
* Fix new circleci check
* Put back upgrade step for backwards comp
* Add store test to test migration directions
* Apply PR suggestions
* Add go-bindata to tools
* Apply PR suggestios
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Allow uploading 10 files per post
* Increase Post.Fileids column size to fit 10 files
* Add Post.Fileids column changes to DB v531
* Fix current_schema_version
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* Update store/sqlstore/upgrade.go
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* Fix schema version
Co-authored-by: Diogo Lima Nicolau <diogo.lima.nicolau@cern.ch>
Co-authored-by: Harrison Healey <harrisonmhealey@gmail.com>
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-30988 - Fix racy test ServerSystemdNotification
The translateFunc is a global variable which was unguarded.
So we convert that into an atomic variable to prevent against races.
https://mattermost.atlassian.net/browse/MM-30988
```release-note
NONE
```
* fix tests
* move call upwards
* use race
* after translations init
* guard with a sync.Once
* missed import
* revert race
* update timeout
It's was including it in the bin folder but when we package we need to include it in the dist/bin so right now the `download_mmctl` script accepts a second parameter to define the path where the binary is going to be stored
* Update to get latest release from S3
* Update mmctl to download to use S3
* Download MMCTL from S3
* Update download_mmctl_release.sh
* Update Makefile
* Update release.mk
* Update the script called, remove call to sub shell and clean up indentation
* Add quotes around THIS_BRANCH var
* Update script to support overriding the OS for packaging
* Add check for release-<Digit> to better match the branch
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Update to get latest release from S3
* Update mmctl to download to use S3
* Download MMCTL from S3
* Update download_mmctl_release.sh
* Update Makefile
* Update release.mk
* Update the script called, remove call to sub shell and clean up indentation
* Add quotes around THIS_BRANCH var
* Update script to support overriding the OS for packaging
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Summary:
Every make call would ping Github to find out the mmctl version to download. Each check made 4 requests to Github. So every make execution resulted in 4 requests to Github. This leads to frequent rate-limit errors from Github.
In this PR we check for the mmctl version only if mmctl doesn't already exist. We also print a more helpful error message.
Reduce the number the number of requests to Github from 4 to 2.
Ticket Link:
https://mattermost.atlassian.net/browse/MM-28303
* MM-27512: Use an authenticated user to bump up request rate limit
An unauthenticated user can only make 60 requests per hour which means 1
request every minute. This can lead to frequent rate limit errors while
getting the latest release.
We change that to use an authenticated user which is already available
in the CI. This moves us to make 5000 requests per hour.
We also add additional logging in the Makefile targets in case
the command fails again so that it's clear what has happened, and not return
cryptic 404 errors again.
Ideally, we should be able to inspect the output of the curl command, but since
the output value of the entire bash script is fed into the variable, it is a bit
difficult to print debug output.
If this still gives error, then we need to either use a cached artifact somehow
or add additional logging and add a retry logic on top of it.
* fix mistake
* new job type created that checks for expired mobile sessions and pushes notifications.
* only send session expired notifications if ExtendSessionLengthWithActivity is enabled.
* includes schema change: field added to Sessions table
* Optimize reactions table
Change reactions primary key to (PostId, UserId, EmojiName) so fetching
reactions for post will use primary key lookup instead of table scan.
* fix db version
* review fixes
* update database schema in scripts/
* Updating build and release process to include externally built mmctl
* Missed last commit before reset
* rename script to align with purpose
* Moving get release script to scripts dir
* Streamlined script to find the correct latest version of an externally built binary from a GitHub Release
* Updated variable names to match
* Updating vars to match
* Updating parameters to named variables for clarity
* Update scripts/get_latest_release.sh
Co-Authored-By: Elisabeth Kulzer <elikul@elikul.de>
Co-authored-by: Elisabeth Kulzer <elikul@elikul.de>
* Updating build and release process to include externally built mmctl
* Missed last commit before reset
* rename script to align with purpose
* Moving get release script to scripts dir
* MM-16888: fix missing indexes
As part of https://mattermost.atlassian.net/browse/MM-16888, we discovered and fixed a number of column and index mismatches between the canonical (i.e. created from scratch) and migrated schemas (i.e migrated from 5.0 through 5.16).
Unfortunately, the migration to fix same was added to `UpgradeDatabaseToVersion514` but never cherry picked to the pending v5.14 release at the time. Customers who upgraded to v5.14 or v5.15 and then get this code as part of v5.16 will never run that migration. Copy it to the UpgradeDatabaseToVersion516 accordingly.
* avoid fixing ChannelMembers.SchemeGuest on MySQL
* synchronize .circleci/config.yml with scripts/mysql-migration-test.sh
* fix circleci invocation
* additional logging on diff
* update build/Jenkinsfile.pr too!
* replace dockerhost with localhost
* remove uneeded setup-max build step (no more dockerhost)
* changes as recommended by @cpanato
* make clean-docker with docker-compose
* added ports to docker-compose.yml (needed for osx). ignore error for ldapadd (when already exists)
* add clean-old-docker to legacy.mk
* docker-compose stop instead of down for `make stop-docker`
* MM-16543 Add mocking for einterfaces packages
* MM-16543 Fix Elasticsearch only returning one page of results
* Remove license checks for einterface mocks
* MM-14030 Add initial CountFrames
* MM-14030 Add format and frame count to image metadata
* Fix tests and stop using image dimensions from OpenGraph
* Fix copyright header
* Move license to NOTICE.txt
* [MM-13341] Add an upgrade test to verify schema
Upgrade test is done by following steps:
1. A mysql dump generated from v4.10.0 is imported in a database and is ran through migration code
2. Another fresh database is generated without any initial data, which gets latest schema.
3. Diff between these 2 databases is generated using mysqldiff tool from mysql utilities
For db setup, version cli command is used which takes care of migration or setup of fresh db
* Using 5.7 tag for mysql docker image which is already used for db setup
* Starting docker containers for db before running tests
* Using db from v5.0.0 for migration test
* Added migration test for psql and made some improvements in running commands in docker
* Add postgres db dump file
* Updated message
* moved dump files to scripts and using error code from diff command
* MM-12083: unittest using externally managed database
* cherry-pick Makefile changes from @cpanato
* Jenkins changes (#9915)
* add docker compose
* udpate
* when using minio dont need to set the region
* update
* add wait for it script
* using old minio
* add new jenkins file
* update makefile
* add dockerfile
* rename the docker-compose proj to avoid colision (#9917)
* rename the docker-compose proj to avoid colision
* enable debug
* enable debug to double checkt the branchs and fix docker-compose name (#9919)
* add ee hash to check (#9920)
* fix name (#9921)
* update jenkins file to push from branch and prs
* if a new push comes in stop the running build
* split mysql and postgres variables
* add script to run jenkins-like env in local dev env
* update docker-compose project name to use uuid to make it more randon
* fix DCNAME definition
* update elasticsearch docker image
* revert test
* tidy up stages, and wait for mysql differently
* update docker image and add check for postgres
* checking if is ready
* update docker compose to have a wait for deps
* add readme and rename dockerfile
* fix -unittest setup
* using mm docker image
* restore parallel unit tests at the package level
Spin up a dedicated database for each package under test to avoid races in accessing the same tables.
Simplify the interface for configuring the test database to just a DSN instead of multiple exports for each field.
* try to work around root mysql access in CI
* update local-test-env.sh too
* MYSQL_ROOT_HOST: %
* fix missing quotes
* setting some memory limits for mysql
* revert memory docker compose does not support
* fix env name for postgres
* expose errors in app/export_test.go
* fix test label, better error checking on teardown
* increase query timeout for tests
* fix export_test
* update local dev script
* add configurable mysql root passwd
* testlib: introduce and leverage
This doesn't yet factor out the individual test helpers: many packages
still rely on `api4` directly to do this, but now wire up the test store
setup through this package. `app` and `store`, in particular, don't use
`testlib` because of circular dependencies at the moment.
* cmd: command_test.go: use api4 testlib
* cmd: plugin_test.go: remove dependence on test-config.json
* cmd: config_test.go use configured database settings
* ensure test-(te|ee) exit with status code
* test-server: run all tests, deprecating test-te/test-ee
* cmd/mattermost/commands: fix unit tests
Instead of relying on (and modifying) a config.json found in the current path, explicitly create a temporary one from defaults for each test. This was likely the source of various bugs over time, but specifically allows us to override the SqlSettings to point at the configured test database for all tests simultaneously.
* wrap run/check into a test helper
It was insufficient to set a config for each invocation of CheckCommand or RunCommand: some tests relied on the config having changed in a subsequent assertion. Instead, create a new test helper embedding api4.TestHelper. This has the nice advantage of cleaning up all the teardown.
* additional TestConfigGet granularity
* customized config path to avoid default location
* be explicit if the storetest initialization fails
* generate safe coverprofile names in the presence of subtests
* additional TestConfigShow granularity
* fix permission_test.go typo
* fix webhook tests
* actually flag.Parse() to skip database setup on os.Execed tests
* fix recent regression in #9962, not caught by unit tests
