mattermost/server/channels/jobs/export_process/worker.go

84 lines
2.6 KiB
Go
Raw Permalink Normal View History

// Copyright (c) 2015-present Mattermost, Inc. All Rights Reserved.
// See LICENSE.txt for license information.
package export_process
import (
"context"
"io"
"path/filepath"
"github.com/mattermost/mattermost/server/public/model"
"github.com/mattermost/mattermost/server/public/shared/configservice"
"github.com/mattermost/mattermost/server/public/shared/mlog"
"github.com/mattermost/mattermost/server/public/shared/request"
"github.com/mattermost/mattermost/server/v8/channels/jobs"
)
Working on refactoring jobs service (#19205) * Working on refactoring jobs service * Making more consistent with the previous existing code * Remove no longer needed functions * Making a base PeridicScheduler to use it in most of the schedulers implementations * Removing accidental complexity from on of the jobs * Removing accidental complexity from expirynotify * Fixing compilation from previous commit * Remove accidental complexity from the export_delete job * Simplifying the workers by making a reusable worker * Using simple worker for export_delete job * Simpliying export process job * Simpliying extract content job * Simpliying import delete job * Simpliying import process job * Simpliying product noticies job * Simpliying fix crt channel unreads job (only removing the uneeded register function) * Simpliying migrations job (only removing the uneeded register function) * fixup * Simpliying plugins job (only removing the uneeded register function) * Simpliying bleve indexing job (only removing the uneeded register function) * Simpliying resend invitation email job (only removing the uneeded register function) * Fixing tests * Simplifying migration tests infrastructure * Adding missed license to files * Adding an empty file to imports package to ensure this package exist even without enterprise repo * Regenerating einterfaces mocks * Adding missed license to files * Updating i18n/en.json file * help fixing enterprise tests compilation * Adding new DailyScheduler * Fixing typo and changing the waitTime type for periodic sechduler * Making the daily scheduler more generic * Adding comments to clarify not used parameters in interface scheduler interface implementations * Using merror to handle multiple errors in jobs workers * Fixing linter errors * Addressing PR review comments * Reverting go.tools.mod changes * Removing the static check for worker type in the model (moving it to the insertion of new jobs * Moving migrations job to the jobs directory * Fixing (and improving a bit) tests * Apply suggestions from code review Co-authored-by: Doug Lauder <wiggin77@warpmail.net> * Fixing enterprise tests * Removing unneeded InitWorkers/InitSchedulers calls * Fix expirenotify job when error happens * Fixing govet errors Co-authored-by: Mattermod <mattermod@users.noreply.github.com> Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
2022-02-14 12:21:18 -05:00
type AppIface interface {
configservice.ConfigService
WriteExportFileContext(ctx context.Context, fr io.Reader, path string) (int64, *model.AppError)
BulkExport(rctx request.CTX, writer io.Writer, outPath string, job *model.Job, opts model.BulkExportOpts) *model.AppError
Log() *mlog.Logger
}
func MakeWorker(jobServer *jobs.JobServer, app AppIface) *jobs.SimpleWorker {
const workerName = "ExportProcess"
Working on refactoring jobs service (#19205) * Working on refactoring jobs service * Making more consistent with the previous existing code * Remove no longer needed functions * Making a base PeridicScheduler to use it in most of the schedulers implementations * Removing accidental complexity from on of the jobs * Removing accidental complexity from expirynotify * Fixing compilation from previous commit * Remove accidental complexity from the export_delete job * Simplifying the workers by making a reusable worker * Using simple worker for export_delete job * Simpliying export process job * Simpliying extract content job * Simpliying import delete job * Simpliying import process job * Simpliying product noticies job * Simpliying fix crt channel unreads job (only removing the uneeded register function) * Simpliying migrations job (only removing the uneeded register function) * fixup * Simpliying plugins job (only removing the uneeded register function) * Simpliying bleve indexing job (only removing the uneeded register function) * Simpliying resend invitation email job (only removing the uneeded register function) * Fixing tests * Simplifying migration tests infrastructure * Adding missed license to files * Adding an empty file to imports package to ensure this package exist even without enterprise repo * Regenerating einterfaces mocks * Adding missed license to files * Updating i18n/en.json file * help fixing enterprise tests compilation * Adding new DailyScheduler * Fixing typo and changing the waitTime type for periodic sechduler * Making the daily scheduler more generic * Adding comments to clarify not used parameters in interface scheduler interface implementations * Using merror to handle multiple errors in jobs workers * Fixing linter errors * Addressing PR review comments * Reverting go.tools.mod changes * Removing the static check for worker type in the model (moving it to the insertion of new jobs * Moving migrations job to the jobs directory * Fixing (and improving a bit) tests * Apply suggestions from code review Co-authored-by: Doug Lauder <wiggin77@warpmail.net> * Fixing enterprise tests * Removing unneeded InitWorkers/InitSchedulers calls * Fix expirenotify job when error happens * Fixing govet errors Co-authored-by: Mattermod <mattermod@users.noreply.github.com> Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
2022-02-14 12:21:18 -05:00
isEnabled := func(cfg *model.Config) bool { return true }
execute := func(logger mlog.LoggerIFace, job *model.Job) error {
defer jobServer.HandleJobPanic(logger, job)
Working on refactoring jobs service (#19205) * Working on refactoring jobs service * Making more consistent with the previous existing code * Remove no longer needed functions * Making a base PeridicScheduler to use it in most of the schedulers implementations * Removing accidental complexity from on of the jobs * Removing accidental complexity from expirynotify * Fixing compilation from previous commit * Remove accidental complexity from the export_delete job * Simplifying the workers by making a reusable worker * Using simple worker for export_delete job * Simpliying export process job * Simpliying extract content job * Simpliying import delete job * Simpliying import process job * Simpliying product noticies job * Simpliying fix crt channel unreads job (only removing the uneeded register function) * Simpliying migrations job (only removing the uneeded register function) * fixup * Simpliying plugins job (only removing the uneeded register function) * Simpliying bleve indexing job (only removing the uneeded register function) * Simpliying resend invitation email job (only removing the uneeded register function) * Fixing tests * Simplifying migration tests infrastructure * Adding missed license to files * Adding an empty file to imports package to ensure this package exist even without enterprise repo * Regenerating einterfaces mocks * Adding missed license to files * Updating i18n/en.json file * help fixing enterprise tests compilation * Adding new DailyScheduler * Fixing typo and changing the waitTime type for periodic sechduler * Making the daily scheduler more generic * Adding comments to clarify not used parameters in interface scheduler interface implementations * Using merror to handle multiple errors in jobs workers * Fixing linter errors * Addressing PR review comments * Reverting go.tools.mod changes * Removing the static check for worker type in the model (moving it to the insertion of new jobs * Moving migrations job to the jobs directory * Fixing (and improving a bit) tests * Apply suggestions from code review Co-authored-by: Doug Lauder <wiggin77@warpmail.net> * Fixing enterprise tests * Removing unneeded InitWorkers/InitSchedulers calls * Fix expirenotify job when error happens * Fixing govet errors Co-authored-by: Mattermod <mattermod@users.noreply.github.com> Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
2022-02-14 12:21:18 -05:00
opts := model.BulkExportOpts{
CreateArchive: true,
}
Working on refactoring jobs service (#19205) * Working on refactoring jobs service * Making more consistent with the previous existing code * Remove no longer needed functions * Making a base PeridicScheduler to use it in most of the schedulers implementations * Removing accidental complexity from on of the jobs * Removing accidental complexity from expirynotify * Fixing compilation from previous commit * Remove accidental complexity from the export_delete job * Simplifying the workers by making a reusable worker * Using simple worker for export_delete job * Simpliying export process job * Simpliying extract content job * Simpliying import delete job * Simpliying import process job * Simpliying product noticies job * Simpliying fix crt channel unreads job (only removing the uneeded register function) * Simpliying migrations job (only removing the uneeded register function) * fixup * Simpliying plugins job (only removing the uneeded register function) * Simpliying bleve indexing job (only removing the uneeded register function) * Simpliying resend invitation email job (only removing the uneeded register function) * Fixing tests * Simplifying migration tests infrastructure * Adding missed license to files * Adding an empty file to imports package to ensure this package exist even without enterprise repo * Regenerating einterfaces mocks * Adding missed license to files * Updating i18n/en.json file * help fixing enterprise tests compilation * Adding new DailyScheduler * Fixing typo and changing the waitTime type for periodic sechduler * Making the daily scheduler more generic * Adding comments to clarify not used parameters in interface scheduler interface implementations * Using merror to handle multiple errors in jobs workers * Fixing linter errors * Addressing PR review comments * Reverting go.tools.mod changes * Removing the static check for worker type in the model (moving it to the insertion of new jobs * Moving migrations job to the jobs directory * Fixing (and improving a bit) tests * Apply suggestions from code review Co-authored-by: Doug Lauder <wiggin77@warpmail.net> * Fixing enterprise tests * Removing unneeded InitWorkers/InitSchedulers calls * Fix expirenotify job when error happens * Fixing govet errors Co-authored-by: Mattermod <mattermod@users.noreply.github.com> Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
2022-02-14 12:21:18 -05:00
includeAttachments, ok := job.Data["include_attachments"]
if ok && includeAttachments == "true" {
opts.IncludeAttachments = true
}
includeArchivedChannels, ok := job.Data["include_archived_channels"]
if ok && includeArchivedChannels == "true" {
opts.IncludeArchivedChannels = true
}
includeProfilePictures, ok := job.Data["include_profile_pictures"]
if ok && includeProfilePictures == "true" {
opts.IncludeProfilePictures = true
}
includeRolesAndSchemes, ok := job.Data["include_roles_and_schemes"]
if ok && includeRolesAndSchemes == "true" {
opts.IncludeRolesAndSchemes = true
}
Working on refactoring jobs service (#19205) * Working on refactoring jobs service * Making more consistent with the previous existing code * Remove no longer needed functions * Making a base PeridicScheduler to use it in most of the schedulers implementations * Removing accidental complexity from on of the jobs * Removing accidental complexity from expirynotify * Fixing compilation from previous commit * Remove accidental complexity from the export_delete job * Simplifying the workers by making a reusable worker * Using simple worker for export_delete job * Simpliying export process job * Simpliying extract content job * Simpliying import delete job * Simpliying import process job * Simpliying product noticies job * Simpliying fix crt channel unreads job (only removing the uneeded register function) * Simpliying migrations job (only removing the uneeded register function) * fixup * Simpliying plugins job (only removing the uneeded register function) * Simpliying bleve indexing job (only removing the uneeded register function) * Simpliying resend invitation email job (only removing the uneeded register function) * Fixing tests * Simplifying migration tests infrastructure * Adding missed license to files * Adding an empty file to imports package to ensure this package exist even without enterprise repo * Regenerating einterfaces mocks * Adding missed license to files * Updating i18n/en.json file * help fixing enterprise tests compilation * Adding new DailyScheduler * Fixing typo and changing the waitTime type for periodic sechduler * Making the daily scheduler more generic * Adding comments to clarify not used parameters in interface scheduler interface implementations * Using merror to handle multiple errors in jobs workers * Fixing linter errors * Addressing PR review comments * Reverting go.tools.mod changes * Removing the static check for worker type in the model (moving it to the insertion of new jobs * Moving migrations job to the jobs directory * Fixing (and improving a bit) tests * Apply suggestions from code review Co-authored-by: Doug Lauder <wiggin77@warpmail.net> * Fixing enterprise tests * Removing unneeded InitWorkers/InitSchedulers calls * Fix expirenotify job when error happens * Fixing govet errors Co-authored-by: Mattermod <mattermod@users.noreply.github.com> Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
2022-02-14 12:21:18 -05:00
outPath := *app.Config().ExportSettings.Directory
exportFilename := job.Id + "_export.zip"
Working on refactoring jobs service (#19205) * Working on refactoring jobs service * Making more consistent with the previous existing code * Remove no longer needed functions * Making a base PeridicScheduler to use it in most of the schedulers implementations * Removing accidental complexity from on of the jobs * Removing accidental complexity from expirynotify * Fixing compilation from previous commit * Remove accidental complexity from the export_delete job * Simplifying the workers by making a reusable worker * Using simple worker for export_delete job * Simpliying export process job * Simpliying extract content job * Simpliying import delete job * Simpliying import process job * Simpliying product noticies job * Simpliying fix crt channel unreads job (only removing the uneeded register function) * Simpliying migrations job (only removing the uneeded register function) * fixup * Simpliying plugins job (only removing the uneeded register function) * Simpliying bleve indexing job (only removing the uneeded register function) * Simpliying resend invitation email job (only removing the uneeded register function) * Fixing tests * Simplifying migration tests infrastructure * Adding missed license to files * Adding an empty file to imports package to ensure this package exist even without enterprise repo * Regenerating einterfaces mocks * Adding missed license to files * Updating i18n/en.json file * help fixing enterprise tests compilation * Adding new DailyScheduler * Fixing typo and changing the waitTime type for periodic sechduler * Making the daily scheduler more generic * Adding comments to clarify not used parameters in interface scheduler interface implementations * Using merror to handle multiple errors in jobs workers * Fixing linter errors * Addressing PR review comments * Reverting go.tools.mod changes * Removing the static check for worker type in the model (moving it to the insertion of new jobs * Moving migrations job to the jobs directory * Fixing (and improving a bit) tests * Apply suggestions from code review Co-authored-by: Doug Lauder <wiggin77@warpmail.net> * Fixing enterprise tests * Removing unneeded InitWorkers/InitSchedulers calls * Fix expirenotify job when error happens * Fixing govet errors Co-authored-by: Mattermod <mattermod@users.noreply.github.com> Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
2022-02-14 12:21:18 -05:00
rd, wr := io.Pipe()
Working on refactoring jobs service (#19205) * Working on refactoring jobs service * Making more consistent with the previous existing code * Remove no longer needed functions * Making a base PeridicScheduler to use it in most of the schedulers implementations * Removing accidental complexity from on of the jobs * Removing accidental complexity from expirynotify * Fixing compilation from previous commit * Remove accidental complexity from the export_delete job * Simplifying the workers by making a reusable worker * Using simple worker for export_delete job * Simpliying export process job * Simpliying extract content job * Simpliying import delete job * Simpliying import process job * Simpliying product noticies job * Simpliying fix crt channel unreads job (only removing the uneeded register function) * Simpliying migrations job (only removing the uneeded register function) * fixup * Simpliying plugins job (only removing the uneeded register function) * Simpliying bleve indexing job (only removing the uneeded register function) * Simpliying resend invitation email job (only removing the uneeded register function) * Fixing tests * Simplifying migration tests infrastructure * Adding missed license to files * Adding an empty file to imports package to ensure this package exist even without enterprise repo * Regenerating einterfaces mocks * Adding missed license to files * Updating i18n/en.json file * help fixing enterprise tests compilation * Adding new DailyScheduler * Fixing typo and changing the waitTime type for periodic sechduler * Making the daily scheduler more generic * Adding comments to clarify not used parameters in interface scheduler interface implementations * Using merror to handle multiple errors in jobs workers * Fixing linter errors * Addressing PR review comments * Reverting go.tools.mod changes * Removing the static check for worker type in the model (moving it to the insertion of new jobs * Moving migrations job to the jobs directory * Fixing (and improving a bit) tests * Apply suggestions from code review Co-authored-by: Doug Lauder <wiggin77@warpmail.net> * Fixing enterprise tests * Removing unneeded InitWorkers/InitSchedulers calls * Fix expirenotify job when error happens * Fixing govet errors Co-authored-by: Mattermod <mattermod@users.noreply.github.com> Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
2022-02-14 12:21:18 -05:00
go func() {
_, appErr := app.WriteExportFileContext(context.Background(), rd, filepath.Join(outPath, exportFilename))
if appErr != nil {
// we close the reader here to prevent a deadlock when the bulk exporter tries to
// write into the pipe while app.WriteFile has already returned. The error will be
// returned by the writer part of the pipe when app.BulkExport tries to call
// wr.Write() on it.
rd.CloseWithError(appErr) // CloseWithError never returns an error
}
Working on refactoring jobs service (#19205) * Working on refactoring jobs service * Making more consistent with the previous existing code * Remove no longer needed functions * Making a base PeridicScheduler to use it in most of the schedulers implementations * Removing accidental complexity from on of the jobs * Removing accidental complexity from expirynotify * Fixing compilation from previous commit * Remove accidental complexity from the export_delete job * Simplifying the workers by making a reusable worker * Using simple worker for export_delete job * Simpliying export process job * Simpliying extract content job * Simpliying import delete job * Simpliying import process job * Simpliying product noticies job * Simpliying fix crt channel unreads job (only removing the uneeded register function) * Simpliying migrations job (only removing the uneeded register function) * fixup * Simpliying plugins job (only removing the uneeded register function) * Simpliying bleve indexing job (only removing the uneeded register function) * Simpliying resend invitation email job (only removing the uneeded register function) * Fixing tests * Simplifying migration tests infrastructure * Adding missed license to files * Adding an empty file to imports package to ensure this package exist even without enterprise repo * Regenerating einterfaces mocks * Adding missed license to files * Updating i18n/en.json file * help fixing enterprise tests compilation * Adding new DailyScheduler * Fixing typo and changing the waitTime type for periodic sechduler * Making the daily scheduler more generic * Adding comments to clarify not used parameters in interface scheduler interface implementations * Using merror to handle multiple errors in jobs workers * Fixing linter errors * Addressing PR review comments * Reverting go.tools.mod changes * Removing the static check for worker type in the model (moving it to the insertion of new jobs * Moving migrations job to the jobs directory * Fixing (and improving a bit) tests * Apply suggestions from code review Co-authored-by: Doug Lauder <wiggin77@warpmail.net> * Fixing enterprise tests * Removing unneeded InitWorkers/InitSchedulers calls * Fix expirenotify job when error happens * Fixing govet errors Co-authored-by: Mattermod <mattermod@users.noreply.github.com> Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
2022-02-14 12:21:18 -05:00
}()
appErr := app.BulkExport(request.EmptyContext(logger), wr, outPath, job, opts)
wr.Close() // Close never returns an error
Working on refactoring jobs service (#19205) * Working on refactoring jobs service * Making more consistent with the previous existing code * Remove no longer needed functions * Making a base PeridicScheduler to use it in most of the schedulers implementations * Removing accidental complexity from on of the jobs * Removing accidental complexity from expirynotify * Fixing compilation from previous commit * Remove accidental complexity from the export_delete job * Simplifying the workers by making a reusable worker * Using simple worker for export_delete job * Simpliying export process job * Simpliying extract content job * Simpliying import delete job * Simpliying import process job * Simpliying product noticies job * Simpliying fix crt channel unreads job (only removing the uneeded register function) * Simpliying migrations job (only removing the uneeded register function) * fixup * Simpliying plugins job (only removing the uneeded register function) * Simpliying bleve indexing job (only removing the uneeded register function) * Simpliying resend invitation email job (only removing the uneeded register function) * Fixing tests * Simplifying migration tests infrastructure * Adding missed license to files * Adding an empty file to imports package to ensure this package exist even without enterprise repo * Regenerating einterfaces mocks * Adding missed license to files * Updating i18n/en.json file * help fixing enterprise tests compilation * Adding new DailyScheduler * Fixing typo and changing the waitTime type for periodic sechduler * Making the daily scheduler more generic * Adding comments to clarify not used parameters in interface scheduler interface implementations * Using merror to handle multiple errors in jobs workers * Fixing linter errors * Addressing PR review comments * Reverting go.tools.mod changes * Removing the static check for worker type in the model (moving it to the insertion of new jobs * Moving migrations job to the jobs directory * Fixing (and improving a bit) tests * Apply suggestions from code review Co-authored-by: Doug Lauder <wiggin77@warpmail.net> * Fixing enterprise tests * Removing unneeded InitWorkers/InitSchedulers calls * Fix expirenotify job when error happens * Fixing govet errors Co-authored-by: Mattermod <mattermod@users.noreply.github.com> Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
2022-02-14 12:21:18 -05:00
if appErr != nil {
return appErr
}
Working on refactoring jobs service (#19205) * Working on refactoring jobs service * Making more consistent with the previous existing code * Remove no longer needed functions * Making a base PeridicScheduler to use it in most of the schedulers implementations * Removing accidental complexity from on of the jobs * Removing accidental complexity from expirynotify * Fixing compilation from previous commit * Remove accidental complexity from the export_delete job * Simplifying the workers by making a reusable worker * Using simple worker for export_delete job * Simpliying export process job * Simpliying extract content job * Simpliying import delete job * Simpliying import process job * Simpliying product noticies job * Simpliying fix crt channel unreads job (only removing the uneeded register function) * Simpliying migrations job (only removing the uneeded register function) * fixup * Simpliying plugins job (only removing the uneeded register function) * Simpliying bleve indexing job (only removing the uneeded register function) * Simpliying resend invitation email job (only removing the uneeded register function) * Fixing tests * Simplifying migration tests infrastructure * Adding missed license to files * Adding an empty file to imports package to ensure this package exist even without enterprise repo * Regenerating einterfaces mocks * Adding missed license to files * Updating i18n/en.json file * help fixing enterprise tests compilation * Adding new DailyScheduler * Fixing typo and changing the waitTime type for periodic sechduler * Making the daily scheduler more generic * Adding comments to clarify not used parameters in interface scheduler interface implementations * Using merror to handle multiple errors in jobs workers * Fixing linter errors * Addressing PR review comments * Reverting go.tools.mod changes * Removing the static check for worker type in the model (moving it to the insertion of new jobs * Moving migrations job to the jobs directory * Fixing (and improving a bit) tests * Apply suggestions from code review Co-authored-by: Doug Lauder <wiggin77@warpmail.net> * Fixing enterprise tests * Removing unneeded InitWorkers/InitSchedulers calls * Fix expirenotify job when error happens * Fixing govet errors Co-authored-by: Mattermod <mattermod@users.noreply.github.com> Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
2022-02-14 12:21:18 -05:00
return nil
}
worker := jobs.NewSimpleWorker(workerName, jobServer, execute, isEnabled)
Working on refactoring jobs service (#19205) * Working on refactoring jobs service * Making more consistent with the previous existing code * Remove no longer needed functions * Making a base PeridicScheduler to use it in most of the schedulers implementations * Removing accidental complexity from on of the jobs * Removing accidental complexity from expirynotify * Fixing compilation from previous commit * Remove accidental complexity from the export_delete job * Simplifying the workers by making a reusable worker * Using simple worker for export_delete job * Simpliying export process job * Simpliying extract content job * Simpliying import delete job * Simpliying import process job * Simpliying product noticies job * Simpliying fix crt channel unreads job (only removing the uneeded register function) * Simpliying migrations job (only removing the uneeded register function) * fixup * Simpliying plugins job (only removing the uneeded register function) * Simpliying bleve indexing job (only removing the uneeded register function) * Simpliying resend invitation email job (only removing the uneeded register function) * Fixing tests * Simplifying migration tests infrastructure * Adding missed license to files * Adding an empty file to imports package to ensure this package exist even without enterprise repo * Regenerating einterfaces mocks * Adding missed license to files * Updating i18n/en.json file * help fixing enterprise tests compilation * Adding new DailyScheduler * Fixing typo and changing the waitTime type for periodic sechduler * Making the daily scheduler more generic * Adding comments to clarify not used parameters in interface scheduler interface implementations * Using merror to handle multiple errors in jobs workers * Fixing linter errors * Addressing PR review comments * Reverting go.tools.mod changes * Removing the static check for worker type in the model (moving it to the insertion of new jobs * Moving migrations job to the jobs directory * Fixing (and improving a bit) tests * Apply suggestions from code review Co-authored-by: Doug Lauder <wiggin77@warpmail.net> * Fixing enterprise tests * Removing unneeded InitWorkers/InitSchedulers calls * Fix expirenotify job when error happens * Fixing govet errors Co-authored-by: Mattermod <mattermod@users.noreply.github.com> Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
2022-02-14 12:21:18 -05:00
return worker
}