diff --git a/CHANGELOG.md b/CHANGELOG.md index 395c7fc..7704548 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,8 +11,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed - Update to Odoo Community 12.0 +## [1.0.1] - 2020-07-14 +### Added +- Improve documentation with development and production environment +- Improve documentation with git repo +- Move default.xml manifest to root, the default location +- Support default.stage.xml to update prod with dev +- Feature to show diff between manifest, between repo of different manifest +- Update manifest + - Muk theme in erplibre_base + - Add draft account invoice approbation in portal + - New module sale_fix_update_price_unit_when_update_qty + - New module account_invoice_approbation + - New module sale_margin_editor + ### Fixed -- Web module display +- Production installation with git_repo ## [1.0.0] - 2020-07-04 ### Added @@ -41,7 +55,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed - Support only python3.6 and python3.7, python3.8 causes error in runtime. -[Unreleased]: http://git.erplibre.ca/ERPLibre/compare/v1.0.0...HEAD +[Unreleased]: http://git.erplibre.ca/ERPLibre/compare/v1.0.1...HEAD +[1.0.1]: http://git.erplibre.ca/ERPLibre/compare/v1.0.0...v1.0.1 [1.0.0]: http://git.erplibre.ca/ERPLibre/compare/v0.1.1...v1.0.0 [0.1.1]: http://git.erplibre.ca/ERPLibre/compare/v0.1.0...v0.1.1 [0.1.0]: http://git.erplibre.ca/ERPLibre/releases/tag/v0.1.0 diff --git a/README.md b/README.md index 0ea50a9..fe65626 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,27 @@ # ERPLibre documentation Select a guide to install your environment! +## Easy way to run locally +Clone the project +```bash +git clone http://git.erplibre.ca/ERPLibre.git +cd ERPLibre +``` +Support Ubuntu 18.04 and OSX +```bash +./script/install_dev.sh +./script/install_locally_prod.sh +``` +Update your configuration if you need to run somehere than 127.0.0.1, file config.conf +``` +#xmlrpc_interface = 127.0.0.1 +#netrpc_interface = 127.0.0.1 +``` +Run +```bash +./run.sh +``` + ## Discover guide [Guide to run ERPLibre in discover to learn it](./doc/DISCOVER.md). diff --git a/manifest/default.xml b/default.xml similarity index 88% rename from manifest/default.xml rename to default.xml index bc968b9..1396a65 100644 --- a/manifest/default.xml +++ b/default.xml @@ -22,7 +22,7 @@ - + @@ -37,35 +37,35 @@ - - + + - + - + - + - - + + - + - + @@ -73,8 +73,8 @@ - - + + @@ -82,11 +82,11 @@ - + - + @@ -96,8 +96,8 @@ - + - + diff --git a/doc/DEVELOPMENT.md b/doc/DEVELOPMENT.md index 2e45e18..01c43de 100644 --- a/doc/DEVELOPMENT.md +++ b/doc/DEVELOPMENT.md @@ -19,12 +19,23 @@ cd ERPLibre ./run.sh ``` +## Develop in Odoo repository +You need to remove +> clone-depth="1" + +from +> ./manifest/default.dev.xml + +file to be able to commit and push. +Commit temporary and regenerate with +> ./script/install_locally_dev.sh + ## Fork project to create a new project independent of ERPLibre (deprecated) ERPLibre was created by this script. It's now deprecated. Use this script when you need to fork directly from the original source. Don't use this script if you want to update from ERPLibre and follow mainstream development. ```bash -./venv/bin/python ./script/fork_project.py --github_token GITHUB_KEY --organization NAME +./script/fork_project.py --github_token GITHUB_KEY --organization NAME ``` # Fork all repo for you own organization @@ -32,7 +43,7 @@ Go to your github account and generate a token to access fork option with your u This command will fork all repos and ERPLibre to your own organization. It keeps track to ERPLibre. ```bash -./venv/bin/python ./script/fork_project_ERPLibre.py --github_token GITHUB_KEY --organization NAME +./script/fork_project_ERPLibre.py --github_token GITHUB_KEY --organization NAME ``` ## Generate manifest from csv repo @@ -40,7 +51,7 @@ Add repo in file [./source_repo_addons.csv](./source_repo_addons.csv) Execute to generate manifest of Repo ```bash -./venv/bin/python ./script/fork_project_ERPLibre.py --skip_fork +./script/fork_project_ERPLibre.py --skip_fork ``` ## Move database prod to dev @@ -52,25 +63,31 @@ Run: # TODO ```bash -./venv/bin/python ./script/git_change_remote.py +./script/git_change_remote.py ``` ## Change git url https to git This will update all urls in git format: ```bash -./venv/bin/python ./script/git_change_remote_https_to_git.py +./script/git_change_remote_https_to_git.py ``` ## Diff repo with another project Tools to display the differences between the repo and another project. ```bash -./venv/bin/python ./script/list_repo_diff.py --sync_to /path/to/project/erplibre --dyr_sync +./script/git_change_remote.py --sync_to /path/to/project/erplibre --dry_sync ``` ## Sync repo with another project Tools to synchronise the repo with another project. This will show differences and try to checkout on the same commit in all repo. ```bash -./venv/bin/python ./script/git_change_remote.py --sync_to /path/to/project/erplibre +./script/git_change_remote.py --sync_to /path/to/project/erplibre +``` + +## Diff code between manifest +To show diff between commits in different manifest +```bash +./script/git_diff_repo_manifest.py --input1 ./manifest/MANIFEST1.xml --input2 ./manifest/MANIFEST2.xml ``` ## Add repo @@ -78,10 +95,17 @@ Access to a new repo, add your URL to file [source_repo_addons.csv](../source_re Fork the repo to be able to push new code: ```bash -./venv/bin/python ./script/fork_project_ERPLibre.py +./script/fork_project_ERPLibre.py ``` To regenerate only manifest.xml. ```bash -./venv/bin/python ./script/fork_project_ERPLibre.py --skip_fork +./script/fork_project_ERPLibre.py --skip_fork +``` + +# Coding +## Create module scaffold (run in the venv) +```bash +source ./venv/bin/activate +python odoo/odoo-bin scaffold MODULE_NAME addons/REPO_NAME/ ``` diff --git a/doc/DISCOVER.md b/doc/DISCOVER.md index 7a1fc0f..7e5ca0e 100644 --- a/doc/DISCOVER.md +++ b/doc/DISCOVER.md @@ -24,8 +24,9 @@ Access to a new repo, add your URL to file [source_repo_addons.csv](../source_re Execute script: ```bash -./venv/bin/python ./script/git_repo_manifest.py -./script/install_locally_dev.sh +./script/git_repo_manifest.py +git checkout -b new_branch git commit -am "Add new repo" +./script/install_locally_dev.sh ``` [Update your repo.](./GIT_REPO.md) diff --git a/doc/GIT_REPO.md b/doc/GIT_REPO.md index a2608c1..95a6991 100644 --- a/doc/GIT_REPO.md +++ b/doc/GIT_REPO.md @@ -10,7 +10,7 @@ curl https://storage.googleapis.com/git-repo-downloads/repo > ./venv/repo ## prod ```bash -./venv/repo init -u http://git.erplibre.ca/ERPLibre -b 12.0_repo +./venv/repo init -u http://git.erplibre.ca/ERPLibre -b master ./venv/repo sync ``` @@ -37,13 +37,25 @@ Freezes all repo, from dev to prod. This will add revision git hash in the Manifest. ```bash -./venv/repo manifest -r -o ./manifest/default.xml +./venv/repo manifest -r -o ./default.xml ``` Do your commit. ```bash git commit -am "[#ticket] subject: short sentence" ``` +### Mix prod and dev to do a stage +When dev contain specific revision with default revision, you want to replace default revision by prod revision and keep specific version, do: +```bash +./script/git_merge_repo_manifest.py --input1 ./manifest/default.dev.xml --input2 ./default.xml --output ./manifest/default.staged.xml +git commit -am "Updated manifest/default.staged.xml" +git daemon --base-path=. --export-all --reuseaddr --informative-errors --verbose & + +./venv/repo init -u git://127.0.0.1:9418/ -b $(git rev-parse --abbrev-ref HEAD) -m ./manifest/default.staged.xml +./venv/repo sync -m ./manifest/default.staged.xml + +./venv/repo manifest -r -o ./default.xml +``` ## Create a dev version ```bash ./venv/repo manifest -o ./manifest/default.dev.xml @@ -51,4 +63,26 @@ git commit -am "[#ticket] subject: short sentence" Do your commit. ```bash git commit -am "[#ticket] subject: short sentence" +``` + +## Useful command +### Search all repo with specific branch name +```bash +./venv/repo forall -pc "git branch -a|grep BRANCH" +``` + +### Search missing branch in all repo +```bash +./venv/repo forall -pc 'git branch -a|(grep /BRANCH$||echo "no match")|grep "no match"' +``` + +### Search change file in all repo +```bash +./venv/repo forall -pc "git status -s" +``` + +### Clean all +Maybe, some version diverge from your manifest. Simply clean all and relaunch your installation. +```bash +./script/clean_repo_manifest.sh ``` \ No newline at end of file diff --git a/doc/HOWTO.md b/doc/HOWTO.md index b337e76..8588b44 100644 --- a/doc/HOWTO.md +++ b/doc/HOWTO.md @@ -1,3 +1,3 @@ # HOW TO ## Update changelog -Please, read documentation (keepachangelog.com)[https://keepachangelog.com] \ No newline at end of file +Please, read documentation [keepachangelog.com](https://keepachangelog.com) \ No newline at end of file diff --git a/doc/PRODUCTION.md b/doc/PRODUCTION.md index 880dfc3..e71fa4e 100644 --- a/doc/PRODUCTION.md +++ b/doc/PRODUCTION.md @@ -4,18 +4,46 @@ ### 1. Clone the project: ```bash -git clone https://git.erplibre.ca/ERPLibre.git +git clone http://git.erplibre.ca/ERPLibre.git +cd ERPLibre ``` ### 2. Modify the parameters Modify the file env_var.sh for production installation. ### 3. Execute the script: +Support Ubuntu 18.04 server ```bash -cd ERPLibre ./script/install_dev.sh ./script/install_production.sh ``` +A service is running by systemd, you can access with the DNS name, found it in env_var.sh + +### 4. SSL: +Generate a ssl certificat +```bash +sudo certbot --nginx +``` + +## Watch log +```bash +sudo systemctl -feu [EL_USER] +``` + +## Run by address ip +Comment following line in /[EL_USER]/erplibre/config.conf +``` +#xmlrpc_interface = 127.0.0.1 +#netrpc_interface = 127.0.0.1 +#proxy_mode = True +``` +Add your address ip in nginx config at server_name in /etc/nginx/sites-available/[EL_WEBSITE_NAME] + +Restart daemon: +```bash +sudo systemctl restart nginx +sudo systemctl restart [EL_USER] +``` ## Production execution ```bash @@ -24,9 +52,14 @@ cd /[EL_USER]/erplibre ``` ## Move database prod to dev - When moving database prod to your dev environment, you want to remove email servers, and install user test to test the database. Run: ```bash ./run.sh --stop-after-init -i user_test,disable_mail_server --dev all -d DATABASE ``` + +## Update production +Simply update all feature. +```bash +./run.sh --limit-time-real 99999 --stop-after-init -u all -d DATABASE +``` diff --git a/doc/RELEASE.md b/doc/RELEASE.md index 9e21f25..e73a371 100644 --- a/doc/RELEASE.md +++ b/doc/RELEASE.md @@ -3,7 +3,7 @@ A guide on how to do a release. ## Generate new prod ```bash -./venv/repo manifest -r -o ./manifest/default.xml +./venv/repo manifest -r -o ./default.xml ``` Do your commit. ```bash @@ -11,11 +11,25 @@ git commit -am "[#ticket] subject: short sentence" ``` ## Merge release -Merge your feature to master. Generate a new tag. Fill CHANGELOG.md +When ready to make a release, create a branch release/#.#.# and create a pull request to master. +Update file CHANGELOG.md and create a section with new version. +Merge it when maintener accept it. + +Add a tag on the commit on branch master with your release. When adding tag, be sure to update default.xml +```bash +git tag v#.#.# +# Push your tags +git push --tags +# Add tags for all repo +./venv/repo forall -pc "git tag ERPLibre/v#.#.#" +./venv/repo forall -pc "git push ERPLibre --tags" +# Get all difference between a tag and HEAD, to update the CHANGELOG.md +./venv/repo forall -pc "git diff ERPLibre/v#.#.#..HEAD" +``` # TIPS ## Compare diff repo with another ERPLibre project To generate a list of differences between repo git commit, do ```bash -./venv/bin/python ./script/git_change_remote.py --sync_to /path/to/directory +./script/git_change_remote.py --sync_to /path/to/directory ``` \ No newline at end of file diff --git a/doc/RUN.md b/doc/RUN.md index c1e1d0b..545af32 100644 --- a/doc/RUN.md +++ b/doc/RUN.md @@ -20,22 +20,30 @@ With arguments ### Method 2 Execute your own python script: ```bash -source ./venv/bin/activate -python odoo/odoo-bin -c config.conf --log-level debug +./run.sh --log-level debug ``` ### Update all Great idea to run it when updating Odoo, it updates database of each modules. ```bash -python odoo/odoo-bin -c config.conf -d [DATABASE] -u all --log-level debug +./run.sh -d [DATABASE] -u all --log-level debug ``` ### Update module ```bash -python odoo/odoo-bin -c config.conf -d [DATABASE] -u [module] --log-level debug +./run.sh -d [DATABASE] -u [module] --log-level debug ``` ### Test +First execution, install you requirements, choose a new database. ```bash -python odoo/odoo-bin -c config.conf -d [DATABASE] -i [module to test] --test-enable --stop-after-init --log-level=test --test-tags [module_name][tags] +./run.sh -d [DATABASE] -i [module to test] --test-enable --stop-after-init --log-level=test +``` +Execute your test on specific module. +```bash +./run.sh -d [DATABASE] -u [module to test] --test-enable --stop-after-init --log-level=test +``` +Execute your test on specific module with tags. +```bash +./run.sh -d [DATABASE] -u [module to test] --test-enable --stop-after-init --log-level=test --test-tags [module_name][tags] ``` \ No newline at end of file diff --git a/env_var.sh b/env_var.sh index d919be9..072a4de 100755 --- a/env_var.sh +++ b/env_var.sh @@ -19,5 +19,6 @@ EL_INSTALL_NGINX="True" # Set the website name EL_WEBSITE_NAME="_" EL_GITHUB_TOKEN="" -EL_MANIFEST_PROD="./manifest/default.xml" +EL_MANIFEST_PROD="./default.xml" EL_MANIFEST_DEV="./manifest/default.dev.xml" +EL_MANIFEST_EXP="./manifest/default.experimental.xml" diff --git a/manifest/default.dev.xml b/manifest/default.dev.xml index 51c95b1..c4493a2 100644 --- a/manifest/default.dev.xml +++ b/manifest/default.dev.xml @@ -22,7 +22,7 @@ - + @@ -37,35 +37,35 @@ - - + + - + - + - + - - + + - + - + @@ -73,8 +73,8 @@ - - + + @@ -82,11 +82,11 @@ - + - + @@ -96,8 +96,8 @@ - + - + diff --git a/manifest/default.experimental.xml b/manifest/default.experimental.xml new file mode 100644 index 0000000..4450cfc --- /dev/null +++ b/manifest/default.experimental.xml @@ -0,0 +1,103 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/manifest/default.staged.xml b/manifest/default.staged.xml new file mode 100644 index 0000000..5a83766 --- /dev/null +++ b/manifest/default.staged.xml @@ -0,0 +1,103 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/run.sh b/run.sh index 28b1df3..a4600ea 100755 --- a/run.sh +++ b/run.sh @@ -1,2 +1,3 @@ #!/usr/bin/env bash -./venv/bin/python ./odoo/odoo-bin -c ./config.conf $@ +source ./venv/bin/activate +python3 ./odoo/odoo-bin -c ./config.conf --limit-time-real 99999 --limit-time-cpu 99999 $@ diff --git a/script/clean_repo_manifest.sh b/script/clean_repo_manifest.sh new file mode 100755 index 0000000..b0833d0 --- /dev/null +++ b/script/clean_repo_manifest.sh @@ -0,0 +1,4 @@ +#!/usr/bin/env bash + +rm -rf ./odoo ./addons/* +mkdir -p addons/addons diff --git a/script/fork_project.py b/script/fork_project.py index c2396a7..b761d83 100644 --- a/script/fork_project.py +++ b/script/fork_project.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!./venv/bin/python import os import sys import argparse diff --git a/script/fork_project_ERPLibre.py b/script/fork_project_ERPLibre.py index fb25b19..3a292cc 100644 --- a/script/fork_project_ERPLibre.py +++ b/script/fork_project_ERPLibre.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!./venv/bin/python import os import sys import argparse @@ -127,7 +127,8 @@ def main(): # Update origin to new repo # git_tool.generate_git_modules(lst_repo_organization, repo_path=config.dir) - git_tool.generate_repo_manifest(lst_repo_organization, repo_path=config.dir) + git_tool.generate_repo_manifest(lst_repo_organization, + output=f"{config.dir}manifest/default.dev.xml") git_tool.generate_install_locally() diff --git a/script/git_change_remote.py b/script/git_change_remote.py index 91b1282..9dbf4f0 100755 --- a/script/git_change_remote.py +++ b/script/git_change_remote.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!./venv/bin/python import os import sys import argparse diff --git a/script/git_change_remote_https_to_git.py b/script/git_change_remote_https_to_git.py index ebdbeec..134177b 100755 --- a/script/git_change_remote_https_to_git.py +++ b/script/git_change_remote_https_to_git.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!./venv/bin/python # © 2020 TechnoLibre (http://www.technolibre.ca) # License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl). diff --git a/script/git_diff_repo_manifest.py b/script/git_diff_repo_manifest.py new file mode 100755 index 0000000..b969bd4 --- /dev/null +++ b/script/git_diff_repo_manifest.py @@ -0,0 +1,95 @@ +#!./venv/bin/python +import os +import sys +import argparse +import logging +from git import Repo + +new_path = os.path.normpath(os.path.join(os.path.dirname(__file__), '..')) +sys.path.append(new_path) + +from script.git_tool import GitTool + +_logger = logging.getLogger(__name__) + + +def get_config(): + """Parse command line arguments, extracting the config file name, + returning the union of config file and command line arguments + + :return: dict of config file settings and command line arguments + """ + # TODO update description + parser = argparse.ArgumentParser( + formatter_class=argparse.RawDescriptionHelpFormatter, + description="""Get git diff between manifest repo revision, + diff revision input1 to input2 """, + epilog='''\ +''' + ) + parser.add_argument('--input1', required=True, + help="Compare input1 to input2. Input1 is older config.") + parser.add_argument('--input2', required=True, + help="Compare input1 to input2. Input2 is newer config.") + # parser.add_argument('--clear', action="store_true", + # help="Create a new manifest and clear old configuration.") + args = parser.parse_args() + return args + + +def main(): + config = get_config() + git_tool = GitTool() + + dct_remote_1, dct_project_1, default_remote_1 = git_tool.get_manifest_xml_info( + filename=config.input1, add_root=True) + dct_remote_2, dct_project_2, default_remote_2 = git_tool.get_manifest_xml_info( + filename=config.input2, add_root=True) + + set_project_1 = set(dct_project_1.keys()) + set_project_2 = set(dct_project_2.keys()) + lst_same_name_normalize = set_project_1.intersection(set_project_2) + lst_missing_name_normalize = set_project_2.difference(set_project_1) + lst_over_name_normalize = set_project_1.difference(set_project_2) + + i = 0 + total = len(lst_same_name_normalize) + for key in lst_missing_name_normalize: + i += 1 + print(f"{i}/{total} - {key} from input1 not in input2.") + + i = 0 + total = len(lst_over_name_normalize) + for key in lst_over_name_normalize: + i += 1 + print(f"{i}/{total} - {key} from input2 not in input1.") + + i = 0 + total = len(lst_same_name_normalize) + for key in lst_same_name_normalize: + value1 = dct_project_1.get(key) + value2 = dct_project_2.get(key) + old_revision = value1.get("@revision", git_tool.default_branch) + new_revision = value2.get("@revision", git_tool.default_branch) + + path1 = value1.get("@path") + path2 = value2.get("@path") + if path1 != path2: + print(f"WARNING id {i}, path of git are different. " + f"Input1 {path1}, input2 {path2}") + continue + + i += 1 + result = "same" if old_revision == new_revision else "diff" + print(f"{i}/{total} - {result} - " + f"{path1} {key} old {old_revision} new {new_revision}") + default_arg = [f"{old_revision}..{new_revision}"] + if old_revision != new_revision: + # get git diff + repo = Repo(path1) + status = repo.git.diff(*default_arg) + print(status) + + +if __name__ == '__main__': + main() diff --git a/script/git_merge_repo_manifest.py b/script/git_merge_repo_manifest.py new file mode 100755 index 0000000..877b836 --- /dev/null +++ b/script/git_merge_repo_manifest.py @@ -0,0 +1,68 @@ +#!./venv/bin/python +import os +import sys +import argparse +import logging +import copy + +new_path = os.path.normpath(os.path.join(os.path.dirname(__file__), '..')) +sys.path.append(new_path) + +from script.git_tool import GitTool + +_logger = logging.getLogger(__name__) + + +def get_config(): + """Parse command line arguments, extracting the config file name, + returning the union of config file and command line arguments + + :return: dict of config file settings and command line arguments + """ + # TODO update description + parser = argparse.ArgumentParser( + formatter_class=argparse.RawDescriptionHelpFormatter, + description="""Replace revision field in input2 from input1 if existing, create an output of new manifest.""", + epilog='''\ +''' + ) + parser.add_argument('--input1', required=True, + help="First manifest to merge into input2.") + parser.add_argument('--input2', required=True, + help="Second manifest, overwrite by input1.") + parser.add_argument('--output', required=True, + help="Output of new manifest") + # parser.add_argument('--clear', action="store_true", + # help="Create a new manifest and clear old configuration.") + args = parser.parse_args() + return args + + +def main(): + config = get_config() + git_tool = GitTool() + + dct_remote_1, dct_project_1, default_remote_1 = git_tool.get_manifest_xml_info( + filename=config.input1, add_root=True) + dct_remote_2, dct_project_2, default_remote_2 = git_tool.get_manifest_xml_info( + filename=config.input2, add_root=True) + + dct_remote_3 = copy.deepcopy(dct_remote_2) + dct_project_3 = copy.deepcopy(dct_project_2) + + for key, value in dct_project_1.items(): + revision = value.get("@revision") + if revision: + dct_project_3[key]["@revision"] = revision + else: + dct_project_3[key]["@upstream"] = "12.0" + dct_project_3[key]["@dest-branch"] = "12.0" + + # Update origin to new repo + git_tool.generate_repo_manifest(dct_remote=dct_remote_3, dct_project=dct_project_3, + output=config.output, + default_remote=default_remote_2) + + +if __name__ == '__main__': + main() diff --git a/script/git_repo_manifest.py b/script/git_repo_manifest.py old mode 100644 new mode 100755 index e2b634a..9076d64 --- a/script/git_repo_manifest.py +++ b/script/git_repo_manifest.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!./venv/bin/python import os import sys import argparse @@ -47,12 +47,13 @@ def main(): # Update origin to new repo if not config.clear: - dct_remote, dct_project = git_tool.get_manifest_xml_info(repo_path=config.dir, - add_root=True) + dct_remote, dct_project, _ = git_tool.get_manifest_xml_info( + repo_path=config.dir, add_root=True) else: dct_remote = {} dct_project = {} - git_tool.generate_repo_manifest(lst_repo_organization, repo_path=config.dir, + git_tool.generate_repo_manifest(lst_repo_organization, + output=f"{config.dir}manifest/default.dev.xml", dct_remote=dct_remote, dct_project=dct_project) git_tool.generate_install_locally() diff --git a/script/git_tool.py b/script/git_tool.py index 8bec5d2..f9e9e37 100644 --- a/script/git_tool.py +++ b/script/git_tool.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!./venv/bin/python # © 2020 TechnoLibre (http://www.technolibre.ca) # License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl). import os @@ -15,6 +15,10 @@ from typing import List CST_FILE_SOURCE_REPO_ADDONS = "source_repo_addons.csv" CST_EL_GITHUB_TOKEN = "EL_GITHUB_TOKEN" +DEFAULT_PROJECT_NAME = "ERPLibre" +DEFAULT_WEBSITE = "erplibre.ca" +DEFAULT_REMOTE_URL = "https://github.com/ERPLibre/ERPLibre.git" +DEFAULT_BRANCH = "12.0" class Struct(object): @@ -23,6 +27,22 @@ class Struct(object): class GitTool: + @property + def default_project_name(self): + return DEFAULT_PROJECT_NAME + + @property + def default_website(self): + return DEFAULT_WEBSITE + + @property + def default_remote_url(self): + return DEFAULT_REMOTE_URL + + @property + def default_branch(self): + return DEFAULT_BRANCH + @staticmethod def get_url(url: str) -> object: """ @@ -125,7 +145,6 @@ class GitTool: Get information about submodule from repo_path :param repo_path: path of repo to get information about submodule :param add_root: add information about root repository - :param upstream: Use this upstream of root :return: [{ "url": original_url, @@ -254,7 +273,13 @@ class GitTool: if add_root: repo_root = Repo(repo_path) - url = repo_root.git.remote("get-url", "origin") + try: + url = repo_root.git.remote("get-url", "origin") + except Exception as e: + print(f"WARNING: Missing origin remote, use default url " + f"{DEFAULT_REMOTE_URL}. Suggest to add a remote origin: \n" + f"> git remote add origin {DEFAULT_REMOTE_URL}") + url = DEFAULT_REMOTE_URL url, url_https, url_git = self.get_url(url) data = { @@ -269,17 +294,19 @@ class GitTool: lst_repo = sorted(lst_repo, key=lambda k: k.get("name")) return lst_repo - def get_manifest_xml_info(self, repo_path: str = "./", + def get_manifest_xml_info(self, repo_path: str = "./", filename=None, add_root: bool = False) -> list: """ Get contain of manifest :param repo_path: path of repo to get information about submodule + :param filename: manifest filename. Default none, or use this instead use repo_path :param add_root: add information about root repository - :return: dct_remote, dct_project + :return: dct_remote, dct_project, default_remote """ - manifest_file = self.get_manifest_file(repo_path=repo_path) - filename = f"{repo_path}/manifest/{manifest_file}" + if filename is None: + manifest_file = self.get_manifest_file(repo_path=repo_path) + filename = f"{repo_path}/{manifest_file}" with open(filename) as xml: xml_as_string = xml.read() xml_dict = xmltodict.parse(xml_as_string, dict_constructor=dict) @@ -289,7 +316,7 @@ class GitTool: lst_project = dct_manifest.get("project") dct_remote = {a.get("@name"): a for a in lst_remote} dct_project = {a.get("@name"): a for a in lst_project} - return dct_remote, dct_project + return dct_remote, dct_project, default_remote @staticmethod def get_project_config(repo_path="./"): @@ -362,8 +389,19 @@ class GitTool: def str_insert(source_str, insert_str, pos): return source_str[:pos] + insert_str + source_str[pos:] - def generate_repo_manifest(self, lst_repo: List[Struct], repo_path: str = "./", - dct_remote={}, dct_project={}): + def generate_repo_manifest(self, lst_repo: List[Struct] = [], output: str = "", + dct_remote={}, dct_project={}, default_remote=None): + """ + Generate repo manifest + :param lst_repo: optional, update manifest with list_repo + :param output: filename to write output + :param dct_remote: dict of remote information + :param dct_project: dict of project information + :param default_remote: name of default remote, optional + :return: + """ + if not output: + raise Exception("Cannot generate manifest with missing output filename.") lst_remote = [] lst_remote_name = [] lst_project = [] @@ -381,14 +419,20 @@ class GitTool: lst_project_info = [ ('@name', dct_value.get("@name")), ('@path', dct_value.get("@path")), - ('@remote', dct_value.get("@remote")) ] + if "@remote" in dct_value.keys(): + lst_project_info.append(('@remote', dct_value.get("@remote"))) if "@revision" in dct_value.keys(): lst_project_info.append(('@revision', dct_value.get("@revision"))) if "@clone-depth" in dct_value.keys(): lst_project_info.append(('@clone-depth', dct_value.get("@clone-depth"))) if "@groups" in dct_value.keys(): lst_project_info.append(('@groups', dct_value.get("@groups"))) + if "@upstream" in dct_value.keys(): + lst_project_info.append(('@upstream', dct_value.get("@upstream"))) + if "@dest-branch" in dct_value.keys(): + lst_project_info.append(('@dest-branch', dct_value.get("@dest-branch"))) + lst_project.append(OrderedDict(lst_project_info)) lst_project_name.append(dct_value.get("@name")) @@ -400,7 +444,7 @@ class GitTool: "Validate why 2 or more is not submodule.") lst_default.append(OrderedDict([ ('@remote', repo.original_organization), - ('@revision', "12.0"), + ('@revision', DEFAULT_BRANCH), ('@sync-j', "4"), ('@sync-c', "true"), ])) @@ -430,6 +474,14 @@ class GitTool: lst_project_info.append(('@groups', "odoo")) lst_project.append(OrderedDict(lst_project_info)) + if default_remote and not lst_default: + lst_default.append(OrderedDict([ + ('@remote', default_remote), + ('@revision', DEFAULT_BRANCH), + ('@sync-j', "4"), + ('@sync-c', "true"), + ])) + # Order in alphabetic lst_order_remote = sorted(lst_remote, key=lambda key: key.get("@name")) lst_order_default = sorted(lst_default, key=lambda key: key.get("@remote")) @@ -444,14 +496,14 @@ class GitTool: str_xml_text = xmltodict.unparse(dct_repo, pretty=True) pos_insert = str_xml_text.rfind("") - if pos_insert: + if pos_insert >= 0: pos_insert += len("") str_xml_text = self.str_insert(str_xml_text, "\n ", pos_insert) pos_insert = str_xml_text.rfind("") - if pos_insert: + if pos_insert >= 0: pos_insert += len("") - str_xml_text = self.str_insert(str_xml_text, "\n ", pos_insert) + str_xml_text = self.str_insert(str_xml_text, "\n ", pos_insert) # pos_insert = str_xml_text.rfind("") # if pos_insert: @@ -465,7 +517,7 @@ class GitTool: str_xml_text = str_xml_text.replace("\t", " ") # create file - with open(f"{repo_path}manifest/default.dev.xml", mode="w") as file: + with open(output, mode="w") as file: file.writelines(str_xml_text + "\n") def generate_git_modules(self, lst_repo: List[Struct], repo_path: str = "./"): @@ -523,7 +575,10 @@ class GitTool: for line in all_lines: # Separate information with path in tuple - line_split = line[:-1].split(',') + line = line.strip() + if not line: + continue + line_split = line.split(',') if len(line_split) != 4: print(f"Error with line {line}, suppose to have only 4 ','.") exit(1) diff --git a/script/install_daemon.sh b/script/install_daemon.sh index 586d95c..2311c04 100755 --- a/script/install_daemon.sh +++ b/script/install_daemon.sh @@ -52,7 +52,7 @@ cat < /tmp/${EL_USER}run.sh #!/usr/bin/env bash cd ${EL_HOME_ERPLIBRE} source ./venv/bin/activate -python3 ${EL_HOME_ERPLIBRE}/odoo/odoo-bin -c ${EL_HOME_ERPLIBRE}/config.conf $@ +python3 ${EL_HOME_ERPLIBRE}/odoo/odoo-bin -c ${EL_HOME_ERPLIBRE}/config.conf --limit-time-real 99999 --limit-time-cpu 99999 $@ EOF echo -e "* Security Run File" diff --git a/script/install_debian_dependancy.sh b/script/install_debian_dependancy.sh index 333a5d9..41b9b1f 100755 --- a/script/install_debian_dependancy.sh +++ b/script/install_debian_dependancy.sh @@ -19,7 +19,7 @@ WKHTMLTOX_X32=https://github.com/wkhtmltopdf/wkhtmltopdf/releases/download/0.12. echo -e "\n---- Update Server ----" # add-apt-repository can install add-apt-repository Ubuntu 18.x -sudo apt-get install software-properties-common -y +sudo apt-get install software-properties-common curl -y # universe package is for Ubuntu 18.x sudo add-apt-repository universe # libpng12-0 dependency for wkhtmltopdf diff --git a/script/install_locally.sh b/script/install_locally.sh index 07cced5..9e36c74 100755 --- a/script/install_locally.sh +++ b/script/install_locally.sh @@ -129,7 +129,7 @@ if [[ $EL_MINIMAL_ADDONS = "False" ]]; then fi printf "\n" >> ${EL_CONFIG_FILE} -printf "workers = 0\n" >> ${EL_CONFIG_FILE} +printf "workers = 2\n" >> ${EL_CONFIG_FILE} printf "max_cron_threads = 2\n" >> ${EL_CONFIG_FILE} if [ ${EL_INSTALL_NGINX} = "True" ]; then diff --git a/script/install_locally_dev.sh b/script/install_locally_dev.sh index 671f74d..af55c91 100755 --- a/script/install_locally_dev.sh +++ b/script/install_locally_dev.sh @@ -19,13 +19,7 @@ EL_CONFIG_FILE="${EL_HOME}/config.conf" ./script/install_locally.sh # Update git-repo -git daemon --base-path=. --export-all --reuseaddr --informative-errors --verbose & -DAEMON_PID=$! - -./venv/repo init -u git://127.0.0.1:9418/ -b $(git rev-parse --abbrev-ref HEAD) -m ${EL_MANIFEST_DEV} -./venv/repo sync -m ${EL_MANIFEST_DEV} - -kill ${DAEMON_PID} +./script/update_manifest_local_dev.sh echo -e "\n---- Install python packages/requirements ----" ${EL_HOME}/venv/bin/pip3 install --upgrade pip diff --git a/script/install_locally_prod.sh b/script/install_locally_prod.sh index 77f158d..dd61b2e 100755 --- a/script/install_locally_prod.sh +++ b/script/install_locally_prod.sh @@ -19,9 +19,7 @@ EL_CONFIG_FILE="${EL_HOME}/config.conf" ./script/install_locally.sh # Update git-repo -./venv/repo init -u http://git.erplibre.ca/ERPLibre -b 12.0_repo -#./venv/repo sync --force-sync -./venv/repo sync +./script/update_manifest_prod.sh echo -e "\n---- Install python packages/requirements ----" ${EL_HOME}/venv/bin/pip3 install --upgrade pip diff --git a/script/update_manifest_dev.sh b/script/update_manifest_dev.sh new file mode 100755 index 0000000..7af5f61 --- /dev/null +++ b/script/update_manifest_dev.sh @@ -0,0 +1,11 @@ +#!/usr/bin/env bash + +. ./env_var.sh + +#EL_MANIFEST_PROD="./default.xml" +#EL_MANIFEST_DEV="./manifest/default.dev.xml" + +# Update git-repo +./venv/repo init -u http://git.erplibre.ca/ERPLibre -b $(git rev-parse --abbrev-ref HEAD) -m ${EL_MANIFEST_DEV} +#./venv/repo sync --force-sync +./venv/repo sync --force-sync -v diff --git a/script/update_manifest_local_dev.sh b/script/update_manifest_local_dev.sh new file mode 100755 index 0000000..8e09e0a --- /dev/null +++ b/script/update_manifest_local_dev.sh @@ -0,0 +1,15 @@ +#!/usr/bin/env bash + +. ./env_var.sh + +#EL_MANIFEST_PROD="./default.xml" +#EL_MANIFEST_DEV="./manifest/default.dev.xml" + +# Update git-repo +git daemon --base-path=. --export-all --reuseaddr --informative-errors --verbose & +DAEMON_PID=$! + +./venv/repo init -u git://127.0.0.1:9418/ -b $(git rev-parse --abbrev-ref HEAD) -m ${EL_MANIFEST_DEV} +./venv/repo sync -v --force-sync -m ${EL_MANIFEST_DEV} + +kill ${DAEMON_PID} diff --git a/script/update_manifest_local_experimental.sh b/script/update_manifest_local_experimental.sh new file mode 100755 index 0000000..c41027e --- /dev/null +++ b/script/update_manifest_local_experimental.sh @@ -0,0 +1,16 @@ +#!/usr/bin/env bash + +. ./env_var.sh + +#EL_MANIFEST_PROD="./default.xml" +#EL_MANIFEST_DEV="./manifest/default.dev.xml" +#EL_MANIFEST_EXP="./manifest/default.experimental.xml" + +# Update git-repo +git daemon --base-path=. --export-all --reuseaddr --informative-errors --verbose & +DAEMON_PID=$! + +./venv/repo init -u git://127.0.0.1:9418/ -b $(git rev-parse --abbrev-ref HEAD) -m ${EL_MANIFEST_EXP} +./venv/repo sync -v --force-sync -m ${EL_MANIFEST_EXP} + +kill ${DAEMON_PID} diff --git a/script/update_manifest_local_prod.sh b/script/update_manifest_local_prod.sh new file mode 100755 index 0000000..c8efb1b --- /dev/null +++ b/script/update_manifest_local_prod.sh @@ -0,0 +1,15 @@ +#!/usr/bin/env bash + +. ./env_var.sh + +#EL_MANIFEST_PROD="./default.xml" +#EL_MANIFEST_DEV="./manifest/default.dev.xml" + +# Update git-repo +git daemon --base-path=. --export-all --reuseaddr --informative-errors --verbose & +DAEMON_PID=$! + +./venv/repo init -u git://127.0.0.1:9418/ -b $(git rev-parse --abbrev-ref HEAD) -m ${EL_MANIFEST_PROD} +./venv/repo sync -v -m ${EL_MANIFEST_PROD} + +kill ${DAEMON_PID} diff --git a/script/update_manifest_prod.sh b/script/update_manifest_prod.sh new file mode 100755 index 0000000..b8d5b0c --- /dev/null +++ b/script/update_manifest_prod.sh @@ -0,0 +1,11 @@ +#!/usr/bin/env bash + +. ./env_var.sh + +#EL_MANIFEST_PROD="./default.xml" +#EL_MANIFEST_DEV="./manifest/default.dev.xml" + +# Update git-repo +./venv/repo init -u http://git.erplibre.ca/ERPLibre -b $(git rev-parse --abbrev-ref HEAD) +#./venv/repo sync --force-sync +./venv/repo sync -v diff --git a/source_repo_addons.csv b/source_repo_addons.csv index 2f64738..a98b03d 100644 --- a/source_repo_addons.csv +++ b/source_repo_addons.csv @@ -1,20 +1,20 @@ url,path,revision,clone-depth https://github.com/ERPLibre/erplibre_addons.git,addons,, https://github.com/MathBenTech/development.git,addons,, -https://github.com/MathBenTech/odoo-business-spending-management-quebec-canada.git,addons,stable_prod_12.0, +https://github.com/MathBenTech/odoo-business-spending-management-quebec-canada.git,addons,12.0_sale_order_line_timesheet_attribution, https://github.com/MathBenTech/scrummer.git,addons,, https://github.com/muk-it/muk_base.git,addons,, https://github.com/muk-it/muk_dms.git,addons,, https://github.com/muk-it/muk_docs.git,addons,, https://github.com/muk-it/muk_misc.git,addons,, https://github.com/muk-it/muk_quality.git,addons,, -https://github.com/muk-it/muk_web.git,addons,, -https://github.com/muk-it/muk_website.git,addons,, +https://github.com/muk-it/muk_web.git,addons,12.0_fix_web_conflict_menu, +https://github.com/muk-it/muk_website.git,addons,12.0_remove_autoinstall, https://github.com/Numigi/odoo-base-addons.git,addons,, https://github.com/Numigi/odoo-entertainment-addons.git,addons,, https://github.com/Numigi/odoo-git-addons.git,addons,, https://github.com/Numigi/odoo-hr-addons.git,addons,, -https://github.com/Numigi/odoo-partner-addons.git,addons,, +https://github.com/Numigi/odoo-partner-addons.git,addons,12.0_partner_no_vat_improve_no_vat, https://github.com/Numigi/odoo-product-addons.git,addons,, https://github.com/Numigi/odoo-project-addons.git,addons,, https://github.com/Numigi/odoo-purchase-addons.git,addons,, @@ -29,7 +29,7 @@ https://github.com/OCA/account-consolidation.git,addons,, https://github.com/OCA/account-financial-reporting.git,addons,, https://github.com/OCA/account-financial-tools.git,addons,, https://github.com/OCA/account-fiscal-rule.git,addons,, -https://github.com/OCA/account-invoicing.git,addons,, +https://github.com/OCA/account-invoicing.git,addons,12.0_account_invoice_approbation, https://github.com/OCA/account-invoice-reporting.git,addons,, https://github.com/OCA/account-payment.git,addons,, https://github.com/OCA/account-reconcile.git,addons,, @@ -45,18 +45,18 @@ https://github.com/OCA/contract.git,addons,, https://github.com/OCA/credit-control.git,addons,, https://github.com/OCA/currency.git,addons,, https://github.com/OCA/data-protection.git,addons,, -https://github.com/OCA/donation.git,addons,, +https://github.com/OCA/donation.git,addons,12.0_module_donation_website, https://github.com/OCA/e-commerce.git,addons,, https://github.com/OCA/edi.git,addons,, https://github.com/OCA/event.git,addons,, -https://github.com/OCA/helpdesk.git,addons,, +https://github.com/OCA/helpdesk.git,addons,12.0_dev, https://github.com/OCA/hr.git,addons,, https://github.com/OCA/knowledge.git,addons,, https://github.com/OCA/l10n-canada.git,addons,, https://github.com/OCA/maintenance.git,addons,, https://github.com/OCA/management-system.git,addons,, https://github.com/OCA/manufacture.git,addons,, -https://github.com/OCA/margin-analysis.git,addons,, +https://github.com/OCA/margin-analysis.git,addons,12.0_sale_margin_editor, https://github.com/OCA/mis-builder.git,addons,, https://github.com/OCA/multi-company.git,addons,, https://github.com/OCA/partner-contact.git,addons,, @@ -64,12 +64,12 @@ https://github.com/OCA/pos.git,addons,, https://github.com/OCA/product-attribute.git,addons,, https://github.com/OCA/product-pack.git,addons,, https://github.com/OCA/product-variant.git,addons,, -https://github.com/OCA/project.git,addons,, +https://github.com/OCA/project.git,addons,12.0_fix_conflict_project_key, https://github.com/OCA/project-reporting.git,addons,, https://github.com/OCA/purchase-workflow.git,addons,, https://github.com/OCA/queue.git,addons,, https://github.com/OCA/reporting-engine.git,addons,, -https://github.com/OCA/sale-workflow.git,addons,, +https://github.com/OCA/sale-workflow.git,addons,12.0_modules_sale_order_line, https://github.com/OCA/server-auth.git,addons,, https://github.com/OCA/server-brand.git,addons,, https://github.com/OCA/server-env.git,addons,, @@ -77,12 +77,12 @@ https://github.com/OCA/server-tools.git,addons,, https://github.com/OCA/server-ux.git,addons,, https://github.com/OCA/social.git,addons,, https://github.com/OCA/stock-logistics-warehouse.git,addons,, -https://github.com/OCA/timesheet.git,addons,, +https://github.com/OCA/timesheet.git,addons,12.0_support_project_agile_timesheet, https://github.com/OCA/web.git,addons,, -https://github.com/OCA/website.git,addons,, +https://github.com/OCA/website.git,addons,12.0_dev, https://github.com/OCA/wms.git,addons,, https://github.com/odoo/odoo.git,.,12.0_feature_5_star_CRM,1 https://github.com/odooaktiv/QuotationRevision.git,addons,, https://github.com/openeducat/openeducat_erp.git,addons,, https://github.com/SanteLibre/santelibre_addons.git,addons,master, -https://github.com/Smile-SA/odoo_addons.git,addons,, +https://github.com/Smile-SA/odoo_addons.git,addons,12.0_remote_autoinstall,