4abd81e218
Summary of changes in this PR:
- Refactor files involved in the `certbot` module to be of a similar structure to every other package; that is, inside a directory inside the main repo root (see below).
- Make repo root README symlink to `certbot` README.
- Pull tests outside of the distributed module.
- Make `certbot/tests` not be a module so that `certbot` isn't added to Python's path for module discovery.
- Remove `--pyargs` from test calls, and make sure to call tests from repo root since without `--pyargs`, `pytest` takes directory names rather than package names as arguments.
- Replace mentions of `.` with `certbot` when referring to packages to install, usually editably.
- Clean up some unused code around executing tests in a different directory.
- Create public shim around main and make that the entry point.
New directory structure summary:
```
repo root ("certbot", probably, but for clarity all files I mention are relative to here)
├── certbot
│ ├── setup.py
│ ├── certbot
│ │ ├── __init__.py
│ │ ├── achallenges.py
│ │ ├── _internal
│ │ │ ├── __init__.py
│ │ │ ├── account.py
│ │ │ ├── ...
│ │ ├── ...
│ ├── tests
│ │ ├── account_test.py
│ │ ├── display
│ │ │ ├── __init__.py
│ │ │ ├── ...
│ │ ├── ... # note no __init__.py at this level
│ ├── ...
├── acme
│ ├── ...
├── certbot-apache
│ ├── ...
├── ...
```
* refactor certbot/ and certbot/tests/ to use the same structure as the other packages
* git grep -lE "\-e(\s+)\." | xargs sed -i -E "s/\-e(\s+)\./-e certbot/g"
* git grep -lE "\.\[dev\]" | xargs sed -i -E "s/\.\[dev\]/certbot[dev]/g"
* git grep -lE "\.\[dev3\]" | xargs sed -i -E "s/\.\[dev3\]/certbot[dev3]/g"
* Remove replacement of certbot into . in install_and_test.py
* copy license back out to main folder
* remove linter_plugin.py and CONTRIBUTING.md from certbot/MANIFEST.in because these files are not under certbot/
* Move README back into main folder, and make the version inside certbot/ a symlink
* symlink certbot READMEs the other way around
* move testdata into the public api certbot zone
* update source_paths in tox.ini to certbot/certbot to find the right subfolder for tests
* certbot version has been bumped down a directory level
* make certbot tests directory not a package and import sibling as module
* Remove unused script cruft
* change . to certbot in test_sdists
* remove outdated comment referencing a command that doesn't work
* Install instructions should reference an existing file
* update file paths in Dockerfile
* some package named in tox.ini were manually specified, change those to certbot
* new directory format doesn't work easily with pyargs according to http://doc.pytest.org/en/latest/goodpractices.html#tests-as-part-of-application-code
* remove other instance of pyargs
* fix up some references in _release.sh by searching for ' . ' and manual check
* another stray . in tox.ini
* fix paths in tools/_release.sh
* Remove final --pyargs call, and now-unnecessary call to modules instead of local files, since that's fixed by certbot's code being one layer deeper
* Create public shim around main and make that the entry point
* without pyargs, tests cannot be run from an empty directory
* Remove cruft for running certbot directly from main
* Have main shim take real arg
* add docs/api file for main, and fix up main comment
* Update certbot/docs/install.rst
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Fix comments in readthedocs requirements files to refer to current package
* Update .[docs] reference in contributing.rst
* Move plugins tests to certbot tests directory
* add certbot tests to MANIFEST.in so packagers can run python setup.py test
* move examples directory inside certbot/
* Move CHANGELOG into certbot, and create a top-level symlink
* Remove unused sys and logging from main shim
* nginx http01 test no longer relies on certbot plugins common test
|
||
|---|---|---|
| .. | ||
| scripts | ||
| apache2_targets.yaml | ||
| multitester.py | ||
| README.md | ||
| requirements.txt | ||
| targets.yaml | ||
| travis-setup.sh | ||
| travis-test-farm.pem.enc | ||
letstest
Simple AWS testfarm scripts for certbot client testing
- Configures (canned) boulder server
- Launches EC2 instances with a given list of AMIs for different distros
- Copies certbot repo and puts it on the instances
- Runs certbot tests (bash scripts) on all of these
- Logs execution and success/fail for debugging
Notes
- Some AWS images, e.g. official CentOS and FreeBSD images require acceptance of user terms on the AWS marketplace website. This can't be automated.
- AWS EC2 has a default limit of 20 t2/t1 instances, if more are needed, they need to be requested via online webform.
Installation and configuration
These tests require Python 2.7, awscli, boto3, PyYAML, and fabric<2.0. If you have Python 2.7 and virtualenv installed, you can use requirements.txt to create a virtual environment with a known set of dependencies by running:
virtualenv --python $(command -v python2.7 || command -v python2 || command -v python) venv
. ./venv/bin/activate
pip install --requirement requirements.txt
You can then configure AWS credentials and create a key by running:
>aws configure --profile <profile name>
[interactive: enter secrets for IAM role]
>aws ec2 create-key-pair --profile <profile name> --key-name <key name> --query 'KeyMaterial' --output text > whatever/path/you/want.pem
Note: whatever you pick for <key name> will be shown to other users with AWS access.
When prompted for a default region name, enter: us-east-1.
Usage
To run tests, activate the virtual environment you created above and run:
>python multitester.py targets.yaml /path/to/your/key.pem <profile name> scripts/<test to run>
You can only run up to two tests at once. The following error is often indicative of there being too many AWS instances running on our account:
NameError: name 'instances' is not defined
If you see this, you can run the following command to shut down all running instances:
aws ec2 terminate-instances --profile <profile name> --instance-ids $(aws ec2 describe-instances --profile <profile name> | grep <key name> | cut -f8)
It will take a minute for these instances to shut down and become available again. Running this will invalidate any in progress tests.
A folder named letest-<timestamp> is also created with a log file from each instance of the test and a file named "results" containing the output above.
The tests take quite a while to run.
Also, the way all of the tests work is to check if there is already a boulder server running and if not start one. The boulder server is left running between tests,
and there are known issues if two instances of boulder attempt to be started. After starting your first test, wait until you see "Found existing boulder server:" or if you see output
about creating a boulder server, wait a minute before starting the 2nd test. You only have to do this after starting your first session of tests or after running
the aws ec2 terminate-instances command above.
Scripts
Example scripts are in the 'scripts' directory, these are just bash scripts that have a few parameters passed to them at runtime via environment variables. test_apache2.sh is a useful reference.
Note that the
test_letsencrypt_auto_*scripts pull code from PyPI using the letsencrypt-auto script, not the local python code. test_apache2 runs the dev venv and does local tests.
See:
- https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
- https://docs.aws.amazon.com/cli/latest/userguide/cli-ec2-keypairs.html
Main repos: