Commit graph

36 commits

Author SHA1 Message Date
Brad Warren
d3b82a4e8e
Fix test farm tests by using a local Pebble instance (#8561)
[As discussed in Mattermost](https://opensource.eff.org/eff-open-source/pl/yhtp4qu4zpfczm5wxmzxhndrto), our Apache test farm tests are failing because the CA certificate in the old version of boulder we have pinned expired over the weekend. This PR fixes that by running a local Pebble instance instead of an external boulder instance.

* switch from external boulder to local pebble

* add --http-01-port to run_acme_server
2020-12-22 10:24:20 -08:00
Brad Warren
7febc18bb0
Make our test farm tests instances self-destruct (#8536)
* remove unused user data

* have instance self-destruct in case cleanup fails

* correct kwargs

* fix param order
2020-12-15 12:00:00 +01:00
Mads Jensen
501df0dc4e
Use in dict rather than "in dict.keys()". Fix linting warnings about "not in". (#8298)
* Fixed a few linting warnings for if not x in y.

These should have been caught by pylint, but weren't.

* Replaced "x in y.keys()" with "x in y".

It's much faster, and more Pythonic.
2020-09-19 11:35:49 +02:00
Brad Warren
085967ad29
Fix test farm tests on macOS and update macOS images (#8219)
* Run one of the test farm tests on macOS.

* it break with 38?

* Remove LOGDIR global

* add comment

* include macOS in name

* Update macOS image.
2020-08-19 18:26:28 -07:00
Adrien Ferrand
8a3a8c7097
Migrate the CI pipeline from Travis to Azure Pipeline (#8098)
Fixes #8071 and fixes https://github.com/certbot/certbot/issues/8110.

This PR migrates every job from Travis in Azure Pipeline.

This PR essentially converts the Travis jobs into Azure Pipeline with a complete iso-fonctionality (or I made a mistake). The jobs are added in the relevant existing pipelines (`main`, `nightly`, `advanced-test`, `release`). A global refactoring thanks to the templating system is done to reduce greatly the verbosity of the pipeline descriptions.

A specific feature (not present in Travis) is added: the stage `On_Failure`. Using directly the Mattermost API, it allows to notify pipeline failure in a Mattermost channel with a link to the failed pipelines without the need to authenticate to Microsoft.

See https://github.com/certbot/certbot/pull/8098#issuecomment-649873641 for the post merge actions to do at the end of this work.
2020-07-02 15:01:21 -07:00
ohemorange
316e4640f8
Upgrade the test farm tests to use Python 3 (#7876)
Fixes #7857.

* stop using urllib2 in test farm tests

* use six for urllib instead

* remove fabric lcd usage

* correct lcd removal

* remove fabric cd

* convert some remote calls to v2

* move more cxns to v2

* get run working with prefix

* get sudo commands working

* remove final fabric v1 references including local

* update requirements and README

* add new venv to gitignore

* update version used in travis

* remove deploy_script unused kwargs

* fix killboulder implementation so I can test creating a new boulder server

* hardcode the gopath due to broken env manamagement in fabric2

* Update letstest readme

* move the comment about hardcoding the ggopath

* catch BaseException instead of Exception

* work around fabric #2007

* use connections as context managers to ensure they're closed

* remove reference to virtualenv
2020-04-09 14:35:47 -07:00
Adrien Ferrand
e048da1e38 Reorganize imports (#7616)
* Isort execution

* Fix pylint, adapt coverage

* New isort

* Fix magic_typing lint

* Second round

* Fix pylint

* Third round. Store isort configuration

* Fix latest mistakes

* Other fixes

* Add newline

* Fix lint errors
2019-12-09 15:50:20 -05:00
Andrzej Górski
9da07590bd Remove --fast from the test farm tests (#7427) 2019-10-08 21:24:55 +02:00
Brad Warren
bf9c681c4f fix backwards logic (#7265) 2019-07-25 10:20:52 +02:00
Brad Warren
a37a4486cf
Add Debian ARM AMI. (#7189)
Inspired by the number of ARM users we have (and because I want to rip out the only 32 bit test we have which without this PR would remove all tests we have on non-x86_64 architectures), this test adds an ARM image to the test farm tests. The image ID was taken from https://wiki.debian.org/Cloud/AmazonEC2Image/Stretch, you can see tests passing at https://travis-ci.com/certbot/certbot/builds/116857897, and I ran test_tests.sh locally and it passed.
2019-06-25 14:03:45 -07:00
Brad Warren
18797dca79 Remove scripts that are never run. (#7111)
* Remove scripts that are never run.

* Update example in multitester.py docstring.
2019-06-03 10:20:20 +03:00
Brad Warren
d1753e46f9 Allow magic profile name none for configuring test farm tests. (#7069)
* Allow magic profile name none.

* Naming is hard.
2019-05-17 20:40:20 +02:00
Brad Warren
16834a0d78 Stop sharing state between processes in test farm tests (#7057)
* Set LOGDIR at top of script.

* Set sentinel at top of script.

* Don't use EC2 global to block on instance start.

* Remove global boto3 state.

* Pass in boulder_url.

* Create main function.

* Add link to reload docs.
2019-05-17 20:36:58 +02:00
Brad Warren
11c3e7107c Exit with a nonzero status when tests fail. (#7065) 2019-05-17 19:44:51 +02:00
Brad Warren
6bcd0415e0 Fix race condition adding tags to instance. (#7080) 2019-05-17 11:27:40 +02:00
Brad Warren
7a6545b747 Regularly print output in the test farm tests (#7079)
* Occasionally print output in test farm tests.

* Flush output.
2019-05-17 11:21:57 +02:00
Brad Warren
f4bbaadd18 Fix cleanup on failure. (#7070)
* Fix cleanup on failure.

* Incrementally build instances list.
2019-05-17 11:17:27 +02:00
ohemorange
f0f5bb4fc0 Update test farm version of boulder to current master (#7002)
Recent changes are no longer compatible with the old version of boulder used in the test farm tests. This PR updates the version of boulder used, and runs it with the new way of running boulder.

A new ami was created and is used here that uses Ubuntu 18.04, so that docker-compose can be installed more properly.

Removed commented-out section about rabbitmq that was already deprecated.

Switched to using the public DNS resolver 8.8.8.8 for the tests because the way to find the correct local resolver changed.
2019-04-30 13:13:37 -07:00
ohemorange
eef4c47633 Add failure message if test farm tests do not run the correct number of tests. (#6771)
Fixes #6748.
2019-02-20 15:20:44 -08:00
Brad Warren
8e5b2ac5b5 Stop multitester.py from eating errors. (#6705) 2019-01-25 23:24:11 +01:00
Brad Warren
cd2edeff1b Fix test farm tests (#6335)
* update CentOS AMI ids

* Remove assumption of usable default subnet
2018-09-05 13:12:05 -07:00
Brad Warren
783b6e4746 Automate EBS cleanup (#6160)
* ensure volume cleanup

* remove volume cleanup

* cleanup function and output
2018-07-17 17:19:04 -07:00
cclauss
09b5927e6a from botocore.exceptions import ClientError (#5507)
Fixes undefined name 'botocore' in flake8 testing of https://github.com/certbot/certbot

$ __flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics__
```
./tests/letstest/multitester.py:144:12: F821 undefined name 'botocore'
    except botocore.exceptions.ClientError as e:
           ^
1     F821 undefined name 'botocore'
```
2018-02-15 11:07:35 -08:00
Brad Warren
3780d068d1 Fix test farm tests 2016-04-19 14:11:17 -04:00
Noah Swartz
842b609dad Merge pull request #2841 from letsencrypt/module-rename
Module rename
2016-04-14 17:35:09 -07:00
Brad Warren
e353f8fabc letstest is a Certbot integration test tool 2016-04-14 17:16:48 -07:00
Brad Warren
b4f6ed8470 rename letstest stuff 2016-04-14 17:10:27 -07:00
Brad Warren
7228c0c9ed 'co' is not a git command 2016-04-14 08:57:34 -07:00
Peter Eckersley
f4c33656a2 Try to finally prevent dangling AWS volumes 2016-02-23 12:08:01 -08:00
Peter Eckersley
1f31cf1a30 Quick test farm fix 2016-02-10 19:09:05 -08:00
Roy Wellington Ⅳ
9c28364477 Make exception syntax Python 3 compatible.
Translate all except and raise statements that are in the old form to
the Python 3 compatible format.
2016-01-30 19:53:50 -08:00
Peter Eckersley
b8281fdd34 Retry in case of ephemeral errors 2016-01-20 14:25:01 -08:00
Peter Eckersley
3e1bc19e0f Optional flag for faster AMIs 2015-12-25 10:43:52 -08:00
Peter Eckersley
e0837ad41f [letstest] create & reuse a persistent boulder server
Reduces minimum multitester.py runtime to just a bit under 10 minutes
2015-12-25 02:39:48 -08:00
Peter Eckersley
49721c7d01 Merge remote-tracking branch 'letstest/upgrade-test' into testfarm 2015-12-23 15:14:30 -08:00
Peter Eckersley
87dfe8c2b2 Move everything into tests/letstest 2015-12-21 11:12:01 -08:00
Renamed from multitester.py (Browse further)