based on the suggestion @bmw made in #10484, this moves nearly
everything from `certbot-apache` and `certbot-nginx` into subdirectories
in `certbot/src/certbot/_internal`, and corresponding "extra"
dependencies are made for the certbot distribution. in their place,
entrypoint shims are made in the old distributions.
this way, installing `certbot[nginx]` will pull in the extra
dependencies needed for the nginx code, and also pull in the shim in
`certbot-nginx`, letting our plugin discovery system work just as it did
before. ditto for apache.
note that this doesn't yet deprecate anything, which was one of the
primary goals of the original issue -- i spun out that work into #10521fixes#10484
---------
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
Co-authored-by: ohemorange <erica@eff.org>
Alternative implementation for #7908.
In this PR:
- set up ruff in CI (add to `tox.ini`, mark dep in `certbot/setup.py`)
- add a `ruff.toml` that ignores particularly annoying errors. I think
line length isn't actually necessary to set with this workflow since
we're not checking it but putting it there for future usage.
- either fix or ignore the rest of the errors that come with the default
linting configuration. fixed errors are mostly unused variables. ignored
are usually where we're doing weird import things for a specific reason.
blast from the past! resurrects
https://github.com/certbot/certbot/pull/9803 with all of @bmw's changes.
i figured instead of force-pushing a basically brand new branch and
obliterating the old review, i'd just start from a clean slate
fixes#8272
---------
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
Co-authored-by: Brad Warren <bmw@eff.org>
Fixes#10252.
See further discussion here: https://github.com/pypa/pip/issues/11457
We are doing option:
> Alternatively, enable the --use-pep517 pip option, possibly with
--no-build-isolation. The --use-pip517 flag will force pip to use the
modern mechanism for editable installs. --no-build-isolation may be
needed if your project has build-time requirements beyond setuptools and
wheel. By passing this flag, you are responsible for making sure your
environment already has the required dependencies to build your package.
Once the legacy mechanism is removed, --use-pep517 will have no effect
and will essentially be enabled by default in this context.
Major changes made here include:
- Add `--use-pep517` to use the modern mechanism, which will be the only
mechanism in future pip releases
- Change to `/src` layout to appease mypy, and because for editable
installs that really is the normal way these days.
- `cd acme && mkdir src && mv acme src/` etc.
- add `where='src'` argument to `find_packages` and add
`package_dir={'': 'src'},` in `setup.py`s
- update `MANIFEST.in` files with new path locations
- Update our many hardcoded filepaths
- Update `importlib-metadata` requirement to fix
double-plugin-entry-point problem in oldest tests
Pebble 2.5.1 supports OCSP stapling, so we can finally replace all boulder tests/harnesses with the much simpler pebble setup.
Closes#9898
* Remove unused `--acme-server` argument
Since this argument is never set and always defaults to 'pebble', just
remove it to simplify assumptions about which test server's being used.
* Remove boulder option from integration tests
Now that pebble supports all of our test cases, we can move off of
the much more complicated boulder test harness.
* pebble_artifacts: bump to latest pebble release
* pebble_artifacts: fix download path
* certbot-ci: unzip pebble assets
* CI: rip out windows tests/jobs
* tox.ini: rm outdated Windows comment
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* ci: rm redundant integration test
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* acme_server: raise error if proxy and http-01 port are both set
* acme_server: rm vestigial preterimate commands stuff
---------
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Drop Python 3.7 support
* Fix lint and test
* Check for venv generation
* Update requirements
* Update oldest constaints and compatibility tests runtime
In addition to the speed improvements in CI, the speed improvements locally with both this https://github.com/certbot/certbot/pull/9666 which this builds on is even more significant. After it's been run once so it's had a chance to set up the different virtual environments, `tox` locally now takes 39 seconds on my laptop when it used to take 137 seconds.
Fixes https://github.com/certbot/certbot/issues/7921.
In all cases when we run `pip_install.py`, we first run `pipstrap.py`. This PR combines these two steps for convenience and to make always doing that less error prone. This will also help me with some of the `tox.ini` refactoring I'm planning to do.
I ran the full test suite on everything and tested the release script changes locally.
This change shouldn't have any effect on cryptography's setup because they install `certbot[test]` which depends on pip, setuptools, and wheel.
* always pipstrap
* use pip_install.py during releases
This is a first step towards implementing the plan I described at https://github.com/certbot/certbot/issues/7909#issuecomment-1448675456 which got a +1 from both Erica and Will. Similar changes for our other packages will be made in followup PRs to try and make this easier to review.
It may be helpful to look at https://github.com/certbot/certbot/pull/7600 when reviewing this PR where we did something similar in the past.
The value of `ignore-paths` in `.pylintrc` should work on Windows based on https://pylint.readthedocs.io/en/latest/user_guide/configuration/all-options.html#ignore-paths and the fact that on macOS/linux, changing path delimiters to `\` still causes these directories to be ignored.
I started testing this for mypy as well, but mypy doesn't current pass for us on Windows so I didn't bother and took this opportunity to remove it from the default environments in `tox.ini`. I'll update https://github.com/certbot/certbot/issues/7803 to mention that the value of `exclude` in `mypy.ini` may need to be tweaked if anyone works on that issue.
* make acme tests internal
* no mypy-win
Adrien and I added this is in https://github.com/certbot/certbot/pull/6590 in response to https://github.com/certbot/certbot/issues/6582 which I wrote. I now personally think these tests are way more trouble than they're worth.
In almost all cases, the versions pinned in `tools/requirements.txt` are used. The two exceptions to this that come to mind are users using OS packages and pip. In the former, the version of our dependencies is picked by the OS and do not change much on most systems. As for pip, [we only "support it on a best effort basis"](https://eff-certbot.readthedocs.io/en/stable/install.html#alternative-2-pip).
Even for pip users, I'm not convinced this buys us much other than frequent test failures. We have our tests configured to error on all Python warnings and [we regularly update `tools/requirements.txt`](https://github.com/certbot/certbot/commits/master/tools/requirements.txt). Due to that, assuming our dependencies follow normal conventions, we should have a chance to fix things in response to planned API changes long before they make their way to our users. I do not think it is necessary for our tests to break immediately after an API is deprecated.
I think almost all other failures due to these tests are caused by upstream bugs. In my experience, almost all of them will sort themselves out pretty quickly. I think that responding to those that are not or planned API changes we somehow missed can be addressed when `tools/requirements.txt` is updated or when someone opens an issue. I personally don't think blocking releases or causing our nightly tests to fail is at all worth it here. I think removing this frequent cause of test failures makes things just a little bit easier for Certbot devs without costing us much of anything.
`lock_test.py` is a weird, heavily customized, standalone testing relic that's giving me trouble because the name currently conflicts with `certbot/tests/lock_test.py`. Moving `certbot/tests` inside the Certbot package as discussed at https://github.com/certbot/certbot/issues/7909#issuecomment-1448675456 would avoid this, however, this is at least somewhat blocked on getting that test code passing lint and mypy checks again because we run those checks on the entirety of the Certbot package 🙃 Since `lock_test.py` could probably stand to be rewritten/refactored anyway, I took this approach.
What I did is I rewrote something largely equivalent to `lock_test.py` inside Certbot's unit tests. I chose not to do this in `certbot-ci` because its not necessary to have an ACME server available. We're no longer explicitly testing things with the nginx plugin here like we were in `lock_test.py`, however, we are checking that `prepare` is called on the plugin at the right time and I added comments about the importance of checking that we lock the directory during the call to `prepare` in the Apache and nginx test code.
As a bonus, this fixes https://github.com/certbot/certbot/issues/8121.
* fix cover tox envs
* make test work on all Pythons
* Remove unused import
Co-authored-by: alexzorin <alex@zorin.id.au>
Co-authored-by: alexzorin <alex@zorin.id.au>
* Remove external mock dependency
This also removes the "external-mock" test environment
* remove superfluous ignores
* remove mock warning ignore from pytest.ini
* drop deps on mock in oldest, drop dep on types-mock
Co-authored-by: Alex Zorin <alex@zorin.id.au>
* Remove Windows 2016 environment, generate 64 bit installer
* Add note to changelog
* Use win_amd64 as installer suffix
* Bump PYTHON_BITNESS to 64
* Require 64 bit Windows for the installer_build job
* Update certbot install path
* update windows test name
* Base installer suffix on PYTHON_BITNESS again
* Update changelog to request users uninstall old version
The reason I want to do this is many of the targets of `test_sdists.sh` use Python 3.6 which [has reached its EOL](https://www.python.org/dev/peps/pep-0494/#lifespan). We could instead just stop running the test on these systems or install a newer version of Python 3 outside of OS packaging, but instead I decided to look into why we have these tests to begin with.
I introduced these tests many years ago in https://github.com/certbot/certbot/pull/4089 as a fix for https://github.com/certbot/certbot/issues/4044. Essentially the problem was the way packagers ran tests and the way we ran tests were slightly different. This difference could cause test failures when distros tried to run tests on our packages.
Since I did this, [we've switched to telling packagers to run tests using `pytest` like we do](5e76669c50/certbot/docs/packaging.rst (notes-for-package-maintainers)) and we've greatly reduced our reliance on OS packaging through things like `snap`.
Because of this, I think we should stop running this test, reducing our reliance on the heavy "test farm tests", and simplifying our CI pipeline. I think future problems here is quite unlikely and even if we have them, it should only affect tests on our non-primary distribution mechanisms which I think is a very minor concern.
When reviewing this PR, it's probably worth noting that I just replaced `targets.yaml` with the contents of `apache2_targets.yaml` since the Apache 2 tests are the only runs we're running with this change.
* Work in progress
* Work in progress
* Work in progress
* Work in progress
* Fix issues around nullability of VirtualHost.path, may discuss that during review
* Work in progress
* Fix remaining types
* Various lint fixes
* Reconfigure tox and mypy to disallow untyped defs globally
* Cleanup compatibility tests
* Use cast for unused v2 logic
* Improve types
* Remove unused comment
* Fix coverage
* Better types
* Fix another type
* Update certbot-apache/certbot_apache/_internal/apacheparser.py
Co-authored-by: alexzorin <alex@zor.io>
* Update certbot-apache/certbot_apache/_internal/assertions.py
Co-authored-by: alexzorin <alex@zor.io>
* Fix type
* Various fixes
* Refactor imports
* Keep naming convention consistent on TypeVars
* Improve types
* Improve types
* Remove remaining Sequence[str] in the project
Co-authored-by: alexzorin <alex@zor.io>
* Add types in all DNS plugins
* Order imports
* Fix type
* Update certbot-dns-route53/certbot_dns_route53/_internal/dns_route53.py
Co-authored-by: alexzorin <alex@zor.io>
* Clean up imports
Co-authored-by: alexzorin <alex@zor.io>
* Start more types
* Second run
* Work in progress
* Types in all acme module
* Various fixes
* Various fixes
* Final fixes
* Disallow untyped defs for acme project
* Fix coverage
* Remote unecessary type ignore
* Use Mapping instead of Dict as input whenever it is possible
* Update acme/acme/client.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Update acme/acme/client.py
Co-authored-by: alexzorin <alex@zor.io>
* Various fixes
* Fix code
* Fix code
* Update acme/acme/client.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Update acme/acme/challenges.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Update acme/acme/client.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Fix deactivate_registration and simplify signature of update_registration
* Do not leak personal data during account deactivation
* Clean more Dicts
* New fix to not leak contact field in the account deactivation payload.
* Add ignore for python 3.6 type check
* Revert "Add ignore for python 3.6 type check"
This reverts commit da7338137b.
* Let's find a smarter way than "type: ignore"
* Update certbot/certbot/_internal/account.py
Co-authored-by: alexzorin <alex@zor.io>
* Fix an annotation
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
Co-authored-by: alexzorin <alex@zor.io>
Fixes#8899
This PR removes the pinning upper limit of mypy currently set to <0.900 and adds the required types-* stub packages to make recent versions of mypy work.
* Unpin mypy
* Improve type in TempHandler
* Add types
