* Base logic
* Various controls when email is None
* Adapt eff tests
* Forward compatibility
* Also for csr
* Explicit regr or meta updates in account objects
* Adapt logic to ask for eff subscription during registering
* Adapt tests
* Move dry-run control
* Add some relevant controls on handle_subscription call checks
This will allow DNS plugin snaps to build if they rely on unreleased acme/certbot, and remove other copy of Certbot from externally snapped plugins. Fixes#8064 and fixes#7946. Implementation is based on the design [here](https://github.com/certbot/certbot/issues/8064#issuecomment-645513120).
To test, see reverted commit 8632064. Steps taken:
- added changes to setup.py and snapcraft.yaml
- successfully snapped, connected, ran `sudo certbot plugins --prepare`
- added temporary changes to have both certbot and certbot-dns-dnsimple use DNSAuthenticator2
- snapped and installed certbot, `certbot plugins` failed as expected.
- snapped and installed certbot-dns-dnsimple, `sudo certbot plugins --prepare` succeeded
- Inspected dns plugin's `bin` and `lib`; no `certbot` or `acme`, as expected.
```
$ ls /snap/certbot-dns-dnsimple/current/lib/python3.6/site-packages/
OpenSSL future-0.18.2.dist-info requests_file.py
PyYAML-5.3.1.dist-info idna setuptools
_cffi_backend.cpython-36m-x86_64-linux-gnu.so idna-2.9.dist-info setuptools-47.3.1.dist-info
certbot_dns_dnsimple lexicon six-1.15.0.dist-info
certbot_dns_dnsimple-1.6.0.dev0-py3.6.egg-info libfuturize six.py
certifi libpasteurize tldextract
certifi-2020.4.5.1.dist-info past tldextract-2.2.2.dist-info
cffi pip urllib3
cffi-1.14.0.dist-info pip-20.1.1.dist-info urllib3-1.25.9.dist-info
chardet pkg_resources wheel
chardet-3.0.4.dist-info pyOpenSSL-19.1.0.dist-info wheel-0.34.2.dist-info
cryptography pycparser yaml
cryptography-2.8.dist-info pycparser-2.20.dist-info zope
dns_lexicon-3.3.26.dist-info requests zope.interface-5.1.0-py3.6-nspkg.pth
easy_install.py requests-2.23.0.dist-info zope.interface-5.1.0.dist-info
future requests_file-1.5.1.dist-info
$ ls /snap/certbot-dns-dnsimple/current/bin/
chardetect futurize lexicon pasteurize tldextract
```
- reset to HEAD^
- snapped and installed certbot to not have the DNSAuthenticator2 changes, `certbot plugins` failed as expected.
* Don't include certbot deps when EXCLUDE_CERTBOT_DEPS is set
* Set EXCLUDE_CERTBOT_DEPS in certbot-dns-dnsimple/snap/snapcraft.yaml
fix branch name
grammar
remove readme link
Remove links to Robie's repo.
say you cant use docker
Commands not command
Update publishing permissions section.
Have Certbot trust plugins.
Do not run snapcraft with sudo.
Upgrade snap to be based on core20
This PR makes several changes to be built on top of the core20 base snap. Fixes#7854.
The main changes are to `snapcraft.yaml`. With Snapcraft 4.0/core20 base, the python plugin is a thin wrapper, basically creating a `venv` and installing the packages from the source. The trouble with this is that it runs pycache, creating caches that conflict from the different parts. So to solve that, we put everything in a single part. Other changes include:
- We use classic confinement, so we need to specify a bunch of python packages to `stage-packages`, as mentioned [here](https://forum.snapcraft.io/t/trouble-bundling-python-with-classic-confinement-in-core20-4-0-4/18234/2).
- The certbot executable now lives in `bin`, so specify running `certbot/bin`.
- Since `python-augeas` is now being pulled into the single part, remove the pinning from constraints so we can use the latest version directly from github.
- Precompile our `cryptography` and `cffi` wheels to be based on python3.8.
Separately, we had to upgrade the snapcraft docker image to be based on focal, due to the thin wrapper situation. This was accomplished [here](https://github.com/adferrand/snapcraft/pull/1).
* Get rid of a whole bunch of error message
* Remove some more overlaps
* don't use certbot from nginx and apache
* use python3 from bin
* certbot needs to be in bin
* try to exclude just the certbot folder
* try a couple things to use the python from the venv bin
* play around with which versions of things we want from each package
* ok, certbot-nginx does need to stage bin
* certbot needs to not stage bin. why does certbot not put certbot in bin?
* fail to inspect more versions of things in the container shell
* take cffi backend from python-augeas
* if we use certbot from bin things should work?
* why is bin not in path? no idea, but let's get it compiled then inspect things in the snap shell
* use snap.certbot instead of bin/certbot
* it does require bin/certbot. I don't know why.
* let's see if we can stick it all in one step
* try installing local subdirectories
* move python-augeas into the single part
* remove after
* put back python-augeas part for now; ERROR: Could not satisfy constraints for 'python-augeas': installation from path or url cannot be constrained to a version
* how was this previously working without git installed? install git.
* maybe it needs to already have python3-wheel installed
* maybe wheel will install first if I change it to -e
* no -e
* maybe try a different python3 package to stage
* this last change wasn't necessary
* remove the bin/ from renew
* nope, it does need bin/certbot
* back to wget
* stage a bare python3
* add all necessary python packages to stage-packages
* pretty sure we don't actually need wheel. let's try removing it!
* remove python-augeas, since we have it pinned to an older version in cb-auto that might work
* stage augeas
* still need libaugeas-dev
* ok let's try building
* combining into one part works! just make sure to unpin python-augeas when generating snap-constraints.txt
* change our scripts to unpin python-augeas
* Use ubuntu 20 in compile_native_wheels.sh
* .travis.yml should use python3-dev instead of python-dev
* jk! we don't need python3-dev in travis
* Update cffi and cryptography wheels for ubuntu20 version of python
* looks like we need python3-dev to build things
* Remove deprecated i386 wheels
I initially added this when the script was doing things like migrating all LXD containers to the snap. I think the external side effects are now pretty minimal thought so I think we can remove the need for this environment variable which makes it easier to use outside of CI for manual testing.
* Tweaks for improved Cloudflare API
* Update docs for dns-cloudflare
* Update tests and changelog
* Fix bad merge
* Fix error code for record add
* Improve error message
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
This PR proposes a way to build the certbot snap for several architectures, using a QEMU-base emulation approach, and several optimizations to keep the build time of each snap below 20 minutes.
Most of the reasoning for the approach proposed here is described in the original PR: https://github.com/basak/certbot-snap-build/pull/27
On top of it, I added a docker pull to a pre-compiled snapcraft docker, instead of compiling it during the Travis pipeline, in order to save 5 to 7 minutes more on each snap build. The snap images are compiled and stored here: https://hub.docker.com/repository/docker/adferrand/snapcraft. Depending on the time the PR will be reviewed, we can:
* continue to use `adferrand/snapcraft`
* move its logic to certbot scope and use something like `certbot/snapcraft`
* wait for https://github.com/snapcore/snapcraft/pull/3144 to be merged, and use `snapcore/snapcraft`.
* Backport https://github.com/basak/certbot-snap-build/pull/27 into Certbot project
* Fix build deps
* Integrate proactively #8012 to fix builds on non-amd64 archs
* Configure jobs on Travis
* Focus on snap builds. Disable temporarily some jobs. Disable deploy actions by security.
* Specify TARGET_ARCH during snap build
* Do not do anything if TOXENV is not set
* Various optimizations
* Use recent version of ubuntu for get correct features on snap out of the box
* Add up to date wheels
* Organizing scripts
* Set dest dir
* Get back original configuration for Travis
* Add comments
* Update common_libs.sh
* Use adferrand/snapcraft
* Test build
* Stable snapcraft
* Update build_and_install.sh
* Move back snap builds to the cron/release pipeline
* Update snap/local/compile_native_wheels.sh
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Update snap/local/compile_native_wheels.sh
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Update snap/local/compile_native_wheels.sh
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Update snap/local/build_and_install.sh
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Enable i386 builds, various optimizations
* Update dependencies
* Configure a simple http server to serve the pre compiled wheels
* Fix wheels compilation
* Relax permissions
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Add snap plugin support
Switch to a PoC branch of certbot that supports the new
CERTBOT_PLUGIN_PATH and wrap the snap to set this variable correctly
based on the content interfaces connected.
(cherry picked from commit 7076a55fd82116d068e2aca7239209b7203917d2)
* Modify certbot.wrapper to append to PYTHONPATH instead of separate CERTBOT_PLUGIN_PATH variable
* Update certbot-wrapper to python3.6 version
* add source field
* Update changelog
* Use bash instead of sh
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Exit if something goes wrong
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* No leading : when modifying empty PYTHONPATH
* Improve bash handling of PYTHONPATH manipulation
Co-authored-by: Robie Basak <robie.basak@canonical.com>
Co-authored-by: Brad Warren <bmw@eff.org>
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
This PR gets its root from an observation I did on current version of Certbot (1.3.0): the `renewal-hooks` directory in Certbot configuration directory is created on Windows with write permissions to everybody.
I thought it was a critical bug since this directory contains hooks that are executed by Certbot, and you certainly do not want this folder to be open to any malicious hook that could be inserted by everyone, then executed with administrator privileges by Certbot.
Turns out for this specific problem that the bug is not critical for the hooks, because the scripts are expected to be in subdirectories of `renewal-hooks` (namely `pre`, `post` and `deploy`), and these subdirectories have proper permissions because we set them explicitly when Certbot is starting.
Still, there is a divergence here between Linux and Windows: on Linux all Certbot directories without explicit permissions have at maximum `0o755` permissions by default, while on Windows it is a `0o777` equivalent. It is not an immediate security risk, but it is definitly error-prone, not expected, and so a potential breach in the future if we forget about it.
Root cause is that umask is not existing in Windows. Indeed under Linux the umask defines the default permissions when you create a file or a directory. Python takes that into account, with an API for `os.open` and `os.mkdir` that expose a `mode` parameter with default value of `0o777`. In practice it is never `0o777` (either you the the `mode` explictly or left the default one) because the effective mode is masked by the current umask value in the system: on Linux it is `0o022`, so files/directories have a maximum mode of `0o755` if you did not set the umask explicitly, and it is what it is observed for Certbot.
However on Windows, the `mode` value passed (and got from default) to the `open` and `mkdir` of `certbot.compat.filesystem` module is taken verbatim, since umask does not exit, and then is used to calculate the DACL of the newly created file/directory. So if the mode is not set explicitly, we end up with files and directories with `0o777` permissions.
This PR fixes this problem by implementing a umask behavior in the `certbot.compat.filesystem` module, that will be applied to any file or directory created by Certbot since we forbid to use the `os` module directly.
The implementation is quite straight-forward. For Linux the behavior is not changed. On Windows a `mask` parameter is added to the function that calculates the DACL, to be invoked appropriately when file or directory are created. The actual value of the mask is taken from an internal class of the `filesystem` module: its default value is `0o755` to match default umasks on Linux, and can be changed with the new method `umask` that have the same behavior than the original `os.umask`. Of course `os.umask` becomes a forbidden function and `filesystem.umask` must be used instead.
Existing code that is impacted have been updated, and new unit tests are created for this new function.
* Implement umask for Windows
* Set umask at the beginning of tests
* Fix lint, update local oldest requirements
* Update certbot-apache/setup.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Improve tests
* Adapt filesystem.makedirs for Windows
* Fix
* Update certbot-apache/setup.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Changelog entries
* Fix lint
* Update certbot/CHANGELOG.md
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
Issue #1123 discusses a feature that allows users to set the cipher
security level. That feature wasn't built. It didn't provide enough
user value to justify the corresponding increase in complexity. The
feature request and the associated discussion threads were closed.
However, the proposed API spec and the TODO section remained in the
cipher docs. They're a vestige of that issue from olden days and this PR
removes those last living traces...
Fixes#8027.
* Add support for NetBSD by telling certbot-nginx where the nginx
configuration directory is.
* Update the CHANGELOG.
* Pass the right type of sequence to "in". Thanks lint.
* Adjust the CHANGELOG.md entry following feedback from ohemorange.
Co-authored-by: Lloyd Parkes <lloyd@must-have-coffee.gen.nz>
In #7771, the Apache configurator gained the ability to identify what
version of OpenSSL Apache's ssl_module is linked against. However, the
detection was only functional if the module was built as a DSO (which is
almost always the case).
This commit covers the case where the ssl_module is statically linked
within the Apache binary. It requires the user to specify the path to
the binary (with --apache-bin) and emits a warning if static linking is
detected but no path has been provided.
This PR upgrades Certbot pinned dependencies through `letsencrypt-auto-source/rebuild_dependencies.py` while taking into account the problems detected in https://github.com/certbot/certbot/pull/8035:
* `cryptography` is pinned to `2.8` to continue to support OpenSSL 1.0.1 on non-x86 ancient Linux distributions (RHEL 6 + Debian 8)
* `parsedatetime` is pinned to `2.5` because of an incompatibility with Python 2.7 (see https://github.com/bear/parsedatetime/issues/246)
* `letsencrypt-auto-source/rebuild_dependencies.py` now takes into account the environment markers that are aded to `AUTHORITATIVE_CONSTRAINTS`: this is used for the `enum34` dependency, to not install it on Python 3.6+ and not break the distribution by swapping the built-in `enum` module during the setup of Certbot venv.
Fixes#8030
* Pin cryptography and parsedatetime
* Upgrade dependencies
* Remove authoritative constraint
* Upgrade dependencies
* Rebuild certbot-auto
* Update letsencrypt-auto-source/rebuild_dependencies.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Honor specific requirements in the AUTHORITATIVE_CONSTRAINTS
* Fix injection
* Update dependencies
* Update rebuild_dependencies.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
Fixes#7988. As described there, the steps involved are:
1. Update our tests so they fail due to this problem.
2. Update the keys used in the tests so they pass with the new changes.
For 1, see a [failing travis run](https://travis-ci.com/github/certbot/certbot/jobs/340710511) with the included change. And for the full output to confirm that this is what is failing, see a [run on debian 10](https://github.com/certbot/certbot/files/4692350/debian_run_log.txt).
This PR adds `rsa4096_key.pem` and `rsa4096_cert.pem`, updates the `TLS-ALPN` test to use those keys in place of the 1024-bit versions, and fixes the README in that `testdata` folder with correct instructions to generate these files.
* export PIP_NO_BINARY in pip install subshell in test_sdists.sh
* set environment variable on the line that installs most packages
* Generate 4096-bit rsa key and cert, and fix README instructions to do so.
* Update TLS_ALPN test to use 4096-bit key instead of 1024-bit key.
* Update changelog
* Older versions of Python have an error when both VIRTUAL_NO_DOWNLOAD and PIP_NO_BINARY are set, so only apply the latter at the install phase.
* Add enum34 constraint manually, since rebuild_dependencies.py seems to be broken.
* only delete key if it exists
* Check OpenSSL version before trying to set PIP_NO_BINARY
* Add comment explaining why we only set PIP_NO_BINARY at the install step
* Add the content interface to Certbot
This commit contains a subset of the changes from 7076a55fd82116d068e2aca7239209b7203917d2.
* Normalise slot parameters
(cherry picked from commit 810941979bcf609c1e0be18e9263abf046b90e82)
Co-authored-by: Robie Basak <robie.basak@canonical.com>
Fixes#7667.
Implements the plan described in #7667.
Here's a terminal log showing that it does so:
```
# sudo snap connect certbot:plugin certbot-dns-dnsimple
error: cannot perform the following tasks:
- Run hook prepare-plug-plugin of snap "certbot" (run hook "prepare-plug-plugin":
-----
Only connect this interface if you trust the plugin author to have root on the system
Run `snap set certbot trust-plugin-with-root=ok` to acknowledge this and then run this command again to perform the connection
-----)
# snap set certbot trust-plugin-with-root=ok
# sudo snap connect certbot:plugin certbot-dns-dnsimple
# sudo snap disconnect certbot:plugin certbot-dns-dnsimple:certbot
# sudo snap connect certbot:plugin certbot-dns-dnsimple
error: cannot perform the following tasks:
- Run hook prepare-plug-plugin of snap "certbot" (run hook "prepare-plug-plugin":
-----
Only connect this interface if you trust the plugin author to have root on the system
Run `snap set certbot trust-plugin-with-root=ok` to acknowledge this and then run this command again to perform the connection
-----)
```
* Add plugin connection hook to accept root trust
* snapctl requires a configure hook to set options
* Add sh notice
* Update changelog
Fixes#7993
This PR uses `os.umask()` during `certbot.compat.filesystem.makedirs()` call to ensure that all directories, and not only the leaf one, have the provided `mode` when created. This ensures a safe and consistent behavior independently from the Python version, since the behavior of `os.makedirs` changed on that matter with Python 3.7.
* Implement logic to apply the same permission on all dirs created by makedirs
* Add a test
* Add comment
* Update certbot/certbot/compat/filesystem.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Error out in apache installer when mod_ssl is not available
* Update to MisconfigurationError and add/fix tests
* Remove error cases we no longer hit and associated test
* mock out function to have consistent error across machines
* improve changelog message
* only check key in modules list, not value
The error message from `python3 -m venv` when you don't have `python3-venv` installed is pretty good, but lets skip the failure and make sure it is installed the first time.
