14dfbdbea5
Snapcraft has a feature name `remote-build`. It allows to compile snaps using the Canonical dedicated build architecture for several architectures. Compared to the QEMU-enabled Docker approach used currently, the remote build has several advantages: * the builds are done on the native architecture, making them basically faster than what can be achieved on QEMU * it avoids to depend on `adferrand/snapcraft` (which could be otherwise be fixed with the merge of https://github.com/snapcore/snapcraft/pull/3144, but this will not happen in the short term) * when everything is good, all snaps build can be run in parallel and then can be orchestrated by one single Azure Pipeline job, since the heavy tasks are done remotely. This PR makes the necessary ajustements to use the remote build feature instead of the QEMU-enabled docker approach. One complex task was to be able to compile the `certbot` snap on `arm64` and `armhf`. Indeed on these architectures the pre-compiled wheel for `cffi` is not available. So it needs to be compiled during the snap build. Sadly, the current version of the python plugin in snapcraft is limited by the fact that `wheels` is not installed in the virtual environment set up to build the python packages, and there is no easy way to change that except by overridding the whole build process. In the long term, I think I will open a PR on `snapcraft` Git repository to provide a consistent solution. But for the short term, I used the possibility to provide arguments to the `venv` module, to add the flag `--system-site-packages`. With it, the virtual environment can use the system site package, where `wheel` is available. The other significant additions are in `tools/snap/build_remote.py` script. If invoking the remote build on a local machine is quite straight-forward, it is another story on the CI because we need build auditability and resiliency during these non-interactive actions. In particular we should avoid as possible inconsistent results on the nightly pipeline and the release pipeline. So this script wraps the `snapcraft` call into a retry logic, and improves its logs in the context of parallel builds. For the minor modifications, it is mainly about ensuring that plugins can be built (some of them also need `cffi` for instance), and simplify the Azure Pipeline since all snaps are retrieved in one go. Please note that the `test-` branches still run only the `amd64` architecture. Indeed I noticed that builds on `arm64` and `armhf` are tending to be very slow to start (up to 40 min) while the `amd64` ones wait at max 10 mins, and usually 30 seconds only when the overall load on Canonical side is low. To work on `certbot/certbot` repository, one secured file needs to be added, because `snapcraft` needs to be authenticated against Launchpad with credentials allowing remote builds. To do so, from a local machine that have this capability, one can extract the existing file at `$HOME/.local/share/snapcraft/provider/launchpad/credentials`, and register it as a secured file in Azure Pipeline with the name `snapcraftRemoteBuildCredentials`. * Define scripts * Setup pipeline to use remote builds * Focus on packaging builds * Set credentials * Setup git * Launch all builds in parallel * Add dev dependencies to build cffi and cryptography * Convert to a python logic * Reorganize the pipeline * Handle the fact that snap builds may be taken from cache * Generate constraints * Exit code * Check existence * Try to handle better non zero exit code * Add --system-site-packages to get wheel in the venv * Add executable permissions * Troubleshoot * Dynamic display, take the maximum timeout for snap build job * Allow retries if the remote build does not start * Trigger only amd64 builds for test branches * Exit properly * Update snapcraft.yaml * Fix snap run * Set secured file name * Update .azure-pipelines/templates/jobs/packaging-jobs.yml Co-authored-by: Brad Warren <bmw@users.noreply.github.com> * Update .azure-pipelines/templates/jobs/packaging-jobs.yml Co-authored-by: Brad Warren <bmw@users.noreply.github.com> * Update .azure-pipelines/templates/jobs/packaging-jobs.yml Co-authored-by: Brad Warren <bmw@users.noreply.github.com> * Move order in deps * Reactivate all builds * Use Manager() as a context manager * Use Pool as a context manager * Some nice refactorings * Check snapcraft execution interruption with exit codes * Use f-string and format expressions * Start log * Consistent use of single/double quotes * Better loop to extract lines * Retry on build failures * Few optimizations Co-authored-by: Brad Warren <bmw@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| packages | ||
| build.sh | ||
| build_dns.sh | ||
| build_remote.py | ||
| common.sh | ||
| compile_native_wheels.sh | ||
| generate_dnsplugins_snapcraft.sh | ||
| README.md | ||
Certbot Plugin Snaps
This is a proof of concept of how a Certbot snap might support plugin snaps that add functionality to Certbot using its existing plugin API.
Architecture
This is a description of how Certbot plugin functionality is exposed via snaps. For information on Certbot's plugin architecture itself, see the Certbot documentation on plugins.
The Certbot snap itself is a classic snap. Plugin snaps are regular confined snaps, but normally do not provide any "apps" themselves. Plugin snaps export loadable Python modules to the Certbot snap via a snap content interface.
Certbot itself accepts a CERTBOT_PLUGIN_PATH environment variable. This
support is currently patched but this is intended to be upstreamed. The
variable, if set, should contain a :-separated list of paths to add to
Certbot's plugin search path.
The Certbot snap runs Certbot via a wrapper which examines its list of
connected interfaces, sets CERTBOT_PLUGIN_PATH accordingly, and then execs
Certbot itself.
Use (Production)
Note: this production use example assumes that these snaps are available in stable channels in the Snap Store, which they aren't yet. See below for development instructions.
To use a Certbot plugin snap, install both the plugin snap and the Certbot snap
as usual. Plugin snaps are confined as normal; the Certbot snap is a classic
snap and thus needs --classic during installation. For example:
snap install --classic certbot
snap set certbot trust-plugin-with-root=ok
snap install certbot-dns-dnsimple
Then connect the plugin snap to the main certbot snap as follows. Note that this connection allows the plugin snap code to run inside the certbot process, which has access to your host system. Only perform this step if you trust the plugin author to have "root" on your system.
sudo snap connect certbot:plugin certbot-dns-dnsimple
Now certbot will automatically load and use the plugin when it is run. To check
that this has worked, certbot plugins should list the plugin.
You can now operate the plugin as normal.
Use (Testing and Development)
To try this out, you'll need to build the snaps (a patched Certbot snap and a plugin snap) manually.
Initial VM Set Up
These steps need to be done once to set up your VM and do not need to be run again to rebuild the snap.
- Start with a Focal VM. You need a full virtual machine using something like DigitalOcean, EC2, or VirtualBox. Docker won't work. Another version of Ubuntu can probably be used, but Focal was used when writing these instructions.
- Set up a user other than root with sudo privileges for use with snapcraft and run all of the following commands with it. A command to do this for a user named certbot looks like
adduser certbot && usermod -aG sudo certbot && su - certbot. - Install git and python with
sudo apt update && sudo apt install -y git python. - Set up lxd for use with snapcraft by running
sudo snap install lxd && sudo /snap/bin/lxd.migrate -yes && sudo /snap/bin/lxd waitready && sudo /snap/bin/lxd init --auto(errors here are ok; it may already have been installed on your system). - Add your current user to the lxd group and update your shell to have the new assignment by running
sudo usermod -a -G lxd ${USER} && newgrp lxd. - Install snapcraft with
sudo snap install --classic snapcraft. cd ~(or any other directory where you want our source files to be)- Run
git clone git://github.com/certbot/certbot cd certbot
Build the Snaps
These are the steps to build and install the snaps. If you have run these steps before, you may want to run the commands in the section below to clean things up before building the snap again.
- Run
snapcraft --use-lxd. - Install the generated snap with
sudo snap install --dangerous --classic certbot_*_amd64.snap. You can transfer the snap to a different machine to run it there instead if you prefer. - Run
tools/merge_requirements.py tools/dev_constraints.txt <(tools/strip_hashes.py letsencrypt-auto-source/pieces/dependency-requirements.txt) > certbot-dns-dnsimple/snap-constraints.txt(this is a workaround for https://github.com/certbot/certbot/issues/8100). cd certbot-dns-dnsimplesnapcraft --use-lxd- Run
sudo snap set certbot trust-plugin-with-root=ok. - Install the generated snap with
sudo snap install --dangerous certbot-dns-dnsimple_*_amd64.snap. Again, you can transfer the snap to a different machine to run it there instead if you prefer. - Connect the plugin with
sudo snap connect certbot:plugin certbot-dns-dnsimple. - Now you can run Certbot as normal. For example,
certbot pluginsshould display the DNSimple plugin as installed.
Reset the Environment
The instructions below clean up the build environment so it can reliably be used again.
cd ~/certbot(or to an alternate path where you put our source files)snapcraft clean --use-lxdrm certbot_*_amd64.snapcd certbot-dns-dnsimplerm certbot-dns-dnsimple_*_amd64.snapsnapcraft clean --use-lxdcd ..
Publishing Permissions
There are security implications to permitting anyone to publish, without review, a plugin into the Snap Store which will then run in Certbot's classic snap context, with full access to the host system.
At a minimum, it is clear that this should happen only with the user's explicit opt-in action.
As implemented, Certbot will only load plugins connected via the snap interface mechanism, so permission is effectively delegated to what interface connections the snap infrastucture will permit.
We have approval from the snap team to use this design as long as we make it explicit what a user is agreeing to when they connect a plugin to the Certbot snap. That work was completed in https://github.com/certbot/certbot/issues/8013.
Outstanding issues
Outstanding items relating to plugin support in Certbot snaps are tracked on GitHub.