upstream/certbot
1
0
Fork 0
mirror of https://github.com/certbot/certbot.git synced 2026-06-05 06:42:10 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions 5
certbot/tools/snap
History
Adrien Ferrand 84178e2773
Do not reuse existing builds on Launchpad when executing snapcraft remote-build (#8719) 
We observed recently several unexpected behavior during the execution of snap jobs in Azure. In particular it seems that `snapcraft remote-build` is tending to reattach to the latest builds on Launchpad triggered by the nightly builds on master, independently from the actual branch, status of the code, or targeted architectures.

Primarily if the builds on Launchpad are stalled for some reason, it blocks effectively any other Azure snap jobs until someone manually cancel the builds on Launchpad. Secondarily it means that the outcome of the builds may be inconsistent, because they can be the result of a build for the master source even if you are on a PR that modifieds these sources (including `snapcraft.yaml`).

After digging in `snapcraft` source code, I realized that the signature computed to understand if a build should be resumed, is not based one some hashes against the snapcraft working directory content, but is simply a hash of the working directory absolute path *itself*. It means that every builds triggered from the working directory `/my/path/certbot` for instance, are recognized as the same unique build on Launchpad side, and may be resumed if they already exist, and so independently from the source code, `snapcraft.yaml` or targeted archs.

For the record, relevant parts in `snapcraft` source code:
82024d3748/snapcraft/project/_project.py (L44)
82024d3748/snapcraft/project/_project.py (L86-L89)
82024d3748/snapcraft/cli/remote.py (L128-L132)

This PR makes effectively the resume build mechanism effectively a noop by moving the source code first in a temporary directory with random name before running `snapcraft remote-build`. This way the signature is never the same and builds are always recognized as brand new builds.

* Invalidate snapcraft remote-build cache by using a temporary workspace.

* Capture one more state in the build
2021-03-22 10:39:09 -07:00
..
build_remote.py Do not reuse existing builds on Launchpad when executing snapcraft remote-build (#8719) 2021-03-22 10:39:09 -07:00
generate_dnsplugins_all.sh Make certbot constraint file independent from certbot-auto + update cryptography (#8649) 2021-02-23 15:29:52 -08:00
generate_dnsplugins_postrefreshhook.sh Update tools/snap/generate_* comments. (#8412) 2020-10-30 13:08:57 -07:00
generate_dnsplugins_snapcraft.sh Pin DNS plugin snap build dependencies (#8553) 2020-12-18 15:02:23 -08:00
README.md Document that logs aren't always created. (#8410) 2020-10-30 13:15:47 -07:00

README.md

Building Certbot Snaps

Local Testing and Development

These instructions are recommended when testing anything about the snap setup for ease of debugging. The architecture of the built snap is limited to the architecture of the system it is built on.

Initial VM Set Up

These steps need to be done once to set up your VM and do not need to be run again to rebuild the snap.

  1. Start with a Focal VM. You need a full virtual machine using something like DigitalOcean, EC2, or VirtualBox. Docker won't work. Another version of Ubuntu can probably be used, but Focal was used when writing these instructions.
  2. Set up a user other than root with sudo privileges for use with snapcraft and run all of the following commands with it. A command to do this for a user named certbot looks like adduser certbot && usermod -aG sudo certbot && su - certbot.
  3. Install git and python with sudo apt update && sudo apt install -y git python.
  4. Set up lxd for use with snapcraft by running sudo snap install lxd && sudo /snap/bin/lxd.migrate -yes; sudo /snap/bin/lxd waitready && sudo /snap/bin/lxd init --auto (errors here are ok; it may already have been installed on your system).
  5. Add your current user to the lxd group and update your shell to have the new assignment by running sudo usermod -a -G lxd ${USER} && newgrp lxd.
  6. Install snapcraft with sudo snap install --classic snapcraft.
  7. cd ~ (or any other directory where you want our source files to be)
  8. Run git clone git://github.com/certbot/certbot
  9. cd certbot (All further instructions are relative to this directory.)

Certbot Snap

Reset the Environment

If the snap has been built before, the instructions below clean up the build environment so it can reliably be used again.

  1. snapcraft clean --use-lxd
  2. [Optional] mv certbot_*_amd64.snap certbot_amd64.snap.bak

Build the Certbot Snap

These are the steps to build and install the Certbot snap. If you have run these steps before, you may want to run the commands in the section above to clean things up or save a previous build before building the snap again (running snapcraft again will overwrite the previous snap).

  1. Run snapcraft --use-lxd.
  2. Install the generated snap with sudo snap install --dangerous --classic certbot_*_amd64.snap. You can transfer the snap to a different machine to run it there instead if you prefer.

Run

Run Certbot as normal. For example, certbot plugins should display the Apache and Nginx plugins.

Certbot Plugin Snaps

These instructions use the certbot-dns-dnsimple plugin as an example, but all of Certbot's other plugin snaps can be built in the same way.

Reset the Environment

If the plugin snap has been built before, the instructions below clean up the build environment so it can reliably be used again.

  1. cd certbot-dns-dnsimple
  2. snapcraft clean --use-lxd
  3. [Optional] mv certbot-dns-dnsimple_*_amd64.snap certbot-dns-simple_amd64.snap.bak
  4. cd ..

Build a Certbot Plugin Snap

These are the steps to build and install the Certbot DNSimple plugin snap. If you have run these steps before, you may want to run the commands in the section above to clean things up or save a previous build before building the snap again (running snapcraft again will overwrite the previous snap).

  1. Run tools/snap/generate_dnsplugins_all.sh to generate all necessary files for all plugin snaps.
  2. cd certbot-dns-dnsimple
  3. snapcraft --use-lxd
  4. Run sudo snap set certbot trust-plugin-with-root=ok.
  5. Install the generated snap with sudo snap install --dangerous certbot-dns-dnsimple_*_amd64.snap. Again, you can transfer the snap to a different machine to run it there instead if you prefer.
  6. Connect the plugin with sudo snap connect certbot:plugin certbot-dns-dnsimple.
  7. Connect the plugin metadata with sudo snap connect certbot-dns-dnsimple:certbot-metadata certbot:certbot-metadata. Install the plugin again to test refresh; if the plugin's hook creates any logs, they are at /var/snap/certbot-dns-dnsimple/current/debuglog.

Run

Run Certbot as normal. For example, certbot plugins should display the DNSimple plugin as installed.

Building for Other Architectures

To build for an unavailable architecture or for multiple architectures simultaneously, we recommend using snapcraft's remote build feature. It is easiest to run this from a local machine.

Initial Local Setup

  1. Create or log into an Ubuntu One account here.
  2. Install git and python with sudo apt update && sudo apt install -y git python.
  3. Install snapcraft with sudo snap install --classic snapcraft.
  4. cd ~ (or any other directory where you want our source files to be)
  5. Run git clone git://github.com/certbot/certbot
  6. cd certbot (All further instructions are relative to this directory.)
  7. To trigger snapcraft to request access to your Launchpad account, run snapcraft remote-build --launchpad-accept-public-upload --status. A URL where you need to grant this access will be printed to your terminal and automatically open in your browser if one is available.

Build Snaps Remotely

Certbot provides a wrapper around snapcraft's remote build to make building all of our plugins easier. To see all available options, run python3 tools/snap/build_remote.py --help.

For example, to build all available snaps for all architectures, run python3 tools/snap/build_remote.py ALL --archs amd64 arm64 armhf.

To build only the certbot snap on only amd64, run python3 tools/snap/build_remote.py certbot --archs armhf.

The command will upload the entire contents of the working directory, so if the remote build appears to hang, try using a clean clone of the certbot repository.