certbot/tools/docker/build.sh

109 lines
3.3 KiB
Bash
Raw Normal View History

#!/bin/bash
Automate Docker builds in Azure (#8193) Fixes https://github.com/certbot/certbot/issues/8022, https://github.com/certbot-docker/certbot-docker/issues/25, and https://github.com/certbot-docker/certbot-docker/issues/20. This PR builds on https://github.com/certbot/certbot/pull/8192 to set up similar builds in Azure to what we currently have at release time as well as nightly builds allowing us to catch problems in these images before a release. It also fully automates our Docker deployments removing a manual step from our release process. We'll need to update our release instructions once this PR lands. If you're not familiar with our `certbot-docker` setup, you can read about how these scripts customized the build process on Docker Hub at https://docs.docker.com/docker-hub/builds/advanced/. You can see the process working properly at: * Nightly build on my fork: https://dev.azure.com/bmw0523/bmw/_build/results?buildId=345&view=logs&j=70ac378a-cb1f-50d1-b328-169807afbcfa * Release build on my fork: https://dev.azure.com/bmw0523/bmw/_build/results?buildId=346&view=logs&j=70ac378a-cb1f-50d1-b328-169807afbcfa * Nightly build on Certbot's Azure setup: https://dev.azure.com/certbot/certbot/_build/results?buildId=2426&view=logs&j=70ac378a-cb1f-50d1-b328-169807afbcfa The builds on my fork pushed to https://hub.docker.com/u/certbottest. The credentials for this account are in our shared vault in 1password if you want to play around with this. While the scripts will (almost?) always be run in CI, I tested the scripts successfully on macOS, Ubuntu 18.04, and Ubuntu 20.04, however, **the scripts do not seem to work when using the Docker snap, at least on Ubuntu 20.04.** It does work with the `docker.io` packages from `apt`. I was able to make things work by no longer setting `DOCKER_BUILDKIT`, but as I described in the code comments, this breaks things on Azure. When writing this PR, I tried to make the minimal modifications to our current set up to get the behavior we want. I'm planning on working on splitting the Docker builds into different Azure jobs so it doesn't increase the overall build time, but this isn't trivial so I figured it would be best done in a separate PR. * Remove license. * update build scripts * write deploy code * Remove unused READMEs. * rewrite readme * Make testing on a fork easier. * Set up Azure automation. * fix typo * Make output more verbose. * clean up cleanup...everybody everywhere * separate build and deploy * Document docker-hub credentials * Use Docker BuildKit when building. * Remove unneeded .gitignore files. * Fix tools/docker/README.md grammar Co-authored-by: ohemorange <ebportnoy@gmail.com> * Clarify <TAG> in README. * no docker snap * rename docker job Co-authored-by: Erica Portnoy <ebportnoy@gmail.com>
2020-08-11 16:09:38 -04:00
set -euxo pipefail
IFS=$'\n\t'
Automate Docker builds in Azure (#8193) Fixes https://github.com/certbot/certbot/issues/8022, https://github.com/certbot-docker/certbot-docker/issues/25, and https://github.com/certbot-docker/certbot-docker/issues/20. This PR builds on https://github.com/certbot/certbot/pull/8192 to set up similar builds in Azure to what we currently have at release time as well as nightly builds allowing us to catch problems in these images before a release. It also fully automates our Docker deployments removing a manual step from our release process. We'll need to update our release instructions once this PR lands. If you're not familiar with our `certbot-docker` setup, you can read about how these scripts customized the build process on Docker Hub at https://docs.docker.com/docker-hub/builds/advanced/. You can see the process working properly at: * Nightly build on my fork: https://dev.azure.com/bmw0523/bmw/_build/results?buildId=345&view=logs&j=70ac378a-cb1f-50d1-b328-169807afbcfa * Release build on my fork: https://dev.azure.com/bmw0523/bmw/_build/results?buildId=346&view=logs&j=70ac378a-cb1f-50d1-b328-169807afbcfa * Nightly build on Certbot's Azure setup: https://dev.azure.com/certbot/certbot/_build/results?buildId=2426&view=logs&j=70ac378a-cb1f-50d1-b328-169807afbcfa The builds on my fork pushed to https://hub.docker.com/u/certbottest. The credentials for this account are in our shared vault in 1password if you want to play around with this. While the scripts will (almost?) always be run in CI, I tested the scripts successfully on macOS, Ubuntu 18.04, and Ubuntu 20.04, however, **the scripts do not seem to work when using the Docker snap, at least on Ubuntu 20.04.** It does work with the `docker.io` packages from `apt`. I was able to make things work by no longer setting `DOCKER_BUILDKIT`, but as I described in the code comments, this breaks things on Azure. When writing this PR, I tried to make the minimal modifications to our current set up to get the behavior we want. I'm planning on working on splitting the Docker builds into different Azure jobs so it doesn't increase the overall build time, but this isn't trivial so I figured it would be best done in a separate PR. * Remove license. * update build scripts * write deploy code * Remove unused READMEs. * rewrite readme * Make testing on a fork easier. * Set up Azure automation. * fix typo * Make output more verbose. * clean up cleanup...everybody everywhere * separate build and deploy * Document docker-hub credentials * Use Docker BuildKit when building. * Remove unneeded .gitignore files. * Fix tools/docker/README.md grammar Co-authored-by: ohemorange <ebportnoy@gmail.com> * Clarify <TAG> in README. * no docker snap * rename docker job Co-authored-by: Erica Portnoy <ebportnoy@gmail.com>
2020-08-11 16:09:38 -04:00
# This script builds certbot docker and certbot dns plugins docker using the
# local Certbot files.
Automate Docker builds in Azure (#8193) Fixes https://github.com/certbot/certbot/issues/8022, https://github.com/certbot-docker/certbot-docker/issues/25, and https://github.com/certbot-docker/certbot-docker/issues/20. This PR builds on https://github.com/certbot/certbot/pull/8192 to set up similar builds in Azure to what we currently have at release time as well as nightly builds allowing us to catch problems in these images before a release. It also fully automates our Docker deployments removing a manual step from our release process. We'll need to update our release instructions once this PR lands. If you're not familiar with our `certbot-docker` setup, you can read about how these scripts customized the build process on Docker Hub at https://docs.docker.com/docker-hub/builds/advanced/. You can see the process working properly at: * Nightly build on my fork: https://dev.azure.com/bmw0523/bmw/_build/results?buildId=345&view=logs&j=70ac378a-cb1f-50d1-b328-169807afbcfa * Release build on my fork: https://dev.azure.com/bmw0523/bmw/_build/results?buildId=346&view=logs&j=70ac378a-cb1f-50d1-b328-169807afbcfa * Nightly build on Certbot's Azure setup: https://dev.azure.com/certbot/certbot/_build/results?buildId=2426&view=logs&j=70ac378a-cb1f-50d1-b328-169807afbcfa The builds on my fork pushed to https://hub.docker.com/u/certbottest. The credentials for this account are in our shared vault in 1password if you want to play around with this. While the scripts will (almost?) always be run in CI, I tested the scripts successfully on macOS, Ubuntu 18.04, and Ubuntu 20.04, however, **the scripts do not seem to work when using the Docker snap, at least on Ubuntu 20.04.** It does work with the `docker.io` packages from `apt`. I was able to make things work by no longer setting `DOCKER_BUILDKIT`, but as I described in the code comments, this breaks things on Azure. When writing this PR, I tried to make the minimal modifications to our current set up to get the behavior we want. I'm planning on working on splitting the Docker builds into different Azure jobs so it doesn't increase the overall build time, but this isn't trivial so I figured it would be best done in a separate PR. * Remove license. * update build scripts * write deploy code * Remove unused READMEs. * rewrite readme * Make testing on a fork easier. * Set up Azure automation. * fix typo * Make output more verbose. * clean up cleanup...everybody everywhere * separate build and deploy * Document docker-hub credentials * Use Docker BuildKit when building. * Remove unneeded .gitignore files. * Fix tools/docker/README.md grammar Co-authored-by: ohemorange <ebportnoy@gmail.com> * Clarify <TAG> in README. * no docker snap * rename docker job Co-authored-by: Erica Portnoy <ebportnoy@gmail.com>
2020-08-11 16:09:38 -04:00
Docker build improvements (#8218) Fixes https://github.com/certbot/certbot/issues/8208. Fixes https://github.com/certbot/certbot/issues/8198. In addition to those two linked issues, this PR: * Splits both the build and deploy steps based on architecture for performance. The Docker builds should no longer be the bottleneck in any stage of the pipeline. * Skips building Docker images for ARM on `test-` branches like [we do for snaps](https://github.com/certbot/certbot/blob/e8a232297d6faedb3b0bffc197a2e98c2e59edcb/.azure-pipelines/templates/jobs/packaging-jobs.yml#L67-L71). I initially didn't want to do this, but the ARM builds take ~18 minutes which is significantly longer than any other job currently running on our `test-` branches. You can see tests running on my fork at: * [Release pipeline](https://dev.azure.com/bmw0523/bmw/_build/results?buildId=387&view=results) * [Test pipeline](https://dev.azure.com/bmw0523/bmw/_build/results?buildId=388&view=results) * [Nightly pipeline](https://dev.azure.com/bmw0523/bmw/_build/results?buildId=390&view=results) * update script intro * update readme * ParseRequestedArch * build all arch in Azure * Build docker images during testing/packaging. * require global variable? * Error if TAG_BASE is empty. * prepare build job * change variable syntax * Update deploy stage. * remove old dockerTag param * add displayName * fix docker images command * split docker_build by arch * Allow deploying a subset of architectures. * deploy in parallel * Skip ARM builds on test- branches. * fix spacing
2020-08-18 13:48:01 -04:00
# Usage: ./build.sh [TAG] [all|amd64|arm32v6|arm64v8]
# with the [TAG] value corresponding the base of the tag to give the Docker
# images and the 2nd value being the architecture to build snaps for.
# Values for the tag should be something like `v0.34.0` or `nightly`. The
# given value is only the base of the tag because the things like the CPU
# architecture are also added to the full tag.
Automate Docker builds in Azure (#8193) Fixes https://github.com/certbot/certbot/issues/8022, https://github.com/certbot-docker/certbot-docker/issues/25, and https://github.com/certbot-docker/certbot-docker/issues/20. This PR builds on https://github.com/certbot/certbot/pull/8192 to set up similar builds in Azure to what we currently have at release time as well as nightly builds allowing us to catch problems in these images before a release. It also fully automates our Docker deployments removing a manual step from our release process. We'll need to update our release instructions once this PR lands. If you're not familiar with our `certbot-docker` setup, you can read about how these scripts customized the build process on Docker Hub at https://docs.docker.com/docker-hub/builds/advanced/. You can see the process working properly at: * Nightly build on my fork: https://dev.azure.com/bmw0523/bmw/_build/results?buildId=345&view=logs&j=70ac378a-cb1f-50d1-b328-169807afbcfa * Release build on my fork: https://dev.azure.com/bmw0523/bmw/_build/results?buildId=346&view=logs&j=70ac378a-cb1f-50d1-b328-169807afbcfa * Nightly build on Certbot's Azure setup: https://dev.azure.com/certbot/certbot/_build/results?buildId=2426&view=logs&j=70ac378a-cb1f-50d1-b328-169807afbcfa The builds on my fork pushed to https://hub.docker.com/u/certbottest. The credentials for this account are in our shared vault in 1password if you want to play around with this. While the scripts will (almost?) always be run in CI, I tested the scripts successfully on macOS, Ubuntu 18.04, and Ubuntu 20.04, however, **the scripts do not seem to work when using the Docker snap, at least on Ubuntu 20.04.** It does work with the `docker.io` packages from `apt`. I was able to make things work by no longer setting `DOCKER_BUILDKIT`, but as I described in the code comments, this breaks things on Azure. When writing this PR, I tried to make the minimal modifications to our current set up to get the behavior we want. I'm planning on working on splitting the Docker builds into different Azure jobs so it doesn't increase the overall build time, but this isn't trivial so I figured it would be best done in a separate PR. * Remove license. * update build scripts * write deploy code * Remove unused READMEs. * rewrite readme * Make testing on a fork easier. * Set up Azure automation. * fix typo * Make output more verbose. * clean up cleanup...everybody everywhere * separate build and deploy * Document docker-hub credentials * Use Docker BuildKit when building. * Remove unneeded .gitignore files. * Fix tools/docker/README.md grammar Co-authored-by: ohemorange <ebportnoy@gmail.com> * Clarify <TAG> in README. * no docker snap * rename docker job Co-authored-by: Erica Portnoy <ebportnoy@gmail.com>
2020-08-11 16:09:38 -04:00
WORK_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
REPO_ROOT="$(dirname "$(dirname "${WORK_DIR}")")"
source "$WORK_DIR/lib/common"
Automate Docker builds in Azure (#8193) Fixes https://github.com/certbot/certbot/issues/8022, https://github.com/certbot-docker/certbot-docker/issues/25, and https://github.com/certbot-docker/certbot-docker/issues/20. This PR builds on https://github.com/certbot/certbot/pull/8192 to set up similar builds in Azure to what we currently have at release time as well as nightly builds allowing us to catch problems in these images before a release. It also fully automates our Docker deployments removing a manual step from our release process. We'll need to update our release instructions once this PR lands. If you're not familiar with our `certbot-docker` setup, you can read about how these scripts customized the build process on Docker Hub at https://docs.docker.com/docker-hub/builds/advanced/. You can see the process working properly at: * Nightly build on my fork: https://dev.azure.com/bmw0523/bmw/_build/results?buildId=345&view=logs&j=70ac378a-cb1f-50d1-b328-169807afbcfa * Release build on my fork: https://dev.azure.com/bmw0523/bmw/_build/results?buildId=346&view=logs&j=70ac378a-cb1f-50d1-b328-169807afbcfa * Nightly build on Certbot's Azure setup: https://dev.azure.com/certbot/certbot/_build/results?buildId=2426&view=logs&j=70ac378a-cb1f-50d1-b328-169807afbcfa The builds on my fork pushed to https://hub.docker.com/u/certbottest. The credentials for this account are in our shared vault in 1password if you want to play around with this. While the scripts will (almost?) always be run in CI, I tested the scripts successfully on macOS, Ubuntu 18.04, and Ubuntu 20.04, however, **the scripts do not seem to work when using the Docker snap, at least on Ubuntu 20.04.** It does work with the `docker.io` packages from `apt`. I was able to make things work by no longer setting `DOCKER_BUILDKIT`, but as I described in the code comments, this breaks things on Azure. When writing this PR, I tried to make the minimal modifications to our current set up to get the behavior we want. I'm planning on working on splitting the Docker builds into different Azure jobs so it doesn't increase the overall build time, but this isn't trivial so I figured it would be best done in a separate PR. * Remove license. * update build scripts * write deploy code * Remove unused READMEs. * rewrite readme * Make testing on a fork easier. * Set up Azure automation. * fix typo * Make output more verbose. * clean up cleanup...everybody everywhere * separate build and deploy * Document docker-hub credentials * Use Docker BuildKit when building. * Remove unneeded .gitignore files. * Fix tools/docker/README.md grammar Co-authored-by: ohemorange <ebportnoy@gmail.com> * Clarify <TAG> in README. * no docker snap * rename docker job Co-authored-by: Erica Portnoy <ebportnoy@gmail.com>
2020-08-11 16:09:38 -04:00
trap Cleanup EXIT
Cleanup() {
Automate Docker builds in Azure (#8193) Fixes https://github.com/certbot/certbot/issues/8022, https://github.com/certbot-docker/certbot-docker/issues/25, and https://github.com/certbot-docker/certbot-docker/issues/20. This PR builds on https://github.com/certbot/certbot/pull/8192 to set up similar builds in Azure to what we currently have at release time as well as nightly builds allowing us to catch problems in these images before a release. It also fully automates our Docker deployments removing a manual step from our release process. We'll need to update our release instructions once this PR lands. If you're not familiar with our `certbot-docker` setup, you can read about how these scripts customized the build process on Docker Hub at https://docs.docker.com/docker-hub/builds/advanced/. You can see the process working properly at: * Nightly build on my fork: https://dev.azure.com/bmw0523/bmw/_build/results?buildId=345&view=logs&j=70ac378a-cb1f-50d1-b328-169807afbcfa * Release build on my fork: https://dev.azure.com/bmw0523/bmw/_build/results?buildId=346&view=logs&j=70ac378a-cb1f-50d1-b328-169807afbcfa * Nightly build on Certbot's Azure setup: https://dev.azure.com/certbot/certbot/_build/results?buildId=2426&view=logs&j=70ac378a-cb1f-50d1-b328-169807afbcfa The builds on my fork pushed to https://hub.docker.com/u/certbottest. The credentials for this account are in our shared vault in 1password if you want to play around with this. While the scripts will (almost?) always be run in CI, I tested the scripts successfully on macOS, Ubuntu 18.04, and Ubuntu 20.04, however, **the scripts do not seem to work when using the Docker snap, at least on Ubuntu 20.04.** It does work with the `docker.io` packages from `apt`. I was able to make things work by no longer setting `DOCKER_BUILDKIT`, but as I described in the code comments, this breaks things on Azure. When writing this PR, I tried to make the minimal modifications to our current set up to get the behavior we want. I'm planning on working on splitting the Docker builds into different Azure jobs so it doesn't increase the overall build time, but this isn't trivial so I figured it would be best done in a separate PR. * Remove license. * update build scripts * write deploy code * Remove unused READMEs. * rewrite readme * Make testing on a fork easier. * Set up Azure automation. * fix typo * Make output more verbose. * clean up cleanup...everybody everywhere * separate build and deploy * Document docker-hub credentials * Use Docker BuildKit when building. * Remove unneeded .gitignore files. * Fix tools/docker/README.md grammar Co-authored-by: ohemorange <ebportnoy@gmail.com> * Clarify <TAG> in README. * no docker snap * rename docker job Co-authored-by: Erica Portnoy <ebportnoy@gmail.com>
2020-08-11 16:09:38 -04:00
rm -rf "$REPO_ROOT"/qemu-*-static || true
for plugin in "${CERTBOT_PLUGINS[@]}"; do
rm -rf "$REPO_ROOT/certbot-$plugin"/qemu-*-static || true
done
}
# Returns the translation from Docker to QEMU architecture
# Usage: GetQemuArch [amd64|arm32v6|arm64v8]
GetQemuArch() {
ARCH=$1
case "$ARCH" in
"amd64")
echo "x86_64"
;;
"arm32v6")
echo "arm"
;;
"arm64v8")
echo "aarch64"
;;
"*")
echo "Not supported build architecture '$1'." >&2
exit 1
esac
}
# Downloads QEMU static binary file for architecture
# Usage: DownloadQemuStatic [x86_64|arm|aarch64]
DownloadQemuStatic() {
ARCH=$1
QEMU_ARCH=$(GetQemuArch "$ARCH")
if [ ! -f "qemu-${QEMU_ARCH}-static" ]; then
QEMU_DOWNLOAD_URL="https://github.com/multiarch/qemu-user-static/releases/download"
QEMU_LATEST_TAG=$(curl -s https://api.github.com/repos/multiarch/qemu-user-static/tags \
| grep 'name.*v[0-9]' \
| head -n 1 \
| cut -d '"' -f 4)
curl -SL "${QEMU_DOWNLOAD_URL}/${QEMU_LATEST_TAG}/x86_64_qemu-$QEMU_ARCH-static.tar.gz" \
| tar xzv
fi
}
Automate Docker builds in Azure (#8193) Fixes https://github.com/certbot/certbot/issues/8022, https://github.com/certbot-docker/certbot-docker/issues/25, and https://github.com/certbot-docker/certbot-docker/issues/20. This PR builds on https://github.com/certbot/certbot/pull/8192 to set up similar builds in Azure to what we currently have at release time as well as nightly builds allowing us to catch problems in these images before a release. It also fully automates our Docker deployments removing a manual step from our release process. We'll need to update our release instructions once this PR lands. If you're not familiar with our `certbot-docker` setup, you can read about how these scripts customized the build process on Docker Hub at https://docs.docker.com/docker-hub/builds/advanced/. You can see the process working properly at: * Nightly build on my fork: https://dev.azure.com/bmw0523/bmw/_build/results?buildId=345&view=logs&j=70ac378a-cb1f-50d1-b328-169807afbcfa * Release build on my fork: https://dev.azure.com/bmw0523/bmw/_build/results?buildId=346&view=logs&j=70ac378a-cb1f-50d1-b328-169807afbcfa * Nightly build on Certbot's Azure setup: https://dev.azure.com/certbot/certbot/_build/results?buildId=2426&view=logs&j=70ac378a-cb1f-50d1-b328-169807afbcfa The builds on my fork pushed to https://hub.docker.com/u/certbottest. The credentials for this account are in our shared vault in 1password if you want to play around with this. While the scripts will (almost?) always be run in CI, I tested the scripts successfully on macOS, Ubuntu 18.04, and Ubuntu 20.04, however, **the scripts do not seem to work when using the Docker snap, at least on Ubuntu 20.04.** It does work with the `docker.io` packages from `apt`. I was able to make things work by no longer setting `DOCKER_BUILDKIT`, but as I described in the code comments, this breaks things on Azure. When writing this PR, I tried to make the minimal modifications to our current set up to get the behavior we want. I'm planning on working on splitting the Docker builds into different Azure jobs so it doesn't increase the overall build time, but this isn't trivial so I figured it would be best done in a separate PR. * Remove license. * update build scripts * write deploy code * Remove unused READMEs. * rewrite readme * Make testing on a fork easier. * Set up Azure automation. * fix typo * Make output more verbose. * clean up cleanup...everybody everywhere * separate build and deploy * Document docker-hub credentials * Use Docker BuildKit when building. * Remove unneeded .gitignore files. * Fix tools/docker/README.md grammar Co-authored-by: ohemorange <ebportnoy@gmail.com> * Clarify <TAG> in README. * no docker snap * rename docker job Co-authored-by: Erica Portnoy <ebportnoy@gmail.com>
2020-08-11 16:09:38 -04:00
TAG_BASE="$1"
Docker build improvements (#8218) Fixes https://github.com/certbot/certbot/issues/8208. Fixes https://github.com/certbot/certbot/issues/8198. In addition to those two linked issues, this PR: * Splits both the build and deploy steps based on architecture for performance. The Docker builds should no longer be the bottleneck in any stage of the pipeline. * Skips building Docker images for ARM on `test-` branches like [we do for snaps](https://github.com/certbot/certbot/blob/e8a232297d6faedb3b0bffc197a2e98c2e59edcb/.azure-pipelines/templates/jobs/packaging-jobs.yml#L67-L71). I initially didn't want to do this, but the ARM builds take ~18 minutes which is significantly longer than any other job currently running on our `test-` branches. You can see tests running on my fork at: * [Release pipeline](https://dev.azure.com/bmw0523/bmw/_build/results?buildId=387&view=results) * [Test pipeline](https://dev.azure.com/bmw0523/bmw/_build/results?buildId=388&view=results) * [Nightly pipeline](https://dev.azure.com/bmw0523/bmw/_build/results?buildId=390&view=results) * update script intro * update readme * ParseRequestedArch * build all arch in Azure * Build docker images during testing/packaging. * require global variable? * Error if TAG_BASE is empty. * prepare build job * change variable syntax * Update deploy stage. * remove old dockerTag param * add displayName * fix docker images command * split docker_build by arch * Allow deploying a subset of architectures. * deploy in parallel * Skip ARM builds on test- branches. * fix spacing
2020-08-18 13:48:01 -04:00
if [ -z "$TAG_BASE" ]; then
echo "We cannot tag Docker images with an empty string!" >&2
exit 1
fi
ParseRequestedArch "${2}"
# Register QEMU handlers
docker run --rm --privileged multiarch/qemu-user-static:register --reset
# Step 1: Certbot core Docker
DOCKER_REPO="${DOCKER_HUB_ORG}/certbot"
Docker build improvements (#8218) Fixes https://github.com/certbot/certbot/issues/8208. Fixes https://github.com/certbot/certbot/issues/8198. In addition to those two linked issues, this PR: * Splits both the build and deploy steps based on architecture for performance. The Docker builds should no longer be the bottleneck in any stage of the pipeline. * Skips building Docker images for ARM on `test-` branches like [we do for snaps](https://github.com/certbot/certbot/blob/e8a232297d6faedb3b0bffc197a2e98c2e59edcb/.azure-pipelines/templates/jobs/packaging-jobs.yml#L67-L71). I initially didn't want to do this, but the ARM builds take ~18 minutes which is significantly longer than any other job currently running on our `test-` branches. You can see tests running on my fork at: * [Release pipeline](https://dev.azure.com/bmw0523/bmw/_build/results?buildId=387&view=results) * [Test pipeline](https://dev.azure.com/bmw0523/bmw/_build/results?buildId=388&view=results) * [Nightly pipeline](https://dev.azure.com/bmw0523/bmw/_build/results?buildId=390&view=results) * update script intro * update readme * ParseRequestedArch * build all arch in Azure * Build docker images during testing/packaging. * require global variable? * Error if TAG_BASE is empty. * prepare build job * change variable syntax * Update deploy stage. * remove old dockerTag param * add displayName * fix docker images command * split docker_build by arch * Allow deploying a subset of architectures. * deploy in parallel * Skip ARM builds on test- branches. * fix spacing
2020-08-18 13:48:01 -04:00
for TARGET_ARCH in "${ALL_REQUESTED_ARCH[@]}"; do
pushd "${REPO_ROOT}"
DownloadQemuStatic "${TARGET_ARCH}"
QEMU_ARCH=$(GetQemuArch "${TARGET_ARCH}")
docker build \
--build-arg TARGET_ARCH="${TARGET_ARCH}" \
--build-arg QEMU_ARCH="${QEMU_ARCH}" \
-f "${WORK_DIR}/core/Dockerfile" \
-t "${DOCKER_REPO}:${TARGET_ARCH}-${TAG_BASE}" \
.
popd
done
Automate Docker builds in Azure (#8193) Fixes https://github.com/certbot/certbot/issues/8022, https://github.com/certbot-docker/certbot-docker/issues/25, and https://github.com/certbot-docker/certbot-docker/issues/20. This PR builds on https://github.com/certbot/certbot/pull/8192 to set up similar builds in Azure to what we currently have at release time as well as nightly builds allowing us to catch problems in these images before a release. It also fully automates our Docker deployments removing a manual step from our release process. We'll need to update our release instructions once this PR lands. If you're not familiar with our `certbot-docker` setup, you can read about how these scripts customized the build process on Docker Hub at https://docs.docker.com/docker-hub/builds/advanced/. You can see the process working properly at: * Nightly build on my fork: https://dev.azure.com/bmw0523/bmw/_build/results?buildId=345&view=logs&j=70ac378a-cb1f-50d1-b328-169807afbcfa * Release build on my fork: https://dev.azure.com/bmw0523/bmw/_build/results?buildId=346&view=logs&j=70ac378a-cb1f-50d1-b328-169807afbcfa * Nightly build on Certbot's Azure setup: https://dev.azure.com/certbot/certbot/_build/results?buildId=2426&view=logs&j=70ac378a-cb1f-50d1-b328-169807afbcfa The builds on my fork pushed to https://hub.docker.com/u/certbottest. The credentials for this account are in our shared vault in 1password if you want to play around with this. While the scripts will (almost?) always be run in CI, I tested the scripts successfully on macOS, Ubuntu 18.04, and Ubuntu 20.04, however, **the scripts do not seem to work when using the Docker snap, at least on Ubuntu 20.04.** It does work with the `docker.io` packages from `apt`. I was able to make things work by no longer setting `DOCKER_BUILDKIT`, but as I described in the code comments, this breaks things on Azure. When writing this PR, I tried to make the minimal modifications to our current set up to get the behavior we want. I'm planning on working on splitting the Docker builds into different Azure jobs so it doesn't increase the overall build time, but this isn't trivial so I figured it would be best done in a separate PR. * Remove license. * update build scripts * write deploy code * Remove unused READMEs. * rewrite readme * Make testing on a fork easier. * Set up Azure automation. * fix typo * Make output more verbose. * clean up cleanup...everybody everywhere * separate build and deploy * Document docker-hub credentials * Use Docker BuildKit when building. * Remove unneeded .gitignore files. * Fix tools/docker/README.md grammar Co-authored-by: ohemorange <ebportnoy@gmail.com> * Clarify <TAG> in README. * no docker snap * rename docker job Co-authored-by: Erica Portnoy <ebportnoy@gmail.com>
2020-08-11 16:09:38 -04:00
# Step 2: Certbot DNS plugins Docker images
for plugin in "${CERTBOT_PLUGINS[@]}"; do
DOCKER_REPO="${DOCKER_HUB_ORG}/${plugin}"
pushd "${REPO_ROOT}/certbot-${plugin}"
# Copy QEMU static binaries downloaded when building the core Certbot image
cp ../qemu-*-static .
Docker build improvements (#8218) Fixes https://github.com/certbot/certbot/issues/8208. Fixes https://github.com/certbot/certbot/issues/8198. In addition to those two linked issues, this PR: * Splits both the build and deploy steps based on architecture for performance. The Docker builds should no longer be the bottleneck in any stage of the pipeline. * Skips building Docker images for ARM on `test-` branches like [we do for snaps](https://github.com/certbot/certbot/blob/e8a232297d6faedb3b0bffc197a2e98c2e59edcb/.azure-pipelines/templates/jobs/packaging-jobs.yml#L67-L71). I initially didn't want to do this, but the ARM builds take ~18 minutes which is significantly longer than any other job currently running on our `test-` branches. You can see tests running on my fork at: * [Release pipeline](https://dev.azure.com/bmw0523/bmw/_build/results?buildId=387&view=results) * [Test pipeline](https://dev.azure.com/bmw0523/bmw/_build/results?buildId=388&view=results) * [Nightly pipeline](https://dev.azure.com/bmw0523/bmw/_build/results?buildId=390&view=results) * update script intro * update readme * ParseRequestedArch * build all arch in Azure * Build docker images during testing/packaging. * require global variable? * Error if TAG_BASE is empty. * prepare build job * change variable syntax * Update deploy stage. * remove old dockerTag param * add displayName * fix docker images command * split docker_build by arch * Allow deploying a subset of architectures. * deploy in parallel * Skip ARM builds on test- branches. * fix spacing
2020-08-18 13:48:01 -04:00
for TARGET_ARCH in "${ALL_REQUESTED_ARCH[@]}"; do
QEMU_ARCH=$(GetQemuArch "${TARGET_ARCH}")
BASE_IMAGE="${DOCKER_HUB_ORG}/certbot:${TARGET_ARCH}-${TAG_BASE}"
docker build \
--build-arg BASE_IMAGE="${BASE_IMAGE}" \
--build-arg QEMU_ARCH="${QEMU_ARCH}" \
-f "${WORK_DIR}/plugin/Dockerfile" \
-t "${DOCKER_REPO}:${TARGET_ARCH}-${TAG_BASE}" \
.
done
popd
done