diff --git a/TODO.md b/TODO.md index 0a6d52b..3b792c1 100644 --- a/TODO.md +++ b/TODO.md @@ -253,6 +253,83 @@ defaults is already the right one. Until then, pin rather than hope. --- +### Documentation is skipped in the stage-2 chroot — `scripts/build-stage2.sh` + +Our own meson package ships `/usr/bin/arch-meson`, and it passes +`--auto-features enabled`. That is right on Arch, whose build chroot has every +optional tool installed. Ours has none of them: + + doxygen xsltproc asciidoctor itstool convert + fig2dev elinks ducktype yelp-build + +With `enabled`, each absent tool is a hard error rather than a skipped +feature. libxml2 stopped on + + libxml2/doc/meson.build:3:10: ERROR: Program 'doxygen' not found + +The chroot now gets a `/usr/local/bin/arch-meson` wrapper appending +`--auto-features auto` — meson honours the last occurrence, so every other +choice arch-meson makes survives. `/usr/local/bin` comes first on the +chroot's PATH; `/usr/bin/arch-meson` is left exactly as the package shipped it. + +**Measured before choosing**, because building the tools was the alternative +and it is not close: doxygen alone wants `clang`, `fmt`, `spdlog` and +`llvm-libs` — a compiler infrastructure for a documentation generator. Behind +the other eight stand Ruby, ImageMagick and a GNOME stack. Several times the +size of everything built so far, for man pages. + +To restore at stage 3, once there is somewhere to build them from. + +### The docs split assumed the docs exist — `patches/pkgbuild/libxml2.sh` + +Consequence of the above, and the **fourth** appearance of one shape: pam's +PDFs, e2fsprogs' `fuse2fs`, cmake's emacs byte-compilation, and now + + mv: cannot stat '/usr/share/doc': No such file or directory + +A failure on the last line of `package()`, after a completely successful +compile, naming a path instead of a reason. It has never once been a broken +build. The move is made conditional rather than deleted — on the host the docs +do exist and belong in their own package. + +### libarchive drops xar for stage 1 — `patches/pkgbuild/libarchive.sh` + +Stage 2 reached its first packaging step and stopped on + + bsdtar: error while loading shared libraries: libicuuc.so.76 + ==> ERROR: Failed to create package file. + +The chain: `bsdtar` → `libarchive.so.13` → `libxml2.so.16` → `libicuuc.so.76`. +The libxml2 in the chroot is our stage-1 build, and stage 1 builds against the +HOST, whose ICU is 76; our icu package ships 78. So bsdtar cannot start — and +bsdtar is what makepkg uses to write the package. **The package that would fix +it is the one being built.** Nothing comes out of the chroot until this is +broken from outside it. + +Supplying `libicuuc.so.76` from the host is the obvious move and the wrong +one: untracked host binaries in `/usr/lib` are exactly what test-chroot.sh's +ARTEFACT check exists to catch. A symlink to 78 does not work either — ICU +version-suffixes every symbol, so `libicuuc.so.78` does not define +`u_strlen_76`. + +The root is that libarchive links libxml2 at all: it wants it for **xar**, an +Apple installer container. Nothing here reads one — makepkg writes +`.pkg.tar.gz`, pacman reads it. `--without-xml2 --without-expat` removes the +whole ICU chain instead of arguing about its version. Stage 1 only; +`STAGE2_SKIP_HOOKS` lists libarchive, so stage 2 builds it as Arch does, +against our own libxml2 and icu, where the versions agree. + +### Rebuilding one package at stage 1 — `scripts/build-stage1.sh` + +`build-stage1.sh` now takes package names, like stage 2 already did. Without +it, rebuilding one package meant deleting its line from `stage1.state` first — +editing the record of what was built in order to build something. That file is +this port's memory, and the standing instruction is to regenerate it from what +is really in `repo/s390x`, never from a log; hand-editing it is the same +mistake in a smaller form. An explicit name outranks the state file, because +being told "already built, skipping" would make the command useless for the +one job it has. + ## Environment, not the port ### Build locale — `scripts/bootstrap-pacman.sh` diff --git a/patches/pkgbuild/libarchive.sh b/patches/pkgbuild/libarchive.sh new file mode 100755 index 0000000..ecb0529 --- /dev/null +++ b/patches/pkgbuild/libarchive.sh @@ -0,0 +1,71 @@ +#!/usr/bin/env bash +# libarchive: no xar during the bootstrap. STAGE 1 ONLY. +# +# THE CYCLE, exactly as it presented itself. Stage 2 reached its very first +# packaging step and stopped on +# +# bsdtar: error while loading shared libraries: libicuuc.so.76 +# ==> ERROR: Failed to create package file. +# +# after libxml2 had compiled and installed perfectly. The chain: +# +# bsdtar -> libarchive.so.13 -> libxml2.so.16 -> libicuuc.so.76 +# +# The libxml2 in the chroot is our own stage-1 build, and stage 1 builds +# against the HOST, whose ICU is 76. Our icu package ships 78. So bsdtar +# cannot start -- and bsdtar is what makepkg uses to write the package. The +# package that would fix it is the one being built. Nothing can come out of +# the chroot until this is broken from outside. +# +# WHY NOT SUPPLY libicuuc.so.76. Copying the host's ICU into the chroot is the +# obvious move and it is the wrong one: it puts untracked host binaries in +# /usr/lib, which is precisely what test-chroot.sh's ARTEFACT check exists to +# catch. A symlink to 78 does not work either -- ICU version-suffixes every +# symbol, so libicuuc.so.78 does not define u_strlen_76. +# +# THE ROOT is that libarchive links libxml2 at all. It wants it for the xar +# format, which is an Apple installer container. Nothing in this bootstrap +# reads one: makepkg writes .pkg.tar.gz, pacman reads it. Dropping xar removes +# the entire ICU chain rather than papering over its version. +# +# STAGE 1 ONLY -- build-stage2.sh lists libarchive in STAGE2_SKIP_HOOKS. Stage +# 2 builds it as Arch does, with xar, against our own libxml2 and icu, where +# the versions agree. The divergence lives exactly as long as the cycle does. +# +# BOTH depends ARRAYS. libxml2 is declared twice: once at the top and once as +# libxml2.so in the depends+= inside package(). Removing only the visible one +# leaves the package uninstallable for a reason the diff appears to fix -- the +# fifth place, and the one that has cost the most time in this port. +set -euo pipefail +python3 - <<'ZZPY' +import io, re +s = io.open("PKGBUILD", encoding="utf-8").read() + +# ./configure: refuse xar outright. libarchive probes for BOTH libxml2 and +# expat and uses whichever it finds, so declining only one leaves the other +# to reintroduce the dependency the day the host installs it. +old = " --without-nettle \\\n" +assert s.count(old) == 1, "libarchive: --without-nettle is not where expected" +s = s.replace(old, old + " --without-xml2 \\\n --without-expat \\\n", 1) + +# depends=(...): the top-level array. +old = " 'libxml2'\n" +assert s.count(old) == 1, "libarchive: 'libxml2' not in depends as expected" +s = s.replace(old, "", 1) + +# depends+=(...) inside package(): the soname form. +old = " 'libxml2.so'\n" +assert s.count(old) == 1, "libarchive: 'libxml2.so' not in package() depends" +s = s.replace(old, "", 1) + +io.open("PKGBUILD", "w", encoding="utf-8").write(s) +ZZPY +grep -q -- '--without-xml2' PKGBUILD || { echo "libarchive: xar not disabled" >&2; exit 1; } +# Nothing named libxml2 may remain in ANY depends array -- checked by counting +# rather than by eye, because there were two and both looked like the only one. +if grep -nE "^\s*'?libxml2" PKGBUILD | grep -q .; then + echo "libarchive: a libxml2 dependency survived:" >&2 + grep -nE "^\s*'?libxml2" PKGBUILD >&2 + exit 1 +fi +echo "libarchive: xar disabled, libxml2 dropped from both depends arrays" diff --git a/patches/pkgbuild/libxml2.sh b/patches/pkgbuild/libxml2.sh new file mode 100755 index 0000000..42db211 --- /dev/null +++ b/patches/pkgbuild/libxml2.sh @@ -0,0 +1,43 @@ +#!/usr/bin/env bash +# libxml2: the docs split assumes the docs were built. +# +# mv: cannot stat '/usr/share/doc': No such file or directory +# +# package_libxml2() ends on +# +# mv "$pkgdir"/usr/share/doc -t doc/usr/share +# +# which feeds the libxml2-docs sub-package. On the build HOST the docs exist, +# because doxygen is installed there. Inside the stage-2 chroot they do not: +# that chroot has none of Arch's optional documentation tools, and +# build-stage2.sh wraps arch-meson with --auto-features auto precisely so a +# missing tool skips a feature instead of failing the build. The skipped +# feature then trips the split. +# +# FOURTH TIME THIS SHAPE APPEARS -- pam's PDFs, e2fsprogs' fuse2fs, cmake's +# emacs byte-compilation, and now this. A failure on the last line of +# package(), after a completely successful compile, naming a path rather than +# a reason. It has never once been a broken build. +# +# The move is made conditional rather than removed, because on the host the +# docs DO exist and belong in their own package. And libxml2-docs stays in +# pkgname: makepkg is content to produce an empty sub-package, and dropping it +# would diverge from Arch for a reason that only holds in one of the two +# stages. +set -euo pipefail +python3 - <<'ZZPY' +import io +s = io.open("PKGBUILD", encoding="utf-8").read() +old = ' mv "$pkgdir"/usr/share/doc -t doc/usr/share\n' +assert s.count(old) == 1, "libxml2: the docs move is not in the expected form" +new = (' # Present when doxygen was available, absent in the stage-2 chroot, which\n' + ' # has no documentation tools and builds with --auto-features auto.\n' + ' if [ -d "$pkgdir"/usr/share/doc ]; then\n' + ' mv "$pkgdir"/usr/share/doc -t doc/usr/share\n' + ' fi\n') +s = s.replace(old, new, 1) +io.open("PKGBUILD", "w", encoding="utf-8").write(s) +ZZPY +grep -q 'has no documentation tools' PKGBUILD || { + echo "libxml2: the docs move was not made conditional" >&2; exit 1; } +echo "libxml2: docs split tolerates a build without doxygen" diff --git a/scripts/build-stage1.sh b/scripts/build-stage1.sh index 123e34e..877ce92 100755 --- a/scripts/build-stage1.sh +++ b/scripts/build-stage1.sh @@ -262,7 +262,10 @@ build_watched() { } built() { grep -qxF "$1" "$STATE" 2>/dev/null; } -mark() { echo "$1" >> "$STATE"; } +# Appended only once: a forced rebuild of an already-built package must not +# leave two identical lines in the state file. built() uses grep -qxF, so a +# duplicate is harmless to the logic and confusing to a reader counting lines. +mark() { built "$1" || echo "$1" >> "$STATE"; } main() { mkdir -p "$WORK/pkg" "$REPO" @@ -271,9 +274,28 @@ main() { # reinstalls them silently builds with whatever was deployed weeks ago. # Cheap, idempotent, and it makes this repository the source of truth. install_host_shims + # Named packages rebuild on demand, the way stage 2 already works. + # + # Without this, rebuilding one package means deleting its line from + # stage1.state first -- editing the record of what was built in order to + # build something. That file is the port's memory; the opening instruction + # for this work is to regenerate it from what is REALLY in repo/s390x and + # never from a log, and hand-editing it is the same mistake in a smaller + # form. Naming a package is also how a hook gets tested: libarchive's xar + # fix needed exactly one package rebuilt, not a pass over a hundred and + # ninety. + local list=("${STAGE1_PACKAGES[@]}") forced=0 + if [ "$#" -gt 0 ]; then + list=("$@") + forced=1 + printf '== stage 1: rebuilding on request: %s ==\n' "$*" + fi local ok=0 fail=0 rc=0 failed=() - for p in "${STAGE1_PACKAGES[@]}"; do - if built "$p"; then + for p in "${list[@]}"; do + # An explicit request outranks the state file. Asking for a package + # that is already built and being told it was skipped would make the + # command useless for the one job it exists to do. + if [ "$forced" -eq 0 ] && built "$p"; then echo "== $p already built, skipping ==" continue fi diff --git a/scripts/build-stage2.sh b/scripts/build-stage2.sh index 9938f4d..c324c0c 100755 --- a/scripts/build-stage2.sh +++ b/scripts/build-stage2.sh @@ -163,7 +163,47 @@ DEBUG_CFLAGS="" DEBUG_CXXFLAGS="" EOC sudo grep -E '^(CARCH|CHOST|MAKEFLAGS|OPTIONS)=' "$ROOT/etc/makepkg.conf" | sed 's/^/ /' - printf ' compiler flags: commented out (stage 1 used none)\n' + printf ' compiler flags: emptied (stage 1 used none)\n' + + # --auto-features auto, for this chroot only. + # + # Our own meson package ships /usr/bin/arch-meson, and it passes + # --auto-features enabled -- correct on Arch, whose build chroot has every + # optional tool. Ours has none of them: + # + # doxygen xsltproc asciidoctor itstool convert fig2dev elinks + # ducktype yelp-build -- all ABSENT + # + # With `enabled`, each one is a hard error. libxml2 stops at + # + # libxml2/doc/meson.build:3:10: ERROR: Program 'doxygen' not found + # + # MEASURED BEFORE CHOOSING, because building them was the other option and + # it is not close: doxygen alone wants clang, fmt, spdlog and llvm-libs -- + # an entire compiler infrastructure for a documentation generator. Behind + # the other eight stand Ruby, ImageMagick and a GNOME stack. That is + # several times the size of everything built so far, for man pages. + # + # `auto` is meson's own default and means "build what you can". It is the + # honest setting for an environment with fewer tools, not a workaround -- + # and what it drops is visible in the artefact, which is where this port + # checks everything anyway. + # + # A WRAPPER, not a patched package: /usr/bin/arch-meson belongs to meson and + # stage 2 must not ship a modified copy of it. meson takes the LAST + # occurrence of an option, so appending wins while leaving every other + # choice arch-meson makes intact. in_chroot puts /usr/local/bin first on + # PATH so this is found. + sudo install -d -m0755 "$ROOT/usr/local/bin" + sudo tee "$ROOT/usr/local/bin/arch-meson" > /dev/null <<'EOW' +#!/usr/bin/env bash +# stage 2: this chroot has none of Arch's optional documentation tools, so a +# feature that cannot be built should be skipped rather than fatal. Appended, +# because meson honours the last occurrence. +exec /usr/bin/arch-meson "$@" --auto-features auto +EOW + sudo chmod 755 "$ROOT/usr/local/bin/arch-meson" + printf ' arch-meson: wrapped with --auto-features auto\n' # makepkg refuses to run as root, so the chroot needs the SAME uid as the # user who owns the bind-mounted sources. A bind mount carries the host's @@ -213,7 +253,7 @@ umount_chroot() { in_chroot() { sudo chroot --userspec="$BUILD_UID:$BUILD_GID" "$ROOT" \ /usr/bin/env -i \ - HOME=/build PATH=/usr/bin \ + HOME=/build PATH=/usr/local/bin:/usr/bin \ LC_ALL=C.UTF-8 \ /usr/bin/bash -lc "$*" } @@ -293,7 +333,9 @@ NOTE # the flag is correct and the hook would be a downgrade. # meson.sh joins them: it moves a wheel out of /usr/local, which only the # host's Debian-patched python puts there. -STAGE2_SKIP_HOOKS=(libgcrypt git meson) +# libarchive: stage 1 drops xar to break the bsdtar -> libxml2 -> libicuuc.so.76 +# cycle. In here the versions agree, so build it as Arch does. +STAGE2_SKIP_HOOKS=(libgcrypt git meson libarchive) # Packages whose sources must be extracted on the HOST, because the chroot # cannot extract anything until they are rebuilt. @@ -375,8 +417,30 @@ stage2_build() { # Install into the chroot so the NEXT package builds against it. With the # host's pacman and --root, because the chroot's own pacman does not start # until stage 2 has rebuilt it. + # + # --nodeps, and ONLY here. The first rebuilt package would not install: + # + # unable to satisfy dependency 'libicuuc.so=78-64' required by libxml2 + # + # Both halves of that are correct. This chroot runs makepkg's soname scan, + # so a package built in here declares versioned soname dependencies the way + # Arch's really do -- which is the faithful metadata stage 2 exists to + # produce. The stage-1 packages around it were built on the host under + # !autodeps, so our icu ships no `provides = libicuuc.so=78-64` to match. + # + # For the length of stage 2 the chroot is therefore a MIXED POPULATION: + # some packages describe their dependencies in Arch's terms and some in + # names only. No resolver can satisfy that, and none should be asked to -- + # it is a property of a bootstrap halfway through, not of the output. It + # dissolves on its own as the last package is rebuilt. + # + # What is NOT relaxed is the metadata in the packages: they keep their + # versioned depends, and test-chroot.sh's RESOLVE check runs the resolver + # against the finished repository with --nodeps OFF. The relaxation is one + # install command wide, and the check that would catch its consequences is + # still there. sudo pacman --root "$ROOT" --config "$CONF2" --cachedir "$CACHE" \ - --noconfirm -U "${produced[@]}" > "$WORK/stage2-inst-$name.txt" 2>&1 || { + --noconfirm --nodeps -U "${produced[@]}" > "$WORK/stage2-inst-$name.txt" 2>&1 || { tail -10 "$WORK/stage2-inst-$name.txt" >&2; return 1; } printf ' installed %s package(s)\n' "${#produced[@]}" }