From f6199bdb16f6c1691e97634a1375a5649cac4904 Mon Sep 17 00:00:00 2001 From: Mathieu Benoit Date: Thu, 20 Aug 2026 01:12:14 -0400 Subject: [PATCH] [FIX] stage 2: produce its first package MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Stage 2 could build and could not deliver. Four obstacles, all in the tail of package(), after a compile that had already succeeded. Our meson ships arch-meson and it passes --auto-features enabled, so each of the nine documentation tools this chroot lacks was a hard error, not a skipped feature. A wrapper appends --auto-features auto; meson honours the last one. Building those tools was the alternative and it is not close -- doxygen alone wants clang, fmt, spdlog, llvm-libs. Then bsdtar would not start: stage-1 libxml2 asks for the host's libicuuc.so.76, our icu ships 78, and bsdtar is what writes the package. The package that would fix it was the one being built. libarchive only links libxml2 for xar, which nothing here reads, so stage 1 drops it. Verified: libxml2 rebuilt against libicuuc.so.78, provides libxml2.so=16-64, zero multiarch paths. --- FR --- L'étage 2 savait bâtir et ne savait pas livrer. Quatre obstacles, tous dans la queue de package(), après une compilation déjà réussie. Notre meson livre arch-meson, qui passe --auto-features enabled : chacun des neuf outils de documentation absents de ce chroot devenait une erreur franche au lieu d'une option écartée. Une enveloppe ajoute --auto-features auto, meson retenant la dernière occurrence. Bâtir ces outils était l'autre voie et l'écart est net -- doxygen seul réclame clang, fmt, spdlog, llvm-libs. Puis bsdtar ne démarrait plus : le libxml2 de l'étage 1 réclame le libicuuc.so.76 de l'hôte, notre icu livre le 78, et bsdtar est ce qui écrit le paquet. Le paquet qui corrigeait cela était celui qu'on bâtissait. libarchive ne lie libxml2 que pour xar, que rien ici ne lit : l'étage 1 l'abandonne. Vérifié : libxml2 rebâti sur libicuuc.so.78, fournit libxml2.so=16-64, aucun chemin multiarch. Assisted-by: Claude Opus 5 --- TODO.md | 77 ++++++++++++++++++++++++++++++++++ patches/pkgbuild/libarchive.sh | 71 +++++++++++++++++++++++++++++++ patches/pkgbuild/libxml2.sh | 43 +++++++++++++++++++ scripts/build-stage1.sh | 28 +++++++++++-- scripts/build-stage2.sh | 72 +++++++++++++++++++++++++++++-- 5 files changed, 284 insertions(+), 7 deletions(-) create mode 100755 patches/pkgbuild/libarchive.sh create mode 100755 patches/pkgbuild/libxml2.sh diff --git a/TODO.md b/TODO.md index 0a6d52b..3b792c1 100644 --- a/TODO.md +++ b/TODO.md @@ -253,6 +253,83 @@ defaults is already the right one. Until then, pin rather than hope. --- +### Documentation is skipped in the stage-2 chroot — `scripts/build-stage2.sh` + +Our own meson package ships `/usr/bin/arch-meson`, and it passes +`--auto-features enabled`. That is right on Arch, whose build chroot has every +optional tool installed. Ours has none of them: + + doxygen xsltproc asciidoctor itstool convert + fig2dev elinks ducktype yelp-build + +With `enabled`, each absent tool is a hard error rather than a skipped +feature. libxml2 stopped on + + libxml2/doc/meson.build:3:10: ERROR: Program 'doxygen' not found + +The chroot now gets a `/usr/local/bin/arch-meson` wrapper appending +`--auto-features auto` — meson honours the last occurrence, so every other +choice arch-meson makes survives. `/usr/local/bin` comes first on the +chroot's PATH; `/usr/bin/arch-meson` is left exactly as the package shipped it. + +**Measured before choosing**, because building the tools was the alternative +and it is not close: doxygen alone wants `clang`, `fmt`, `spdlog` and +`llvm-libs` — a compiler infrastructure for a documentation generator. Behind +the other eight stand Ruby, ImageMagick and a GNOME stack. Several times the +size of everything built so far, for man pages. + +To restore at stage 3, once there is somewhere to build them from. + +### The docs split assumed the docs exist — `patches/pkgbuild/libxml2.sh` + +Consequence of the above, and the **fourth** appearance of one shape: pam's +PDFs, e2fsprogs' `fuse2fs`, cmake's emacs byte-compilation, and now + + mv: cannot stat '/usr/share/doc': No such file or directory + +A failure on the last line of `package()`, after a completely successful +compile, naming a path instead of a reason. It has never once been a broken +build. The move is made conditional rather than deleted — on the host the docs +do exist and belong in their own package. + +### libarchive drops xar for stage 1 — `patches/pkgbuild/libarchive.sh` + +Stage 2 reached its first packaging step and stopped on + + bsdtar: error while loading shared libraries: libicuuc.so.76 + ==> ERROR: Failed to create package file. + +The chain: `bsdtar` → `libarchive.so.13` → `libxml2.so.16` → `libicuuc.so.76`. +The libxml2 in the chroot is our stage-1 build, and stage 1 builds against the +HOST, whose ICU is 76; our icu package ships 78. So bsdtar cannot start — and +bsdtar is what makepkg uses to write the package. **The package that would fix +it is the one being built.** Nothing comes out of the chroot until this is +broken from outside it. + +Supplying `libicuuc.so.76` from the host is the obvious move and the wrong +one: untracked host binaries in `/usr/lib` are exactly what test-chroot.sh's +ARTEFACT check exists to catch. A symlink to 78 does not work either — ICU +version-suffixes every symbol, so `libicuuc.so.78` does not define +`u_strlen_76`. + +The root is that libarchive links libxml2 at all: it wants it for **xar**, an +Apple installer container. Nothing here reads one — makepkg writes +`.pkg.tar.gz`, pacman reads it. `--without-xml2 --without-expat` removes the +whole ICU chain instead of arguing about its version. Stage 1 only; +`STAGE2_SKIP_HOOKS` lists libarchive, so stage 2 builds it as Arch does, +against our own libxml2 and icu, where the versions agree. + +### Rebuilding one package at stage 1 — `scripts/build-stage1.sh` + +`build-stage1.sh` now takes package names, like stage 2 already did. Without +it, rebuilding one package meant deleting its line from `stage1.state` first — +editing the record of what was built in order to build something. That file is +this port's memory, and the standing instruction is to regenerate it from what +is really in `repo/s390x`, never from a log; hand-editing it is the same +mistake in a smaller form. An explicit name outranks the state file, because +being told "already built, skipping" would make the command useless for the +one job it has. + ## Environment, not the port ### Build locale — `scripts/bootstrap-pacman.sh` diff --git a/patches/pkgbuild/libarchive.sh b/patches/pkgbuild/libarchive.sh new file mode 100755 index 0000000..ecb0529 --- /dev/null +++ b/patches/pkgbuild/libarchive.sh @@ -0,0 +1,71 @@ +#!/usr/bin/env bash +# libarchive: no xar during the bootstrap. STAGE 1 ONLY. +# +# THE CYCLE, exactly as it presented itself. Stage 2 reached its very first +# packaging step and stopped on +# +# bsdtar: error while loading shared libraries: libicuuc.so.76 +# ==> ERROR: Failed to create package file. +# +# after libxml2 had compiled and installed perfectly. The chain: +# +# bsdtar -> libarchive.so.13 -> libxml2.so.16 -> libicuuc.so.76 +# +# The libxml2 in the chroot is our own stage-1 build, and stage 1 builds +# against the HOST, whose ICU is 76. Our icu package ships 78. So bsdtar +# cannot start -- and bsdtar is what makepkg uses to write the package. The +# package that would fix it is the one being built. Nothing can come out of +# the chroot until this is broken from outside. +# +# WHY NOT SUPPLY libicuuc.so.76. Copying the host's ICU into the chroot is the +# obvious move and it is the wrong one: it puts untracked host binaries in +# /usr/lib, which is precisely what test-chroot.sh's ARTEFACT check exists to +# catch. A symlink to 78 does not work either -- ICU version-suffixes every +# symbol, so libicuuc.so.78 does not define u_strlen_76. +# +# THE ROOT is that libarchive links libxml2 at all. It wants it for the xar +# format, which is an Apple installer container. Nothing in this bootstrap +# reads one: makepkg writes .pkg.tar.gz, pacman reads it. Dropping xar removes +# the entire ICU chain rather than papering over its version. +# +# STAGE 1 ONLY -- build-stage2.sh lists libarchive in STAGE2_SKIP_HOOKS. Stage +# 2 builds it as Arch does, with xar, against our own libxml2 and icu, where +# the versions agree. The divergence lives exactly as long as the cycle does. +# +# BOTH depends ARRAYS. libxml2 is declared twice: once at the top and once as +# libxml2.so in the depends+= inside package(). Removing only the visible one +# leaves the package uninstallable for a reason the diff appears to fix -- the +# fifth place, and the one that has cost the most time in this port. +set -euo pipefail +python3 - <<'ZZPY' +import io, re +s = io.open("PKGBUILD", encoding="utf-8").read() + +# ./configure: refuse xar outright. libarchive probes for BOTH libxml2 and +# expat and uses whichever it finds, so declining only one leaves the other +# to reintroduce the dependency the day the host installs it. +old = " --without-nettle \\\n" +assert s.count(old) == 1, "libarchive: --without-nettle is not where expected" +s = s.replace(old, old + " --without-xml2 \\\n --without-expat \\\n", 1) + +# depends=(...): the top-level array. +old = " 'libxml2'\n" +assert s.count(old) == 1, "libarchive: 'libxml2' not in depends as expected" +s = s.replace(old, "", 1) + +# depends+=(...) inside package(): the soname form. +old = " 'libxml2.so'\n" +assert s.count(old) == 1, "libarchive: 'libxml2.so' not in package() depends" +s = s.replace(old, "", 1) + +io.open("PKGBUILD", "w", encoding="utf-8").write(s) +ZZPY +grep -q -- '--without-xml2' PKGBUILD || { echo "libarchive: xar not disabled" >&2; exit 1; } +# Nothing named libxml2 may remain in ANY depends array -- checked by counting +# rather than by eye, because there were two and both looked like the only one. +if grep -nE "^\s*'?libxml2" PKGBUILD | grep -q .; then + echo "libarchive: a libxml2 dependency survived:" >&2 + grep -nE "^\s*'?libxml2" PKGBUILD >&2 + exit 1 +fi +echo "libarchive: xar disabled, libxml2 dropped from both depends arrays" diff --git a/patches/pkgbuild/libxml2.sh b/patches/pkgbuild/libxml2.sh new file mode 100755 index 0000000..42db211 --- /dev/null +++ b/patches/pkgbuild/libxml2.sh @@ -0,0 +1,43 @@ +#!/usr/bin/env bash +# libxml2: the docs split assumes the docs were built. +# +# mv: cannot stat '/usr/share/doc': No such file or directory +# +# package_libxml2() ends on +# +# mv "$pkgdir"/usr/share/doc -t doc/usr/share +# +# which feeds the libxml2-docs sub-package. On the build HOST the docs exist, +# because doxygen is installed there. Inside the stage-2 chroot they do not: +# that chroot has none of Arch's optional documentation tools, and +# build-stage2.sh wraps arch-meson with --auto-features auto precisely so a +# missing tool skips a feature instead of failing the build. The skipped +# feature then trips the split. +# +# FOURTH TIME THIS SHAPE APPEARS -- pam's PDFs, e2fsprogs' fuse2fs, cmake's +# emacs byte-compilation, and now this. A failure on the last line of +# package(), after a completely successful compile, naming a path rather than +# a reason. It has never once been a broken build. +# +# The move is made conditional rather than removed, because on the host the +# docs DO exist and belong in their own package. And libxml2-docs stays in +# pkgname: makepkg is content to produce an empty sub-package, and dropping it +# would diverge from Arch for a reason that only holds in one of the two +# stages. +set -euo pipefail +python3 - <<'ZZPY' +import io +s = io.open("PKGBUILD", encoding="utf-8").read() +old = ' mv "$pkgdir"/usr/share/doc -t doc/usr/share\n' +assert s.count(old) == 1, "libxml2: the docs move is not in the expected form" +new = (' # Present when doxygen was available, absent in the stage-2 chroot, which\n' + ' # has no documentation tools and builds with --auto-features auto.\n' + ' if [ -d "$pkgdir"/usr/share/doc ]; then\n' + ' mv "$pkgdir"/usr/share/doc -t doc/usr/share\n' + ' fi\n') +s = s.replace(old, new, 1) +io.open("PKGBUILD", "w", encoding="utf-8").write(s) +ZZPY +grep -q 'has no documentation tools' PKGBUILD || { + echo "libxml2: the docs move was not made conditional" >&2; exit 1; } +echo "libxml2: docs split tolerates a build without doxygen" diff --git a/scripts/build-stage1.sh b/scripts/build-stage1.sh index 123e34e..877ce92 100755 --- a/scripts/build-stage1.sh +++ b/scripts/build-stage1.sh @@ -262,7 +262,10 @@ build_watched() { } built() { grep -qxF "$1" "$STATE" 2>/dev/null; } -mark() { echo "$1" >> "$STATE"; } +# Appended only once: a forced rebuild of an already-built package must not +# leave two identical lines in the state file. built() uses grep -qxF, so a +# duplicate is harmless to the logic and confusing to a reader counting lines. +mark() { built "$1" || echo "$1" >> "$STATE"; } main() { mkdir -p "$WORK/pkg" "$REPO" @@ -271,9 +274,28 @@ main() { # reinstalls them silently builds with whatever was deployed weeks ago. # Cheap, idempotent, and it makes this repository the source of truth. install_host_shims + # Named packages rebuild on demand, the way stage 2 already works. + # + # Without this, rebuilding one package means deleting its line from + # stage1.state first -- editing the record of what was built in order to + # build something. That file is the port's memory; the opening instruction + # for this work is to regenerate it from what is REALLY in repo/s390x and + # never from a log, and hand-editing it is the same mistake in a smaller + # form. Naming a package is also how a hook gets tested: libarchive's xar + # fix needed exactly one package rebuilt, not a pass over a hundred and + # ninety. + local list=("${STAGE1_PACKAGES[@]}") forced=0 + if [ "$#" -gt 0 ]; then + list=("$@") + forced=1 + printf '== stage 1: rebuilding on request: %s ==\n' "$*" + fi local ok=0 fail=0 rc=0 failed=() - for p in "${STAGE1_PACKAGES[@]}"; do - if built "$p"; then + for p in "${list[@]}"; do + # An explicit request outranks the state file. Asking for a package + # that is already built and being told it was skipped would make the + # command useless for the one job it exists to do. + if [ "$forced" -eq 0 ] && built "$p"; then echo "== $p already built, skipping ==" continue fi diff --git a/scripts/build-stage2.sh b/scripts/build-stage2.sh index 9938f4d..c324c0c 100755 --- a/scripts/build-stage2.sh +++ b/scripts/build-stage2.sh @@ -163,7 +163,47 @@ DEBUG_CFLAGS="" DEBUG_CXXFLAGS="" EOC sudo grep -E '^(CARCH|CHOST|MAKEFLAGS|OPTIONS)=' "$ROOT/etc/makepkg.conf" | sed 's/^/ /' - printf ' compiler flags: commented out (stage 1 used none)\n' + printf ' compiler flags: emptied (stage 1 used none)\n' + + # --auto-features auto, for this chroot only. + # + # Our own meson package ships /usr/bin/arch-meson, and it passes + # --auto-features enabled -- correct on Arch, whose build chroot has every + # optional tool. Ours has none of them: + # + # doxygen xsltproc asciidoctor itstool convert fig2dev elinks + # ducktype yelp-build -- all ABSENT + # + # With `enabled`, each one is a hard error. libxml2 stops at + # + # libxml2/doc/meson.build:3:10: ERROR: Program 'doxygen' not found + # + # MEASURED BEFORE CHOOSING, because building them was the other option and + # it is not close: doxygen alone wants clang, fmt, spdlog and llvm-libs -- + # an entire compiler infrastructure for a documentation generator. Behind + # the other eight stand Ruby, ImageMagick and a GNOME stack. That is + # several times the size of everything built so far, for man pages. + # + # `auto` is meson's own default and means "build what you can". It is the + # honest setting for an environment with fewer tools, not a workaround -- + # and what it drops is visible in the artefact, which is where this port + # checks everything anyway. + # + # A WRAPPER, not a patched package: /usr/bin/arch-meson belongs to meson and + # stage 2 must not ship a modified copy of it. meson takes the LAST + # occurrence of an option, so appending wins while leaving every other + # choice arch-meson makes intact. in_chroot puts /usr/local/bin first on + # PATH so this is found. + sudo install -d -m0755 "$ROOT/usr/local/bin" + sudo tee "$ROOT/usr/local/bin/arch-meson" > /dev/null <<'EOW' +#!/usr/bin/env bash +# stage 2: this chroot has none of Arch's optional documentation tools, so a +# feature that cannot be built should be skipped rather than fatal. Appended, +# because meson honours the last occurrence. +exec /usr/bin/arch-meson "$@" --auto-features auto +EOW + sudo chmod 755 "$ROOT/usr/local/bin/arch-meson" + printf ' arch-meson: wrapped with --auto-features auto\n' # makepkg refuses to run as root, so the chroot needs the SAME uid as the # user who owns the bind-mounted sources. A bind mount carries the host's @@ -213,7 +253,7 @@ umount_chroot() { in_chroot() { sudo chroot --userspec="$BUILD_UID:$BUILD_GID" "$ROOT" \ /usr/bin/env -i \ - HOME=/build PATH=/usr/bin \ + HOME=/build PATH=/usr/local/bin:/usr/bin \ LC_ALL=C.UTF-8 \ /usr/bin/bash -lc "$*" } @@ -293,7 +333,9 @@ NOTE # the flag is correct and the hook would be a downgrade. # meson.sh joins them: it moves a wheel out of /usr/local, which only the # host's Debian-patched python puts there. -STAGE2_SKIP_HOOKS=(libgcrypt git meson) +# libarchive: stage 1 drops xar to break the bsdtar -> libxml2 -> libicuuc.so.76 +# cycle. In here the versions agree, so build it as Arch does. +STAGE2_SKIP_HOOKS=(libgcrypt git meson libarchive) # Packages whose sources must be extracted on the HOST, because the chroot # cannot extract anything until they are rebuilt. @@ -375,8 +417,30 @@ stage2_build() { # Install into the chroot so the NEXT package builds against it. With the # host's pacman and --root, because the chroot's own pacman does not start # until stage 2 has rebuilt it. + # + # --nodeps, and ONLY here. The first rebuilt package would not install: + # + # unable to satisfy dependency 'libicuuc.so=78-64' required by libxml2 + # + # Both halves of that are correct. This chroot runs makepkg's soname scan, + # so a package built in here declares versioned soname dependencies the way + # Arch's really do -- which is the faithful metadata stage 2 exists to + # produce. The stage-1 packages around it were built on the host under + # !autodeps, so our icu ships no `provides = libicuuc.so=78-64` to match. + # + # For the length of stage 2 the chroot is therefore a MIXED POPULATION: + # some packages describe their dependencies in Arch's terms and some in + # names only. No resolver can satisfy that, and none should be asked to -- + # it is a property of a bootstrap halfway through, not of the output. It + # dissolves on its own as the last package is rebuilt. + # + # What is NOT relaxed is the metadata in the packages: they keep their + # versioned depends, and test-chroot.sh's RESOLVE check runs the resolver + # against the finished repository with --nodeps OFF. The relaxation is one + # install command wide, and the check that would catch its consequences is + # still there. sudo pacman --root "$ROOT" --config "$CONF2" --cachedir "$CACHE" \ - --noconfirm -U "${produced[@]}" > "$WORK/stage2-inst-$name.txt" 2>&1 || { + --noconfirm --nodeps -U "${produced[@]}" > "$WORK/stage2-inst-$name.txt" 2>&1 || { tail -10 "$WORK/stage2-inst-$name.txt" >&2; return 1; } printf ' installed %s package(s)\n' "${#produced[@]}" }