Mathieu Benoit
c046e027e9
Relecture adversaire du commit précédent : il avait CRÉÉ un danger. Le rapport
s'écrivant maintenant VM par VM, celui de la descente EN COURS est le plus
récent, et « --detruire » l'aurait choisi — qm destroy --purge sur l'arbre que
le processus installait encore. Deux garde-fous : un PID dans le rapport, et
un refus net tant qu'un autre deep_proxmox.py tourne. Le second est nécessaire
car une descente déjà lancée a l'ancien module en mémoire.
Reconnu par ARGUMENT, pas par sous-chaîne : mon propre « pgrep -f
deep_proxmox.py » de surveillance donnait deux faux positifs sur trois.
Trois autres, mêmes preuves :
- un rapport vide plus récent masquait celui qui nommait les VM réelles ;
- detruire() ne créditait jamais l'étage 1 : le décompte était décalé de un
dans tous les cas, donc l'avertissement sortait toujours ;
- _ssh_config_drop_hosts prenait l'indentation pour de la syntaxe. Sur un bloc
au corps non indenté, seule la ligne Host partait et ssh rattachait
« StrictHostKeyChecking no » au bloc précédent — un serveur de production.
Et un bloc partagé (« Host prod-db vm-a ») partait en entier.
Les cinq correctifs meurent sous mutation.
--- EN ---
Adversarial review of the previous commit: it had CREATED a hazard. With the
report now written VM by VM, the RUNNING descent's is the most recent, and
"--detruire" would have picked it — qm destroy --purge on the tree the process
was still installing. Two guards: a PID in the report, and a flat refusal
while another deep_proxmox.py runs. The second is needed because an
already-running descent holds the old module in memory.
Matched by ARGUMENT, not substring: my own monitoring "pgrep -f
deep_proxmox.py" produced two false positives out of three.
Three more, same evidence:
- a newer empty report masked the one naming the real VMs;
- detruire() never credited level 1: the count was off by one in every case,
so the warning always fired;
- _ssh_config_drop_hosts took indentation for syntax. On a block with an
unindented body only the Host line went, and ssh attached
"StrictHostKeyChecking no" to the preceding block — a production server.
And a shared block ("Host prod-db vm-a") went entirely.
All five fixes die under mutation.
Assisted-by: claude-opus-5
(cherry picked from commit 7d348d976f96e420b4fec4d879911b658a730ffa)
|
||
|---|---|---|
| .. | ||
| deep_proxmox.py | ||
| README.base.md | ||
| README.fr.md | ||
| README.md | ||
LongTest — tests that create real machines
These are not unit tests. They create virtual machines, install systems on
them, and take hours. They live here and not in test/, which the unit
runner sweeps: ./script/test/run_unit_test.sh must stay runnable in seconds
on any machine, including one without virtualisation.
Run them from the menu — TODO › Execute › Test › Long tests — or directly.
deep_proxmox.py — how deep does Proxmox-in-Proxmox go?
The practicable nesting depth cannot be deduced, only measured — and one measurement is not a measurement.
A manual look at one fourth-level VM found a guest 36 times slower than real time (583 seconds of wall clock for 16 seconds of guest time, each ACPI line taking a second) and then a frozen kernel: identical RIP across three samples two minutes apart, and not one byte written to disk.
Running this script refuted the conclusion drawn from it. Its own fourth-level VM — 2 vCPU where the manual one had 12 — booted, installed, and wrote gigabytes. What looked like a nesting ceiling was a parallelism ceiling under nesting. That is exactly what the algorithm caps, and this is how it stopped being a guess.
Which is the point of the script: a number obtained once, on one machine, in one chain, is an anecdote.
./LongTest/deep_proxmox.py --depth 10 --dry-run # the plan, nothing created
./LongTest/deep_proxmox.py --depth 10 # hours
./LongTest/deep_proxmox.py --detruire # undo it
The descent is uniform. Every level, the first included, goes through the
same six steps: create, wait for ssh, install Proxmox, reboot and check the
kernel, bring pmxcfs back up, check the storage. Only creation differs —
libvirt locally, qm afterwards.
It sends our install_proxmox.sh over scp instead of letting the VM clone
the repository: it is our code we want to exercise, and the remote is often
behind the checkout — a fix absent from the remote made the same defect "come
back" on three VMs in a row.
The resource algorithm — sized from the bottom up
The first version handed down whatever the parent could spare, and a real descent showed what that costs. Level 4 ended up with 44 GB of memory and 2 vCPU on a host that had 2 — a hundred percent overcommit, at every level, with the hypervisor itself to serve on top. Its install ran past two and a half hours against thirteen minutes for level 3, and extrapolating that ratio gave five years for the tenth.
So the direction is reversed. The deepest level gets what a test Proxmox actually asks for — 4 GB of memory, 25 GB of disk, 2 vCPU — and every parent above it adds its own overhead and nothing else: one vCPU, 2 GiB, 10 GB. A ten-level descent therefore asks its first level for 11 vCPU, 22 GB and 115 GB, where handing resources down wanted 50 GB of memory for the same depth.
Three budgets can bound the depth, and script/proxmox/nesting.py names the
one that ran out:
- memory — every level must run its own daemons (
pve-cluster,pvestatd,pvedaemon,pveproxy) and hold its child; - disk — the child's disk lives inside the parent's, which must also hold its own system;
- processor — each level wants one vCPU more than its child, so ten levels ask eleven of the first. Half the physical cores is the ceiling: the orchestrator runs on that machine too.
That third budget is measured, not assumed. Twelve vCPU at the fourth level froze the guest kernel in early boot — same instruction pointer at three readings two minutes apart — while two progressed. The number was not the culprit: that VM had twelve vCPU on a host with two, six times wider than its own machine. Overcommit freezes, not the twelve.
Memory is not the lever. On that same manual VM, dropping it from 9 GB to 2 GB moved nothing — it stopped after reading the same 32 MiB, which is simply the size of the boot files.
The plan is printed before anything is created, and the script never promises a depth it knows will not fit — better to announce six levels and reach six than to promise ten and die at the seventh without knowing why.