Mathieu Benoit
b2b3f36026
Ma propre conclusion de ce matin était fausse, et une mesure l'a réfutée. J'avais écrit — code, README, commit — que le gel à 12 vCPU venait du SURENGAGEMENT : cette VM avait douze vCPU sur un hôte qui en avait deux. Descente réelle : l'étage 4 à huit vCPU, sur un parent qui en avait NEUF, charge 1,47, aucun surengagement. Gelé pareil. 32 Mio lus en 106 minutes, même RIP à trois relevés espacés de cinq minutes. C'est le nombre de vCPU de l'invité imbriqué, et rien d'autre. Le dimensionnement de bas en haut donnait 8 vCPU à l'étage 4, 7 au 5 : il rendait larges précisément les étages qui doivent rester étroits. Trois largeurs fixes le remplacent — métal, intermédiaire, fond. La mémoire et le disque, eux, restent dimensionnés depuis le bas. VCPU_INTERMEDIAIRE = 3 est une hypothèse assumée : deux démarre au quatrième étage, huit gèle, rien n'est mesuré entre les deux. --- EN --- My own conclusion from this morning was wrong, and a measurement refuted it. I had written — code, README, commit — that the 12-vCPU freeze came from OVERCOMMIT: that VM had twelve vCPU on a host with two. Real descent: level 4 at eight vCPU, on a parent with NINE, load 1.47, no overcommit whatsoever. Frozen all the same. 32 MiB read in 106 minutes, same RIP at three readings five minutes apart. It is the nested guest's vCPU count, nothing else. Bottom-up sizing gave level 4 eight vCPU and level 5 seven: it made wide exactly the levels that must stay narrow. Three fixed widths replace it — metal, intermediate, floor. Memory and disk stay sized from the bottom. VCPU_INTERMEDIAIRE = 3 is an owned hypothesis: two boots at the fourth level, eight freezes, nothing is measured in between. Assisted-by: claude-opus-5 (cherry picked from commit ee45ff4f333c69e3862e277334cb34efa39bbd57)
103 lines
5.2 KiB
Markdown
103 lines
5.2 KiB
Markdown
|
||
# LongTest — tests that create real machines
|
||
|
||
These are not unit tests. They create virtual machines, install systems on
|
||
them, and take hours. They live here and **not** in `test/`, which the unit
|
||
runner sweeps: `./script/test/run_unit_test.sh` must stay runnable in seconds
|
||
on any machine, including one without virtualisation.
|
||
|
||
Run them from the menu — `TODO › Execute › Test › Long tests` — or directly.
|
||
|
||
## deep_proxmox.py — how deep does Proxmox-in-Proxmox go?
|
||
|
||
The practicable nesting depth cannot be deduced, only measured — and one
|
||
measurement is not a measurement.
|
||
|
||
A manual look at one fourth-level VM found a guest **36 times slower than real
|
||
time** (583 seconds of wall clock for 16 seconds of guest time, each ACPI line
|
||
taking a second) and then a frozen kernel: identical RIP across three samples
|
||
two minutes apart, and **not one byte written** to disk.
|
||
|
||
Running this script **refuted the conclusion drawn from it**. Its own
|
||
fourth-level VM — 2 vCPU where the manual one had 12 — booted, installed, and
|
||
wrote gigabytes. What looked like a nesting ceiling was a *parallelism*
|
||
ceiling under nesting. That is exactly what the algorithm caps, and this is
|
||
how it stopped being a guess.
|
||
|
||
Which is the point of the script: a number obtained once, on one machine, in
|
||
one chain, is an anecdote.
|
||
|
||
```
|
||
./LongTest/deep_proxmox.py --depth 10 --dry-run # the plan, nothing created
|
||
./LongTest/deep_proxmox.py --depth 10 # hours
|
||
./LongTest/deep_proxmox.py --detruire # undo it
|
||
```
|
||
|
||
The descent is **uniform**. Every level, the first included, goes through the
|
||
same six steps: create, wait for ssh, install Proxmox, reboot and check the
|
||
kernel, bring pmxcfs back up, check the storage. Only creation differs —
|
||
libvirt locally, `qm` afterwards.
|
||
|
||
It sends **our** `install_proxmox.sh` over scp instead of letting the VM clone
|
||
the repository: it is our code we want to exercise, and the remote is often
|
||
behind the checkout — a fix absent from the remote made the same defect "come
|
||
back" on three VMs in a row.
|
||
|
||
### The resource algorithm — sized from the bottom up
|
||
|
||
The first version handed down whatever the parent could spare, and a real
|
||
descent showed what that costs. Level 4 ended up with 44 GB of memory and
|
||
2 vCPU **on a host that had 2** — a hundred percent overcommit, at every
|
||
level, with the hypervisor itself to serve on top. Its install ran past two
|
||
and a half hours against thirteen minutes for level 3, and extrapolating that
|
||
ratio gave five years for the tenth.
|
||
|
||
So the direction is reversed for **memory and disk**. The deepest level gets
|
||
what a test Proxmox actually asks for — 4 GB of memory, 25 GB of disk — and
|
||
every parent above it adds its own overhead and nothing else: 2 GiB and 10 GB.
|
||
A ten-level descent therefore asks its first level for 22 GB and 115 GB, where
|
||
handing resources down wanted 50 GB of memory for the same depth. The
|
||
processor follows a different rule; see below.
|
||
|
||
Three budgets can bound the depth, and `script/proxmox/nesting.py` names the
|
||
one that ran out:
|
||
|
||
* **memory** — every level must run its own daemons (`pve-cluster`,
|
||
`pvestatd`, `pvedaemon`, `pveproxy`) *and* hold its child;
|
||
* **disk** — the child's disk lives *inside* the parent's, which must also
|
||
hold its own system;
|
||
* **processor** — it does *not* grow with depth. Every nested level keeps a
|
||
fixed, narrow width; only the first level counts against the physical cores.
|
||
Either the machine can carry that first level or it can carry nothing.
|
||
|
||
That third rule is measured, and it cost two descents to get right. A nested
|
||
guest at the **fourth** level freezes in early boot as soon as it is wide:
|
||
twelve vCPU the first time, eight the second — same instruction pointer at
|
||
three readings five minutes apart, 32 MiB read and not one byte more for 106
|
||
minutes. Two vCPU boots.
|
||
|
||
The first freeze was blamed on **overcommit**: that VM had twelve vCPU on a
|
||
host with two. The second measurement refuted it — eight vCPU on a parent with
|
||
**nine**, load 1.47, no overcommit at all, and the same freeze. It is the
|
||
nested guest's vCPU count, not its ratio to its host's.
|
||
|
||
At the third level, 9 vCPU boots in 117 s. The threshold sits between the
|
||
third and fourth level, so no nested level is ever made wide. An earlier
|
||
version of this algorithm gave each parent one vCPU more than its child, which
|
||
made level 4 eight wide — exactly the frozen case. The rule made wide what
|
||
must stay narrow.
|
||
|
||
Hence three fixed widths: `VCPU_METAL` for level 1 (on bare metal, no freeze
|
||
risk — eleven vCPU booted there in 42 s), `VCPU_IMBRIQUE` for the deepest, and
|
||
`VCPU_INTERMEDIAIRE` in between, wide enough to host its child without being
|
||
as narrow as it. That middle number is a **hypothesis**: two is proven to boot
|
||
at the fourth level and eight is proven to freeze, with nothing measured in
|
||
between. The descent decides.
|
||
|
||
Memory is not the lever. On that same manual VM, dropping it from 9 GB to 2 GB
|
||
moved nothing — it stopped after reading the same 32 MiB, which is simply the
|
||
size of the boot files.
|
||
|
||
The plan is printed **before** anything is created, and the script never
|
||
promises a depth it knows will not fit — better to announce six levels and
|
||
reach six than to promise ten and die at the seventh without knowing why.
|