Mathieu Benoit
8e5f5b9097
Le défaut promettait dix étages qu'aucune machine ne tient. Une descente
complète par ligne, sur 28 cœurs :
étage 1 : 0 s d'amorçage, 200 s d'installation, 280 s en tout
étage 2 : 37 s, 344 s, 495 s
étage 3 : 93 s, 777 s, 1 064 s
étage 4 : 15 608 s, 26 306 s, n'a pas abouti
Trois étages coûtent une demi-heure. Le quatrième a coûté 4 h 20 d'amorçage et
7 h 18 d'installation sur la même machine : tout y est 15 à 30 fois plus lent,
pas une seule étape. C'est aussi là que les fabricants s'arrêtent — le
quatrième étage est le troisième hyperviseur imbriqué, AMD en documente deux.
La profondeur reste le seul paramètre, et le README dit désormais ce qu'on
achète en la montant.
--- EN ---
The default promised ten levels no machine can hold. One full descent per row,
on 28 cores:
level 1: 0 s boot, 200 s install, 280 s total
level 2: 37 s, 344 s, 495 s
level 3: 93 s, 777 s, 1 064 s
level 4: 15 608 s, 26 306 s, did not finish
Three levels cost half an hour. The fourth cost 4 h 20 of boot and 7 h 18 of
install on the same machine: everything there is 15 to 30 times slower, not one
step. It is also where the vendors stop — level 4 is the third nested
hypervisor, and AMD documents two.
Depth remains the only parameter, and the README now says what raising it buys.
Assisted-by: claude-opus-5
(cherry picked from commit 226d6ffa3c664ba9e1a6dfdf48d52027b52eda23)
128 lines
6.3 KiB
Markdown
128 lines
6.3 KiB
Markdown
|
||
# LongTest — tests that create real machines
|
||
|
||
These are not unit tests. They create virtual machines, install systems on
|
||
them, and take hours. They live here and **not** in `test/`, which the unit
|
||
runner sweeps: `./script/test/run_unit_test.sh` must stay runnable in seconds
|
||
on any machine, including one without virtualisation.
|
||
|
||
Run them from the menu — `TODO › Execute › Test › Long tests` — or directly.
|
||
|
||
## deep_proxmox.py — how deep does Proxmox-in-Proxmox go?
|
||
|
||
The practicable nesting depth cannot be deduced, only measured — and one
|
||
measurement is not a measurement.
|
||
|
||
A manual look at one fourth-level VM found a guest **36 times slower than real
|
||
time** (583 seconds of wall clock for 16 seconds of guest time, each ACPI line
|
||
taking a second) and then a frozen kernel: identical RIP across three samples
|
||
two minutes apart, and **not one byte written** to disk.
|
||
|
||
Running this script **refuted the conclusion drawn from it**. Its own
|
||
fourth-level VM — 2 vCPU where the manual one had 12 — booted, installed, and
|
||
wrote gigabytes. What looked like a nesting ceiling was a *parallelism*
|
||
ceiling under nesting. That is exactly what the algorithm caps, and this is
|
||
how it stopped being a guess.
|
||
|
||
Which is the point of the script: a number obtained once, on one machine, in
|
||
one chain, is an anecdote.
|
||
|
||
```
|
||
./LongTest/deep_proxmox.py # three levels, ~30 minutes
|
||
./LongTest/deep_proxmox.py --dry-run # the plan, nothing created
|
||
./LongTest/deep_proxmox.py --depth 5 # ask for more, knowingly
|
||
./LongTest/deep_proxmox.py --detruire # undo it
|
||
```
|
||
|
||
### How deep is worth asking for
|
||
|
||
The depth is the only setting, and **three** is the default because three
|
||
works. Measured on a 28-core machine, one full descent per row:
|
||
|
||
| level | boot (ssh) | install | total |
|
||
|------:|-----------:|--------:|------:|
|
||
| 1 | 0 s | 200 s | 280 s |
|
||
| 2 | 37 s | 344 s | 495 s |
|
||
| 3 | 93 s | 777 s | 1 064 s |
|
||
| 4 | **15 608 s** | **26 306 s** | did not finish |
|
||
|
||
Three levels cost half an hour. The **fourth** cost 4 h 20 of boot and 7 h 18
|
||
of install on the same machine — everything there is 15 to 30 times slower, not
|
||
just one step. And it lands exactly where the hardware vendors stop: level 4 is
|
||
the *third* nested hypervisor, and AMD documents two.
|
||
|
||
A wider guest makes it worse, sharply: at level 4, one extra vCPU multiplied
|
||
the boot by 9.4 (1 664 s at two vCPU, 15 608 s at three), and at eight vCPU the
|
||
guest read 32 MiB in 106 minutes with a static instruction pointer. At levels 2
|
||
and 3 that same vCPU costs nothing.
|
||
|
||
So: three by default, five if you want to know, ten only to watch the wall.
|
||
|
||
The descent is **uniform**. Every level, the first included, goes through the
|
||
same six steps: create, wait for ssh, install Proxmox, reboot and check the
|
||
kernel, bring pmxcfs back up, check the storage. Only creation differs —
|
||
libvirt locally, `qm` afterwards.
|
||
|
||
It sends **our** `install_proxmox.sh` over scp instead of letting the VM clone
|
||
the repository: it is our code we want to exercise, and the remote is often
|
||
behind the checkout — a fix absent from the remote made the same defect "come
|
||
back" on three VMs in a row.
|
||
|
||
### The resource algorithm — sized from the bottom up
|
||
|
||
The first version handed down whatever the parent could spare, and a real
|
||
descent showed what that costs. Level 4 ended up with 44 GB of memory and
|
||
2 vCPU **on a host that had 2** — a hundred percent overcommit, at every
|
||
level, with the hypervisor itself to serve on top. Its install ran past two
|
||
and a half hours against thirteen minutes for level 3, and extrapolating that
|
||
ratio gave five years for the tenth.
|
||
|
||
So the direction is reversed for **memory and disk**. The deepest level gets
|
||
what a test Proxmox actually asks for — 4 GB of memory, 25 GB of disk — and
|
||
every parent above it adds its own overhead and nothing else: 2 GiB and 10 GB.
|
||
A ten-level descent therefore asks its first level for 22 GB and 115 GB, where
|
||
handing resources down wanted 50 GB of memory for the same depth. The
|
||
processor follows a different rule; see below.
|
||
|
||
Three budgets can bound the depth, and `script/proxmox/nesting.py` names the
|
||
one that ran out:
|
||
|
||
* **memory** — every level must run its own daemons (`pve-cluster`,
|
||
`pvestatd`, `pvedaemon`, `pveproxy`) *and* hold its child;
|
||
* **disk** — the child's disk lives *inside* the parent's, which must also
|
||
hold its own system;
|
||
* **processor** — it does *not* grow with depth. Every nested level keeps a
|
||
fixed, narrow width; only the first level counts against the physical cores.
|
||
Either the machine can carry that first level or it can carry nothing.
|
||
|
||
That third rule is measured, and it cost two descents to get right. A nested
|
||
guest at the **fourth** level freezes in early boot as soon as it is wide:
|
||
twelve vCPU the first time, eight the second — same instruction pointer at
|
||
three readings five minutes apart, 32 MiB read and not one byte more for 106
|
||
minutes. Two vCPU boots.
|
||
|
||
The first freeze was blamed on **overcommit**: that VM had twelve vCPU on a
|
||
host with two. The second measurement refuted it — eight vCPU on a parent with
|
||
**nine**, load 1.47, no overcommit at all, and the same freeze. It is the
|
||
nested guest's vCPU count, not its ratio to its host's.
|
||
|
||
At the third level, 9 vCPU boots in 117 s. The threshold sits between the
|
||
third and fourth level, so no nested level is ever made wide. An earlier
|
||
version of this algorithm gave each parent one vCPU more than its child, which
|
||
made level 4 eight wide — exactly the frozen case. The rule made wide what
|
||
must stay narrow.
|
||
|
||
Hence three fixed widths: `VCPU_METAL` for level 1 (on bare metal, no freeze
|
||
risk — eleven vCPU booted there in 42 s), `VCPU_IMBRIQUE` for the deepest, and
|
||
`VCPU_INTERMEDIAIRE` in between, wide enough to host its child without being
|
||
as narrow as it. That middle number is a **hypothesis**: two is proven to boot
|
||
at the fourth level and eight is proven to freeze, with nothing measured in
|
||
between. The descent decides.
|
||
|
||
Memory is not the lever. On that same manual VM, dropping it from 9 GB to 2 GB
|
||
moved nothing — it stopped after reading the same 32 MiB, which is simply the
|
||
size of the boot files.
|
||
|
||
The plan is printed **before** anything is created, and the script never
|
||
promises a depth it knows will not fit — better to announce six levels and
|
||
reach six than to promise ten and die at the seventh without knowing why.
|