Mathieu Benoit
84ec78a61d
LongTest était le SEUL répertoire en CamelCase que nous ayons créé. Les deux exceptions sous script/ — OCA_maintainer-tools, OCA_odoo-module-migrator — sont des noms de dépôts amont tirés par Google Repo, pas les nôtres. Tout le reste est en minuscules avec des soulignés : image_db, code_generator, fork_github_repo, shell_script_odoo. Le nom avait été repris tel qu'il m'avait été dicté, sans être confronté à la convention — le contrôle même que le reste de ce travail applique partout. 50 occurrences dans 8 fichiers. Le menu TODO résout le nouveau chemin, l'essai à blanc passe, et les 125 tests des trois fichiers touchés restent verts. --- EN --- LongTest was the ONLY CamelCase directory we created. The two exceptions under script/ — OCA_maintainer-tools, OCA_odoo-module-migrator — are upstream repo names pulled by Google Repo, not ours. Everything else is lowercase with underscores: image_db, code_generator, fork_github_repo, shell_script_odoo. The name had been taken as dictated, without being checked against the convention — the very check the rest of this work applies everywhere. 50 occurrences across 8 files. The TODO menu resolves the new path, the dry run passes, and the 125 tests in the three touched files stay green. Assisted-by: claude-opus-5 (cherry picked from commit 170ee61e50dfeff638c84c07eadac00c62526e4d)
128 lines
6.3 KiB
Markdown
128 lines
6.3 KiB
Markdown
|
||
# long_test — tests that create real machines
|
||
|
||
These are not unit tests. They create virtual machines, install systems on
|
||
them, and take hours. They live here and **not** in `test/`, which the unit
|
||
runner sweeps: `./script/test/run_unit_test.sh` must stay runnable in seconds
|
||
on any machine, including one without virtualisation.
|
||
|
||
Run them from the menu — `TODO › Execute › Test › Long tests` — or directly.
|
||
|
||
## deep_proxmox.py — how deep does Proxmox-in-Proxmox go?
|
||
|
||
The practicable nesting depth cannot be deduced, only measured — and one
|
||
measurement is not a measurement.
|
||
|
||
A manual look at one fourth-level VM found a guest **36 times slower than real
|
||
time** (583 seconds of wall clock for 16 seconds of guest time, each ACPI line
|
||
taking a second) and then a frozen kernel: identical RIP across three samples
|
||
two minutes apart, and **not one byte written** to disk.
|
||
|
||
Running this script **refuted the conclusion drawn from it**. Its own
|
||
fourth-level VM — 2 vCPU where the manual one had 12 — booted, installed, and
|
||
wrote gigabytes. What looked like a nesting ceiling was a *parallelism*
|
||
ceiling under nesting. That is exactly what the algorithm caps, and this is
|
||
how it stopped being a guess.
|
||
|
||
Which is the point of the script: a number obtained once, on one machine, in
|
||
one chain, is an anecdote.
|
||
|
||
```
|
||
./long_test/deep_proxmox.py # three levels, ~30 minutes
|
||
./long_test/deep_proxmox.py --dry-run # the plan, nothing created
|
||
./long_test/deep_proxmox.py --depth 5 # ask for more, knowingly
|
||
./long_test/deep_proxmox.py --detruire # undo it
|
||
```
|
||
|
||
### How deep is worth asking for
|
||
|
||
The depth is the only setting, and **three** is the default because three
|
||
works. Measured on a 28-core machine, one full descent per row:
|
||
|
||
| level | boot (ssh) | install | total |
|
||
|------:|-----------:|--------:|------:|
|
||
| 1 | 0 s | 200 s | 280 s |
|
||
| 2 | 37 s | 344 s | 495 s |
|
||
| 3 | 93 s | 777 s | 1 064 s |
|
||
| 4 | **15 608 s** | **26 306 s** | did not finish |
|
||
|
||
Three levels cost half an hour. The **fourth** cost 4 h 20 of boot and 7 h 18
|
||
of install on the same machine — everything there is 15 to 30 times slower, not
|
||
just one step. And it lands exactly where the hardware vendors stop: level 4 is
|
||
the *third* nested hypervisor, and AMD documents two.
|
||
|
||
A wider guest makes it worse, sharply: at level 4, one extra vCPU multiplied
|
||
the boot by 9.4 (1 664 s at two vCPU, 15 608 s at three), and at eight vCPU the
|
||
guest read 32 MiB in 106 minutes with a static instruction pointer. At levels 2
|
||
and 3 that same vCPU costs nothing.
|
||
|
||
So: three by default, five if you want to know, ten only to watch the wall.
|
||
|
||
The descent is **uniform**. Every level, the first included, goes through the
|
||
same six steps: create, wait for ssh, install Proxmox, reboot and check the
|
||
kernel, bring pmxcfs back up, check the storage. Only creation differs —
|
||
libvirt locally, `qm` afterwards.
|
||
|
||
It sends **our** `install_proxmox.sh` over scp instead of letting the VM clone
|
||
the repository: it is our code we want to exercise, and the remote is often
|
||
behind the checkout — a fix absent from the remote made the same defect "come
|
||
back" on three VMs in a row.
|
||
|
||
### The resource algorithm — sized from the bottom up
|
||
|
||
The first version handed down whatever the parent could spare, and a real
|
||
descent showed what that costs. Level 4 ended up with 44 GB of memory and
|
||
2 vCPU **on a host that had 2** — a hundred percent overcommit, at every
|
||
level, with the hypervisor itself to serve on top. Its install ran past two
|
||
and a half hours against thirteen minutes for level 3, and extrapolating that
|
||
ratio gave five years for the tenth.
|
||
|
||
So the direction is reversed for **memory and disk**. The deepest level gets
|
||
what a test Proxmox actually asks for — 4 GB of memory, 25 GB of disk — and
|
||
every parent above it adds its own overhead and nothing else: 2 GiB and 10 GB.
|
||
A ten-level descent therefore asks its first level for 22 GB and 115 GB, where
|
||
handing resources down wanted 50 GB of memory for the same depth. The
|
||
processor follows a different rule; see below.
|
||
|
||
Three budgets can bound the depth, and `script/proxmox/nesting.py` names the
|
||
one that ran out:
|
||
|
||
* **memory** — every level must run its own daemons (`pve-cluster`,
|
||
`pvestatd`, `pvedaemon`, `pveproxy`) *and* hold its child;
|
||
* **disk** — the child's disk lives *inside* the parent's, which must also
|
||
hold its own system;
|
||
* **processor** — it does *not* grow with depth. Every nested level keeps a
|
||
fixed, narrow width; only the first level counts against the physical cores.
|
||
Either the machine can carry that first level or it can carry nothing.
|
||
|
||
That third rule is measured, and it cost two descents to get right. A nested
|
||
guest at the **fourth** level freezes in early boot as soon as it is wide:
|
||
twelve vCPU the first time, eight the second — same instruction pointer at
|
||
three readings five minutes apart, 32 MiB read and not one byte more for 106
|
||
minutes. Two vCPU boots.
|
||
|
||
The first freeze was blamed on **overcommit**: that VM had twelve vCPU on a
|
||
host with two. The second measurement refuted it — eight vCPU on a parent with
|
||
**nine**, load 1.47, no overcommit at all, and the same freeze. It is the
|
||
nested guest's vCPU count, not its ratio to its host's.
|
||
|
||
At the third level, 9 vCPU boots in 117 s. The threshold sits between the
|
||
third and fourth level, so no nested level is ever made wide. An earlier
|
||
version of this algorithm gave each parent one vCPU more than its child, which
|
||
made level 4 eight wide — exactly the frozen case. The rule made wide what
|
||
must stay narrow.
|
||
|
||
Hence three fixed widths: `VCPU_METAL` for level 1 (on bare metal, no freeze
|
||
risk — eleven vCPU booted there in 42 s), `VCPU_IMBRIQUE` for the deepest, and
|
||
`VCPU_INTERMEDIAIRE` in between, wide enough to host its child without being
|
||
as narrow as it. That middle number is a **hypothesis**: two is proven to boot
|
||
at the fourth level and eight is proven to freeze, with nothing measured in
|
||
between. The descent decides.
|
||
|
||
Memory is not the lever. On that same manual VM, dropping it from 9 GB to 2 GB
|
||
moved nothing — it stopped after reading the same 32 MiB, which is simply the
|
||
size of the boot files.
|
||
|
||
The plan is printed **before** anything is created, and the script never
|
||
promises a depth it knows will not fit — better to announce six levels and
|
||
reach six than to promise ten and die at the seventh without knowing why.
|