thttpd forwards the CGI's stderr to the client *ahead* of its stdout:
the socket receives the HTTP status line, then any stderr log line, and
only then the CGI headers, turning the log line into a bogus response
header; on an error path the same mechanism could push garbage in front
of the "Status:" header and corrupt the response entirely.
All diagnostics are already carried by the response itself (die()'s
body, the in-band "warning:" line with git's captured error), so the
duplicated stderr logging brings nothing and only risks breaking the
channel it leaks into: drop it, and state the constraint in a comment
above die() so it doesn't come back. The usage text for a bad command
line is folded into the 500 response body, which is also what a shell
user sees when testing by hand.
The "git commit failed" warning said nothing about the cause, leaving
the admin to guess between a missing identity, an ownership refusal, a
git binary absent from the restricted PATH the web server gives to its
CGIs, etc. run_git() now captures the command's stdout and stderr
through a pipe and the warning line carries git's own message (first
255 bytes, control chars flattened), so the admin directly sees the
cause; as the command runs through /bin/sh, an unfindable git yields
status 127 and the shell's message, reworded as "cannot execute git:
..." to directly point at the typical PATH issue. Capturing also
guarantees that git output can never corrupt the CGI response nor leak
to the client on servers which wire the CGI's stderr to the socket.
When the git commit fails after a save (typically a missing committer
identity in the storage repository, or an ownership/permission issue),
the failure was only logged to stderr, which lands in the web server's
error log at best: the file kept being updated but the history silently
stopped being recorded. Report it as a "warning: git commit failed ..."
line appended to the response, where the page can show it to the user,
on top of the stderr log. Also stop treating a no-op as a failure:
re-pushing identical content stages nothing, so the commit is now
simply skipped when "git diff --cached --quiet" reports no staged
change, instead of letting "git commit" fail on an empty commit.
Notes are append-only on the wire, which makes concurrent edits
conflict-free but leaves no way to revise or clean up a note from the
page: fixing a note requires a hand-edit of the storage file. This adds
the replacement directive that the design had reserved:
<cid> setnotes <hash> <replacement text>
Unlike the other directives, a replacement must carry a token of the
base it was computed from, or it could silently destroy a concurrent
update (one reviewer's append landing between another's read and
replace). The token is the SDBM hash (8 hex chars) of the note blob
the client based its edit on: the server only applies the replacement
if it still matches the stored blob (empty-string hash for a commit
without notes). On mismatch the directive is dropped, the line is left
exactly as found, and a "conflict <cid>" line is emitted in the
response before the resulting lines so that the client can point the
user at what needs manual reconciliation; other directives from the
same POST are still applied, and nothing is written nor committed when
everything conflicted. SDBM is trivially computed on both sides (a
concurrency token, not a security feature, and JS crypto is unavailable
on plain http anyway), and its small multiplier keeps the whole hash
computation exact in awk's double-precision arithmetic, which is
precisely why it was chosen over wider-multiplier hashes.
An empty replacement deletes the notes (and the line if no state is
left), finally allowing obsolete notes to be removed without editing
the file by hand. Replacements are capped to 4000 chars instead of the
500-char append cap, since a coalesced blob may legitimately have grown
beyond a single addition.
This adds the read side of the review persistence CGI: GET
update.cgi?branch=X.Y now returns the current overlay for that branch
as a JSON array of {"cid", "state", "notes"} objects with absent fields
omitted; a missing or empty file yields "[]". The raw storage format
never travels: the notes are unescaped by the parser and JSON-escaped
on output, so the client can JSON.parse() the response and insert the
notes via textContent directly. Unparseable lines or fields are
silently skipped as everywhere else.
Reads are lockless: the atomic rename on the write side guarantees
that the file is always a complete valid version. The response carries
Cache-Control: no-store so that a browser never reuses a stale overlay
on refresh.
The backport review page keeps the human edits (verdict overrides and
notes) only in the loaded DOM: they are lost on reload and never shared
between reviewers. This adds the server side of the shared persistence
design: update.awk, a GNU awk CGI script which stores these edits into
one file per major branch (e.g. "3.5") inside a dedicated git
repository, one line per touched commit:
<commit_id> [state <n|u|w|y>] [notes "<quoted notes>"]
The overlay only ever stores human edits keyed on the commit id; the AI
verdict and explanation stay in the generated HTML. Commit ids are
length-agnostic and matched by symmetric prefix (first match wins), so
the current 8-char pipeline and a future 12-char one both work without
any migration.
POST applies line-oriented directives ("<cid> state <n|u|w|y|revert>",
"<cid> notes <text to append>"), none of which carries a base value:
states are last-write-wins and notes are append-only (capped to 500
chars per push and sanitised so that no newline may ever enter a stored
line), which keeps concurrent edits conflict-free. Broken directives,
fields or lines are silently ignored, never fatal, and lines not being
modified are preserved byte-for-byte so that admin hand-edits survive.
Writers are serialised by a mkdir lock at an obvious place (<repo>/lock)
with PID-gated crash takeover via atomic rename, the file itself is
replaced by an atomic rename from a temp file inside the lock dir, and
every resulting state is committed to git, which acts as the event log
(git blame/log -L provide the full history). A crash at any point leaves
at worst a stale lock (reclaimed on the next write) or a valid but
uncommitted tree (folded into the next commit), never a broken file.
The takeover races are covered: the staleness decision and the takeover
rename are not one atomic operation, so the thief verifies after the
rename, discarding the stolen dir only if it still carries the pid that
was judged dead and renaming it back in place untouched otherwise; the
victim redoes the whole locked cycle from a fresh read when its final
rename fails, since nothing was applied to the branch file yet; a writer
finding its own pid in the lock adopts it as stale; and the release only
removes the lock after checking that it still contains our own pid.
A few awk specifics are worth noting: external commands (git, mkdir,
mv) go through /bin/sh, so everything interpolated into a command line
is shell-quoted (single-quote is escaped and the argument placed inside
single quotes); the -b (bytes) flag keeps all string operations byte-
based regardless of the locale; and a manual argv parsing due to gawk
silently consumes a leading "-r" argument, ignoring ours.
Also note that gawk uses a file cache for getline() and co, which opens
lots of traps so we need to be extremely careful about properly closing
files if we want to check for changes (e.g. lock's pid file).
Finally, writing through a redirection whose target cannot be opened is
a fatal awk error terminating the script without even a response, so the
writes into the (stealable) lock dir are arranged to resist a takeover:
the pid is written through the shell, where a vanished dir is a plain
command failure, and the temp file is opened the very instant the lock
is acquired, its descriptor surviving a later theft. The update.cgi
wrapper considers any >0 return code as a failure and returns a generic
error as it will indicate that the awk script itself couldn't produce
a valid response.
For now, only the POST ("save changes") action is implemented.
The "Get updates" and "Save changes" buttons only existed at the top
right of the page, while a review session ends at the bottom of the
table: with no button left in sight there, it was way too easy to
forget to save the work. Emit a copy of both buttons and of the status
line at the bottom right, sharing the same handlers; the status
message and the save-button graying are applied to both instances at
once so the two spots always tell the same story.
Some server setups leak the CGI's stderr into the response body (e.g.
inetd-style servers where fd 2 is the client socket): since stderr is
unbuffered and stdout is buffered, a git error message then lands
*before* the "OK <n>" line, and the applied-count check failed with a
cryptic "server applied only ? of N changes" although everything had
been applied. Scan the whole response for the lines of interest (the
"OK <n>" count, the "conflict" and "warning" lines) instead of assuming
they come first, dump the raw response to the console when no count is
found at all to ease diagnosis, and display the server's warning lines
(such as the new "git commit failed" one) next to the save status so
that a recording problem is visible from the page instead of being
buried in a server log.
The save handler treated any HTTP 200 as a full success and advanced
the local reference for everything it had sent, only special-casing the
reported conflicts. But the server legitimately drops directives it
cannot parse, and answers "OK <n> directives applied" with what it
really did. The typical case is an outdated update scripton the server
which ignores the whole "setnotes" directive, applies nothing, and
the client still displayed the edit as saved... until the next "Get
updates" reverted it.
Let's count the directives sent, and when the server's applied count
plus the reported conflicts don't add up, believe the server, not
ourselves: advance nothing, keep every edit local (boxes open, save
button lit) and tell the user how many changes were ignored,
suggesting a version mismatch.
The "Save changes" button used to remain active all the time, giving no
hint about whether anything was pending. It is now disabled whenever
nothing differs from the reference: no verdict change, no non-empty
note addition, no note edition differing from its base. It gets
re-evaluated after every action which may change that (verdict clicks,
typing in a note input, opening/cancelling a box, updates and saves),
bailing out at the first pending change so the common case stays cheap.
As a side effect, the button lighting up right after a reload confirms
at a glance that the browser restored unsaved local edits.
An "[edit note]" link now appears next to "[add note]" whenever a line
has shared notes: it presents the whole note blob in the input box for
edition, and the save sends it as a replacement (the "setnotes"
directive, carrying the hash of the blob the edit was based on).
Emptying the box deletes the note. Clicking "add note" first and then
"edit note" merges the reference notes with the pending addition so
nothing typed so far is lost, and a "[cancel]" link aborts an edition
opened by mistake without touching anything.
An open input only disappears once its content is synchronized with the
reference: a successful save, an update proving an exact match, or an
explicit cancel. To that end, "Get updates" first silently closes the
no-op boxes (opened but nothing changed), then after applying the
fetched state it closes the boxes it made redundant (an addition
someone already pushed, an edition matching the current notes), and
re-bases an edition whose base moved, turning it red: red uniformly
means "the reference changed under your edit, review the notes above
against your text before saving". A save refused by the server (the
"conflict <cid>" response lines) turns the input red the same way,
keeping it in edition; states and additions from the same save are
unaffected. After a reload, a browser-restored pending note reopens in
append mode, the only safe assumption since a lost replacement base
cannot be recovered.
This is the write side of the review syncing: a "Save changes" button
next to "Get updates" collects the local edits and pushes them to
update.cgi. An edit is a radio button change differing from the
reference state (so clicking around and coming back to the reference
sends nothing), or a note typed in the per-line input revealed by the
new "[add note]" link under the AI explanation (500 chars max, matching
the server-side cap).
No directive carries a base value: states are last-write-wins and notes
are append-only server-side, so two reviewers saving concurrently
cannot conflict. On success the reference advances to the pushed values
and the note inputs are cleared, so the page is clean without needing a
refetch; on error (server busy or unreachable) everything stays local
and a later click simply retries.
This adds the read side of the review syncing to the generated page: a
"Get updates" button at the top right retrieves the shared state from
update.cgi (reached by a bare relative URL, so it must be in a cgi-bin
directory next to the page) and applies it. Nothing is fetched
automatically, not even at load time: it's up to the user to explicitly
click to resynchronize, and without it (or with the server down) the
page keeps behaving fully standalone as today.
Three states exist per line to make this work. The original state is
the verdict the bot chose, captured at load time and constant. The
reference state is the last known shared state, on top of which the
user's edits sit; it starts equal to the original. The local state is
the DOM itself (the checked radios). Applying a fetched overlay
recomputes every line's reference as "the server's entry if any,
otherwise the bot's verdict", so a removed override properly falls back
to the original; the reference always advances but the displayed state
only moves where the user had no local edit: local edits win, and
re-applying the same overlay twice changes nothing.
The received commit ids are resolved exactly first, then by symmetric
prefix (one id being a prefix of the other, for mixed-length ids), first
line wins. The shared notes land in a dedicated container below the AI
explanation, rendered via innerText (no HTML injection) and replaced
wholesale so the operation stays idempotent; entries for commits absent
from the page are simply ignored.
The whole exchange was tested with the real generated page's scripts
running against a stubbed DOM covering load capture, prefix resolution,
adopt-vs-keep on both changed and disappeared entries, idempotent
re-application, and silent degradation on fetch failure.
The review page will need to exchange its state with the update.cgi
service sitting next to it, and for this it must know which branch it
covers since all branches' pages may share a directory. post-ai.sh
takes a new "-v <version>" argument and emits it as a "branch" JS
variable; when absent the variable is empty and the page will simply
not offer syncing, keeping the output standalone as today.
update-3.0.sh deduces the version from its own name (update-3.0.sh ->
3.0), so that adding a symlink with another version for a new branch
continues to work with no other change, and passes it to post-ai.sh.
The commit id column doesn't need to show more than 8 chars to stay
unambiguous within a single page, and longer ids needlessly widen the
table. Everything that is keyed on the id (the commit link, the row's
name= attribute and the cid[] JS array) keeps the full id produced by
the pipeline, whatever its length, so this is a display-only change
which also gets us ready for a possible future move of the pipeline to
longer ids.
When the page is reloaded, the browser restores the "review" radio
column to the user's last selection (e.g. "All"), but the "review" JS
variable is regenerated to the default first line to review: the
listing then restarts from that line while "All" still appears
selected, and one has to click a random line then "All" again to
really see everything.
Give an id to each line's review radio and resynchronize the variable
from the actually checked radio when the page loads: a restored "All"
(or any restored line) now behaves as selected, and on a fresh load the
checked radio is the generated one so nothing changes.
The patchbot stopped on a previous ultra-rare forced push due to wanting
the user's name and e-mail before proceeding. We don't want merges nor
rebases anyway, only to reset the tree to the next one, so let's do that.
It's useful to instantly see how many patches of each category have
already been backported and are still pending, let's count them and
report them at the top of the page.
Some rare commit messages area really too large because they contain
code excerpts in the message body or are release commits with their
changelog. In this case, instead of leaving an empty file that will
be silently ignored, let's produce an output message indicating that
the verdict is uncertain, with an explanation stating that there was
an error.
In order to spot old patches marked "wait" that have not yet been
backported, it's convenient to be able to click "all" to start the
review from the first patch, deselect "no", "uncertain" and "backported",
leaving only "wait" and "yes". This will reveal all pending patches that
have still not yet been backported, including those prior to the last
review, allowing to reconsider older patches marked "wait" that have
not yet been picked.
The statuses[] table was pre-filled from the shell code during
initialization based on the evaluation and the buttons pre-checked
accordingly, but upon reload, the checked buttons are preserved and
the statuses reinitialized, leading to a different status and color
on lines that were changed.
In practice we don't need this table and we can directly check each
button's state. This makes sure that displayed state is consistent
with checked buttons and allows to preserve the statuses upon reloads
to benefit from updates. Only the start of the review is reset upon
reload now (this allows to consider latest backport state). Of course,
a full reload (shift-ctrl-R) continues to reset the form.
This is a set of scripts, prompts and howtos to have an LLM read commit
messages and determine with great accuracy whether the patch's author
intended for the patch to be backported ASAP, backported after some time,
not backported, or unknown state. It provides all this in an interactive
interface making it easy to adjust choices and proceed with what was
selected. This has been improving over the last 9 months, as helped to
spot patches for a handful of backport sessions, and was only limited by
usability issues (UI). Now that these issues are solved, let's commit the
tool in its current working state. It currently runs every hour in a
crontab for me and started to prove useful since the last update, so it
should be considered in a usable state now, especially since this latest
update reaches close to 100% accuracy compared to a human choice, so it
saves precious development time and may allow stable releases to be
emitted more regularly.
There's detailed readme, please read it before complaining about the
ugliness of the UI :-)
