Update the README to cover the new syncing feature: the deployment
section explains the three optional pieces and their relations. We
also explain the special cases of "Get updates" and "Save updates"
when combined with pending/conflicting changes (particularly in
edition mode).
An explanation of the difficulties to set git permission is also
provided.
Every save used to be committed as a bare "update 3.5", which makes the
storage history useless to navigate. The subject now names the branch
and the first commit whose review was touched, followed by "+ N more"
when several, and the body lists all of them one per line:
update 3.5: 6a7b27a0 + 2 more
6a7b27a0
d13aaf05
b12dd0b5
This is what to grep for when hand-editing the storage repository, for
example to locate the change to revert or rebase. A commit touched by
several directives of the same save (state plus notes) is only listed
once.
thttpd forwards the CGI's stderr to the client *ahead* of its stdout:
the socket receives the HTTP status line, then any stderr log line, and
only then the CGI headers, turning the log line into a bogus response
header; on an error path the same mechanism could push garbage in front
of the "Status:" header and corrupt the response entirely.
All diagnostics are already carried by the response itself (die()'s
body, the in-band "warning:" line with git's captured error), so the
duplicated stderr logging brings nothing and only risks breaking the
channel it leaks into: drop it, and state the constraint in a comment
above die() so it doesn't come back. The usage text for a bad command
line is folded into the 500 response body, which is also what a shell
user sees when testing by hand.
The "git commit failed" warning said nothing about the cause, leaving
the admin to guess between a missing identity, an ownership refusal, a
git binary absent from the restricted PATH the web server gives to its
CGIs, etc. run_git() now captures the command's stdout and stderr
through a pipe and the warning line carries git's own message (first
255 bytes, control chars flattened), so the admin directly sees the
cause; as the command runs through /bin/sh, an unfindable git yields
status 127 and the shell's message, reworded as "cannot execute git:
..." to directly point at the typical PATH issue. Capturing also
guarantees that git output can never corrupt the CGI response nor leak
to the client on servers which wire the CGI's stderr to the socket.
When the git commit fails after a save (typically a missing committer
identity in the storage repository, or an ownership/permission issue),
the failure was only logged to stderr, which lands in the web server's
error log at best: the file kept being updated but the history silently
stopped being recorded. Report it as a "warning: git commit failed ..."
line appended to the response, where the page can show it to the user,
on top of the stderr log. Also stop treating a no-op as a failure:
re-pushing identical content stages nothing, so the commit is now
simply skipped when "git diff --cached --quiet" reports no staged
change, instead of letting "git commit" fail on an empty commit.
Notes are append-only on the wire, which makes concurrent edits
conflict-free but leaves no way to revise or clean up a note from the
page: fixing a note requires a hand-edit of the storage file. This adds
the replacement directive that the design had reserved:
<cid> setnotes <hash> <replacement text>
Unlike the other directives, a replacement must carry a token of the
base it was computed from, or it could silently destroy a concurrent
update (one reviewer's append landing between another's read and
replace). The token is the SDBM hash (8 hex chars) of the note blob
the client based its edit on: the server only applies the replacement
if it still matches the stored blob (empty-string hash for a commit
without notes). On mismatch the directive is dropped, the line is left
exactly as found, and a "conflict <cid>" line is emitted in the
response before the resulting lines so that the client can point the
user at what needs manual reconciliation; other directives from the
same POST are still applied, and nothing is written nor committed when
everything conflicted. SDBM is trivially computed on both sides (a
concurrency token, not a security feature, and JS crypto is unavailable
on plain http anyway), and its small multiplier keeps the whole hash
computation exact in awk's double-precision arithmetic, which is
precisely why it was chosen over wider-multiplier hashes.
An empty replacement deletes the notes (and the line if no state is
left), finally allowing obsolete notes to be removed without editing
the file by hand. Replacements are capped to 4000 chars instead of the
500-char append cap, since a coalesced blob may legitimately have grown
beyond a single addition.
This adds the read side of the review persistence CGI: GET
update.cgi?branch=X.Y now returns the current overlay for that branch
as a JSON array of {"cid", "state", "notes"} objects with absent fields
omitted; a missing or empty file yields "[]". The raw storage format
never travels: the notes are unescaped by the parser and JSON-escaped
on output, so the client can JSON.parse() the response and insert the
notes via textContent directly. Unparseable lines or fields are
silently skipped as everywhere else.
Reads are lockless: the atomic rename on the write side guarantees
that the file is always a complete valid version. The response carries
Cache-Control: no-store so that a browser never reuses a stale overlay
on refresh.
The backport review page keeps the human edits (verdict overrides and
notes) only in the loaded DOM: they are lost on reload and never shared
between reviewers. This adds the server side of the shared persistence
design: update.awk, a GNU awk CGI script which stores these edits into
one file per major branch (e.g. "3.5") inside a dedicated git
repository, one line per touched commit:
<commit_id> [state <n|u|w|y>] [notes "<quoted notes>"]
The overlay only ever stores human edits keyed on the commit id; the AI
verdict and explanation stay in the generated HTML. Commit ids are
length-agnostic and matched by symmetric prefix (first match wins), so
the current 8-char pipeline and a future 12-char one both work without
any migration.
POST applies line-oriented directives ("<cid> state <n|u|w|y|revert>",
"<cid> notes <text to append>"), none of which carries a base value:
states are last-write-wins and notes are append-only (capped to 500
chars per push and sanitised so that no newline may ever enter a stored
line), which keeps concurrent edits conflict-free. Broken directives,
fields or lines are silently ignored, never fatal, and lines not being
modified are preserved byte-for-byte so that admin hand-edits survive.
Writers are serialised by a mkdir lock at an obvious place (<repo>/lock)
with PID-gated crash takeover via atomic rename, the file itself is
replaced by an atomic rename from a temp file inside the lock dir, and
every resulting state is committed to git, which acts as the event log
(git blame/log -L provide the full history). A crash at any point leaves
at worst a stale lock (reclaimed on the next write) or a valid but
uncommitted tree (folded into the next commit), never a broken file.
The takeover races are covered: the staleness decision and the takeover
rename are not one atomic operation, so the thief verifies after the
rename, discarding the stolen dir only if it still carries the pid that
was judged dead and renaming it back in place untouched otherwise; the
victim redoes the whole locked cycle from a fresh read when its final
rename fails, since nothing was applied to the branch file yet; a writer
finding its own pid in the lock adopts it as stale; and the release only
removes the lock after checking that it still contains our own pid.
A few awk specifics are worth noting: external commands (git, mkdir,
mv) go through /bin/sh, so everything interpolated into a command line
is shell-quoted (single-quote is escaped and the argument placed inside
single quotes); the -b (bytes) flag keeps all string operations byte-
based regardless of the locale; and a manual argv parsing due to gawk
silently consumes a leading "-r" argument, ignoring ours.
Also note that gawk uses a file cache for getline() and co, which opens
lots of traps so we need to be extremely careful about properly closing
files if we want to check for changes (e.g. lock's pid file).
Finally, writing through a redirection whose target cannot be opened is
a fatal awk error terminating the script without even a response, so the
writes into the (stealable) lock dir are arranged to resist a takeover:
the pid is written through the shell, where a vanished dir is a plain
command failure, and the temp file is opened the very instant the lock
is acquired, its descriptor surviving a later theft. The update.cgi
wrapper considers any >0 return code as a failure and returns a generic
error as it will indicate that the awk script itself couldn't produce
a valid response.
For now, only the POST ("save changes") action is implemented.
The "Get updates" and "Save changes" buttons only existed at the top
right of the page, while a review session ends at the bottom of the
table: with no button left in sight there, it was way too easy to
forget to save the work. Emit a copy of both buttons and of the status
line at the bottom right, sharing the same handlers; the status
message and the save-button graying are applied to both instances at
once so the two spots always tell the same story.
Some server setups leak the CGI's stderr into the response body (e.g.
inetd-style servers where fd 2 is the client socket): since stderr is
unbuffered and stdout is buffered, a git error message then lands
*before* the "OK <n>" line, and the applied-count check failed with a
cryptic "server applied only ? of N changes" although everything had
been applied. Scan the whole response for the lines of interest (the
"OK <n>" count, the "conflict" and "warning" lines) instead of assuming
they come first, dump the raw response to the console when no count is
found at all to ease diagnosis, and display the server's warning lines
(such as the new "git commit failed" one) next to the save status so
that a recording problem is visible from the page instead of being
buried in a server log.
The save handler treated any HTTP 200 as a full success and advanced
the local reference for everything it had sent, only special-casing the
reported conflicts. But the server legitimately drops directives it
cannot parse, and answers "OK <n> directives applied" with what it
really did. The typical case is an outdated update scripton the server
which ignores the whole "setnotes" directive, applies nothing, and
the client still displayed the edit as saved... until the next "Get
updates" reverted it.
Let's count the directives sent, and when the server's applied count
plus the reported conflicts don't add up, believe the server, not
ourselves: advance nothing, keep every edit local (boxes open, save
button lit) and tell the user how many changes were ignored,
suggesting a version mismatch.
The "Save changes" button used to remain active all the time, giving no
hint about whether anything was pending. It is now disabled whenever
nothing differs from the reference: no verdict change, no non-empty
note addition, no note edition differing from its base. It gets
re-evaluated after every action which may change that (verdict clicks,
typing in a note input, opening/cancelling a box, updates and saves),
bailing out at the first pending change so the common case stays cheap.
As a side effect, the button lighting up right after a reload confirms
at a glance that the browser restored unsaved local edits.
An "[edit note]" link now appears next to "[add note]" whenever a line
has shared notes: it presents the whole note blob in the input box for
edition, and the save sends it as a replacement (the "setnotes"
directive, carrying the hash of the blob the edit was based on).
Emptying the box deletes the note. Clicking "add note" first and then
"edit note" merges the reference notes with the pending addition so
nothing typed so far is lost, and a "[cancel]" link aborts an edition
opened by mistake without touching anything.
An open input only disappears once its content is synchronized with the
reference: a successful save, an update proving an exact match, or an
explicit cancel. To that end, "Get updates" first silently closes the
no-op boxes (opened but nothing changed), then after applying the
fetched state it closes the boxes it made redundant (an addition
someone already pushed, an edition matching the current notes), and
re-bases an edition whose base moved, turning it red: red uniformly
means "the reference changed under your edit, review the notes above
against your text before saving". A save refused by the server (the
"conflict <cid>" response lines) turns the input red the same way,
keeping it in edition; states and additions from the same save are
unaffected. After a reload, a browser-restored pending note reopens in
append mode, the only safe assumption since a lost replacement base
cannot be recovered.
This is the write side of the review syncing: a "Save changes" button
next to "Get updates" collects the local edits and pushes them to
update.cgi. An edit is a radio button change differing from the
reference state (so clicking around and coming back to the reference
sends nothing), or a note typed in the per-line input revealed by the
new "[add note]" link under the AI explanation (500 chars max, matching
the server-side cap).
No directive carries a base value: states are last-write-wins and notes
are append-only server-side, so two reviewers saving concurrently
cannot conflict. On success the reference advances to the pushed values
and the note inputs are cleared, so the page is clean without needing a
refetch; on error (server busy or unreachable) everything stays local
and a later click simply retries.
This adds the read side of the review syncing to the generated page: a
"Get updates" button at the top right retrieves the shared state from
update.cgi (reached by a bare relative URL, so it must be in a cgi-bin
directory next to the page) and applies it. Nothing is fetched
automatically, not even at load time: it's up to the user to explicitly
click to resynchronize, and without it (or with the server down) the
page keeps behaving fully standalone as today.
Three states exist per line to make this work. The original state is
the verdict the bot chose, captured at load time and constant. The
reference state is the last known shared state, on top of which the
user's edits sit; it starts equal to the original. The local state is
the DOM itself (the checked radios). Applying a fetched overlay
recomputes every line's reference as "the server's entry if any,
otherwise the bot's verdict", so a removed override properly falls back
to the original; the reference always advances but the displayed state
only moves where the user had no local edit: local edits win, and
re-applying the same overlay twice changes nothing.
The received commit ids are resolved exactly first, then by symmetric
prefix (one id being a prefix of the other, for mixed-length ids), first
line wins. The shared notes land in a dedicated container below the AI
explanation, rendered via innerText (no HTML injection) and replaced
wholesale so the operation stays idempotent; entries for commits absent
from the page are simply ignored.
The whole exchange was tested with the real generated page's scripts
running against a stubbed DOM covering load capture, prefix resolution,
adopt-vs-keep on both changed and disappeared entries, idempotent
re-application, and silent degradation on fetch failure.
The review page will need to exchange its state with the update.cgi
service sitting next to it, and for this it must know which branch it
covers since all branches' pages may share a directory. post-ai.sh
takes a new "-v <version>" argument and emits it as a "branch" JS
variable; when absent the variable is empty and the page will simply
not offer syncing, keeping the output standalone as today.
update-3.0.sh deduces the version from its own name (update-3.0.sh ->
3.0), so that adding a symlink with another version for a new branch
continues to work with no other change, and passes it to post-ai.sh.
The commit id column doesn't need to show more than 8 chars to stay
unambiguous within a single page, and longer ids needlessly widen the
table. Everything that is keyed on the id (the commit link, the row's
name= attribute and the cid[] JS array) keeps the full id produced by
the pipeline, whatever its length, so this is a display-only change
which also gets us ready for a possible future move of the pipeline to
longer ids.
When the page is reloaded, the browser restores the "review" radio
column to the user's last selection (e.g. "All"), but the "review" JS
variable is regenerated to the default first line to review: the
listing then restarts from that line while "All" still appears
selected, and one has to click a random line then "All" again to
really see everything.
Give an id to each line's review radio and resynchronize the variable
from the actually checked radio when the page loads: a restored "All"
(or any restored line) now behaves as selected, and on a fresh load the
checked radio is the generated one so nothing changes.
Three functions are provided here:
fd_dump: lists all FDs
fd_dump_conn: lists all FDs holding a connection
fd_dump_listener: lists all FDs holding a listener
They take no argument, and dump some of the known info. E.g. for
a connection, ctrl, xprt, flags, mux, sessions, frontend's name
and session's age are reported. Example:
(gdb) fd_dump_conn
fd 31: rm=0 tm=0x2 um=0 st=0x21 refc=0x1 tkov=0 gen=0 conn=0x7fffe803b600: flg=0x300 err=0 ctrl=0xdf51c0 xprt=0xdf5c80 mux=0xbaeee0 sess=0x7ffff003b570: fe=0x1e45b00 id=foo age=0ms
They are particularly slow because they iterate over all possible FDs,
so better limit them to the desired types.
The thread_dump function dumps the list of known threads and a few info
on them (pointer, current run queue, flags etc). This should help more
easily spot a particular one and find stuck ones.
E.g:
(gdb) thread_dump
Tid 0: pth=0x7ffff7e797c0 mono=2222322327950732 now_ms=4294947291 fl=0x38 rq=-1 cq=0 current=(nil)
Tid 1: pth=0x7ffff78d8640 mono=2222322327928085 now_ms=4294947291 fl=0x38 rq=-1 cq=0 current=(nil)
Tid 2: pth=0x7ffff6b7e640 mono=2222322327927150 now_ms=4294947291 fl=0x38 rq=-1 cq=0 current=(nil)
Tid 3: pth=0x7ffff637d640 mono=2222322327924878 now_ms=4294947291 fl=0x38 rq=-1 cq=0 current=(nil)
Tid 4: pth=0x7ffff5b7c640 mono=2222322327925676 now_ms=4294947291 fl=0x38 rq=-1 cq=0 current=(nil)
Tid 5: pth=0x7ffff537b640 mono=2222322327929524 now_ms=4294947291 fl=0x38 rq=-1 cq=0 current=(nil)
Tid 6: pth=0x7ffff4b7a640 mono=2222322327926817 now_ms=4294947291 fl=0x38 rq=-1 cq=0 current=(nil)
Tid 7: pth=0x7fffdffff640 mono=2222322327947960 now_ms=4294947291 fl=0x38 rq=-1 cq=0 current=(nil)
New functions task_dump_wq and task_dump_rq can be used to dump tasks
in a wait queue or in a run queue respectively. For the wait queue (the
most common usage), one needs to pass either the thread-local's timers,
or the thread group ones for shared tasks:
task_dump_wq &ha_tgroup_ctx[0].timers
task_dump_wq &ha_thread_ctx[0].timers
For the run queue, task_dump_rq will take the thread's rqueue:
task_dump_rq &ha_thread_ctx[0].rqueue
The output is the task pointer and a dump of the task* struct per line,
then a total count at the end.
The ebtree descent functions currently use $arg0 as is and it's up to
the user to manually type the required casts that are never obvious
(particularly when coming from a pointer). Let's put the eb_root* cast
in the function to be more user-friendly.
This utility takes in argument the path to a core dump, and it looks
for the archive signature of libraries embedded with "set-dumpable libs",
and either emits the offset and size of stdout, or directly dumps the
contents so that the tar file can be extracted directly by piping the
output to tar xf.
Handshakes events were not properly decoded. Only send errors were decoded
as expected, other events were reported with a '-'. It is now fixes.
This patch could be backported as far as 3.2.
The pools memory usage calculation was done using ints by default, making
it harder to identify large ones. Let's switch to unsigned long for the
size calculations.
More and more often, core dumps retrieved on systems that build with
-fPIE by default are becoming unexploitable. Even functions and global
symbols get relocated and gdb cannot figure their final position.
Ironically the post_mortem struct lying in its own section that was
meant to ease its finding is not exempt from this problem.
The only remaining way is to inspect the core to search for the
post-mortem magic, figure its offset from the file and look up the
corresponding virtual address with objdump. This is quite a hassle.
This patch implements a simple utility that opens a 64-bit core dump,
scans the program headers looking for a data segment which contains
the post-mortem magic, and prints it on stdout. It also places the
"pm_init" command alone on its own line to ease copy-pasting into the
gdb console. With this, at least the other commands in this directory
work again and allow to inspect the program's state. E.g:
$ ./getpm core.57612
Found post-mortem magic in segment 5:
Core File Offset: 0xfc600 (0xd5000 + 0x27600)
Runtime VAddr: 0x5613e52b6600 (0x5613e528f000 + 0x27600)
Segment Size: 0x28000
In gdb, copy-paste this line:
pm_init 0x5613e52b6600
It's worth noting that the program has so few dependencies that it even
builds with nolibc, allowing to upload a static executable into containers
being debugged and lacking development tools and compilers. The build
procedure is indicated inthe source code.
- Use the automatic alignment feature instead of hardcoding 64 all over
the code.
- This also converts a few bare __attribute__((aligned(X))) to using the
ALIGNED macro.
The patchbot stopped on a previous ultra-rare forced push due to wanting
the user's name and e-mail before proceeding. We don't want merges nor
rebases anyway, only to reset the tree to the next one, so let's do that.
Commit 41f28b3c53 ("DEV: phash: Update 414 and 431 status codes to phash")
accidentally committed a.out, resulting in build/checkout issues when
locally rebuilt. Let's drop it.
This should be backported to 3.1.
Nick Ramirez reported the following error while testing the h2-tracer.lua
script:
Lua filter 'h2-tracer' : [state-id 0] runtime error: /etc/haproxy/h2-tracer.lua:227: attempt to index a nil value (field '?') from /etc/haproxy/h2-tracer.lua:227: in function line 109.
It is caused by h2ff indexing with an out of bound value. Indeed, h2ff
is indexed with the frame type, which can potentially be > 9 (not common
nor observed during Willy's tests), while h2ff only defines indexes from
0 to 9.
The fix was provided by Willy, it consists in skipping h2ff indexing if
frame type is > 9. It was confirmed that doing so fixes the error.
This is also needed in order to make the requested number of CPUs
appear. For now we don't reroute to the original sysconf() call so
we return -1,EINVAL for all other info.
The following config is sufficient to trace H2 exchanges between a client
and a server:
global
lua-load "dev/h2/h2-tracer.lua"
listen h2_sniffer
mode tcp
bind :8002
filter lua.h2-tracer #hex
server s1 127.0.0.1:8003
The commented "hex" argument will also display full frames in hex (not
recommended). The connections are prefixed with a 3-hex digit number in
order to also support a bit of multiplexing without impacting the reading
too much. The screen is split in two, with the request on the left and
the response on the right. Here's an example of what it does between an
haproxy backend and an haproxy frontend both in H2, when submitted a
curl request for /?s=30k handled by httpterm:
[001] ### req start
[001] [PREFACE len=24]
[001] [SETTINGS sid=0 len=24 (bytes=24)]
[001] | ### res start
[001] | [SETTINGS sid=0 len=18 (bytes=27)]
[001] | [SETTINGS ACK sid=0 len=0 (bytes=0)]
[001] [SETTINGS ACK sid=0 len=0 (bytes=56)]
[001] [HEADERS EH+ES sid=1 len=47 (bytes=47)]
[001] | [HEADERS EH sid=1 len=101 (bytes=15351)]
[001] | [DATA sid=1 len=15126 (bytes=15241)]
[001] | [DATA sid=1 len=1258 (bytes=106)]
[001] | ... -106 = 1152
[001] | ... -1152 = 0
[001] [WINDOW_UPDATE sid=1 len=4 (bytes=43)]
[001] [WINDOW_UPDATE sid=0 len=4 (bytes=30)]
[001] [WINDOW_UPDATE sid=1 len=4 (bytes=17)]
[001] [WINDOW_UPDATE sid=0 len=4 (bytes=4)]
[001] | [DATA ES sid=1 len=14336 (bytes=14336)]
[001] [WINDOW_UPDATE sid=0 len=4 (bytes=4)]
[001] ### req end: 31080 bytes total
[001] | [GOAWAY sid=0 len=8 (bytes=8)]
[001] | ### res end: 31097 bytes total
It deserves some improvements. For instance at the moment it does not
verify the preface, any 24 bytes will work. It does not perform any
protocol validation either. Detecting some issues such as out-of-sequence
frames could be helpful. But it already helps as-is.
Connection errors can be detected via connect/recv/send syscall, but also
because it was reported by the poller. So dedicated events, at the FD level,
are introduced to make the difference.
term_events tool was updated accordingly.
This development tool can be used to convert a string representing a
termination event logs to its human redable representation. Several string
may be converting at a time. To do so, several arguments can be specified on
the commeand line or they can be provided on STDIN, using "-" argument.
Here is an exemple:
> term_events f2x2f4x4 m2m4m1 e2e1 s2s1S1 E1 M1 F1
### f2x2f4x4 : fd:shutr > xprt:shutr > fd:snd_err > xprt:snd_err
### m2m4m1 : muxc:shutr > muxc:snd_err > muxc:shutw
### e2e1 : se:eos > se:shutw
### s2s1S1 : strm:eos > strm:shutw > STRM:shutw
### E1 : SE:shutw
### M1 : MUXC:shutw
### F1 : FD:shutw
The make target "dev/term_events/term_events" must be used to compile it.
It's convenient to have a share lib be able to also work as a wrapper.
But recent glibc broke support for this dual-mode thing some time ago:
https://patchwork.ozlabs.org/project/glibc/patch/20190312130235.8E82C89CE49C@oldenburg2.str.redhat.com/https://stackoverflow.com/questions/59074126/loading-executable-or-executing-a-library
Trying to preload such an executable indeed returns:
ERROR: ld.so: object '/path/to/ncpu.so' from LD_PRELOAD cannot be preloaded (cannot dynamically load position-independent executable): ignored.
Note that the code still supports it since libc.so is both an executable
and a lib. The approach taken here is the same as in the nousr.so wrapper.
It consists in dropping the DF_1_PIE flag from the resulting executable
since it's what the dynamic linker is looking for. This flag is found in
FLAGS_1 in the .dynamic section. As readelf -a suggests, it's after the
tag 0x6ffffffb. The value is 0x08000000. We're using objdump to figure the
length and offset of the struct, dd to extract the 3 parts, and sed to
patch the binary.
It's likely that it will only work on 64-bit little endian, though tests
should be performed to see what to do on other platforms. At least on
x86_64, ld.so is happy and it continues to be possible to use the binary
as a .so, and that the platform where most of the development happens so
that's fine.
In any case the wrapper and the standard shared lib are still made two
distinct files so that it's possible to use the non-patched version on
unsupported OSes or architectures.
The wrapper mode allows to present itself as LD_PRELOAD before loading
haproxy, which is often more convenient since it allows to pass the
number of CPUs in argument. However, this mode is no longer supported by
modern glibcs, so a future patch will come to implement a trick that was
tested to work at least on x86.
Collecting captures of /sys isn't sufficient for NUMA development because
haproxy detects the number of CPUs at boot time and will not be able to
inspect more than this number. Let's just have a small utility to report
a fake number of CPUs, that will be loaded using LD_PRELOAD. It checks
the NCPU variable if it exists and will present this number of CPUs, or
if it does not exist, will expose the maximum supported number.
These is a collection of functions I'm occasionally using to navigate
in core dumps. Only working ones were extracted.
Those requiring knowledge of global variables (e.g. pools, proxy list)
use the one extracted from the post_mortem struct. That one is defined
in post-mortem.gdb and needs to be initialized using "pm_init post_mortem"
or "pm_init <pointer>". From this point a number of global variables are
accessible even if symbols are missing; those ones are then used by other
functions to dump streams, threads, pools, proxies etc.
The files can be sourced or copy-pasted into a gdb session. It's worth
trying to keep them up-to-date, as the old ones used to navigate through
tasks are no longer usable due to massive changes.
