Now that log-profile parsing logic has been implemented in "MINOR: log:
add log-profile parsing logic" and is actually effective since "MEDIUM:
log: handle log-profile in process_send_log()", let's document the feature
and add some examples.
Log-profile section is declared like this:
log-profile myprof
log-tag "custom-tag"
on error format "%ci: error"
on any format "(custom httplog) ${HAPROXY_HTTP_LOG_FMT}" sd "[exampleSDID@1234 step=\"accept\" id=\"%ID\"]"
(check out the documentation for the full list of options, some options
are only relevant under specific contexts)
And used this way (from usual "log" directive lines):
global
log stdout format rfc5424 profile myprof local0
--------------
For now, the use of log-profiles is somewhat limited because we lack
the ability to explicitly trigger the log building process at specific
steps during the stream handling, but it should gain more traction over
the time as the feature evolves and new mechanisms allowing the emission
of logs at expected processing steps will be added.
It should partially fix GH #401
'%OG' logformat alias may be used to report the log origin (when/where)
that triggered log generation using sess_build_logline().
Possible values are:
- "sess_error": log was generated during session error handling
- "sess_killed": log was generated during session abortion (killed
embryonic session)
- "txn_accept": log was generated right after frontend conn was accepted
- "txn_request": log was generated after client request was received
- "txn_connect": log was generated after backend connection establishment
- "txn_response": log was generated during server response handling
- "txn_close": log was generated at the final txn step, before closing
- "unspec": unknown or not specified
Documentation was updated.
In commit f8642ee82 ("MEDIUM: resolvers: rename dns extra counters to
resolvers extra counters"), we renamed "dns" counters to "resolvers", but
we forgot to update the documentation accordingly.
This may be backported to all stable versions.
Add a documentation about the history of the master-worker and how it
was implemented in its first version and how it is currently working.
This is a global view of the architecture, and not an exhaustive
explanation of all mechanisms.
To stay consistent with the work started in 54627f991 ("DOC: config: add
context hint for proxy keywords") and 3d4e1e682 ("DOC: config: add context
hint for server keywords"), we add missing context hint for "guid" (both
proxy and server) keyword and "hash-key" server keyword that were added
during 3.0 development.
This may be backported in 3.0.
"guid" proxy keyword added in da754b45 ("MINOR: proxy: implement GUID
support") was lacking the section hint in the keyword description, let's
fix that.
It could be backported in 3.0 with da754b45.
As reported by Ashley Morris, "hash-key" keyword which was introduced in
commit faa8c3e0 ("MEDIUM: lb-chash: Deterministic node hashes based on
server address") doesn't belong to proxy keywords and should be found in
5.2 "Server and default-server options" instead.
It should be backported in 3.0 with faa8c3e0
Using CertCache.set() from init context wasn't explicitly supported and
caused the process to crash:
crash.lua:
core.register_init(function()
CertCache.set{filename="reg-tests/ssl/set_cafile_client.pem", ocsp=""}
end)
crash.conf:
global
lua-load crash.lua
listen front
bind localhost:9090 ssl crt reg-tests/ssl/set_cafile_client.pem ca-file reg-tests/ssl/set_cafile_interCA1.crt verify none
./haproxy -f crash.conf
[NOTICE] (267993) : haproxy version is 3.0-dev2-640ff6-910
[NOTICE] (267993) : path to executable is ./haproxy
[WARNING] (267993) : config : missing timeouts for proxy 'front'.
| While not properly invalid, you will certainly encounter various problems
| with such a configuration. To fix this, please ensure that all following
| timeouts are set to a non-zero value: 'client', 'connect', 'server'.
[1] 267993 segmentation fault (core dumped) ./haproxy -f crash.conf
This is because in hlua_ckch_set/hlua_ckch_commit_yield, we always
consider that we're being called from a yield-capable runtime context.
As such, hlua_gethlua() is never checked for NULL and we systematically
try to wake hlua->task and yield every 10 instances.
In fact, if we're called from the body or init context (that is, during
haproxy startup), hlua_gethlua() will return NULL, and in this case we
shouldn't care about yielding because it is ok to commit all instances
at once since haproxy is still starting up.
Also, when calling CertCache.set() from a non-yield capable runtime
context (such as hlua fetch context), we kept doing as if the yield
succeeded, resulting in unexpected function termination (operation
would be aborted and the CertCache lock wouldn't be released). Instead,
now we explicitly state in the doc that CertCache.set() cannot be used
from a non-yield capable runtime context, and we raise a runtime error
if it is used that way.
These bugs were discovered by reading the code when trying to address
Svace report documented by @Bbulatov GH #2586.
It should be backported up to 2.6 with 30fcca18 ("MINOR: ssl/lua:
CertCache.set() allows to update an SSL certificate file")
In ticket #785, people are still confused about how to use the crt-store
load parameters in a crt-list.
This patch adds an example.
This must be backported in 3.0
This patch removes the old README file and replaces it with a more
modern markdown version which allows clickable links on the github page.
It also adds some of the Github Actions worfklow Status.
This patch includes the HAProxy png in the doc directory.
Released version 3.0.0 with the following main changes :
- MINOR: sample: implement the uptime sample fetch
- CI: scripts: fix build of vtest regarding option -C
- CI: scripts: build vtest using multiple CPUs
- MINOR: log: rename 'log-format tag' to 'log-format alias'
- DOC: config: document logformat item naming and typecasting features
- BUILD: makefile: yearly reordering of objects by build time
- BUILD: fd: errno is also needed without poll()
- DOC: config: fix two typos "RST_STEAM" vs "RST_STREAM"
- DOC: config: refer to the non-deprecated keywords in ocsp-update on/off
- DOC: streamline http-reuse and connection naming definition
- REGTESTS: complete http-reuse test with pool-conn-name
- DOC: config: add %ID logformat alias alternative
- CLEANUP: ssl/ocsp: readable ifdef in ssl_sock_load_ocsp
- BUG/MINOR: ssl/ocsp: init callback func ptr as NULL
- CLEANUP: ssl_sock: move dirty openssl-1.0.2 wrapper to openssl-compat
- BUG/MINOR: activity: fix Delta_calls and Delta_bytes count
- CI: github: upgrade the WolfSSL job to 5.7.0
- DOC: install: update quick build reminders with some missing options
- DOC: install: update the range of tested openssl version to cover 3.3
- DEV: patchbot: prepare for new version 3.1-dev
- MINOR: version: mention that it's 3.0 LTS now.
With the introduction of "pool-conn-name", documentation related to
http-reuse was rendered more complex than already, notably with multiple
cross-references between "pool-conn-name" and "sni" server keywords.
Took the opportunity to improve all http-reuse related documentation.
First, "http-reuse" keyword general purpose has been greatly expanded
and reordered.
Then, "pool-conn-name" and "sni" have been clarified, in particular the
relation between them, with the foremost being an advanced usage to the
default SSL SNI case in the context of http-reuse. Also update
attach-srv rule documentation as its name parameter is directly linked
to both "pool-conn-name" and "sni".
The ability to give a name to a logformat_node (known as logformat item in
the documentation) implemented in 2ed6068f2a ("MINOR: log: custom name for
logformat node") wasn't documented.
The same goes for the ability to force the logformat_node's output type to
a specific type implemented in 1448478d62 ("MINOR: log: explicit
typecasting for logformat nodes")
Let's quickly describe such new usages at the start of the custom log
format section.
In 2.9 we started to introduce an ambiguity in the documentation by
referring to historical log-format variables ('%var') as log-format
tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return
%Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add
sample fetches for timing events").
In fact, we've had this confusion between log-format tag and log-format
var for more than 10 years now, but in 2.9 it was the first time the
confusion was exposed in the documentation.
Indeed, both 'log-format variable' and 'log-format tag' actually refer
to the same feature (that is: '%B' and friends that can be used for
direct access to some log-oriented predefined fetches instead of using
%[expr] with generic sample expressions).
This feature was first implemented in 723b73ad75 ("MINOR: config: Parse
the string of the log-format config keyword") and later documented in
4894040fa ("DOC: log-format documentation"). At that time, it was clear
that we used to name it 'log-format variable'.
But later the same year, 'log-format tag' naming started to appear in
some commit messages (while still referring to the same feature), for
instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version
in logs using logformat %sslc/%sslv").
Unfortunately in 2.9 when we added (and documented) new log-format
variables we officially started drifting to the misleading 'log-format
tag' naming (perhaps because it was the most recent naming found for
this feature in git log history, or because the confusion has always
been there)
Even worse, in 3.0 this confusion led us to rename all 'var' occurrences
to 'tag' in log-format related code to unify the code with the doc.
Hopefully William quickly noticed that we made a mistake there, but
instead of reverting to historical naming (log-format variable), it was
decided that we must use a different name that is less confusing than
'tags' or 'variables' (tags and variables are keywords that are already
used to designate other features in the code and that are not very
explicit under log-format context today).
Now we refer to '%B' and friends as a logformat alias, which is
essentially a handy way to print some log oriented information in the
log string instead of leveraging '%[expr]' with generic sample expressions
made of fetches and converters. Of course, there are some subtelties, such
as a few log-format aliases that still don't have sample fetch equivalent
for historical reasons, and some aliases that may be a little faster than
their generic sample expression equivalents because most aliases are
pretty much hardcoded in the log building function. But in general
logformat aliases should be simply considered as an alternative to using
expressions (with '%[expr']')
Also, under log-format context, when we want to refer to either an alias
('%alias') or an expression ('%[expr]'), we should use the generic term
'logformat item', which in fact designates a single item within the
logformat string provided by the user. Indeed, a logformat item (whether
is is an alias or an expression) always starts with '%' and may accept
optional flags / arguments
Both the code and the documentation were updated in that sense, hopefully
this will clarify things and prevent future confusions.
Released version 3.0-dev13 with the following main changes :
- CLEANUP: ssl/cli: remove unused code in dump_crtlist_conf
- MINOR: ssl: check parameter in ckch_conf_cmp()
- BUG/MINOR: ring: free ring's allocated area not ring's usable area when using maps
- DOC: configuration: rework the crt-store load documentation
- DEBUG: tools: add vma_set_name() helper
- DEBUG: shctx: name shared memory using vma_set_name()
- DEBUG: sink: add name hint for memory area used by memory-backed sinks
- DEBUG: pollers: add name hint for large memory areas used by pollers
- DEBUG: errors: add name hint for startup-logs memory area
- DEBUG: fd: add name hint for large memory areas
- MEDIUM: ssl: don't load file by discovering them in crt-store
- DOC: configuration: update the crt-list documentation
- DOC: configuration: add the supported crt-store options in crt-list
- BUG/MEDIUM: proto: fix fd leak in <proto>_connect_server
- MINOR: sock: set conn->err_code in case of EPERM
- BUG/MINOR: http-ana: Don't crush stream termination condition on internal error
- MAJOR: spoe: Let the SPOE back into the game
- BUG/MINOR: connection: parse PROXY TLV for LOCAL mode
- BUG/MINOR: server: free PROXY v2 TLVs on srv drop
- MINOR: rhttp: add log on connection allocation failure
- BUG/MEDIUM: rhttp: fix preconnect on single-thread
- BUG/MINOR: rhttp: prevent listener suspend
- BUG/MINOR: rhttp: fix task_wakeup state
- MINOR: session: define flag to explicitely release listener on free
- MEDIUM: rhttp: create session for active preconnect
- MINOR: rhttp: support PROXY emission on preconnect
- MINOR: connection: support PROXY v2 TLV emission without stream
- MINOR: traces: enumerate the list of levels/verbosities when not found
- BUG/MINOR: sock: fix sock_create_server_socket
- MINOR: proto: fix coding style
- BUG/MAJOR: quic: Crash with TLS_AES_128_CCM_SHA256 (libressl only)
- REGTESTS: scripts: allow to change the vtest timeout
- BUG/MEDIUM: quic_tls: prevent LibreSSL < 4.0 from negotiating CHACHA20_POLY1305
- CI: scripts/build-ssl.sh: loudly fail on unsupported platforms
- BUG/MEDIUM: mux-quic: Create sedesc in same time of the QUIC stream
- MINOR: mux-quic: Set abort info for SC-less QCS on STOP_SENDING frame
- CI: scripts/build-ssl: add a DESTDIR and TMPDIR variable
- CI: scripts/buil-ssl: cleanup the boringssl and quictls build
- MINOR: config: add thread-hard-limit to set an upper bound to nbthread
- BUILD: quic: fix unused variable warning when threads are disabled
- BUG/MEDIUM: stick-tables: Fix race with peers when trashing oldest entries
- BUG/MEDIUM: stick-tables: Fix race with peers when killing a sticky session
- BUG/MEDIUM: stick-tables: make sure never to create two same remote entries
- CLEANUP: stick-tables: remove a few unneeded tests for use_wrlock
- MINOR: stick-tables: remove the uneeded read lock in stksess_free()
- CLEANUP: tools: fix vma_set_name() function comment
- DEBUG: tools: add vma_set_name_id() helper
- DEBUG: pollers/fd: add thread id suffix to per-thread memory areas name hints
- DOC: config: fix aes_gcm_enc() description text
- BUILD: trace: fix warning on null dereference
- MEDIUM: config: prevent communication with privileged ports
- MAJOR: config: prevent QUIC with clients privileged port by default
- BUG/MINOR: quic: adjust restriction for stateless reset emission
- MINOR: quic: clarify doc for quic_recv()
- MINOR: server: generalize sni expr parsing
- MINOR: server: define pool-conn-name keyword
- MEDIUM: connection: use pool-conn-name instead of sni on reuse
- BUG/MINOR: rhttp: initialize session origin after preconnect reversal
- BUG/MEDIUM: server/dns: preserve server's port upon resolution timeout or error
- BUG/MINOR: http-htx: Support default path during scheme based normalization
- BUG/MINOR: server: Don't reset resolver options on a new default-server line
- DOC: quic: specify that connection migration is not supported
- DOC: config: fix incorrect section reference about custom log format
- DOC: config: uniformize the naming and description of custom log format args
- DOC: config: clarify the fact that custom log format is not just for logging
- REGTESTS: acl_cli_spaces: avoid a warning caused by undefined logs
The wording in the Custom log format section was still extremely centered
on logging, but it's about time to mention that these are usable for other
actions as well, otherwise it's very confusing for newcomers who try to
define a variable or header. The updated text also reminds about the risks
of safe encodings that may (rarely) mangle an output string, and encourages
to migrate away from the unquoted definition which is full of backslashes.
It would definitely deserve further improvements and refinements.
A significant number of actions now take arguments that are evaluated as
log-format expressions. Some of them are called "fmt", others "string".
The description of the argument sometimes just says "the log-format
string" or "log format" or "custom log format" etc. Most of them do not
mention the section to visit, and section 8.2 speaking about log-format
is very centric on logs usage (the primary use case), making all of this
very confusing for newcomers.
Since section 8.2.6 is titled "Custom log format" and describes the syntax
to be used with the "log-format" (and other) directives, let's call this
"Custom log format" everywhere and mention section 8.2.6. When the field
was called "string", it was also renamed to "fmt".
It doesn't seem worth backporting this, unless it applies fine.
Since 2.5 with commit 98b930d043 ("MINOR: ssl: Define a default https
log format"), some log-format sections were shifted a bit without having
been renumberred, causing 8.2.4 to be referenced as the custom log
format while it's in fact 8.2.6. This patch fixes the affected
locations.
In addition two places mentioned 8.2.6 instead of 8.2.5 for the error
log format.
This can be backported to 2.6.
Currently haproxy does not support QUIC connection migration. This is
advertized to clients on their connections. Document this in the first
QUIC related paragraph.
This should be backported up to 2.6.
Define a new server keyword pool-conn-name. The purpose of this keyword
will be to identify connections inside the idle connections pool,
replacing SNI in case SSL is not wanted.
This keyword uses a sample expression argument. It thus can reuse
existing function parse_srv_expr() for parsing. In the future, it may be
necessary to define a keyword variant which uses a logformat for
extensability.
This patch only implement parsing. Argument is stored inside new server
field <pool_conn_name> and expression is generated in
_srv_parse_finalize() into <pool_conn_name_expr>.
If pool-conn-name is not set but SNI is, the latter is reused
automatically as pool-conn-name via _srv_parse_finalize(). This ensures
current reuse behavior remains compatible and idle connection reuse will
not mix connections with different SNIs by mistake.
Main usage will be for rhttp when SSL is not wanted between the two
haproxy instances. Previously, it was possible to use "sni" keyword even
without SSL on a server line which have a similar effect. However,
having a dedicated "pool-conn-name" keyword is deemed clearer. Besides,
it would allow for more complex configuration where pool-conn-name and
SNI are use in parallel with different values.
Previous commit introduce new protection mechanism to forbid
communications with clients which use a privileged source port. By
default, this mechanism is disabled for every protocols.
This patch changes the default value and activate the protection
mechanism for QUIC protocol. This is justified as it is a probable sign
of DNS/NTP amplification attack.
This is labelled as major as it can be a breaking change with some
network environments.
This commit introduces a new global setting named
harden.reject_privileged_ports.{tcp|quic}. When active, communications
with clients which use privileged source ports are forbidden. Such
behavior is considered suspicious as it can be used as spoofing or
DNS/NTP amplication attack.
Value is configured per transport protocol. For each TCP and QUIC
distinct code locations are impacted by this setting. The first one is
in sock_accept_conn() which acts as a filter for all TCP based
communications just after accept() returns a new connection. The second
one is dedicated for QUIC communication in quic_recv(). In both cases,
if a privileged source port is used and setting is disabled, received
message is silently dropped.
By default, protection are disabled for both protocols. This is to be
able to backport it without breaking changes on stable release.
This should be backported as it is an interesting security feature yet
relatively simple to implement.
On todays large systems, it's not always desired to run on all threads
for light loads, and usually users enforce nbthread to a lower value
(e.g. 8). The problem is that this is a fixed value, and moving such
configs to smaller machines continues to enforce the value and this
becomes extremely unproductive due to having more threads than CPUs.
This also happens quite a bit in VMs, containers, or cloud instances
of various sizes.
This commit introduces the thread-hard-limit setting that allows to only
set an upper bound to the number of threads without raising a lower value.
This means that using "thread-hard-limit 8" will make sure that no more
than 8 threads will be used when available, but it will remain two when
run on a dual-core machine.
It's quite frustrating, particularly on the command line, not to have
access to the list of available levels and verbosities when one does
not exist for a given source, because there's no easy way to find them
except by starting without and connecting to the CLI. Let's enumerate
the list of supported levels and verbosities when a name does not match.
For example:
$ ./haproxy -db -f quic-repro.cfg -dt h2:help
[NOTICE] (9602) : haproxy version is 3.0-dev12-60496e-27
[NOTICE] (9602) : path to executable is ./haproxy
[ALERT] (9602) : -dt: no such trace level 'help', available levels are 'error', 'user', 'proto', 'state', 'data', and 'developer'.
$ ./haproxy -db -f quic-repro.cfg -dt h2:user:help
[NOTICE] (9604) : haproxy version is 3.0-dev12-60496e-27
[NOTICE] (9604) : path to executable is ./haproxy
[ALERT] (9604) : -dt: no such trace verbosity 'help' for source 'h2', available verbosities for this source are: 'quiet', 'clean', 'minimal', 'simple', 'advanced', and 'complete'.
The same is done for the CLI where the existing help message is always
displayed when entering an invalid verbosity or level.
This reverts commits 885e40494c and
dff9807188.
We decided to spend some time to refactor and rationnalize the SPOE for the
3.1. Thus there is no reason to still consider it as deprecated for the
3.0. Compatibility between the both versions will be maintained.
See #2502 for more info.
The load keyword from the documentation has its own section to be
readable (like the server or bind options section).
The ocsp-update keyword was move from the bind section to the crt-list
load one.
Released version 3.0-dev12 with the following main changes :
- CI: drop asan.log umbrella completely
- BUG/MINOR: log: fix leak in add_sample_to_logformat_list() error path
- BUG/MINOR: log: smp_rgs array issues with inherited global log directives
- MINOR: rhttp: Don't require SSL when attach-srv name parsing
- REGTESTS: ssl: be more verbose with ocsp_compat_check.vtc
- DOC: Update UUID references to RFC 9562
- MINOR: hlua: add hlua_nb_instruction getter
- MEDIUM: hlua: take nbthread into account in hlua_get_nb_instruction()
- BUG/MEDIUM: server: clear purgeable conns before server deletion
- BUG/MINOR: mux-quic: fix error code on shutdown for non HTTP/3
- BUG/MINOR: qpack: fix error code reported on QPACK decoding failure
- BUG/MEDIUM: htx: mark htx_sl as packed since it may be realigned
- BUG/MEDIUM: stick-tables: properly mark stktable_data as packed
- SCRIPTS: run-regtests: fix a few occurrences of extended regexes
- BUG/MINOR: ssl_sock: fix xprt_set_used() to properly clear the TASK_F_USR1 bit
- MINOR: dynbuf: provide a b_dequeue() variant for multi-thread
- BUG/MEDIUM: muxes: enforce buf_wait check in takeover()
- BUG/MINOR: h1: Check authority for non-CONNECT methods only if a scheme is found
- BUG/MEDIUM: h1: Reject CONNECT request if the target has a scheme
- BUG/MAJOR: h1: Be stricter on request target validation during message parsing
- MINOR: qpack: prepare error renaming
- MINOR: h3/qpack: adjust naming for errors
- MINOR: h3: adjust error reporting on sending
- MINOR: h3: adjust error reporting on receive
- MINOR: mux-quic: support glitches
- MINOR: h3: report glitch on RFC violation
- BUILD: stick-tables: better mark the stktable_data as 32-bit aligned
- MINOR: ssl: rename tune.ssl.ocsp-update.mode in ocsp-update.mode
- REGTESTS: update the ocsp-update tests
- BUILD: stats: remove non portable getline() usage
- MEDIUM: ssl: add ocsp-update.mindelay and ocsp-update.maxdelay
- BUILD: log: get rid of non-portable strnlen() func
- BUG/MEDIUM: fd: prevent memory waste in fdtab array
- CLEANUP: compat: make the MIN/MAX macros more reliable
- Revert: MEDIUM: evports: permit to report multiple events at once"
- BUG/MINOR: stats: Don't state the 303 redirect response is chunked
- MINOR: mux-h1: Add a flag to ignore the request payload
- REORG: mux-h1: Group H1S_F_BODYLESS_* flags
- CLEANUP: mux-h1: Remove unused H1S_F_ERROR_MASK mask value
- MEDIUM: mux-h1: Support C-L/T-E header suppressions when sending messages
- MINOR: ssl: ckch_store_new_load_files_conf() loads filenames from ckch_conf
- MEDIUM: ssl/crtlist: loading crt-store keywords from a crt-list
- CLEANUP: ssl/ocsp: remove the deprecated parsing code for "ocsp-update"
- MINOR: ssl: pass ckch_store instead of ckch_data to ssl_sock_load_ocsp()
- MEDIUM: ssl: ckch_conf_parse() uses -1/0/1 for off/default/on
- MINOR: ssl: handle PARSE_TYPE_INT and PARSE_TYPE_ONOFF in ckch_store_load_files()
- MINOR: ssl/ocsp: use 'ocsp-update' in crt-store
- MINOR: ssl: ckch_conf_clean() utility function for ckch_conf
- MEDIUM: ssl: add ocsp-update.disable global option
- MEDIUM: ssl/cli: handle crt-store keywords in crt-list over the CLI
- MINOR: ssl: ckch_conf_cmp() compare multiple ckch_conf structures
- MEDIUM: ssl: temporarily load files by detecting their presence in crt-store
- REGTESTS: ocsp-update: change the reg-test to support the new crt-store mode
- DOC: capabilities: fix chapter header rendering
The header of a new management guide chapter, "13.1. Linux capabilities
support", is not rendered in HTML format in a proper way, because of missing
dots at the end of this chapter's number.
This option allow to disable completely the ocsp-update.
To achieve this, the ocsp-update.mode global keyword don't rely anymore
on SSL_SOCK_OCSP_UPDATE_OFF during parsing to call
ssl_create_ocsp_update_task().
Instead, we will inherit the SSL_SOCK_OCSP_UPDATE_* value from
ocsp-update.mode for each certificate which does not specify its own
mode.
To disable completely the ocsp without editing all crt entries,
ocsp-update.disable is used instead of "ocsp-update.mode" which is now
only used as the default value for crt.
Since the ocsp-update is not strictly a tuning of the SSL stack, but a
feature of its own, lets rename the option.
The option was also missing from the index.
Implement basic support for glitches on QUIC multiplexer. This is mostly
identical too glitches for HTTP/2.
A new configuration option named tune.quic.frontend.glitches-threshold
is defined to limit the number of glitches on a connection before
closing it.
Glitches counter is incremented via qcc_report_glitch(). A new
qcc_app_ops callback <report_susp> is defined. On threshold reaching, it
allows to set an application error code to close the connection. For
HTTP/3, value H3_EXCESSIVE_LOAD is returned. If not defined, default
code INTERNAL_ERROR is used.
For the moment, no glitch are reported for QUIC or HTTP/3 usage. This
will be added in future patches as needed.
Based on Willy's idea (from 3.0-dev6 announcement message): in this patch
we try to reduce the max latency that can be caused by running lua scripts
with default settings.
Indeed, by default, hlua engine is allowed to process up to 10k
instructions per batch. While this value was found to be the optimal one
for a single thread, it turns out that keeping a thread busy for 10k lua
instructions could increase thread contention. This is especially true
when the script is loaded with 'lua-load', because in that case the
current thread owns the main lua lock and prevent other threads from
making any progress if they're also waiting on the main lock.
Thanks to Thierry Fournier's work, we know that performance-wise we can
reach optimal performance by sticking between 500 and 10k instructions
per batch. Given that, when the script is loaded using 'lua-load', if no
"tune.lua.forced-yield" was set by the user, we automatically divide the
default value (10K) by the number of threads haproxy can use to reduce
thread contention (given that all threads could compete for the main lua
lock), however we make sure not to return a value below 500, because
Thierry's work showed that this would come with a significant performance
loss.
The historical behavior may still be enforced by setting
"tune.lua.forced-yield" to 10000 in the global config section.
When support for UUIDv7 was added in commit
aab6477b67
the specification still was a draft.
It has since been published as RFC 9562.
This patch updates all UUID references from the obsoleted RFC 4122 and the
draft for RFC 9562 to the published RFC 9562.
Released version 3.0-dev11 with the following main changes :
- BUILD: clock: improve check for pthread_getcpuclockid()
- CI: add Illumos scheduled workflow
- CI: netbsd: limit scheduled workflow to parent repo only
- OPTIM: log: resolve logformat options during postparsing
- BUG/MINOR: haproxy: only tid 0 must not sleep if got signal
- REGTEST: add tests for acl() sample fetch
- BUG/MINOR: acl: support built-in ACLs with acl() sample
- BUG/MINOR: cfgparse: use curproxy global var from config post validation
- MEDIUM: stconn/muxes: Add an abort reason for SE shutdowns on muxes
- MINOR: mux-h2: Set the SE abort reason when a RST_STREAM frame is received
- MEDIUM: mux-h2: Forward h2 client cancellations to h2 servers
- MINOR: mux-quic: Set tha SE abort reason when a STOP_SENDING frame is received
- MINOR: stconn: Add samples to retrieve about stream aborts
- MINOR: mux-quic: Add .ctl callback function to get info about a mux connection
- MINOR: muxes: Add ctl commands to get info on streams for a connection
- MINOR: connection: Add samples to retrieve info on streams for a connection
- BUG/MEDIUM: log/ring: broken syslog octet counting
- BUG/MEDIUM: mux-quic: fix crash on STOP_SENDING received without SD
- DOC: lua: fix filters.txt file location
- MINOR: dynbuf: pass a criticality argument to b_alloc()
- MINOR: dynbuf: add functions to help queue/requeue buffer_wait fields
- MINOR: dynbuf: use the b_queue()/b_requeue() functions everywhere
- MEDIUM: dynbuf: make the buffer_wq an array of list heads
- CLEANUP: tinfo: better align fields in thread_ctx
- MINOR: dynbuf: provide a b_dequeue() function to detach a bw from the queue
- MEDIUM: dynbuf: generalize the use of b_dequeue() to detach buffer_wait
- MEDIUM: dynbuf/stream: re-enable queueing upon failed buffer allocation
- MEDIUM: dynbuf/stream: do not allocate the buffers in the callback
- MEDIUM: applet: make appctx_buf_available() only wake the applet up, not allocate
- MINOR: applet: set the blocking flag in the buffer allocation function
- MINOR: applet: adjust the allocation criticity based on the requested buffer
- MINOR: dynbuf/mux-h1: use different criticalities for buffer allocations
- MEDIUM: dynbuf/mux-h1: do not allocate the buffers in the callback
- MEDIUM: dynbuf: refrain from offering a buffer if more critical ones are waiting
- MINOR: stconn: report that a buffer allocation succeeded
- MINOR: stream: report that a buffer allocation succeeded
- MINOR: applet: report about buffer allocation success
- MINOR: mux-h1: report that a buffer allocation succeeded
- MEDIUM: stream: allocate without queuing when retrying
- MEDIUM: channel: allocate without queuing when retrying
- MEDIUM: mux-h1: allocate without queuing when retrying
- MEDIUM: dynbuf: implement emergency buffers
- MEDIUM: dynbuf: use emergency buffers upon failed memory allocations
The buffer reserve set by tune.buffers.reserve has long been unused, and
in order to deal gracefully with failed memory allocations we'll need to
resort to a few emergency buffers that are pre-allocated per thread.
These buffers are only for emergency use, so every time their count is
below the configured number a b_free() will refill them. For this reason
their count can remain pretty low. We changed the default number from 2
to 4 per thread, and the minimum value is now zero (e.g. for low-memory
systems). The tune.buffers.limit setting has always been a problem when
trying to deal with the reserve but now we could simplify it by simply
pushing the limit (if set) to match the reserve. That was already done in
the past with a static value, but now with threads it was a bit trickier,
which is why the per-thread allocators increment the limit on the fly
before allocating their own buffers. This also means that the configured
limit is saner and now corresponds to the regular buffers that can be
allocated on top of emergency buffers.
At the moment these emergency buffers are not used upon allocation
failure. The only reason is to ease bisecting later if needed, since
this commit only has to deal with resource management.
The goal is to indicate how critical the allocation is, between the
least one (growing an existing buffer ring) and the topmost one (boot
time allocation for the life of the process).
The 3 tcp-based muxes (h1, h2, fcgi) use a common allocation function
to try to allocate otherwise subscribe. There's currently no distinction
of direction nor part that tries to allocate, and this should be revisited
to improve this situation, particularly when we consider that mux-h2 can
reduce its Tx allocations if needed.
For now, 4 main levels are planned, to translate how the data travels
inside haproxy from a producer to a consumer:
- MUX_RX: buffer used to receive data from the OS
- SE_RX: buffer used to place a transformation of the RX data for
a mux, or to produce a response for an applet
- CHANNEL: the channel buffer for sync recv
- MUX_TX: buffer used to transfer data from the channel to the outside,
generally a mux but there can be a few specificities (e.g.
http client's response buffer passed to the application,
which also gets a transformation of the channel data).
The other levels are a bit different in that they don't strictly need to
allocate for the first two ones, or they're permanent for the last one
(used by compression).
At the beginning of the filter class section, we encourage the user to
check out filters.txt file to get to know how the filters API works
within haproxy.
However the file location is incorrect. The proper directory to look for
the file is: doc/internals/api.
It should be backported up to 2.5.
Thanks to the previous fix, it is now possible to get the number of opened
streams for a connection and the negociated limit. Here, corresponding
sample feches are added, in fc_ and bc_ scopes.
On frontend side, the limit of streams is imposed by HAProxy. But on the
backend side, the limit is defined by the server. it may be useful for
debugging purpose because it may explain slow-downs on some processing.
It is now possible to retrieve some info about the abort received for a
server or a client stream, if any.
* fs.aborted and bs.aborted can be used to know if an abort was received
on frontend or backend side. A boolean is returned.
* fs.rst_code and bs.rst_code return the code of the received RESET_STREAM
frame for a H2 stream or the code of the received STOP_SENDING frame for
a QUIC stream. In both cases, the error code attached to the frame is
returned. The sample fetch fails if no such frame was received or if the
stream is not an H2/QUIC stream.
Released version 3.0-dev10 with the following main changes :
- BUG/MEDIUM: cache: Vary not working properly on anything other than accept-encoding
- REGTESTS: cache: Add test on 'vary' other than accept-encoding
- BUG/MINOR: stats: replace objt_* by __objt_* macros
- CLEANUP: tools/cbor: rename cbor_encode_ctx struct members
- MINOR: log/cbor: _lf_cbor_encode_byte() explicitly requires non-NULL ctx
- BUG/MINOR: log: fix global lf_expr node options behavior
- CLEANUP: log: add a macro to know if a lf_node is configurable
- MINOR: httpclient: allow to use absolute URI with new flag HC_F_HTTPROXY
- MINOR: ssl: introduce ocsp_update.http_proxy for ocsp-update keyword
- BUG/MINOR: log/encode: consider global options for key encoding
- BUG/MINOR: log/encode: fix potential NULL-dereference in LOGCHAR()
- BUG/MINOR: log: fix global lf_expr node options behavior (2nd try)
- MINOR: log/cbor: _lf_cbor_encode_byte() explicitly requires non-NULL ctx (again)
- BUG/MEDIUM: log: don't ignore disabled node's options
- BUG/MINOR: stconn: don't wake up an applet waiting on buffer allocation
- MINOR: sock: rename sock to sock_fd in sock_create_server_socket
- MEDIUM: proto_uxst: take in account server namespace
- MEIDUM: unix sock: use my_socketat to create bind socket
- MINOR: sock_set_mark: take sock family in account
- MEDIUM: proto: make common fd checks in sock_create_server_socket
- MINOR: sock: add EPERM case in sock_handle_system_err
- MINOR: capabilities: add cap_sys_admin support
- CLEANUP: ssl: clean the includes in ssl_ocsp.c
- CLEANUP: ssl: move the global ocsp-update options parsing to ssl_ocsp.c
- MINOR: stats: fix visual alignment for stat_cols_px definition
- MINOR: stats: convert req_tot as generic column
- MINOR: stats: prepare stats-file support for values other than FN_COUNTER
- MINOR: counters: move freq-ctr from proxy/server into counters struct
- MINOR: stats: support rate in stats-file
- MINOR: stats: convert rate as generic column for proxy stats
- MINOR: counters: move last_change into counters struct
- MINOR: stats: support age in stats-file
- MINOR: stats: convert age as generic column for proxy stat
- CLEANUP: ssl: rename new_ckch_store_load_files_path() to ckch_store_new_load_files_path()
- MINOR: ssl: rename ocsp_update.http_proxy into ocsp-update.httpproxy
- REORG: stats: define stats-proxy source module
- MINOR: stats: extract proxy clear-counter in a dedicated function
- REGTESTS: stats: add test stats-file counters preload
- CI: netbsd: adjust packages after NetBSD-10 released
- CLEANUP: assorted typo fixes in the code and comments
- REGTESTS: replace REQUIRE_VERSION by version_atleast
- MEDIUM: log: optimizing tmp->type handling in sess_build_logline()
- BUG/MINOR: log: prevent double spaces emission in sess_build_logline()
- OPTIM: log: declare empty buffer as global variable
- OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack
- OPTIM: log: use lf_buildctx's buffer instead of temporary stack buffers
- OPTIM: log: speedup date printing in sess_build_logline() when no encoding is used
If 'namespace' keyword is used in the backend server settings or/and in the
bind string, it means that haproxy process will call setns() to change its
default namespace to the configured one and then, it will create a
socket in this new namespace. setns() syscall requires CAP_SYS_ADMIN
capability in the process Effective set (see man 2 setns). Otherwise, the
process must be run as root.
To avoid to run haproxy as root, let's add cap_sys_admin capability in the
same way as we already added the support for some other network capabilities.
As CAP_SYS_ADMIN belongs to CAP_SYS_* capabilities type, let's add a separate
flag LSTCHK_SYSADM for it. This flag is set, if the 'namespace' keyword was
found during configuration parsing. The flag may be unset only in
prepare_caps_for_setuid() or in prepare_caps_from_permitted_set(), which
inspect process EUID/RUID and Effective and Permitted capabilities sets.
If system doesn't support Linux capabilities or 'cap_sys_admin' was not set
in 'setcap', but 'namespace' keyword is presented in the configuration, we
keep the previous strict behaviour. Process, that has changed uid to the
non-priviledged user, will terminate with alert. This alert invites the user
to recheck its configuration.
In the case, when haproxy will start and run under a non-root user and
'cap_sys_admin' is not set, but 'namespace' keyword is presented, this patch
does not change previous behaviour as well. We'll still let the user to try
its configuration, but we inform via warning, that unexpected things, like
socket creation errors, may occur.
The ocsp_update.http_proxy global option allows to set an HTTP proxy
address which will be used to send the OCSP update request with an
absolute form URI.
Released version 3.0-dev9 with the following main changes :
- BUILD: ssl: use %zd for sizeof() in ssl_ckch.c
- MINOR: backend: use be_counters for health down accounting
- BUG/MINOR: backend: use cum_sess counters instead of cum_conn
- BUG/MINOR: stats: fix stot metric for listeners
- REGTESTS: use -dI for insecure fork by default in the regtest scripts
- MINOR: stats: rename proxy stats
- MINOR: stats: rename ambiguous stat_l and stat_count
- MINOR: stats: rename info stats
- MINOR: stats: use stricter naming stats/field/line
- MINOR: stats: use STAT_F_* prefix for flags
- BUG/MEDIUM: applet: Let's applets decide if they have more data to deliver
- BUILD: stick-tables: silence build warnings when threads are disabled
- MINOR: tools: Rename `ha_generate_uuid` to `ha_generate_uuid_v4`
- MINOR: Add `ha_generate_uuid_v7`
- MINOR: Add support for UUIDv7 to the `uuid` sample fetch
- MEDIUM: shctx: Naming shared memory context
- BUG/MINOR: h1: fix detection of upper bytes in the URI
- MINOR: intops: add a pair of functions to check multi-byte ranges
- TESTS: add a unit test for the multi-byte range checks
- CLEANUP: h1: make use of the multi-byte matching functions
- REGTESTS: ssl: Remove "sleep" calls from ocsp auto update test
- BUG/MEDIUM: peers: Automatically start to learn on local peer
- BUG/MEDIUM: peers: Reprocess peer state after all session shutdowns
- MINOR: peers: Remove unused PEERS_F_RESYNC_REQUESTED flag
- MINOR: peers: Don't set TEACH flags on a peer from the sync task
- MINOR: peers: Use a peer flag to block the applet waiting ack of the sync task
- BUG/MEDIUM: peers: Wait for sync task ack when a resynchro is finished
- MINOR: peers: Remove unused PEERS_F_RESYNC_PROCESS flag
- MINOR: applet: Add a function to know the side where an applet was created
- MEDIUM: peers: Simplify the peer flags dealing with the connection state
- MEDIUM: peers: Use true states for the peer applets as seen from outside
- MEDIUM: peers: Use true states for the learn state of a peer
- MINOR: peers: Start learning for local peer before receiving messages
- MINOR: peers: Rename PEERS_F_TEACH_COMPLETE to PEERS_F_LOCAL_TEACH_COMPLETE
- MINOR: peers: Reorder and slightly rename PEER flags
- MINOR: peers: Reorder and rename PEERS flags
- REORG: peers: Move peer and peers flags in the corresponding header file
- DEV: flags/peers: Decode PEER and PEERS flags
- MINOR: peers: Add comment on processing functions of the sync task
- MINOR: peers: Use a static variable to wait a resync on reload
- BUG/MEDIUM: peers: Use atomic operations on peers flags when necessary
- REORG: peers: Rename all occurrences to 'ps' variable
- BUG/MINOR: peers: Don't wait for a remote resync if there no remote peer
- MINOR: stats: update ambiguous "metrics" naming to "stat_cols"
- MINOR: stats: introduce a more expressive stat definition method
- MINOR: stats: implement automatic metric generation from stat_col
- MINOR: stats: hide some columns in output
- MEDIUM: stats: convert counters to new column definition
- MINOR: stats: define stats-file output format support
- MEDIUM: stats: implement dump stats-file CLI
- MINOR: ist: define iststrip() new function
- MINOR: guid: define guid_is_valid_fmt()
- MINOR: stats: apply stats-file on process startup
- MINOR: stats: parse header lines from stats-file
- MINOR: stats: parse values from stats-file
- MEDIUM: stats: define stats-file keyword
- BUG/MINOR: mworker: reintroduce way to disable seamless reload with -x /dev/null
- CLEANUP: log: remove unused checks for encode_{chunk,string}
- MINOR: log: store lf_expr nodes inside substruct
- MINOR: log: global lf_expr node options
- CLEANUP: log: simplify complex values usages in sess_build_logline()
- MINOR: log: skip custom logformat_node name if empty
- MINOR: log: add lf_int() wrapper to print integers
- MINOR: log: add lf_rawtext{_len}() functions
- MEDIUM: log: pass date strings to lf_rawtext()
- MEDIUM: log: write raw strings using lf_rawtext()
- MEDIUM: log: use lf_rawtext for lf_ip() and lf_port() hex strings
- MINOR: log: explicitly handle %ts and %tsc as text strings
- MINOR: log: use LOG_VARTEXT_{START,END} to enclose text strings
- MINOR: log: make all lf_* sess build helper static
- MINOR: log: merge lf_encode_string() and lf_encode_chunk() logic
- MEDIUM: log: lf_* build helpers now take a ctx argument
- MINOR: log: expose node typecast in lf_buildctx struct
- MINOR: log: postpone conversion for sample expressions in sess_build_logline()
- MINOR: log: add LOG_OPT_NONE flag
- MINOR: log: add no_escape_map to bypass escape with _lf_encode_bytes()
- MINOR: log: add +bin logformat node option
- MINOR: log: add +json encoding option
- MINOR: tools: add cbor encode helpers
- MINOR: log: add +cbor encoding option
- MINOR: log: support true cbor binary encoding
- CLEANUP: dynbuf: move the reserve and limit parsers to dynbuf.c
- MINOR: list: add a macro to detect that a list contains at most one element
- MINOR: cli/wait: rename the condition "srv-unused" to "srv-removable"
As previously discussed, "srv-unused" is sufficiently ambiguous to cause
some trouble over the long term. Better use "srv-removable" to indicate
that the server is removable, and if the conditions to delete a server
change over time, the wait condition will be adjusted without renaming
it.
CBOR in hex format as implemented in previous commit is convenient because
the produced output is portable and can easily be embedded in regular
syslog payloads.
However, one of the goal of CBOR implementation is to be able to produce
"Concise Binary" object representation. Here is an excerpt from cbor.io
website:
"Some applications also benefit from CBOR itself being encoded in
binary. This saves bulk and allows faster processing."
Currently we don't offer that with '+cbor', quite the opposite actually
since a text string encoded with '+cbor' option will be larger than a
text string encoded with '+json' or without encoding at all, because for
each CBOR binary byte, 2 characters will be emitted.
Hopefully, the sink/log API allows for binary data to be passed as
parameter, this is because all relevant functions in the chain don't rely
on the terminating NULL byte and take a string pointer + string length as
parameter. We can actually rely on this property to support the '+bin'
option when combined with '+cbor' to produce RAW binary CBOR output.
Be careful though, as this is only intended for use with set-var-fmt or to
send binary data to capable UDP/ring endpoints.
Example:
log-format "%{+cbor,+bin}o %(test)[bin(00AABB)]"
Will produce:
bf64746573745f4300aabbffff
(output was piped to `hexdump -ve '1/1 "%.2x"'` to dump raw bytes as HEX
characters)
With cbor.me pretty printer, it gives us:
BF # map(*)
64 # text(4)
74657374 # "test"
5F # bytes(*)
43 # bytes(3)
00AABB # "\u0000\xAA\xBB"
FF # primitive(*)
FF # primitive(*)
In this patch, we make use of the CBOR (RFC8949) encode helper functions
from the previous commit to implement '+cbor' encoding option for log-
formats. The logic behind it is pretty similar to '+json' encoding option,
except that the produced output is a CBOR payload written in HEX format so
that it remains compatible to use this with regular syslog endpoints.
Example:
log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]"
Will produce:
BF6B6E616D65645F6669656C64626F6BFF
Detailed view (from cbor.me):
BF # map(*)
6B # text(11)
6E616D65645F6669656C64 # "named_field"
62 # text(2)
6F6B # "ok"
FF # primitive(*)
If the option isn't set globally, but on a specific node instead, then
only the value will be encoded according to CBOR specification.
Example:
log-format "test cbor bool: %{+cbor}[bool(true)]"
Will produce:
test cbor bool: F5
In this patch, we add the "+json" log format option that can be set
globally or per log format node.
What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the
current context which is provided to all lf_* log building function.
This way, all lf_* are now aware of this option and try to comply with
JSON specification when the option is set.
If the option is set globally, then sess_build_logline() will produce a
map-like object with key=val pairs for named logformat nodes.
(logformat nodes that don't have a name are simply ignored).
Example:
log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]"
Will produce:
{"named_field": "ok"}
If the option isn't set globally, but on a specific node instead, then
only the value will be encoded according to JSON specification.
Example:
log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }"
Will produce:
{"manual_key": true}
When the option is set, +E option will be ignored, and partial numerical
values (ie: because of logasap) will be encoded as-is.
Support '+bin' option argument on logformat nodes to try to preserve
binary output type with binary sample expressions.
For this, we rely on the log/sink API which is capable of conveying binary
data since all related functions don't search for a terminating NULL byte
in provided log payload as they take a string pointer and a string length
as argument.
Example:
log-format "%{+bin}o %[bin(00AABB)]"
Will produce:
00aabb
(output was piped to `hexdump -ve '1/1 "%.2x"'` to dump raw bytes as HEX
characters)
This should be used carefully, because many syslog endpoints don't expect
binary data (especially NULL bytes). This is mainly intended for use with
set-var-fmt actions or with ring/udp log endpoints that know how to deal
with such binary payloads.
Also, this option is only supported globally (for use with '%o'), it will
not have any effect when set on an individual node. (it makes no sense to
have binary data in the middle of log payload that was started without
binary data option)
Since the introduction of the automatic seamless reload using the
internal socketpair, there is no way of disabling the seamless reload.
Previously we just needed to remove -x from the startup command line,
and remove any "expose-fd" keyword on stats socket lines.
This was introduced in 2be557f7c ("MEDIUM: mworker: seamless reload use
the internal sockpairs").
The patch copy /dev/null again and pass it to the next exec so we never
try to get socket from the -x.
Must be backported as far as 2.6.
This commit is the final to implement preloading of haproxy internal
counters via stats-file parsing.
Define a global keyword "stats-file". It allows to specify the path to
the stats-file which will be parsed on process startup.
Define a new CLI command "dump stats-file" with its handler
cli_parse_dump_stat_file(). It will loop twice on proxies_list to dump
first frontend and then backend side. It reuses the common function
stats_dump_stat_to_buffer(), using STAT_F_BOUND to restrict on the
correct side.
A new module stats-file.c is added to regroup function specifics to
stats-file. It defines two main functions :
* stats_dump_file_header() to generate the list of column list prefixed
by the line context, either "#fe" or "#be"
* stats_dump_fields_file() to generate each stat lines. Object without
GUID are skipped. Each stat entry is separated by a comma.
For the moment, stats-file does not support statistics modules. As such,
stats_dump_*_line() functions are updated to prevent looping over stats
module on stats-file output.
Released version 3.0-dev8 with the following main changes :
- BUG/MINOR: cli: Don't warn about a too big command for incomplete commands
- BUG/MINOR: listener: always assign distinct IDs to shards
- BUG/MINOR: log: fix lf_text_len() truncate inconsistency
- BUG/MINOR: tools/log: invalid encode_{chunk,string} usage
- BUG/MINOR: log: invalid snprintf() usage in sess_build_logline()
- CLEANUP: log: lf_text_len() returns a pointer not an integer
- MINOR: quic: simplify qc_send_hdshk_pkts() return
- MINOR: quic: uniformize sending methods for handshake
- MINOR: quic: improve sending API on retransmit
- MINOR: quic: use qc_send_hdshk_pkts() in handshake IO cb
- MEDIUM: quic: remove duplicate hdshk/app send functions
- OPTIM: quic: do not call qc_send() if nothing to emit
- OPTIM: quic: do not call qc_prep_pkts() if everything sent
- BUG/MEDIUM: http-ana: Deliver 502 on keep-alive for fressh server connection
- BUG/MINOR: http-ana: Fix TX_L7_RETRY and TX_D_L7_RETRY values
- BUILD: makefile: warn about unknown USE_* variables
- BUILD: makefile: support USE_xxx=0 as well
- BUG/MINOR: guid: fix crash on invalid guid name
- BUILD: atomic: fix peers build regression on gcc < 4.7 after recent changes
- BUG/MINOR: debug: make sure DEBUG_STRICT=0 does work as documented
- BUILD: cache: fix non-inline vs inline declaration mismatch to silence a warning
- BUILD: debug: make DEBUG_STRICT=1 the default
- BUILD: pools: make DEBUG_MEMORY_POOLS=1 the default option
- CI: update the build options to get rid of unneeded DEBUG options
- BUILD: makefile: get rid of the config CFLAGS variable
- BUILD: makefile: allow to use CFLAGS to append build options
- BUILD: makefile: drop the SMALL_OPTS settings
- BUILD: makefile: move -O2 from CPU_CFLAGS to OPT_CFLAGS
- BUILD: makefile: get rid of the CPU variable
- BUILD: makefile: drop the ARCH variable and better document ARCH_FLAGS
- BUILD: makefile: extract ARCH_FLAGS out of LDFLAGS
- BUILD: makefile: move the fwrapv option to STD_CFLAGS
- BUILD: makefile: make the ERR variable also support 0
- BUILD: makefile: add FAILFAST to select the -Wfatal-errors behavior
- BUILD: makefile: extract -Werror/-Wfatal-errors from automatic CFLAGS
- BUILD: makefile: split WARN_CFLAGS from SPEC_CFLAGS
- BUILD: makefile: rename SPEC_CFLAGS to NOWARN_CFLAGS
- BUILD: makefile: do not pass warnings to VERBOSE_CFLAGS
- BUILD: makefile: also drop DEBUG_CFLAGS
- CLEANUP: makefile: make the output of the "opts" target more readable
- DOC: install: clarify the build process by splitting it into subsections
- BUG/MINOR: server: fix slowstart behavior
- BUG/MEDIUM: cache/stats: Handle inbuf allocation failure in the I/O handler
- MINOR: ssl: add the section parser for 'crt-store'
- DOC: configuration: Add 3.12 Certificate Storage
- REGTESTS: ssl: test simple case of crt-store
- MINOR: ssl: rename ckchs_load_cert_file to new_ckch_store_load_files_path
- MINOR: ssl/crtlist: alloc ssl_conf only when a valid keyword is found
- BUG/MEDIUM: stick-tables: fix the task's next expiration date
- CLEANUP: stick-tables: always respect the to_batch limit when trashing
- BUG/MEDIUM: peers/trace: fix crash when listing event types
- BUG/MAJOR: stick-tables: fix race with peers in entry expiration
- DEBUG: pool: improve decoding of corrupted pools
- REORG: pool: move the area dump with symbol resolution to tools.c
- DEBUG: pools: report the data around the offending area in case of mismatch
- MINOR: listener/protocol: add proto name in alerts
- MINOR: proto_quic: add proto name in alert
- BUG/MINOR: lru: fix the standalone test case for invalid revision
- DOC: management: fix typos
- CI: revert kernel addr randomization introduced in 3a0fc864
- MINOR: ring: clarify the usage of ring_size() and add ring_allocated_size()
- BUG/MAJOR: ring: use the correct size to reallocate startup_logs
- MINOR: ring: always check that the old ring fits in the new one in ring_dup()
- CLEANUP: ssl: remove dead code in cfg_parse_crtstore()
- MINOR: ssl: supports crt-base in crt-store
- MINOR: ssl: 'key-base' allows to load a 'key' from a specific path
- MINOR: net_helper: Add support for floats/doubles.
- BUG/MEDIUM: grpc: Fix several unaligned 32/64 bits accesses
- MINOR: peers: Split resync process function to separate running/stopping states
- MINOR: peers: Add 2 peer flags about the peer learn status
- MINOR: peers: Add flags to report the peer state to the resync task
- MINOR: peers: sligthly adapt part processing the stopping signal
- MINOR: peers: Add functions to commit peer changes from the resync task
- BUG/MINOR: peers: Report a resync was explicitly requested from a thread-safe manner
- BUG/MAJOR: peers: Update peers section state from a thread-safe manner
- MEDIUM: peers: Only lock one peer at a time in the sync process function
- MINOR: peer: Restore previous peer flags value to ease debugging
- BUG/MEDIUM: stconn: Don't forward channel data if input data must be filtered
- BUILD: cache: fix a build warning with gcc < 7
- BUILD: xxhash: silence a build warning on Solaris + gcc-5.5
- CI: reduce ASAN log redirection umbrella size
- CLEANUP: assorted typo fixes in the code and comments
- BUG/MEDIUM: evports: do not clear returned events list on signal
- MEDIUM: evports: permit to report multiple events at once
- MEDIUM: ssl: support aliases in crt-store
- BUG/MINOR: ssl: check on forbidden character on wrong value
- BUG/MINOR: ssl: fix crt-store load parsing
- BUG/MEDIUM: applet: Fix applet API to put input data in a buffer
- BUG/MEDIUM: spoe: Always retry when an applet fails to send a frame
- BUG/MEDIUM: peers: Fix exit condition when max-updates-at-once is reached
- BUILD: linuxcap: Properly declare prepare_caps_from_permitted_set()
- BUG/MEDIUM: peers: fix localpeer regression with 'bind+server' config style
- MINOR: peers: stop relying on srv->addr to find peer port
- MEDIUM: ssl: support a named crt-store section
- MINOR: stats: remove implicit static trash_chunk usage
- REORG: stats: extract HTML related functions
- REORG: stats: extract JSON related functions
- MEDIUM: ssl: crt-base and key-base local keywords for crt-store
- MINOR: stats: Get the right prototype for stats_dump_html_end().
- MAJOR: ssl: use the msg callback mecanism for backend connections
- MINOR: ssl: implement keylog fetches for backend connections
- BUG/MINOR: stconn: Fix sc_mux_strm() return value
- MINOR: mux-pt: Test conn flags instead of sedesc ones to perform a full close
- MINOR: stconn/connection: Move shut modes at the SE descriptor level
- MINOR: stconn: Rewrite shutdown functions to simplify the switch statements
- MEDIUM: stconn: Use only one SC function to shut connection endpoints
- MEDIUM: stconn: Explicitly pass shut modes to shut applet endpoints
- MEDIUM: stconn: Use one function to shut connection and applet endpoints
- MEDIUM: muxes: Use one callback function to shut a mux stream
- BUG/MINOR: sock: handle a weird condition with connect()
- BUG/MINOR: fd: my_closefrom() on Linux could skip contiguous series of sockets
- BUG/MEDIUM: peers: Don't set PEERS_F_RESYNC_PROCESS flag on a peer
- BUG/MEDIUM: peers: Fix state transitions of a peer
- MINOR: init: use RLIMIT_DATA instead of RLIMIT_AS
- CI: modernize macos matrix
Limiting total allocatable process memory (VSZ) via setting RLIMIT_AS limit is
no longer effective, in order to restrict memory consumption at run time.
We can see from process memory map below, that there are many holes within
the process VA space, which bumps its VSZ to 1.5G. These holes are here by
many reasons and could be explaned at first by the full randomization of
system VA space. Now it is usually enabled in Linux kernels by default. There
are always gaps around the process stack area to trap overflows. Holes before
and after shared libraries could be explained by the fact, that on many
architectures libraries have a 'preferred' address to be loaded at; putting
them elsewhere requires relocation work, and probably some unshared pages.
Repetitive holes of 65380K are most probably correspond to the header that
malloc has to allocate before asked a claimed memory block. This header is
used by malloc to link allocated chunks together and for its internal book
keeping.
$ sudo pmap -x -p `pidof haproxy`
127136: ./haproxy -f /home/haproxy/haproxy/haproxy_h2.cfg
Address Kbytes RSS Dirty Mode Mapping
0000555555554000 388 64 0 r---- /home/haproxy/haproxy/haproxy
00005555555b5000 2608 1216 0 r-x-- /home/haproxy/haproxy/haproxy
0000555555841000 916 64 0 r---- /home/haproxy/haproxy/haproxy
0000555555926000 60 60 60 r---- /home/haproxy/haproxy/haproxy
0000555555935000 116 116 116 rw--- /home/haproxy/haproxy/haproxy
0000555555952000 7872 5236 5236 rw--- [ anon ]
00007fff98000000 156 36 36 rw--- [ anon ]
00007fff98027000 65380 0 0 ----- [ anon ]
00007fffa0000000 156 36 36 rw--- [ anon ]
00007fffa0027000 65380 0 0 ----- [ anon ]
00007fffa4000000 156 36 36 rw--- [ anon ]
00007fffa4027000 65380 0 0 ----- [ anon ]
00007fffa8000000 156 36 36 rw--- [ anon ]
00007fffa8027000 65380 0 0 ----- [ anon ]
00007fffac000000 156 36 36 rw--- [ anon ]
00007fffac027000 65380 0 0 ----- [ anon ]
00007fffb0000000 156 36 36 rw--- [ anon ]
00007fffb0027000 65380 0 0 ----- [ anon ]
...
00007ffff7fce000 4 4 0 r-x-- [ anon ]
00007ffff7fcf000 4 4 0 r---- /usr/lib/x86_64-linux-gnu/ld-2.31.so
00007ffff7fd0000 140 140 0 r-x-- /usr/lib/x86_64-linux-gnu/ld-2.31.so
...
00007ffff7ffe000 4 4 4 rw--- [ anon ]
00007ffffffde000 132 20 20 rw--- [ stack ]
ffffffffff600000 4 0 0 --x-- [ anon ]
---------------- ------- ------- -------
total kB 1499288 75504 72760
This exceeded VSZ makes impossible to start an haproxy process with 200M
memory limit, set at its initialization stage as RLIMIT_AS. We usually
have in this case such cryptic output at stderr:
$ haproxy -m 200 -f haproxy_quic.cfg
(null)(null)(null)(null)(null)(null)
At the same time the process RSS (a memory really used) is only 75,5M.
So to make process memory accounting more realistic let's base the memory
limit, set by -m option, on RSS measurement and let's use RLIMIT_DATA instead
of RLIMIT_AS.
RLIMIT_AS was used before, because earlier versions of haproxy always allocate
memory buffers for new connections, but data were not written there
immediately. So these buffers were not instantly counted in RSS, but were
always counted in VSZ. Now we allocate new buffers only in the case, when we
will write there some data immediately, so using RLIMIT_DATA becomes more
appropriate.
This patch implements the backend side of the keylog fetches.
The code was ready but needed the SSL message callbacks.
This could be used like this:
log-format "CLIENT_EARLY_TRAFFIC_SECRET %[ssl_bc_client_random,hex] %[ssl_bc_client_early_traffic_secret]\n
CLIENT_HANDSHAKE_TRAFFIC_SECRET %[ssl_bc_client_random,hex] %[ssl_bc_client_handshake_traffic_secret]\n
SERVER_HANDSHAKE_TRAFFIC_SECRET %[ssl_bc_client_random,hex] %[ssl_bc_server_handshake_traffic_secret]\n
CLIENT_TRAFFIC_SECRET_0 %[ssl_bc_client_random,hex] %[ssl_bc_client_traffic_secret_0]\n
SERVER_TRAFFIC_SECRET_0 %[ssl_bc_client_random,hex] %[ssl_bc_server_traffic_secret_0]\n
EXPORTER_SECRET %[ssl_bc_client_random,hex] %[ssl_bc_exporter_secret]\n
EARLY_EXPORTER_SECRET %[ssl_bc_client_random,hex] %[ssl_bc_early_exporter_secret]"
Add support for crt-base and key-base local keywords for the crt-store.
current_crtbase and current_keybase are filed with a copy of the global
keyword argument when a crt-store is declared, and updated with a new
path when the keywords are in the crt-store section.
The ckch_conf_kws[] array was updated with ¤t_crtbase and
¤t_keybase instead of the global_ssl ones so the parser can use
them.
The keyword must be used before any "load" line in a crt-store section.
Example:
crt-store web
crt-base /etc/ssl/certs/
key-base /etc/ssl/private/
load crt "site3.crt" alias "site3"
load crt "site4.crt" key "site4.key"
frontend in2
bind *:443 ssl crt "@web/site3" crt "@web/site4.crt"
This patch introduces named crt-store section. A named crt-store allows
to add a scope to the crt name.
For example, a crt named "foo.crt" in a crt-store named "web" will
result in a certificate called "@web/foo.crt".
The crt-store load line now allows to put an alias. This alias is used
as the key in the ckch_tree instead of the certificate. This way an
alias can be referenced in the configuration with the '@/' prefix.
This can only be define with a crt-store.
The global 'key-base' keyword allows to read the 'key' parameter of a
crt-store load line using a path prefix.
This is the equivalent of the 'crt-base' keyword but for 'key'.
It only applies on crt-store.
Released version 3.0-dev7 with the following main changes :
- BUG/MINOR: ssl: Wrong ocsp-update "incompatibility" error message
- BUG/MINOR: ssl: Detect more 'ocsp-update' incompatibilities
- MEDIUM: ssl: Add 'tune.ssl.ocsp-update.mode' global option
- REGTESTS: ssl: Add OCSP update compatibility tests
- REGTESTS: ssl: Add functional test for global ocsp-update option
- BUG/MINOR: server: reject enabled for dynamic server
- BUG/MINOR: server: fix persistence cookie for dynamic servers
- MINOR: server: allow cookie for dynamic servers
- REGTESTS: Fix script about OCSP update compatibility tests
- BUG/MINOR: cli: Report an error to user if command or payload is too big
- MINOR: sc_strm: Add generic version to perform sync receives and sends
- MEDIUM: stream: Use generic version to perform sync receives and sends
- MEDIUM: buf: Add b_getline() and b_getdelim() functions
- MEDIUM: applet: Handle applets with their own buffers in put functions
- MEDIUM: cli/applet: Stop to test opposite SC in I/O handler of CLI commands
- MINOR: applet: Always use applet API to set appctx flags
- BUG/MEDIUM: applet: State appctx have more data if its EOI/EOS/ERROR flag is set
- MAJOR: cli: Update the CLI applet to handle its own buffers
- MINOR: applet: Let's applets .snd_buf function deal with full input buffers
- MINOR: stconn: Add a connection flag to notify sending data are the last ones
- MAJOR: cli: Use a custom .snd_buf function to only copy the current command
- DOC: config: balance 'first' not usable in LOG mode
- BUG/MINOR: log/balance: detect if user tries to use unsupported algo
- MINOR: lbprm: implement true "sticky" balance algo
- MEDIUM: log/balance: leverage lbprm api for log load-balancing
- BUG/BUILD: debug: fix unused variable error
- MEDIUM: lb-chash: Deterministic node hashes based on server address
- BUG/MEDIUM: stick-tables: fix a small remaining race in expiration task
- REGTESTS: Do not use REQUIRE_VERSION for HAProxy 2.5+ (4)
- REGTESTS: Remove REQUIRE_VERSION=1.9 from all tests (2)
- CLEANUP: Reapply ist.cocci (3)
- CLEANUP: Reapply strcmp.cocci (2)
- CLEANUP: Reapply xalloc_cast.cocci
- CLEANUP: Reapply ha_free.cocci
- CI: vtest: show coredumps if any
- REGTESTS: ssl: disable ssl/ocsp_auto_update.vtc
- BUG/MINOR: backend: properly handle redispatch 0
- MINOR: quic: HyStart++ implementation (RFC 9406)
- BUG/MEDIUM: stconn: Don't forward shutdown to SE if iobuf is not empty
- BUG/MEDIUM: stick-table: use the update lock when reading tables from peers
- BUG/MAJOR: applet: fix a MIN vs MAX usage in appctx_raw_rcv_buf()
- OPTIM: peers: avoid the locking dance around peer_send_teach_process_msgs()
- BUILD: quic: 32 bits compilation issue (QUIC_MIN() usage)
- BUG/MEDIUM: server/lbprm: fix crash in _srv_set_inetaddr_port()
- MEDIUM: mworker: get rid of libsystemd
- BUILD: systemd: fix build error on non-systemd systems with USE_SYSTEMD=1
- BUG/MINOR: bwlim/config: fix missing '\n' after error messages
- MINOR: stick-tables: mark the seen stksess with a flag "seen"
- OPTIM: stick-tables: check the stksess without taking the read lock
- MAJOR: stktable: split the keys across multiple shards to reduce contention
- CI: extend Fedora Rawhide, add m32 mode
- BUG/MINOR: stick-tables: Missing stick-table key nullity check
- BUILD: systemd: enable USE_SYSTEMD by default with TARGET=linux-glibc
- MINOR: systemd: Include MONOTONIC_USEC field in RELOADING=1 message
- BUG/MINOR: proxy: fix logformat expression leak in use_backend rules
- MEDIUM: log: rename logformat var to logformat tag
- MINOR: log: expose logformat_tag struct
- MEDIUM: log: carry tag context in logformat node
- MEDIUM: tree-wide: add logformat expressions wrapper
- MINOR: proxy: add PR_FL_CHECKED flag
- MAJOR: log: implement proper postparsing for logformat expressions
- MEDIUM: log: add compiling logic to logformat expressions
- MEDIUM: proxy/log: leverage lf_expr API for logformat preparsing
- MINOR: guid: introduce global UID module
- MINOR: guid: restrict guid format
- MINOR: proxy: implement GUID support
- MINOR: server: implement GUID support
- MINOR: listener: implement GUID support
- DOC: configuration: grammar fixes for strict-sni
- BUG/MINOR: init: relax LSTCHK_NETADM checks for non root
- MEDIUM: capabilities: check process capabilities sets
- CLEANUP: global: remove LSTCHK_CAP_BIND
- BUG/MEDIUM: quic: don't blindly rely on unaligned accesses
Since the Linux capabilities support add-on (see the commit bd84387beb
("MEDIUM: capabilities: enable support for Linux capabilities")), we can also
check haproxy process effective and permitted capabilities sets, when it
starts and runs as non-root.
Like this, if needed network capabilities are presented only in the process
permitted set, we can get this information with capget and put them in the
process effective set via capset. To do this properly, let's introduce
prepare_caps_from_permitted_set().
First, it checks if binary effective set has CAP_NET_ADMIN or CAP_NET_RAW. If
there is a match, LSTCHK_NETADM is removed from global.last_checks list to
avoid warning, because in the initialization sequence some last configuration
checks are based on LSTCHK_NETADM flag and haproxy process euid may stay
unpriviledged.
If there are no CAP_NET_ADMIN and CAP_NET_RAW in the effective set, permitted
set will be checked and only capabilities given in 'setcap' keyword will be
promoted in the process effective set. LSTCHK_NETADM will be also removed in
this case by the same reason. In order to be transparent, we promote from
permitted set only capabilities given by user in 'setcap' keyword. So, if
caplist doesn't include CAP_NET_ADMIN or CAP_NET_RAW, LSTCHK_NETADM would not
be unset and warning about missing priviledges will be emitted at
initialization.
Need to call it before protocol_bind_all() to allow binding to priviledged
ports under non-root and 'setcap cap_net_bind_service' must be set in the
global section in this case.
This commit is similar with the two previous ones. Its purpose is to add
GUID support on listeners. Due to bind_conf and listeners configuration,
some specifities were required.
Its possible to define several listeners on a single bind line, for
example by specifying multiple addresses. As such, it's impossible to
support a "guid" keyword on a bind line. The problem is exacerbated by
the cloning of listeners when sharding is used.
To resolve this, a new keyword "guid-prefix" is defined for bind lines.
It allows to specify a string which will be used as a prefix for
automatically generated GUID for each listeners attached to a bind_conf.
Automatic GUID listeners generation is implemented via a new function
bind_generate_guid(). It is called on post-parsing, after
bind_complete_thread_setup(). For each listeners on a bind_conf, a new
GUID is generated with bind_conf prefix and the index of the listener
relative to other listeners in the bind_conf. This last value is stored
in a new bind_conf field named <guid_idx>. If a GUID cannot be inserted,
for example due to a non-unique value, an error is returned, startup is
interrupted with configuration rejected.
This commit is similar to previous one, except that it implements GUID
support for server instances. A guid_node field is inserted into server
structure. A new "guid" server keyword is defined.
Implement proxy identiciation through GUID. As such, a guid_node member
is inserted into proxy structure. A proxy keyword "guid" is defined to
allow user to fix its value.
This is a simple algorithm to replace the classic slow start phase of the
congestion control algorithms. It should reduce the high packet loss during
this step.
Implemented only for Cubic.
Motivation: When services are discovered through DNS resolution, the order in
which DNS records get resolved and assigned to servers is arbitrary. Therefore,
even though two HAProxy instances using chash balancing might agree that a
particular request should go to server3, it is likely the case that they have
assigned different IPs and ports to the server in that slot.
This patch adds a server option, "hash-key <key>" which can be set to "id" (the
existing behaviour, default), "addr", or "addr-port". By deriving the keys for
the chash tree nodes from a server's address and port we ensure that independent
HAProxy instances will agree on routing decisions. If an address is not known
then the key is derived from the server's puid as it was previously.
When adjusting a server's weight, we now check whether the server's hash has
changed. If it has, we have to remove all its nodes first, since the node keys
will also have to change.
log load-balancing implementation was not seamlessly integrated within
lbprm API. The consequence is that it could become harder to maintain
over time since it added some specific cases just for the log backend.
Moreover, it resulted in some code duplication since balance algorithms
that are common to logs and regular (tcp, http) backends were specifically
rewritten for log backends.
Thanks to the previous commit, we now have all the prerequisites to make
log load-balancing fully leverage lbprm logic. Thus in this patch we make
__do_send_log_backend() use existing lbprm algorithms, and we no longer
require log-specific lbprm initialization in cfgparse.c and in
postcheck_log_backend().
As a bonus, for log backends this allows weighed algorithms to properly
support weights (ie: roundrobin, random and log-hash) since we now
leverage the same lb algorithms that we use for tcp/http backends
(doc was updated).
As previously mentioned in cd352c0db ("MINOR: log/balance: rename
"log-sticky" to "sticky""), let's define a sticky algorithm that may be
used from any protocol. Sticky algorithm sticks on the same server as
long as it remains available.
The documentation was updated accordingly.
b61147fd ("MEDIUM: log/balance: merge tcp/http algo with log ones")
introduced an ambiguity because 'first' algorithm is not usable in
LOG mode but it was not specified in the doc.
This should be backported in 2.9 with b61147fd.
This commit allows "cookie" keyword for dynamic servers. After code
review, nothing was found which could prevent a dynamic server to use
it. An extra warning is added under cli_parse_add_server() if cookie
value is ignored due to a non HTTP backend.
This patch is not considered a bugfix. However, it may backported if
needed as its impact seems minimal.
Since their first implementation, dynamic servers are created into
maintenance state. This has been done purposely to avoid immediate
activation of a newly inserted server.
However, this principle is incompatible if "enabled" keyword is used on
"add server". The newly created instance will be unreacheable as proxy
load-balancing algorithm is not informed of its presence via
srv_lb_propagate(). The new server could be unblocked by toggling its
state with "disable server" / "enable server" commands, which will
trigger srv_lb_propagate() invocation.
To avoid this unexpected state, simply forbid "enabled" keyword for
dynamic servers. In the long-term, it could be possible to re authorize
it but at least this requires to call srv_lb_propagate() on dynamic
server creation.
This should fix github issue #2497.
This patch should not be backported as-is, to avoid breaking dynamic
servers API on stable versions. "enabled" should instead be ignored for
them. This will be implemented in a dedicated patch on top of 2.9.
This option can be used to set a default ocsp-update mode for all
certificates of a given conf file. It allows to activate ocsp-update on
certificates without the need to create separate crt-lists. It can still
be superseded by the crt-list 'ocsp-update' option. It takes either "on"
or "off" as value and defaults to "off".
Since setting this new parameter to "on" would mean that we try to
enable ocsp-update on any certificate, and also certificates that don't
have an OCSP URI, the checks performed in ssl_sock_load_ocsp were
softened. We don't systematically raise an error when trying to enable
ocsp-update on a certificate that does not have an OCSP URI, be it via
the global option or the crt-list one. We will still raise an error when
a user tries to load a certificate that does have an OCSP URI but a
missing issuer certificate (if ocsp-update is enabled).
Released version 3.0-dev6 with the following main changes :
- MINOR: mux-h2: always use h2c_report_glitch()
- MEDIUM: mux-h2: allow to set the glitches threshold to kill a connection
- MINOR: quic: simplify rescheduling for handshake
- MINOR: quic: remove qc_treat_rx_crypto_frms()
- DOC: configuration: clarify ciphersuites usage (V2)
- MINOR: tools: use public interface for FreeBSD get_exec_path()
- BUG/MINOR: ssl: fix possible ctx memory leak in sample_conv_aes_gcm()
- BUG/MINOR: ssl: do not set the aead_tag flags in sample_conv_aes_gcm()
- BUG/MINOR: server: fix first server template not being indexed
- MEDIUM: ssl: initialize the SSL stack explicitely
- MEDIUM: ssl: allow to change the OpenSSL security level from global section
- CLEANUP: ssl: remove useless #ifdef in openssl-compat.h
- CI: github: add -DDEBUG_LIST to the default builds
- BUG/MINOR: hlua: segfault when loading the same filter from different contexts
- BUG/MINOR: hlua: missing lock in hlua_filter_new()
- BUG/MINOR: hlua: fix missing lock in hlua_filter_delete()
- DEBUG: lua: precisely identify if stream is stuck inside lua or not
- MINOR: hlua: use accessors for stream hlua ctx
- BUG/MEDIUM: hlua: streams don't support mixing lua-load with lua-load-per-thread (2nd try)
- MINOR: debug: enable insecure fork on the command line
- CI: github: add -dI to haproxy arguments
- BUG/MINOR: listener: Wake proxy's mngmt task up if necessary on session release
- BUG/MINOR: listener: Don't schedule frontend without task in listener_release()
- MINOR: session: rename private conns elements
- BUG/MAJOR: server: do not delete srv referenced by session
- BUG/MEDIUM: spoe: Don't rely on stream's expiration to detect processing timeout
- BUG/MINOR: spoe: Be sure to be able to quickly close IDLE applets on soft-stop
- MAJOR: spoe: Deprecate the SPOE filter
- MINOR: cfgparse: Add a global option to expose deprecated directives
- MINOR: spoe: Add SPOE filters in the exposed deprecated directives
- CLEANUP: assorted typo fixes in the code and comments
- CI: temporarily adjust kernel entropy to work with ASAN/clang
- BUG/MEDIUM: spoe: Return an invalid frame on recv if size is too small
- BUG/MINOR: session: ensure conn owner is set after insert into session
- BUG/MEDIUM: http_ana: ignore NTLM for reuse aggressive/always and no H1
- BUG/MAJOR: connection: fix server used_conns with H2 + reuse safe
- BUG/MAJOR: ocsp: Separate refcount per instance and per store
- REGTESTS: ssl: Add OCSP related tests
- BUG/MEDIUM: ssl: Fix crash when calling "update ssl ocsp-response" when an update is ongoing
- BUG/MEDIUM: ssl: Fix crash in ocsp-update log function
- MEDIUM: ssl: Change output of ocsp-update log
- MINOR: ssl: Change level of ocsp-update logs
- CLEANUP: ssl: Remove undocumented ocsp fetches
- REGTESTS: ssl: Add checks on ocsp-update log format
- MINOR: connection: implement conn_release()
- MINOR: connection: extend takeover with release option
- MEDIUM: server: close idle conn on server deletion
- MEDIUM: mux: prepare for takeover on private connections
- MEDIUM: server: close private idle connection before server deletion
- BUG/MINOR: mux-quic: close all QCS before freeing QCC tasklet
- BUG/MEDIUM: mux-fcgi: Properly handle EOM flag on end-of-trailers HTX block
- BUILD: server: fix build regression on old compilers (<= gcc-4.4)
- OPTIM: http_ext: avoid useless copy in http_7239_extract_{ipv4,ipv6}
- MINOR: debug: add "debug dev trace" to flood with traces
- MINOR: atomic: add a read-specific variant of __ha_cpu_relax()
- MINOR: applet: add new function applet_append_line()
- MINOR: log/applet: add new function syslog_applet_append_event()
- MEDIUM: ring/sink: use applet_append_line()/syslog_applet_append_event() for readers
- REORG: dns/ring: split the ring between the generic one and the DNS one
- MEDIUM: ring: move the ring reader code to ring_dispatch_messages()
- MEDIUM: sink: move the generic ring forwarder code use ring_dispatch_messages()
- MEDIUM: log/sink: make the log forwarder code use ring_dispatch_messages()
- MINOR: buf: add b_add_ofs() to add a count to an absolute position
- MINOR: buf: add b_rel_ofs() to turn an absolute offset into a relative one
- MINOR: buf: add b_putblk_ofs() to copy a block at a specific position
- MINOR: buf: add b_getblk_ofs() that works relative to area and not head
- MINOR: ring: make the ring reader use only absolute offsets
- MINOR: ring: reserve one special value for the readers count
- MINOR: vecpair: add new vector pair based data manipulation mechanisms
- MINOR: vecpair: add necessary functions to use vecpairss from/to ring APIs
- MINOR: ring: rename totlen vs msglen in ring_write()
- MINOR: ring: add ring_data() to report the amount of data in a ring
- MINOR: ring: add ring_size() to return the ring's size
- MINOR: ring: add ring_dup() to copy a ring into another one
- MINOR: ring: also add ring_area(), ring_head(), ring_tail()
- MINOR: ring: make callers use ring_data() and ring_size(), not ring->buf
- MINOR: errors: use ring_dup() to duplicate the startup_logs
- MINOR: ring: use ring_size(), ring_area(), ring_head() and ring_tail()
- MINOR: ring: add a flag to indicate a mapped file
- MAJOR: ring: insert an intermediary ring_storage level
- MINOR: ring: resize only under thread isolation
- MINOR: ring: allow to reduce a ring size
- MEDIUM: ring: replace the buffer API in ring_write() with the vec<->ring API
- MEDIUM: ring: change the ring reader to use the new vector-based API now
- MEDIUM: ring: remove the struct buffer from the ring
- MEDIUM: ring: align the head and tail fields in the ring_storage structure
- MINOR: ring: make the reader check the readers count before inc/dec
- MEDIUM: ring: lock the tail's readers counters before proceeding with the changes
- MEDIUM: ring: protect the reader's positions against writers
- MEDIUM: ring: use the topmost bit of the tail as a lock
- MEDIUM: move the ring's lock to only protect the readers list
- MEDIUM: ring: unlock the ring's tail earlier
- MINOR: ring: don't take the readers lock if there are no readers
- MEDIUM: ring/applet: turn the wait_entry list to an mt_list instead
- MEDIUM: ring: protect the initialization of the initial reader offset
- MINOR: ring: make sure ring_dispatch waits when facing a changing message
- MAJOR: ring: drop the now unneeded lock
- OPTIM: ring: don't even try to update offset when failed to read
- OPTIM: ring: have only one thread at a time wake up all readers
- MINOR: ring: keep a few frequently used pointers in the local stack
- MINOR: ring: add the definition of a ring waiting cell
- MINOR: ring: make the number of queues configurable
- MAJOR: ring: implement a waiting queue in front of the ring
- MEDIUM: ring: significant boost in the loop by checking the ring queue ptr first
- MEDIUM: ring: improve speed in the queue waiting loop on x86_64
- MINOR: ring: simplify the write loop a little bit
- CLEANUP: ring: further simplify the write loop
- MINOR: ring: it's not x86 but all non-ARMv8.1 which needs the read before OR
- MINOR: ring: avoid writes to cells during copy
- OPTIM: ring: use relaxed stores to release the threads
- CLEANUP: ring: use only curr_cell and not next_cell in the main write loop
- BUILD: ssl: fix build error on older compilers with openssl-3.2
- BUG/MINOR: server: 'source' interface ignored from 'default-server' directive
- BUG/MAJOR: ring: free the ring storage not the ring itself when using maps
Now the rings have one wait queue per group. This should limit the
contention on systems such as EPYC CPUs where the performance drops
dramatically when using more than one CCX.
Tests were run with different numbers and it was showed that value
6 outperforms all other ones at 12, 24, 48, 64 and 80 threads on an
EPYC, a Xeon and an Ampere CPU. Value 7 sometimes comes close and
anything around these values degrades quickly. The value has been
left tunable in the global section.
This commit only introduces everything needed to set up the queue count
so that it's easier to adjust it in the forthcoming patches, but it was
initially added after the series, making it harder to compare.
It was also shown that trying to group the threads in queues by their
thread groups is counter-productive and that it was more efficient to
do that by applying a modulo on the thread number. As surprising as it
seems, it does have the benefit of well balancing any number of threads.
Since commit "BUG/MEDIUM: ssl: Fix crash in ocsp-update log function",
some information from the log line are "faked" because they can be
actually retrieved anymore (or never could). We should then remove them
from the logline all along instead of providing some useless fields.
We then only keep pure OCSP-update information in the log line:
"<certname> <status> <status str> <fail count> <success count>"
Backend connections can be marked as private to prevent their sharing by
multiple clients. Now, this has become an exception as only two reasons
for data traffic can trigger this (checks are ignored here) :
* http-reuse never
* HTTP response with NTLM header
The first case is easy to manage as the connection is flagged as private
since its inception. However, the second case is dynamic as the
connection can be flagged anytime during its lifetime. When using a
backend protocol such as HTTP/2 with reuse mode aggressive or always, we
face a design issue as the connection would be marked as private,
despite potentially being shared by several clients at the same time.
This is conceptually invalid, but worst it can trigger crashes on MUX
stream detach callback depending on the order of release of the streams,
by calling session_check_idle_conn() with a NULL session. It could also
be possible to have several NTLM responses on a single connection for
different sessions. In this case, connection owner is still being
updated without attaching the connection to its correct session, which
ultimately would cause a crash on session_check_idle_conn with an
invalid session.
Here are two backtrace examples from GDB for such cases :
Thread 1 (Thread 0x7ff73e9fc700 (LWP 648859)):
#0 session_check_idle_conn (conn=0x7ff72f597800, sess=0x0) at include/haproxy/session.h:209
#1 h2_detach (sd=<optimized out>) at src/mux_h2.c:4520
#2 0x000056151742be24 in sc_detach_endp (scp=scp@entry=0x7ff73e9f0f18) at src/stconn.c:376
#3 0x000056151742c208 in sc_destroy (sc=<optimized out>) at src/stconn.c:444
#4 0x0000561517370871 in stream_free (s=s@entry=0x7ff72a2dbd80) at src/stream.c:728
#5 0x000056151737541f in process_stream (t=t@entry=0x7ff72d5e2620, context=0x7ff72a2dbd80, state=<optimized out>) at src/stream.c:2645
#6 0x0000561517456cbb in run_tasks_from_lists (budgets=budgets@entry=0x7ff73e9f10d0) at src/task.c:632
#7 0x00005615174576b9 in process_runnable_tasks () at src/task.c:876
#8 0x000056151742275a in run_poll_loop () at src/haproxy.c:2996
#9 0x0000561517422db1 in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3195
#10 0x00007ff789e081ca in start_thread () from /lib64/libpthread.so.0
#11 0x00007ff789a39e73 in clone () from /lib64/libc.so.6
(gdb)
Thread 1 (Thread 0x7ff52e7fc700 (LWP 681458)):
#0 0x0000556ebd6e7e69 in session_check_idle_conn (conn=0x7ff5787ff100, sess=0x7ff51d2539a0) at include/haproxy/session.h:209
#1 h2_detach (sd=<optimized out>) at src/mux_h2.c:4520
#2 0x0000556ebd7f3e24 in sc_detach_endp (scp=scp@entry=0x7ff52e7f0f18) at src/stconn.c:376
#3 0x0000556ebd7f4208 in sc_destroy (sc=<optimized out>) at src/stconn.c:444
#4 0x0000556ebd738871 in stream_free (s=s@entry=0x7ff520e28200) at src/stream.c:728
#5 0x0000556ebd73d41f in process_stream (t=t@entry=0x7ff565783700, context=0x7ff520e28200, state=<optimized out>) at src/stream.c:2645
#6 0x0000556ebd81ecbb in run_tasks_from_lists (budgets=budgets@entry=0x7ff52e7f10d0) at src/task.c:632
#7 0x0000556ebd81f6b9 in process_runnable_tasks () at src/task.c:876
#8 0x0000556ebd7ea75a in run_poll_loop () at src/haproxy.c:2996
#9 0x0000556ebd7eadb1 in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3195
#10 0x00007ff5752081ca in start_thread () from /lib64/libpthread.so.0
#11 0x00007ff574e39e73 in clone () from /lib64/libc.so.6
(gdb)
To solve this issue, simply ignore NTLM responses when using a
multiplexer with streams support and the connection is not already
attached to the session. The connection is not marked as private and
will continue to be shared freely accross clients. This is considered
conceptually valid as NTLM usage (rfc 4559) with HTTP is broken and was
designed only with HTTP/1.1 in mind. A side-effect of the change is that
SESS_FL_PREFER_LAST is also not set anymore on NTLM detection, which
allows following requests to be load-balanced accross several server
instances.
The original behavior is kept for HTTP/1 or if the connection is already
attached to the session. This last case happens when using HTTP/2 with
default http-reuse safe mode since the following patch :
0d21deaded
MEDIUM: backend: add reused conn to sess if mux marked as HOL blocking
This should be backported up to all stable releases. Up until 2.4, it
can be taken as-is. For lesser versions, above patch is not present. In
this case the condition should be restricted only to HTTP/1 usage :
if (srv_conn && strcmp(srv_conn->mux->name, "H1") == 0) {
It is the first deprecated directive exposed via the
'expose-deprecated-directives' global option. This way, it is possible to
silent the warning about the SPOE uses.
Similarly to "expose-exprimental-directives" option, there is no a global
option to expose some deprecated directives. Idea is to have a way to silent
warnings about deprecated directives when there is no alternative solution.
Of course, deprecated directives covered by this option are not listed and
may change. It is only a best effort to let users upgrade smoothly.
As announced on the ML few weeks (months ?) ago and on several GH issues,
the SPOE is now deprecated. Sadly, this filter should be refactored to work
properly. It was implemented as a functionnal PoC for the 1.7 and since
then, no time was invest to improve it and make it truly maintainable in
time. Worst, other parts of HAProxy evolve, especially applets part, making
maintenance ever more expensive.
Instead of keeping the SPOE filter in a this state and always reply to users
encountering issues or limitations that it is far from perfect but we cannot
work on it for now, we decided to deprecate it.
We can still change our mind before the 3.0.0 release if the situation
evolves. Otherwise the filter will be removed or marked as unmaintained for
the 3.1. If the situation does not change, it means the 3.0 will be the last
version with a true SPOE support.
-dI allow to enable "insure-fork-wanted" directly from the command line,
which is useful when you want to run ASAN with addr2line with a lot of
configuration files without editing them.
The new "ssl-security-level" option allows one to change the OpenSSL
security level without having to change the openssl.cnf global file of
your distribution. This directives applies on every SSL_CTX context.
People sometimes change their security level directly in the ciphers
directive, however there are some cases when the security level change
is not applied in the right order (for example when applying a DH
param).
Before this patch, it was to possible to trick by using a specific
openssl.cnf file and start haproxy this way:
OPENSSL_CONF=./openssl.cnf ./haproxy -f bug-2468.cfg
Values for the security level can be found there:
https://www.openssl.org/docs/man1.1.1/man3/SSL_CTX_set_security_level.html
This was discussed in github issue #2468.
The previous attempt removed the TLSv1.3 version for the
"ciphersuites" keywords. However it looks like the TLSv1.2 support for
SSL_CTX_set_ciphersuites() is a bug, and can have inconsistent behavior.
This patch revert the previous attempt and add explaining about this
problem and clear examples on how to configure TLSv1.2 ciphers + TLSv1.3
ciphersuites.
Revert "DOC: configuration: clarify ciphersuites usage"
This reverts commit e2a44d6c94.
This must be backported to all stable branches.
Fixes issue #2459.
Till now it was still needed to write rules to eliminate bad behaving
H2 clients, while most of the time it would be desirable to just be able
to set a threshold on the level of anomalies on a connection.
This is what this patch does. By setting a glitches threshold for frontend
and backend, it allows to automatically turn a connection to the error
state when the threshold is reached so that the connection dies by itself
without having to write possibly complex rules.
One subtlety is that we still have the error state being exclusive to the
parser's state so this requires the h2c_report_glitches() function to return
a status indicating if the threshold was reached or not so that processing
can instantly stop and bypass the state update, otherwise the state could
be turned back to a valid one (e.g. after parsing CONTINUATION); we should
really contemplate the possibility to use H2_CF_ERROR for this. Fortunately
there were very few places where a glitch was reported outside of an error
path so the changes are quite minor.
Now by setting the front value to 1000, a client flooding with short
CONTINUATION frames is instantly stopped.
Released version 3.0-dev5 with the following main changes :
- BUG/MEDIUM: applet: Fix HTX .rcv_buf callback function to release outbuf buffer
- BUG/MAJOR: ssl/ocsp: crash with ocsp when old process exit or using ocsp CLI
- BUG/MEDIUM: server: fix dynamic servers initial settings
- BUG/MINOR: ssl/cli: duplicate cleaning code in cli_parse_del_crtlist
- LICENSE: event_hdl: fix GPL license version
- LICENSE: http_ext: fix GPL license version
- BUG/MEDIUM: mux-h1: Fix again 0-copy forwarding of chunks with an unknown size
- BUG/MINOR: mux-h1: Properly report when mux is blocked during a nego
- MINOR: mux-h1: Move checks performed before a shutdown in a dedicated function
- MINOR: mux-h1: Move all stuff to detach a stream in an internal function
- MAJOR: mux-h1: Drain requests on client side before shut a stream down
- MEDIUM: htx/http-ana: No longer close connection on early HAProxy response
- MINOR: quic: filter show quic by address
- MINOR: quic: specify show quic output fields
- MINOR: quic: add MUX output for show quic
- CLEANUP: mux-h2: Fix h2s_make_data() comment about the return value
- DOC: configuration: clarify ciphersuites usage
- BUG/MINOR: config/quic: Alert about PROXY protocol use on a QUIC listener
- BUG/MINOR: hlua: Fix log level to the right value when set via TXN:set_loglevel
- MINOR: hlua: Be able to disable logging from lua
- BUG/MINOR: tools: seed the statistical PRNG slightly better
- BUG/MINOR: hlua: fix unsafe lua_tostring() usage with empty stack
- BUG/MINOR: hlua: don't use lua_tostring() from unprotected contexts
- BUG/MINOR: hlua: fix possible crash in hlua_filter_new() under load
- BUG/MINOR: hlua: improper lock usage in hlua_filter_callback()
- BUG/MINOR: hlua: improper lock usage in hlua_filter_new()
- BUG/MEDIUM: hlua: improper lock usage with SET_SAFE_LJMP()
- BUG/MAJOR: hlua: improper lock usage with hlua_ctx_resume()
- BUG/MINOR: hlua: don't call ha_alert() in hlua_event_subscribe()
- MINOR: hlua: use SEND_ERR to report errors in hlua_event_runner()
- CLEANUP: hlua: txn class functions may LJMP
- BUG/MINOR: sink: fix a race condition in the TCP log forwarding code
- BUILD: thread: move lock label definitions to thread-t.h
- BUILD: tree-wide: fix a few missing includes in a few files
- BUILD: buf: make b_ncat() take a const for the source
- CLEANUP: assorted typo fixes in the code and comments
- CLEANUP: fix typo in naming for variable "unused"
- CI: run more smoke tests on config syntax to check memory related issues
- CI: enable monthly build only test on netbsd-9.3
- CI: skip scheduled builds on forks
- BUG/MINOR: ssl/cli: typo in new ssl crl-file CLI description
- BUG/MEDIUM: quic: fix connection freeze on post handshake
- BUG/MINOR: mux-quic: fix crash on aborting uni remote stream
- CLEANUP: log: fix obsolete comment for add_sample_to_logformat_list()
- CLEANUP: tree-wide: use proper ERR_* return values for PRE_CHECK fcts
- BUG/MINOR: cfgparse: report proper location for log-format-sd errors
- MINOR: vars: export var_set and var_unset functions
- MINOR: Add aes_gcm_enc converter
- BUG/MEDIUM: quic: fix handshake freeze under high traffic
- MINOR: quic: always use ncbuf for rx CRYPTO
- BUILD: ssl: define EVP_CTRL_AEAD_GET_TAG for older versions
- DOC: design: write first notes about ring-v2
- OPTIM: sink: try to merge "dropped" messages faster
- OPTIM: sink: drop the sink lock used to count drops
- DEV: haring: make haring not depend on the struct ring itself
- DEV: haring: split the code between ring and buffer
- DEV: haring: automatically use the advertised ring header size
- BUILD: solaris: fix compilation errors
The converter can be used to encrypt the raw byte input using the
AES-GCM algorithm, using provided nonce and key.
Co-authored-by: Dragan Dosen (ddosen@haproxy.com)
This is 39th iteration of typo fixes
The naming issue on the argument called "unsued" instead of "unused"
in two functions from resolvers and stick-tables was put into a second
patch so that it can be omitted if it were to cause backport issues.
Add core.silent (-1) value to be able to disable logging via
TXN:set_loglevel() call. Otherwise, there is no way to do so and it may be
handy. This special value cannot be used with TXN:log() function.
This patch may be backported if necessary.
Ciphersuites can be used with any TLS/SSL protocol version and are not
specific to TLSv1.3. However you can only specify the TLSv1.3 ciphers in
ciphersuite format.
Should fix issue #2459.
Backport to every stable branches.
Extend "show quic" to be able to dump MUX related information. This is
done via the new function qcc_show_quic(). This replaces the old streams
dumping list which was incomplete.
These info are displayed on full output or by specifying "mux" field.
Add the possibility to customize show quic full output with only a
specific set of printed fields. This is specified as a comma-separated
list. Here are the currently supported values :
* tp: transport parameters
* sock: connection addresses and socket FD
* pktns: packet number space with ack ranges and in flight bytes
* cc: congestion controler and loss information
Note that streams output is not filtered by this mechanism. It's because
it will be replaced soon by an output generated from the MUX which will
use its owned field name.
Add the possibilty to restrict show quic output to only a single
connection. This is done by specifying a quic_conn address pointer.
Default format selection has evolved with it. Indeed, it seems more
fitting to use full format by default when filtering on a connection.
However, it's still possible to revert to the original oneline format
with it by specifying it explicitely.
Released version 3.0-dev4 with the following main changes :
- BUG/MEDIUM: ssl: Fix crash when calling "update ssl ocsp-response" when an update is ongoing
- BUG/MEDIUM: quic: Wrong K CUBIC calculation.
- MINOR: quic: Update K CUBIC calculation (RFC 9438)
- MINOR: quic: Dynamic packet reordering threshold
- MINOR: quic: Add a counter for reordered packets
- BUG/MAJOR: mux-h1: Fix zero-copy forwarding when sending chunks of unknown size
- MINOR: stats: Use a dedicated function to check if output is almost full
- BUG/MEDIUM: applet: Add a flag to state an applet is using zero-copy forwarding
- BUG/MEDIUM: stconn/applet: Block 0-copy forwarding if producer needs more room
- MINOR: applet: Remove uselelss test on SE_FL_SHR/SHW flags
- MEDIUM: applet: Add notion of shutdown for write for applets
- MINOR: cli: No longer check SC for shutdown to interrupt wait command
- BUG/MEDIUM: stconn: Allow expiration update when READ/WRITE event is pending
- BUG/MEDIUM: stconn: Don't check pending shutdown to wake an applet up
- CLEANUP: stconn: Move SE flags set by app layer at the end of the bitfield
- MINOR: stconn: Rename SE_FL_MAY_FASTFWD and reorder bitfield
- MINOR: stconn: Add SE flag to announce zero-copy forwarding on consumer side
- MINOR: muxes: Announce support for zero-copy forwarding on consumer side
- BUG/MAJOR: stconn: Check support for zero-copy forwarding on both sides
- MINOR: muxes/applet: Simplify checks on options to disable zero-copy forwarding
- BUG/MINOR: quic: reject unknown frame type
- MINOR: quic: handle all frame types on reception
- BUG/MINOR: quic: reject HANDSHAKE_DONE as server
- BUG/MINOR: qpack: reject invalid increment count decoding
- BUG/MINOR: qpack: reject invalid dynamic table capacity
- DOC/MINOR: userlists: mention solutions to high cpu with hashes
- DOC: quic: Missing tuning setting in "Global parameters"
- BUG/MEDIUM: applet: Immediately free appctx on early error
- BUG/MEDIUM: hlua: Be able to garbage collect uninitialized lua sockets
- BUG/MEDIUM: hlua: Don't loop if a lua socket does not consume received data
- BUG/MEDIUM: quic: fix transient send error with listener socket
- MINOR: log: custom name for logformat node
- MINOR: sample: add type_to_smp() helper function
- MINOR: log: explicit typecasting for logformat nodes
- MINOR: log: simplify last_isspace in sess_build_logline()
- MINOR: log: simplify quotes handling in sess_build_logline()
- MINOR: log: print metadata prefixes separately in sess_build_logline()
- MINOR: log: automate string array construction in sess_build_logline()
- DOC: quic: fix recommandation for bind on multiple address
- MINOR: quic: warn on bind on multiple addresses if no IP_PKTINFO support
- OPTIM: quic: improve slightly qc_snd_buf() internal
- MINOR: quic: move IP_PKTINFO on send on a dedicated function
- MINOR: quic: remove sendto() usage variant
- MINOR: quic: only use sendmsg() syscall variant
- BUILD: applet: fix build on some 32-bit archs
- BUG/MINOR: quic: initialize msg_flags before sendmsg
- BUG/MEDIUM: mux-h1: Don't emit 0-CRLF chunk in h1_done_ff() when iobuf is empty
- CLEANUP: proxy/log: remove unused proxy flag
- CLEANUP: log: fix process_send_log() indentation
- CLEANUP: log: use free_logformat_list() in parse_logformat_string()
- MINOR: log: add free_logformat_node() helper function
- BUG/MINOR: log: fix potential lf->name memory leak
- BUG/MINOR: ist: allocate nul byte on istdup
- BUG/MINOR: stats: drop srv refcount on early release
- BUG/MAJOR: promex: fix crash on deleted server
- BUG/MAJOR: server: fix stream crash due to deleted server
- BUG/MEDIUM: mux-quic: do not crash on qcs_destroy for connection error
- MINOR: cli: Remove useless loop on commands to find unescaped semi-colon
- BUG/MEDIUM: cli: Warn if pipelined commands are delimited by a \n
- BUG/MAJOR: cli: Restore non-interactive mode behavior with pipelined commands
- BUG/MINOR: quic: fix output of show quic
- MINOR: ssl: Call callback function after loading SSL CRL data
- BUG/MINOR: ist: only store NUL byte on succeeded alloc
Documentation falsely mentions that binding on multiple addresses is
forbidden for QUIC listeners. This is not the case. Moreover, this
behavior is reliable when using destination address retrieval on receive
via IP_PKTINFO, which allows to determine the proper source address for
response.
This should be backported up to 2.7. On 2.6 specific source address
definition on sendmsg via IP_PKTINFO is not implemented. As such, bind
on multiple addresses should remain forbidden for this release.
Should have come with this previous commit:
MINOR: quic: Add a counter for reordered packets
Must be backported where the previous commit was backported.
This change adds a paragraph to the documentation regarding "userlists"
and the use of hashed password value.
It indicates what a user can do to address the high CPU cost of
having to calculate the hash at each request, such as reducing the
number of rounds or the cost complexity, if the algorithm allows for it.
I believe it is necessary to mention how the musl C library
impacts performance of hashing functions, as this has already led to a
few issues:
https://github.com/haproxy/haproxy/issues/1298https://github.com/haproxy/haproxy/issues/2008https://github.com/haproxy/haproxy/issues/2251
The performance impact is significant enough to mention it.
Acked-by: Lukas Tribus <lukas@ltri.eu>
Let's say that the largest packet number acknowledged by the peer is #10, when inspecting
the non already acknowledged packets to detect if they are lost or not, this is the
case a least if the difference between this largest packet number and and their
packet numbers are bigger or equal to the packet reordering threshold as defined
by the RFC 9002. This latter must not be less than QUIC_LOSS_PACKET_THRESHOLD(3).
Which such a value, packets #7 and oldest are detected as lost if non acknowledged,
contrary to packet number #8 or #9.
So, the packet loss detection is very sensitive to such a network characteristic
where non acknowledged packets are distant from each others by their packet number
differences.
Do not use this static value anymore for the packet reordering threshold which is used
as a criteria to detect packet loss. In place, make it depend on the difference
between the number of the last transmitted packet and the number of the oldest
one among the packet which are still in flight before being inspected to be
deemed as lost.
Add new tune.quic.reorder-ratio setting to apply a ratio in percent to this
dynamic packet reorder threshold.
Should be backported to 2.6.
Released version 3.0-dev3 with the following main changes :
- DOC: configuration: clarify http-request wait-for-body
- BUG/MAJOR: ssl_sock: Always clear retry flags in read/write functions
- MINOR: h3: add traces for stream sending function
- BUG/MEDIUM: h3: do not crash on invalid response status code
- BUG/MEDIUM: qpack: allow 6xx..9xx status codes
- BUG/MEDIUM: quic: fix crash on invalid qc_stream_buf_free() BUG_ON
- CLEANUP: log: deinitialization of the log buffer in one function
- BUG/MINOR: h1: Don't support LF only at the end of chunks
- BUG/MEDIUM: h1: Don't support LF only to mark the end of a chunk size
- MINOR: ssl: add HAVE_SSL_0RTT constant
- MINOR: ssl: rename HA_OPENSSL_HAVE_0RTT_SUPPORT constant to HAVE_SSL_0RTT_QUIC
- MEDIUM: ssl/quic: always compile the ssl_conf.early_data test
- DOC: httpclient: add dedicated httpclient section
- BUG/MINOR: h1-htx: properly initialize the err_pos field
- BUG/MEDIUM: h1: always reject the NUL character in header values
- CLEANUP: h1: remove unused function h1_measure_trailers()
- BUG/MINOR: ssl/quic: fix 0RTT define
- MINOR: mux-quic: prepare for earlier flow control update
- MINOR: mux-quic: define a flow control related type
- MEDIUM: mux-quic: limit stream flow control on snd_buf
- MEDIUM: mux-quic: limit conn flow control on snd_buf
- MINOR: mux-quic: remove unneeded sent-offset fields
- MINOR: mux-quic: check fctl during STREAM frame build
- MAJOR: mux-quic: remove intermediary Tx buffer
- MEDIUM: mux-quic: simplify sending API
- MEDIUM: mux-quic: release Tx buf on too small room
- MEDIUM: mux-quic: properly handle conn Tx buf exhaustion
- MINOR: mux-quic: realign Tx buffer if possible
- CLEANUP: connection: remove obsolete comment in header file
- OPTIM: connection: progressive hash for conn_calculate_hash()
- MINOR: tcp_act: fix alphabetical ordering of tcp request content actions
- MINOR: tcp-act: Rename "set-{mark,tos}" to "set-fc-{mark,tos}"
- MINOR: hlua: Rename set_{tos, mark} to set_fc_{tos, mark}
- MEDIUM: tcp-act: <expr> support for set-fc-{mark,tos} actions
- MEDIUM: tcp-act/backend: support for set-bc-{mark,tos} actions
- MINOR: stats: Be able to access to registered stats modules from anywhere
- MEDIUM: stats: Be able to access a specific field into a stats module
- MINOR: promex: Add a param to override the description when a metric is dumped
- MINOR: promex: Add info in the promex context to dump extra counters
- MEDIUM: promex: Dump frontends extra counters if requested
- MEDIUM: promex: Dump backends extra counters if requested
- MEDIUM: promex: Dump servers extra counters if requested
- MEDIUM: promex: Dump listeners extra counters if requested
- DOC: promex: Add documentation about extra-counters
- MINOR: promex: Always limit the number of labels dumped for each metric
- MEDIUM: promex: Simplify the context using generic pointers for restart points
- MINOR: promex: Remove unsued htx parameter when a metric is dumped
- MEDIUM: promex: Add a registration mechanism to support modules
- MEDIUM: promex: Dump metrics of registered modules with a way to filter them
- MEDIUM: promex/stick-table: Dump stick-table metrics via a promex module
- MEDIUM: promex/resolvers: Dump resolvers metrics via a promex module
- MINOR: promex: Rename dump functions to use the right wording
- MINOR: promex: Always pass the final name and description to promex_dmp_ts()
- MEDIUM: promex: Add support for filters on metric names
- REGTESTS: promex: Adapt script to be less verbose
- MINOR: compiler: add a new DO_NOT_FOLD() macro to prevent code folding
- MINOR: debug: make sure calls to ha_crash_now() are never merged
- MINOR: debug: make ABORT_NOW() store the caller's line number when using abort
- BUG/MINOR: diag: always show the version before dumping a diag warning
- BUG/MINOR: diag: run the final diags before quitting when using -c
- MINOR: acl: add extra diagnostics about suspicious string patterns
- BUG/MINOR: quic: Wrong ack ranges handling when reaching the limit.
- BUILD: quic: Variable name typo inside a BUG_ON().
- DOC: config: fix typo for '%ms' log format alternative
- DOC: config: fix ordering for "txn.*" fetches
- MINOR: stream: add "txn.redispatch" fetch
- BUILD: debug: remove leftover parentheses in ABORT_NOW()
- MINOR: debug: make BUG_ON() catch build errors even without DEBUG_STRICT
- BUG/MINOR: ssl: Fix error message after ssl_sock_load_ocsp call
- MINOR: debug: support passing an optional message in ABORT_NOW()
- MINOR: debug: add an optional message argument to the BUG_ON() family
- DEBUG: make the "debug dev {debug|warn|check}" command print a message
- CLEANUP: quic: Code clarifications for QUIC CUBIC (RFC 9438)
- BUG/MINOR: quic: fix possible integer wrap around in cubic window calculation
- MINOR: quic: Stop using 1024th of a second.
- CI: github: abandon asan matrix.py helper
- CI: ssl: add yet another OpenSSL download fallback
- DOC: install: clarify WolfSSL chroot requirements
- MINOR: task: Move wait_event in the task header file
- MINOR: stconn: Be able to detect applets using HTX
- MINOR: stconn: Explicitly use an appctx to attach a stconn on it
- MINOR: stconn: Be prepared to handle error when a SC is attached to an applet
- MINOR: applet: Add dedicated IN/OUT buffers for appctx
- MINOR: applet: Add traces to debug receive/send and block/wake events
- MINOR: applet: Add support for callback functions to exchange data with channels
- MINOR: applet: Implement default functions to exchange data with channels
- MEDIUM: stconn: Add functions to handle applets I/O from the SC layer
- MEDIM: applet: Add the applet handler based on IN/OUT buffers
- MINOR: applet: Show IN/OUT buffers in trace messages when used
- MINOR: applet: Add flags on the appctx and stop abusing its state
- MINIOR: applet: Add flags to deal with ends of input, ends of stream and errors
- MINOR: applet: Remove appctx state field to only used the flags
- MINOR: applet: Add an appctx flag to report shutdown to applets
- MEDIUM: applet: Use appctx flags to report EOS/EOI/ERROR to SE
- MINOR: applet: Add callback function to deal with zero-copy forwarding
- MEDIUM: applet: Add support for zero-copy forwarding from an applet
- MINOR: applet: Automatically handle applets having more data for the stream
- MEDIUM: stats: Don't interrupt processing on partial post
- MAJOR: stats: Update HTTP stats applet to handle its own buffers
- MEDIUM: cache: Temporarily remove zero-copy forwarding support
- MAJOR: cache: Update HTTP cache applet to handle its own buffers
- MAJOR: cache: Send cached objects using zero-copy forwarding
- MINOR: stconn: Add support for flags during zero-copy forwarding negotiation
- MINOR: mux-h1: Be able to define the length of a chunk size when it is prepended
- MEDIUM: stconn: Nofify requested size during zero-copy forwarding nego is exact
- MINOR: mux-h1: Stop zero-copy forwarding during nego for too big requested size
- MEDIUM: mux-h1: Support zero-copy forwarding for chunks with an unknown size
- MAJOR: stats: Send stats dump over HTTP using zero-copy forwarding
- MEDIUM: applet: Simplify a bit API to exchange data with applets
- MINOR: cache: Remove unsed .data_sent field from the cache applet context
- MINOR: applet: Use an option to disable zero-copy forwarding for all applets
- MINOR: applet: Identify applets using their own buffers via a flag
- BUG/MINOR: ssl: Duplicate ocsp update mode when dup'ing ckch
- MINOR: ssl: Use OCSP_CERTID instead of ckch_store in ckch_store_build_certid
- BUG/MINOR: ssl: Clear the ckch instance when deleting a crt-list line
- BUG/MEDIUM: ocsp: Separate refcount per instance and per store
- BUG/MINOR: ssl: Destroy ckch instances before the store during deinit
- BUG/MINOR: ssl: Reenable ocsp auto-update after an "add ssl crt-list"
- REGTESTS: ssl: Add OCSP related tests
- REGTESTS: ssl: Fix empty line in cli command input
- DOC: install: recommend pcre2
- DOC: config: fix misplaced "txn.conn_retries"
- DOC: config: fix typos for "bytes_{in,out}"
- DOC: config: fix misplaced "bytes_{in,out}"
- DOC: config: add more custom log format table alternatives
- MINOR: stream: rename "txn.redispatch" to "txn.redispatched"
- MINOR: sample: implement bc_{be,srv}_queue samples
- BUG/MINOR: mux-h2: count rejected DATA frames against the connection's flow control
- MINOR: mux-h2: count excess of CONTINUATION frames as a glitch
- MINOR: mux-h2: count late reduction of INITIAL_WINDOW_SIZE as a glitch
- DOC: internal: update missing data types in peers-v2.0.txt
- MEDIUM: stick-tables: add a new stored type for glitch_cnt and glitch_rate
- MINOR: session: add the necessary functions to update the per-session glitches
- MEDIUM: mux-h2: update session trackers with number of glitches
- BUG/MINOR: server/cli: add missing LF at the end of certain notice/error lines
- BUG/MINOR: vars/cli: fix missing LF after "get var" output
- BUG/MEDIUM: cli: fix once for all the problem of missing trailing LFs
- MINOR: cli: make sure to always print a pending message after release()
- MINOR: cli: always reset the applet task's timeout
- MINOR: cli: add a new "wait" command to wait for a certain delay
- BUG/MINOR: applet: Always release empty appctx buffers after processing
- MINOR: server: split the server deletion code in two parts
- MINOR: cli/wait: make the wait command support a more detailed help message
- MINOR: cli/wait: also support an unrecoverable failure status
- MINOR: cli/wait: also pass up to 4 arguments to the external conditions
- MINOR: cli/wait: add a condition to wait on a server to become unused
- CI: Update to actions/cache@v4
- BUILD: address a few remaining calloc(size, n) cases
- BUG/MEDIUM: pool: fix rare risk of deadlock in pool_flush()
The "wait" command now supports a condition, "srv-unused", which waits
for the designated server to become totally unused, indicating that it
is removable. Upon each wakeup it calls srv_check_for_deletion() to
verify if conditions are met, if not if it's recoverable, or if it's
not recoverable, and proceeds according to this, never waiting for a
final decision longer than the configured delay.
The purpose is to make it possible to remove servers from the CLI after
waiting for their sessions to be terminated:
$ socat -t5 /path/to/socket - <<< "
disable server px/srv1
shutdown sessions server px/srv1
wait 2s srv-unused px/srv1
del server px/srv1"
Or even wait for connections to terminate themselves:
$ socat -t70 /path/to/socket - <<< "
disable server px/srv1
wait 1m srv-unused px/srv1
del server px/srv1"
This allows to insert delays between commands, i.e. to collect a same
set of metrics at a fixed interval. E.g:
$ socat -t20 /path/to/socket <<< "show activity; wait 10s; show activity"
The goal will be to extend the feature to optionally support waiting on
certain conditions. For this reason the struct definitions and enums were
placed into cli-t.h.
This adds a new pair of stored types in the stick-tables:
- glitch_cnt
- glitch_rate
These keep count of the number of glitches reported on a front connection,
in order to decide how to act with a badly defective client or a potential
attacker. For now nothing updates these counters, but all the infrastructure
needed to configure, update and retrieve them was added, including the doc.
No regtest was added yet since they're not filled yet.
This is apparently the only location where the stored data types are
documented, but it was quite outdated as it stopped at gpc1 rate. This
patch adds the missing types (up to and including gpc_rate).
The fetch will return true if the stream was redispatched: this is a
past action, thus we rename the fetch to better reflect its true
meaning and prevent confusions.
Documentation was updated.
While at it, the fetch was moved from internal states section to Layer 4
section, which is where it belongs.
No backport needed unless 92b2edb (" MINOR: stream: add "txn.redispatch"
fetch") gets backported.
Counters are managed at the stream level and also work in TCP mode.
They were found in the Layer 7 section, moving them to the Layer 4
section instead.
This could be backported in 2.9 with fa0a304f3 ("DOC: config: add an
index of sample fetch keywords")
An extra space was placed at the start of "bytes_out" description,
and dconv was having a hard time to properly render the text in html
format because of that.
Finally, remove an extra line feed.
This should be backported in 2.9 with c7424a1ba ("MINOR: samples:
implement bytes_in and bytes_out samples")
txn.conn_retries was inserted in the internal states sample table, but it
should belong to Layer 4 sample table instead (SMP_USE_L4SRV)
This should be backported in 2.9 with fa0a304f3 ("DOC: config: add an
index of sample fetch keywords")
At the beginning of the 3.0-dev cycle, the zero-copy forwarding support was
added only for the cache applet with an option to disable it. This was a
hack, waiting for a better integration with applets. It is now possible to
implement the zero-copy forwarding for any applets. So the specific option
for the cache applet was renamed to be used for all applets. And this option
is now also checked for the stats applet.
Concretely, 'tune.cache.zero-copy-forwarding' was renamed to
'tune.applet.zero-copy-forwarding'.
Fetch will return true if the stream underwent a redispatch according to
"option redispatch" setting upon retries.
Documentation was added, and the "%rc" logformat alternative now mentions
the new fetch to properly emulate the logformat behavior.
set-bc-{mark,tos} actions are pretty similar to set-fc-{mark,tos} to set
mark/tos on packets sent from haproxy to server: set-bc-{mark,tos} actions
act on the whole backend/srv connection: from connect() to connection
teardown, thus they may only be used before the connection to the server
is instantiated, meaning that they are only relevant for request-oriented
rules such as tcp-request or http-request rules. For now their use is
limited to content request rules, because tos and mark informations are
stored directly within the stream, thus it is required that the stream
already exists.
stream flags are used in combination with dedicated stream struct members
variables to pass 'tos' and 'mark' informations so that they are correctly
considered during stream connection assignment logic (prior to connecting
to actually connecting to the server)
'tos' and 'mark' fd sockopts are taken into account in conn hash
parameters for connection reuse mechanism.
The documentation was updated accordingly.
In this patch we add the possibility to use sample expression as argument
for set-fc-{mark,tos} actions. To make it backward compatible with
previous behavior, during parsing we first try to parse the value as
as integer (decimal or hex notation), and then fallback to expr parsing
in case of failure.
The documentation was updated accordingly.
This is a complementary patch to "MINOR: tcp-act: Rename "set-{mark,tos}"
to "set-fc-{mark,tos}"", but for the Lua API.
set_mark and set_tos were kept as aliases for set_fc_mark and set_fc_tos
but they were marked as deprecated.
Using this opportunity to reorder set_mark and set_tos by alphabetical
order.
"set-mark" and "set-tos" only alter packets from haproxy to client
(frontend connection). Since we may add support for equivalent keywords
on server side, we rename them with an explicit name to prevent
confusions.
Thus, we rename:
- "set-mark" to "set-fc-mark"
- "set-tos" to "set-fc-tos"
"set-mark" and "set-tos" were kept as aliases (to "set-fc-mark" and
"set-fc-tos" respectively) for now to prevent config breakage, but they
have been marked as deprecated so they can be removed in future version.
Move httpclient keywords into its own section and explain adding
an introductory paragraph.
Also see Github issue #2409
Should be backported to 2.6 ; but note that:
2.7 does not have httpclient.resolvers.disabled
2.6 does not have httpclient.retries and httpclient.timeout.connect
Make it more explicit what happens in the various scenarios that cause
HAProxy to stop waiting when "http-request wait-for-body" is used.
Also fix a couple of grammatical errors.
Fixes: #2410
Signed-Off-By: Thayne McCombs <astrothayne@gmail.com>
Released version 3.0-dev2 with the following main changes :
- MINOR: ot: logsrv struct becomes logger
- MINOR: ssl: Update ssl_fc_curve/ssl_bc_curve to use SSL_get0_group_name
- CLEANUP: ssl: fix indentation in smp_fetch_ssl_fc_ec()
- DEV: patchbot: produce a verdict for too long commit messages
- CLEANUP: ssl: fix indentation in smp_fetch_ssl_fc_ec() (part 2)
- CLEANUP: quic: Double quic_dgram_parse() prototype declaration.
- BUG/MINOR: map: list-based matching potential ordering regression
- REGTESTS: add a test to ensure map-ordering is preserved
- DOC: config: fix typo about map_*_key converters
- DOC: configuration: corrected description of keyword tune.ssl.ocsp-update.mindelay
- MINOR: map: mapfile ordering also matters for tree-based match types
- DEV: phash: add a trivial perfect hash generator for integers
- OPTIM: http: simplify http_get_status_idx() using a hash
- CLEANUP: http: avoid duplicating literals in find_http_meth()
- MINOR: http: add infrastructure to choose status codes for err / fail
- MEDIUM: http_act: check status codes against the bit fields for err/fail
- MEDIUM: http: add the ability to redefine http-err-codes and http-fail-codes
- CI: codespell: ignore some words in URLs
- CI: codespell: add more words to whitelist
- CLEANUP: fix spelling of "occured" in src/h3.c
- BUILD: quic: missing include for quic_tp
- BUG/MINOR: mux-quic: do not prevent non-STREAM sending on flow control
- MEDIUM: ssl: allow multiple fallback certificate to allow ECDSA/RSA selection
- MEDIUM: ssl: generate '*' SNI filters for default certificates
- MEDIUM: ssl: does not use default_ctx for 'generate-certificate' option
- REORG: ssl: move 'generate-certificates' code to ssl_gencert.c
- DOC: configuration: update configuration on how to have multiple default certs
- MEDIUM: ssl: implements 'default-crt' keyword for bind Lines
- CI: github: update wolfSSL to 5.6.6
- DOC: INSTALL: require at least WolfSSL 5.6.6
- DEV: h2: add support for multiple flags in mkhdr
- DEV: h2: support hex-encoded data sequences in mkhdr
- BUG/MINOR: mux-h2: also count streams for refused ones
- BUG/MEDIUM: quic: keylog callback not called (USE_OPENSSL_COMPAT)
- MINOR: vars: fix indentation in var_clear_buffer()
- DOC: configuration: fix set-dst in actions keywords matrix
- BUG/MEDIUM: mux-h2: refine connection vs stream error on headers
- MINOR: mux-h2/traces: add a missing trace on connection WU with negative inc
- MINOR: mux-h2: add a counter of "glitches" on a connection
- MINOR: connection: add a new mux_ctl to report number of connection glitches
- MINOR: mux-h2: implement MUX_CTL_GET_GLITCHES
- MINOR: connection: add sample fetches to report per-connection glitches
- BUILD: stick-table: fix build error on 32-bit platforms
- MINOR: quic: Transport parameters encoding without version_information
- MINOR: quic: Enable early data at SSL session level (aws-lc)
- MINOR: ssl_sock: Early data disabled during SSL_CTX switching (aws-lc)
- MINOR: quic: Correctly wait for the completion of handshakes with early data (aws-lc)
- BUG/MEDIUM: cli: some err/warn msg dumps add LR into CSV output on stat's CLI
- BUG/MINOR: jwt: fix jwt_verify crash on 32-bit archs
- BUILD: quic: fix build error when using the compatibility layer
- BUILD: quic: Fix build error when building QUIC against wolfssl.
- BUILD: quic: Fix build error when building QUIC against libressl.
- BUG/MINOR: hlua: fix uninitialized var in hlua_core_get_var()
- CLEANUP: hlua: fix indent, remove extra return in hlua_core_get_var()
- BUG/MEDIUM: cache: Fix crash when deleting secondary entry
- BUG/MINOR: quic: newreno QUIC congestion control algorithm no more available
- CLEANUP: quic: Remove unused CUBIC_BETA_SCALE_FACTOR_SHIFT macro.
- MINOR: quic: Stop hardcoding a scale shifting value (CUBIC_BETA_SCALE_FACTOR_SHIFT)
- MINOR: quic: extract qc_stream_buf free in a dedicated function
- BUG/MEDIUM: quic: remove unsent data from qc_stream_desc buf
- CLEANUP: fix spelling of "elemt"
- CI: extend spell check white list
- CI: enable spell check on git push
- BUILD: makefile: also define cmd_CXX to pretty-print C++ build commands
- BUILD/MEDIUM: deviceatlas: addon build rework.
- DOC: deviceatlas: update to be in line with the v3 api.
- BUILD/MEDIUM: deviceatlas: updating the addon part.
- BUILD: deviceatlas: remove unneeded depenency on libcurl / libzip
- BUILD: deviceatlas: fix empty "-I" left on CFLAGS
- Revert "CI: enable spell check on git push"
Now with fc_glitches and bc_glitches we can retrieve the number of
detected glitches on a front or back connection. On the backend it
can indicate a bug in a server that may induce frequent reconnections
hence CPU usage in TLS reconnections, and on the frontend it may
indicate an abusive client that may be trying to attack the stack
or to fingerprint it. Small non-zero values are definitely expected
and can be caused by network glitches for example, as well as rare
bugs in the other component (or maybe even in haproxy). These should
never be considered as alarming as long as they remain low (i.e.
much less than one per request). A reg-test is provided.
Since d54e8f8107 ("DOC: config: reorganize actions into their own section")
dconv-generated shortlink for "set-dst" in actions keywords matrix is
broken.
This is due to trailing "<expr>" which should not be specified in the
matrix, but only in the actual keyword prototype and description.
This should be backported in 2.9 with d54e8f8107.
The new global keywords "http-err-codes" and "http-fail-codes" allow to
redefine which HTTP status codes indicate a client-induced error or a
server error, as tracked by stick-table counters. This is only done
globally, though everything was done so that it could easily be extended
to a per-proxy mechanism if there was a real need for this (but it would
eat quite more RAM then).
A simple reg-test was added (http-err-fail.vtc).
Deleted the text paragraph in the description of keyword
tune.ssl.ocsp-update.mindelay, which was added in the commit 5843237
"MINOR: ssl: Add global options to modify ocsp update min/max delay",
because it was a copy of the description of tune.ssl.ssl-ctx-cache-size.
Fix a doc typo that was introduced with ca4758378 ("MINOR: map: add
map_*_key converters to provide the matching key").
No backport needed unless ca4758378 is.
Released version 3.0-dev1 with the following main changes :
- MINOR: channel: Use dedicated functions to deal with STREAMER flags
- MEDIUM: applet: Handle channel's STREAMER flags on applets size
- MINOR: applets: Use channel's field to compute amount of data received
- MEDIUM: cache: Save body size of cached objects and track it on delivery
- MEDIUM: cache: Add support for endp-to-endp fast-forwarding
- MINOR: cache: Add global option to enable/disable zero-copy forwarding
- MINOR: pattern: Use reference name as filename to read patterns from a file
- MEDIUM: pattern: Add support for virtual and optional files for patterns
- DOC: config: Add section about name format for maps and ACLs
- DOC: management/lua: Update commands about map and acl
- MINOR: promex: Add support for specialized front/back/li/srv metric names
- MINOR: promex: Export active/backup metrics per-server
- BUG/MINOR: ssl: Double free of OCSP Certificate ID
- MINOR: ssl/cli: Add ha_(warning|alert) msgs to CLI ckch callback
- BUG/MINOR: ssl: Wrong OCSP CID after modifying an SSL certficate
- BUG/MINOR: lua: Wrong OCSP CID after modifying an SSL certficate (LUA)
- DOC: configuration: typo req.ssl_hello_type
- MINOR: hq-interop: add fastfwd support
- CLEANUP: mux_quic: rename ffwd function with prefix qmux_strm_
- MINOR: mux-quic: add traces for 0-copy/fast-forward
- BUG/MINOR: mworker/cli: fix set severity-output support
- CLEANUP: mworker/cli: add comments about pcli_find_and_exec_kw()
- BUG/MEDIUM: quic: Possible buffer overflow when building TLS records
- BUILD: ssl: update types in wolfssl cert selection callback
- MINOR: ssl: activate the certificate selection callback for WolfSSL
- CI: github: switch to wolfssl git-c4b77ad for new PR
- BUG/MEDIUM: map/acl: pat_ref_{set,delete}_by_id regressions
- BUG/MINOR: ext-check: cannot use without preserve-env
- CLEANUP: mux-quic: remove unused prototype
- MINOR: mux-quic: clean up qcs Rx buffer allocation API
- MINOR: mux-quic: clean up qcs Tx buffer allocation API
- CLEANUP: mux-quic: clean up app ops callback definitions
- MINOR: mux-quic: factorize QC_SF_UNKNOWN_PL_LENGTH set
- MINOR: h3: complete traces for sending
- MINOR: h3: adjust zero-copy sending related code
- MINOR: hq-interop: use zero-copy to transfer single HTX data block
- BUG/MEDIUM: quic: QUIC CID removed from tree without locking
- BUG/MEDIUM: stconn: Block zero-copy forwarding if EOS/ERROR on consumer side
- BUG/MEDIUM: mux-h1: Cound data from input buf during zero-copy forwarding
- BUG/MEDIUM: mux-h1: Explicitly skip request's C-L header if not set originally
- CLEANUP: mux-h1: Fix a trace message about C-L header addition
- BUG/MEDIUM: mux-h2: Report too large HEADERS frame only when rxbuf is empty
- BUG/MEDIUM: mux-quic: report early error on stream
- DOC: config: add arguments to sample fetch methods in the table
- DOC: config: also add arguments to the converters in the table
- BUG/MINOR: resolvers: default resolvers fails when network not configured
- SCRIPTS: mk-patch-list: produce a list of patches
- DEV: patchbot: add the AI-based bot to pre-select candidate patches to backport
- BUG/MEDIUM: mux-h2: Switch pending error to error if demux buffer is empty
- BUG/MEDIUM: mux-h2: Only Report H2C error on read error if demux buffer is empty
- BUG/MEDIUM: mux-h2: Don't report error on SE if error is only pending on H2C
- BUG/MEDIUM: mux-h2: Don't report error on SE for closed H2 streams
- DOC: config: Update documentation about local haproxy response
- DEV: patchbot: use checked buttons as reference instead of internal table
- DEV: patchbot: allow to show/hide backported patches
- MINOR: h3: remove quic_conn only reference
- BUG/MINOR: server: Use the configured address family for the initial resolution
- MINOR: mux-quic: remove qcc_shutdown() from qcc_release()
- MINOR: mux-quic: use qcc_release in case of init failure
- MINOR: mux-quic: adjust error code in init failure
- MINOR: h3: add traces for connection init stage
- BUG/MINOR: h3: properly handle alloc failure on finalize
- MINOR: h3: use INTERNAL_ERROR code for init failure
- BUG/MAJOR: stconn: Disable zero-copy forwarding if consumer is shut or in error
- MINOR: stats: store the parent proxy in stats ctx (http)
- BUG/MEDIUM: stats: unhandled switching rules with TCP frontend
- MEDIUM: proxy: set PR_O_HTTP_UPG on implicit upgrades
- MINOR: proxy: monitor-uri works with tcp->http upgrades
- OPTIM: server: eb lookup for server_find_by_name()
- OPTIM: server: ebtree lookups for findserver_unique_* functions
- MINOR: server/event_hdl: add server_inetaddr struct to facilitate event data usage
- MINOR: server/event_hdl: update _srv_event_hdl_prepare_inetaddr prototype
- BUG/MINOR: server/event_hdl: propagate map port info through inetaddr event
- MINOR: server: ensure connection cleanup on server addr changes
- CLEANUP: server/event_hdl: remove purge_conn hint in INETADDR event
- MEDIUM: server: merge srv_update_addr() and srv_update_addr_port() logic
- CLEANUP: server: remove unused server_parse_addr_change_request() function
- CLEANUP: resolvers: remove duplicate func prototype
- MINOR: resolvers: add unique numeric id to nameservers
- MEDIUM: server: make server_set_inetaddr() updater serializable
- MINOR: server/event_hdl: expose updater info through INETADDR event
- MINOR: server: add dns hint in server_inetaddr_updater struct
- MEDIUM: server/dns: clear RMAINT when addr resolves again
- BUG/MINOR: server/dns: use server_set_inetaddr() to unset srv addr from DNS
- BUG/MEDIUM: server/dns: perform svc_port updates atomically from SRV records
- MEDIUM: peers: use server as stream target
- CLEANUP: peers: remove unused sock_init_arg struct member
- CLEANUP: peers: remove unused "proto" and "xprt" struct members
- MINOR: peers: rely on srv->addr and remove peer->addr
- DOC: config: add context hint for server keywords
- MINOR: stktable: add table_process_entry helper function
- MINOR: stktable: use {show,set,clear} table with ptr
- MINOR: map: add map_*_key converters to provide the matching key
- DOC: fix typo for fastfwd QUIC option
- BUG/MINOR: mux-quic: always report error to SC on RESET_STREAM emission
- MEDIUM: mux-quic: add BUG_ON if sending on locally closed QCS
- BUG/MINOR: mux-quic: disable fast-fwd if connection on error
- BUG/MINOR: quic: Wrong keylog callback setting.
- BUG/MINOR: quic: Missing call to TLS message callbacks
- MINOR: h3: check connection error during sending
- BUG/MINOR: h3: close connection on header list too big
- BUG/MINOR: h3: close connection on sending alloc errors
- BUG/MINOR: h3: disable fast-forward on buffer alloc failure
- Revert "MINOR: mux-quic: Disable zero-copy forwarding for send by default"
- MINOR: stktable: stktable_data_ptr() cannot fail in table_process_entry()
- CLEANUP: assorted typo fixes in the code and comments
- CI: use semantic version compare for determing "latest" OpenSSL
- CLEANUP: server: remove ambiguous check in srv_update_addr_port()
- CLEANUP: resolvers: remove unused RSLV_UPD_OBSOLETE_IP flag
- CLEANUP: resolvers: remove some more unused RSLV_UDP flags
- MEDIUM: server: simplify snr_set_srv_down() to prevent confusions
- MINOR: backend: export get_server_*() functions
- MINOR: tcpcheck: export proxy_parse_tcpcheck()
- MEDIUM: udp: allow to retrieve the frontend destination address
- MINOR: global: export a way to list build options
- MINOR: debug: add features and build options to "show dev"
- BUG/MINOR: server: fix server_find_by_name() usage during parsing
- REGTESTS: check attach-srv out of order declaration
- CLEANUP: quic: Remaining useless code into server part
- BUILD: quic: Missing quic_ssl.h header protection
- BUG/MEDIUM: h3: fix incorrect snd_buf return value
- MINOR: h3: do not consider missing buf room as error on trailers
- BUG/MEDIUM: stconn: Forward shutdown on write timeout only if it is forwardable
- BUG/MEDIUM: stconn: Set fsb date if zero-copy forwarding is blocked during nego
- BUG/MEDIUM: spoe: Never create new spoe applet if there is no server up
- MINOR: mux-h2: support limiting the total number of H2 streams per connection
- CLEANUP: mux-h2: remove the printfs from previous commit on h2 streams limit.
- DEV: h2: add the ability to emit literals in mkhdr
- DEV: h2: add the preface as well in supported output types
- DEV: h2: support passing raw data for a frame
- IMPORT: ebtree: implement and use flsnz_long() to count bits
- IMPORT: ebtree: switch the sizes and offsets to size_t and ssize_t
- IMPORT: ebtree: rework the fls macros to better deal with arch-specific ones
- IMPORT: ebtree: make string_equal_bits turn back to unsigned char
- IMPORT: ebtree: use unsigned ints for flznz()
- IMPORT: ebtree: make string_equal_bits() return an unsigned
This patch introduces a new setting: tune.h2.fe.max-total-streams. It
sets the HTTP/2 maximum number of total streams processed per incoming
connection. Once this limit is reached, HAProxy will send a graceful GOAWAY
frame informing the client that it will close the connection after all
pending streams have been closed. In practice, clients tend to close as fast
as possible when receiving this, and to establish a new connection for next
requests. Doing this is sometimes useful and desired in situations where
clients stay connected for a very long time and cause some imbalance inside a
farm. For example, in some highly dynamic environments, it is possible that
new load balancers are instantiated on the fly to adapt to a load increase,
and that once the load goes down they should be stopped without breaking
established connections. By setting a limit here, the connections will have
a limited lifetime and will be frequently renewed, with some possibly being
established to other nodes, so that existing resources are quickly released.
The default value is zero, which enforces no limit beyond those implied by
the protocol (2^30 ~= 1.07 billion). Values around 1000 were found to
already cause frequent enough connection renewal without causing any
perceptible latency to most clients. One notable exception here is h2load
which reports errors for all requests that were expected to be sent over
a given connection after it receives a GOAWAY. This is an already known
limitation: https://github.com/nghttp2/nghttp2/issues/981
The patch was made in two parts inside h2_frt_handle_headers():
- the first one, at the end of the function, which verifies if the
configured limit was reached and if it's needed to emit a GOAWAY ;
- the second, just before decoding the stream frame, which verifies if
a previously configured limit was ignored by the client, and closes
the connection if this happens. Indeed, one reason for a connection
to stay alive for too long definitely comes from a stupid bot that
periodically fetches the same resource, scans lots of URLs or tries
to brute-force something. These ones are more likely to just ignore
the last stream ID advertised in GOAWAY than a regular browser, or
a well-behaving client such as curl which respects it. So in order
to make sure we can close the connection we need to enforce the
advertised limit.
Note that a regular client will not face a problem with that because in
the worst case it will have max_concurrent_streams in flight and this
limit is taken into account when calculating the advertised last
acceptable stream ID.
Just a note: it may also be possible to move the first part above to
h2s_frt_stream_new() instead so that it's not processed for trailers,
though it doesn't seem to be more interesting, first because it has
two return points.
This is something that may be backported to 2.9 and 2.8 to offer more
control to those dealing with dynamic infrastructures, especially since
for now we cannot force a connection to be cleanly closed using rules
(e.g. github issues #946, #2146).
This reverts commit 18f2ccd244.
Found issues related to QUIC fast-forward were resolved (see github
issue #2372). Reenable it by default. If any issue arises, it can be
disabled using the global statement :
tune.quit.zero-copy-fwd-send off
This can be backported to 2.9, but only after a sensible period of
observation.
All map_*_ converters now have an additional output type: key. Such
converters will return the matched entry's key (as found in the map file)
as a string instead of the value.
Consider this example map file:
|example.com value1
|haproxy value2
With the above map file:
str(test.example.com/url),map_dom_key(file.map) will return "example.com"
str(running haproxy),map_sub_key(file.map) will return "haproxy"
This should address GH #1446.
Documentation about 'L' state in the termination state was outdated. Today,
not only the request may be intercepted, but also the response.
Documentation about 'L' must be more generic.
However, documentation about possible 2-letter termination states was also
extended to add 'LC' and 'LH' in the list. And 'LR' was adapted too.
This patch should fix the issue #2384. It may be backported to every stable
versions. Note that on 2.8 and lowers, we talk about session and not stream.
rep_ssl_hello_type was renamed in res.ssl_hello_type a long time ago.
This patch fixes a typo where an example was renamed
"rep.ssl_hello_type" instead of "res.ssl_hello_type"
fixes issue #2377 and #2379.
Must be backported in all maintained versions.
Because maps and list of ACLs are no longer necessarily referenced by
filenames, CLI commands to manipulate them were updated accordingly. Instead
of "filename" we talk about "name" now.
The same is performed in the LUA documentation.
Maps and list of ACLs can now reference something else than regular files
and can have prefix to set the type of the list (file, virutal file or
optional file). So, the configuration manual was updated accordingly.
The section 2.7. about name format for maps and ACLs was added (the former
2.7. sections with some examples was moved to 2.8.) and references to map or
ACLs files were updated.
tune.cache.zero-copy-forwarding parameter can now be used to enable or
disable the zero-copy fast-forwarding for the cache applet only. It is
enabled ('on') by default. It can be disabled by setting the parameter to
'off'.
Released version 2.9.0 with the following main changes :
- DOC: config: add missing colon to "bytes_out" sample fetch keyword (2)
- BUG/MINOR: cfgparse-listen: fix warning being reported as an alert
- DOC: config: add matrix entry for "max-session-srv-conns"
- DOC: config: fix monitor-fail typo
- DOC: config: add context hint for proxy keywords
- DEBUG: stream: Report lra/fsb values for front end back SC in stream dump
- REGTESTS: sample: Test the behavior of consecutive delimiters for the field converter
- BUG/MINOR: sample: Make the `word` converter compatible with `-m found`
- DOC: Clarify the differences between field() and word()
- BUG/MINOR: server/event_hdl: properly handle AF_UNSPEC for INETADDR event
- BUILD: http_htx: silence uninitialized warning on some gcc versions
- MINOR: acme.sh: don't use '*' in the filename for wildcard domain
- MINOR: global: Use a dedicated bitfield to customize zero-copy fast-forwarding
- MINOR: mux-pt: Add global option to enable/disable zero-copy forwarding
- MINOR: mux-h1: Add global option to enable/disable zero-copy forwarding
- MINOR: mux-h2: Add global option to enable/disable zero-copy forwarding
- MINOR: mux-quic: Add global option to enable/disable zero-copy forwarding
- MINOR: mux-quic: Disable zero-copy forwarding for send by default
- DOC: config: update the reminder on the HTTP model and add some terminology
- DOC: config: add a few more differences between HTTP/1 and 2+
- DOC: config: clarify session vs stream
- DOC: config: fix typo abandonned -> abandoned
- DOC: management: fix two latest typos (optionally, exception)
- BUG/MEDIUM: peers: fix partial message decoding
- DOC: management: update stream vs session
No backport needed, these were introduced by latest commits 3dd55fa13
("MINOR: mworker/cli: implement hard-reload over the master CLI") and
cef29d370 ("MINOR: trace: define simple -dt argument").
It was really necessary to try to clear the confusion between sessions
and streams, so let's first lift a little bit the HTTP model part to
better consider new protocols, and explain what a stream is and how this
differs from the earlier sessions.
There is at least an bug for now in this part and it is still unstable. Thus
it is better to disable it for now by default. It can be enable by setting
tune.quic.zero-copy-fwd-send to 'on'.
tune.quic.zero-copy-fwd-send can now be used to enable or disable the
zero-copy fast-forwarding for the QUIC mux only, for sends. For now, there
is no option to disable it for receives because it is not supported yet.
It is enabled ('on') by default.
tune.h2.zero-copy-fwd-send can now be used to enable or disable the
zero-copy fast-forwarding for the H2 mux only, for sends. For now, there is
no option to disable it for receives because it is not supported yet.
It is enabled ('on') by default.
tune.h1.zero-copy-fwd-recv and tune.h1.zero-copy-fwd-send can now be used to
enable or disable the zero-copy fast-forwarding for the H1 mux only, for
receives or sends. Unlike the PT mux, there are 2 options here because
client and server sides can use difference muxes.
Both are enabled ('on') by default.
tune.pt.zero-copy-forwarding parameter can now be used to enable or disable
the zero-copy fast-forwarding for the PT mux only. It is enabled ('on') by
default. It can be disabled by setting the parameter to 'off'. In this case,
this disables receive and send side.
Zero-copy fast-forwading feature is a quite new and is a bit sensitive.
There is an option to disable it globally. However, all protocols have not
the same maturity. For instance, for the PT multiplexer, there is nothing
really new. The zero-copy fast-forwading is only another name for the kernel
splicing. However, for the QUIC/H3, it is pretty new, not really optimized
and it will evolved. And soon, the support will be added for the cache
applet.
In this context, it is usefull to be able to enable/disable zero-copy
fast-forwading per-protocol and applet. And when it is applicable, on sends
or receives separately. So, instead of having one flag to disable it
globally, there is now a dedicated bitfield, global.tune.no_zero_copy_fwd.
word() mentions that delimiters at the start and end are ignored, but
it does not mention that consecutive delimiters are merged.
May be backported as far as the patch applies.
Add a small list of contexts where each proxy keyword is expected to be
employed. (Similar to the defaults/frontend/backend/listen compatibility
grid).
Following 4039329 ("DOC: config: specify supported sections for
"max-session-srv-conns"), "max-session-srv-conns" was also missing
from the proxy keyword matrix.
Released version 2.9-dev12 with the following main changes :
- BUG/MINOR: global: Fix tune.disable-(fast-forward/zero-copy-forwarding) options
- DOC: config: removing "log-balance" references
- MINOR: server/event_hdl: add SERVER_INETADDR event
- MINOR: tools: use const for read only pointers in ip{cmp,cpy}
- MINOR: server/ip: centralize server ip updates
- MINOR: backend: remove invalid mode test for "hash-balance-factor"
- Revert "MINOR: cfgparse-listen: warn when use-server rules is used in wrong mode"
- MINOR: proxy: add free_logformat_list() helper function
- MINOR: proxy: add free_server_rules() helper function
- MINOR: log/backend: prevent "use-server" rules use with LOG mode
- MINOR: log/balance: set lbprm tot_weight on server on queue/dequeue
- DOC: config: specify supported sections for "max-session-srv-conns"
- DOC: config: fix timeout check inheritance restrictions
- REGTESTS: connection: disable http_reuse_be_transparent.vtc if !TPROXY
- DOC: lua: add sticktable class reference from Proxy.stktable
- DOC: lua: fix Proxy.get_mode() output
- DOC: lua: add "syslog" to Proxy.get_mode() output
- MEDIUM: ssl: implement rsa/ecdsa selection with WolfSSL
- MINOR: ssl: replace 'trash.area' by 'servername' in ssl_sock_switchctx_cbk()
- MINOR: ssl: move certificate selection in a dedicate function
- MEDIUM: ssl: use ssl_sock_chose_sni_ctx() in the clienthello callback
- MINOR: mworker/cli: implement hard-reload over the master CLI
- BUG/MEDIUM: mux-h1: Properly ignore trailers when a content-length is announced
- MINOR: task/profiling: do not record task_drop_running() as a caller
- OPTIM: pattern: save memory and time using ebst instead of ebis
- BUILD: map: fix build warning
- MINOR: trace: define simple -dt argument
- MINOR: trace: parse level in a function
- MINOR: trace: parse verbosity in a function
- MINOR: trace: support -dt optional format
- OPTIM: mux-h2/zero-copy: don't allocate more buffers per connections than streams
- BUG/MINOR: quic: fix CONNECTION_CLOSE_APP encoding
- BUG/MEDIUM: stconn: Don't perform zero-copy FF if opposite SC is blocked
- BUG/MEDIUM: mux-h2: Remove H2_SF_NOTIFIED flag for H2S blocked on fast-forward
- CLEANUP: quic: Remove dead definitions/declarations
- REORG: quic: Move some QUIC CLI code to its C file
- REORG: quic: Add a new module to handle QUIC connection IDs
- REORG: quic: QUIC connection types header cleaning
- BUILD: quic: Missing RX header inclusions
- REORG: quic: Move CRYPTO data buffer defintions to QUIC TLS module
- REORG: quic: Move QUIC CRYPTO stream definitions/declarations to QUIC TLS
- REORG: quic: Move several inlined functions from quic_conn.h
- REORG: quic: Move QUIC SSL BIO method related functions to quic_ssl.c
- REORG: quic: Move the QUIC DCID parser to quic_sock.c
- REORG: quic: Rename some functions used upon ACK receipt
- REORG: quic: Move QUIC path definitions/declarations to quic_cc module
- REORG: quic: Move qc_handle_conn_migration() to quic_conn.c
- REORG: quic: Move quic_build_post_handshake_frames() to quic_conn module
- REORG: quic: Move qc_may_probe_ipktns() to quic_tls.h
- REORG: quic: Move qc_pkt_long() to quic_rx.h
- REORG: quic: Rename some (quic|qc)_conn* objects to quic_conn_closed
- REORG: quic: Move NEW_CONNECTION_ID frame builder to quic_cid
- REORG: quic: Move ncbuf related function from quic_rx to quic_conn
- REORG: quic: Add a new module for QUIC retry
- BUILD: quic: Several compiler warns fixes after retry module creation
- REORG: quic: Move qc_notify_send() to quic_conn
- REORG: quic: Add a new module for retransmissions
- REORG: quic: Remove qc_pkt_insert() implementation
- REORG: quic: Move quic_increment_curr_handshake() to quic_sock
- BUG/MINOR: cache: Remove incomplete entries from the cache when stream is closed
- MEDIUM: cli: allow custom pattern for payload
- CLEANUP: mworker/cli: use a label to return errors
- MINOR: mworker/cli: implements the customized payload pattern for master CLI
- DOC: management: add documentation about customized payload pattern
- BUG/MEDIUM: server/event_hdl: memory overrun in _srv_event_hdl_prepare_inetaddr()
- MINOR: event_hdl: add global tunables
- BUG/MAJOR: server/addr: fix a race during server addr:svc_port updates
- MEDIUM: log/balance: support FQDN for UDP log servers
- BUG/MINOR: compression: possible NULL dereferences in comp_prepare_compress_request()
- BUG/MEDIUM: master/cli: Properly pin the master CLI on thread 1 / group 1
- BUG/MEDIUM: mux-quic: Stop zero-copy FF during nego if input is not empty
- CLEANUP: log: Fix %rc comment in sess_build_logline()
- BUG/MINOR: h3: fix TRAILERS encoding
- BUG/MINOR: h3: always reject PUSH_PROMISE
- MINOR: h3: use correct error code for missing SETTINGS
- MINOR: http-fetch: Add a sample to retrieve the server status code
- DOC: config: Improve 'status' sample documentation
- MINOR: http-fetch: Add a sample to get the transaction status code
- MEDIUM: http-ana: Set termination state before returning haproxy response
- MINOR: stream: Expose session terminate state via a new sample fetch
- MINOR: stream: add a sample fetch to get the number of connection retries
- MINOR: stream: Expose the stream's uniq_id via a new sample fetch
- MINOR: muxes: Rename mux_ctl_type values to use MUX_CTL_ prefix
- MINOR: muxes: Add a callback function to send commands to mux streams
- MINOR: muxes: Implement ->sctl() callback for muxes and return the stream id
- MINOR: Add sample fetches to get the frontend and backend stream ID
- BUG/MEDIUM: cli: Don't look for payload pattern on empty commands
- DOC: config: Add argument for tune.lua.maxmem
- DOC: config: fix mention of request slot in http-response capture
- DOC: config: fix remaining mention of @reverse for attach-srv action
- DOC: config: fix missing characters in set-spoe-group action
- DOC: config: reorganize actions into their own section
- BUG/MINOR: acme.sh: update the deploy script
- MINOR: rhttp: mark reverse HTTP as experimental
- CLEANUP: quic_cid: remove unused listener arg
- BUG/MINOR: quic_tp: fix preferred_address decoding
- MINOR: quic_tp: use in_addr/in6_addr for preferred_address
- MINOR: acme.sh: use the master CLI for hot update
- DOC: config: move the cache-use and cache-store actions to the proper section
- DOC: config: fix alphabetical ordering of converter keywords
- DOC: config: add missing colon to "bytes_out" sample fetch keyword
- DOC: config: add an index of converter keywords
- DOC: config: add an index of sample fetch keywords
- BUG/MINOR: config: Stopped parsing upon unmatched environment variables
- DEBUG: unstatify a few functions that are often present in backtraces
- BUILD: server: shut a bogus gcc warning on certain ubuntu
Now we're adding a table for each section, it allows to more easily
spot the list of available sample fetch functions and their types.
For now the arguments are not mentioned in the table because they'd
break indexing but they can be added back later.
- rfc7239_* were misplaced and incorrectly ordered
- table_gpt was placed before some table_gpc*
- capture-req/res were misplaced
- htonl was misplaced
- upper/url_* were misplaced
- x509_v_err_str was misplaced
Let's fix these since poor ordering complicates their finding.
Actions were grouped by previous commit d54e8f810 ("DOC: config: reorganize
actions into their own section") but cache-use and cache-store were still
making reference to the cache section. This moves the text back to their
respective keywords in the actions section and leaves the example and an
explanation of how to use the keywords in the cache section.
Mark the reverse HTTP feature as experimental. This will allow to adjust
if needed the configuration mechanism with future developments without
maintaining retro-compatibility.
Concretely, each config directives linked to it now requires to specify
first global expose-experimental-directives before. This is the case for
the following directives :
- rhttp@ prefix uses in bind and server lines
- nbconn bind keyword
- attach-srv tcp rule
Each documentation section refering to these keywords are updated to
highlight this new requirement.
Note that this commit has duplicated on several places the code from the
global function check_kw_experimental(). This is because the latter only
work with cfg_keyword type. This is not adapted with bind_kw or
action_kw types. This should be improve in a future patch.
The split of the rulesets from their respective actions has long been
overdue so it's time to do it because it has become extremely difficult
to add simple actions in the documentation, as well as it's hard to find
them.
This commit creates two new sections "4.3 Actions keywords matrix" and
"4.4 Alphabetically sorted actions reference" that enumerates all known
actions, with a check indicating for which rule sets they're valid. This
removes all the repetition, occurrences of "see http-request blah for
details" and significantly reduces the number of keywords listed in the
proxies section. This removes 2245 lines from the proxies section in
exchange of 1608 in these new sections.
"fc.id" and "bc.id" sample fetches can now be used to get, respectively, the
frontend or the backend stream ID. They rely on ->sctl() callback function
on the mux attached to the corresponding SC.
It means these sample fetches work only for connection, not applets, and
from the time a multiplexer is installed.
"txn.conn_retries" can now be used to get the number of connection
retries. This value is only stable once the connection is fully
established. For HTTP sessions, L7-retries must also be passed.
It is now possible to retrieve the session terminate state, using
"txn.sess_term_state". The sample fetch returns the 2-character session
termation state.
Of course, the result of this sample fetch is volatile. It is subject to
change. It is also most of time useless because no termation state is set
except at the end. It should only be useful in http-after-response rule
sets. It may also be used to customize the logs using a log-format
directive.
This patch should fix the issue #2221.
It was possible get the status code in the HTTP response and the one
received from the server. Thanks to 'txn.status', it is now possible to get
the transaction status code. It is equivalent to '%ST' in log-format.
Most of time, it is the same than 'status', except if the status code of the
HTTP reply does not match the one used to interrupt the transaction. For
instance, an error file use mapped on 400 containing a 404.
We clearly state the 'status' sample returns the status code the client will
receive, if no change happens on the HTTP response. This should avoid
ambiguities with the 'server-status' sample fetch.
The code returned by the "status" sample fetch is the one in the HTTP
response at the moment the sample is evaluated. It may be the status code in
the server response or the one of the HAProxy reply in case of error, deny,
redirect...
However, it could be handy to retrieve the status code returned by the
server, when a HTTP response was really received from it. It is the purpose
of the "server_status" sample fetch. The server status code itself is stored
in the HTTP txn.
The local variable "event_hdl_async_max_notif_at_once" which was
introduced with the event_hdl API was left as is but with a TODO note
telling that we should make it a global tunable.
Well, we're doing this now. To prepare for upcoming tunables related to
event_hdl API, we add a dedicated struct named event_hdl_tune which is
globally exposed through the event_hdl header file so that it may be used
from everywhere. The struct is automatically initialized in
event_hdl_init() according to defaults.h.
"event_hdl_async_max_notif_at_once" now becomes
"event_hdl_tune.max_events_at_once" with it's dedicated
configuation keyword: "tune.events.max-events-at-once".
We're also taking this opportunity to raise the default value from 10
to 100 since it's seems quite reasonnable given existing async event_hdl
users.
The documentation was updated accordingly.
Add an optional argument for "-dt". This argument is interpreted as a
list of several trace statement separated by comma. For each statement,
a specific trace name can be specifed, or none to act on all sources.
Using double-colon separator, it is possible to add specifications on
the wanted level and verbosity.
Add '-dt' haproxy process argument. This will automatically activate all
trace sources on stderr with the error level. This could be useful to
troubleshoot issues such as protocol violations.
The mworker mode never had a proper 'hard-stop' (-st) for the reload,
this is a mode which was commonly used with the daemon mode, but it was
never implemented in mworker mode.
This patch fixes the problem by implementing a "hard-reload" command
over the master CLI. It does the same as the "reload" command, but
instead of waiting for the connections to stop in the previous process,
it immediately quits the previous process after binding.
Following previous commit: in this patch we add the "syslog" output as
possible return value for Proxy.get_mode() function since log backend
may now be enumerated from lua with 9a74a6c ("MAJOR: log: introduce log
backends")
Proxy.get_mode() function internally relies on proxy_mode_str() to return
the proxy mode. The current function description is exhaustive about the
possible outputs for the function. I can't tell if it's relevant or not
but it's subject to changes. Here it is the case, the documentation
indicates that "health" mode may be returned, which cannot happen
since 77e0daef9 ("MEDIUM: proxy: remove obsolete "mode health"").
This should be backported up to 2.4
In 6e0425b718 ("DOC: config: Add documentation about TCP/HTTP rules in
defaults section") an error was made: the restriction note about the
setting not being inherited from anonymous default section was added
by mistake in the "timeout check" documentation. But it is wrong,
"timeout check" behaves like other "timeout" directives for proxy
sections.
This should be backported up to 2.6.
There was no info about supported sections for "max-session-srv-conns"
proxy directive. A quick look at the code tells us that it may be used
in proxies with the FE capability set.
"log-balance" keyword was removed by b61147f ("MEDIUM: log/balance: merge
tcp/http algo with log ones") but it was still documented.
Removing "log-balance" references in the documentation where needed.
Released version 2.9-dev11 with the following main changes :
- BUG/MINOR: startup: set GTUNE_SOCKET_TRANSFER correctly
- BUG/MINOR: sock: mark abns sockets as non-suspendable and always unbind them
- BUILD: cache: fix build error on older compilers
- BUG/MAJOR: quic: complete thread migration before tcp-rules
- BUG/MEDIUM: quic: Possible crash for connections to be killed
- MINOR: quic: remove unneeded QUIC specific stopping function
- MINOR: acl: define explicit HTTP_3.0
- DEBUG: connection/flags: update flags for reverse HTTP
- BUILD: log: silence a build warning when threads are disabled
- MINOR: quic: Add traces to debug frames handling during retransmissions
- BUG/MEDIUM: quic: Possible crash during retransmissions and heavy load
- BUG/MINOR: quic: Possible leak of TX packets under heavy load
- BUG/MINOR: quic: Possible RX packet memory leak under heavy load
- BUG/MINOR: server: do not leak default-server in defaults sections
- DEBUG: tinfo: store the pthread ID and the stack pointer in tinfo
- MINOR: debug: start to create a new struct post_mortem
- MINOR: debug: add OS/hardware info to the post_mortem struct
- MINOR: debug: report in port_mortem whether a container was detected
- MINOR: debug: report in post_mortem if the container techno used is docker
- MINOR: debug: detect CPU model and store it in post_mortem
- MINOR: debug: report any detected hypervisor in post_mortem
- MINOR: debug: collect some boot-time info related to the process
- MINOR: debug: copy the thread info into the post_mortem struct
- MINOR: debug: dump the mapping of the libs into post_mortem
- MINOR: debug: add the ability to enter components in the post_mortem struct
- MINOR: init: add info about the main program to the post_mortem struct
- DOC: management: document "show dev"
- CLEANUP: assorted typo fixes in the code and comments
- CI: limit codespell checks to main repo, not forks
- DOC: 51d: updated 51Degrees repo URL for v3.2.10
- DOC: install: update the list of openssl versions
- MINOR: ext-check: add an option to preserve environment variables
- BUG/MEDIUM: mux-h1: Don't set CO_SFL_MSG_MORE flag on last fast-forward send
- MINOR: rhttp: rename proto_reverse_connect
- MINOR: rhttp: large renaming to use rhttp prefix
- MINOR: rhttp: add count of active conns per thread
- MEDIUM: rhttp: support multi-thread active connect
- MINOR: listener: allow thread kw for rhttp bind
- DOC: rhttp: replace maxconn by nbconn
- MINOR: log/balance: rename "log-sticky" to "sticky"
- MEDIUM: mux-quic: Add consumer-side fast-forwarding support
- MAJOR: h3: Implement zero-copy support to send DATA frame
After giving it some thought, it could pretty well happen that other
protocols benefit from the sticky algorithm that some used to emulate
using a "stick-on int(0)" or things like this previously. So better
rename it to "sticky" right now instead of having to keep that "log-"
prefix forever. It's still limited to logs, of course, only the algo
is renamed in the config.
Usage of existing "maxconn" for rhttp listeners configuration was
replaced recently by a new dedicating "nbconn" keyword. Update the
documentation part to reflect this.
No need to backport.
Thanks to previous commit, a reverse HTTP listener is able to distribute
actively opened connections accross its threads. To be able to exploit
this, allow "thread" keyword for such a listener.
An extra check is added to explicitely forbids a reverse bind to span
multiple thread groups. Without this, multiple listeners instances will
be created, each with its owned "nbconn" value. This may surprise users
so for now, better to deactivate this possibility.
In Github issue #2128, @jvincze84 explained the complexity of using
external checks in some advanced setups due to the systematic purge of
environment variables, and expressed the desire to preserve the
existing environment. During the discussion an agreement was found
around having an option to "external-check" to do that and that
solution was tested and confirmed to work by user @nyxi.
This patch just cleans this up, implements the option as
"preserve-env" and documents it. The default behavior does not change,
the environment is still purged, unless "preserve-env" is passed. The
choice of not using "import-env" instead was made so that we could
later use it to name specific variables that have to be imported
instead of keeping the whole environment.
The patch is simple enough that it could be backported if needed (and
was in fact tested on 2.6 first).
Released version 2.9-dev10 with the following main changes :
- CLEANUP: Re-apply xalloc_size.cocci (3)
- BUG/MEDIUM: stconn: Report send activity during mux-to-mux fast-forward
- BUG/MEDIUM: stconn: Don't report rcv/snd expiration date if SC cannot epxire
- MINOR: stconn: Don't queue stream task in past in sc_notify()
- BUG/MEDIUM: Don't apply a max value on room_needed in sc_need_room()
- BUG/MINOR: stconn: Sanitize report for read activity
- CLEANUP: htx: Properly indent htx_reserve_max_data() function
- DOC: stconn: Improve comments about lra and fsb usage
- BUG/MEDIUM: quic: fix actconn on quic_conn alloc failure
- BUG/MEDIUM: quic: fix sslconns on quic_conn alloc failure
- BUG/MEDIUM: mux-h1: Be sure xprt support splicing to use it during fast-forward
- MINOR: proto_reverse_connect: use connect timeout
- BUG/MINOR: mux-h1: Release empty ibuf during data fast-forwarding
- BUG/MINOR: stick-table/cli: Check for invalid ipv4 key
- MEDIUM: stktable/cli: simplify entry key handling
- MINOR: stktable/cli: support v6tov4 and v4tov6 conversions
- BUG/MINOR: mux-h1: Properly handle http-request and http-keep-alive timeouts
- BUG/MEDIUM: freq-ctr: Don't report overshoot for long inactivity period
- BUG/MEDIUM: pool: fix releasable pool calculation when overloaded
- BUG/MINOR: pool: check one other random bucket on alloc conflict
- BUG/MEDIUM: pool: try once to allocate from another bucket if empty
- MEDIUM: stconn/muxes: Loop on data fast-forwarding to forward at least a buffer
- MINOR: stconn/mux-h2: Use a iobuf flag to report EOI to consumer side during FF
- MEDIUM: quic: Heavy task mode during handshake
- MEDIUM: quic: Heavy task mode with non contiguously bufferized CRYPTO data
- MINOR: quic: release the TLS context asap from quic_conn_release()
- MINOR: quic: Add idle timer task pointer to traces
- BUG/MINOR: quic: idle timer task requeued in the past
- CLEANUP: quic: Indentation fix in qc_do_build_pkt()
- MINOR: quic: Avoid zeroing frame structures
- BUG/MEDIUM: quic: Too short Initial packet sent (enc. level allocation failed)
- BUG/MEDIUM: quic: Avoid trying to send ACK frames from an empty ack ranges tree
- BUG/MEDIUM: quic: Possible crashes when sending too short Initial packets
- BUG/MEDIUM: quic: Avoid some crashes upon TX packet allocation failures
- BUG/MEDIUM: quic: Possible crashes during secrets allocations (heavy load)
- BUG/MEDIUM: stconn: Don't update stream expiration date if already expired
- MINOR: errors: ha_alert() and ha_warning() uses warn_exec_path()
- MINOR: errors: does not check MODE_STARTING for log emission
- MEDIUM: errors: move the MODE_QUIET test in print_message()
- DOC: management: -q is quiet all the time
- MEDIUM: mworker: -W is mandatory when using -S
- BUG/MEDIUM: mux-h1: Exit early if fast-forward is not supported by opposite SC
- MEDIUM: quic: adjust address validation
- MINOR: quic: reduce half open counters scope
- MEDIUM: quic: limit handshake per listener
- MEDIUM: quic: define an accept queue limit
- BUG/MINOR: quic: fix retry token check inconsistency
- MINOR: task/debug: explicitly support passing a null caller to wakeup functions
- MINOR: task/debug: make task_queue() and task_schedule() possible callers
- OPTIM: mux-h2: don't allocate more buffers per connections than streams
- BUG/MINOR: quic: remove dead code in error path
- MEDIUM: quic: respect closing state even on soft-stop
- MEDIUM: quic: release conn socket before using quic_cc_conn
- DOC: config: use the word 'backend' instead of 'proxy' in 'track' description
- BUG/MEDIUM: applet: Remove appctx from buffer wait list on release
- MINOR: tools: make str2sa_range() directly return type hints
- BUG/MEDIUM: server: invalid address (post)parsing checks
- BUG/MINOR: sink: don't learn srv port from srv addr
- CLEANUP: sink: bad indent in sink_new_from_logger()
- CLEANUP: sink: useless leftover in sink_add_srv()
- BUG/MINOR: quic: Useless use of non-contiguous buffer for in order CRYPTO data
- MINOR: server: always initialize pp_tlvs for default servers
- BUG/MEDIUM: proxy: always initialize the default settings after init
- MEDIUM: startup: 'haproxy -c' is quiet when valid
- BUG/MINOR: sample: Fix bytes converter if offset is bigger than sample length
- BUG/MINOR: log: keep the ref in dup_logger()
- BUG/MINOR: quic: fix crash on qc_new_conn alloc failure
- BUG/MINOR: quic: fix decrement of half_open counter on qc alloc failure
- BUG/MEDIUM: quic: fix FD for quic_cc_conn
- DOC: config: Fix name for tune.disable-zero-copy-forwarding global param
- REGTESTS: startup: -conf-OK requires -V with current VTest
- BUG/MEDIUM: quic: Non initialized CRYPTO data stream deferencing
- MINOR: quic: Add a max window parameter to congestion control algorithms
- MINOR: quic: Maximum congestion control window for each algo
- DOC: quic: Wrong syntax for "quic-cc-algo" keyword.
- DOC: quic: Maximum congestion control window configuration
- BUG/MINOR: quic: maximum window limits do not match the doc
- BUG/MEDIUM: connection: report connection errors even when no mux is installed
- BUG/MINOR: stconn: Handle abortonclose if backend connection was already set up
- MINOR: connection: Add a CTL flag to notify mux it should wait for reads again
- MEDIUM: mux-h1: Handle MUX_SUBS_RECV flag in h1_ctl() and susbscribe for reads
- BUG/MEDIUM: stream: Properly handle abortonclose when set on backend only
- MINOR: stconn: Use SC to detect frontend connections in sc_conn_recv()
- REGTESTS: http: Improve script testing abortonclose option
- MINOR: activity: report profiling duration and age in "show profiling"
- BUG/MEDIUM: mworker: set the master variable earlier
- BUG/MEDIUM: stream: Don't call mux .ctl() callback if not implemented
- MINOR: connection: update rhttp flags usage
- BUG/MINOR: mux_h2: reject passive reverse conn if error on add to idle
- MINOR: server: force add to idle on reverse
- MINOR: shctx: Set last_append to NULL when reserving block in hot list
- MEDIUM: shctx: Move list between hot and avail list in O(1)
- MEDIUM: shctx: Simplify shctx_row_reserve_hot loop
- MINOR: shctx: Remove explicit 'from' param from shctx_row_data_append
- MEDIUM: cache: Use dedicated cache tree lock alongside shctx lock
- MINOR: cache: Remove expired entry delete in "show cache" command
- MINOR: cache: Add option to avoid removing expired entries in lookup function
- MEDIUM: cache: Use rdlock on cache in cache_use
- MEDIUM: shctx: Remove 'hot' list from shared_context
- MINOR: cache: Use dedicated trash for "show cache" cli command
- MEDIUM: cache: Switch shctx spinlock to rwlock and restrict its scope
- MEDIUM: cache: Add refcount on cache_entry
- MEDIUM: shctx: Descend shctx_lock calls into the shctx_row_reserve_hot
- MINOR: shctx: Add new reserve_finish callback call to shctx_row_reserve_hot
- MAJOR: cache: Delay cache entry delete in reserve_hot function
- MINOR: shctx: Remove redundant arg from free_block callback
- MINOR: shctx: Remove 'use_shared_mem' variable
- DOC: cache: Specify when function expects a cache lock
- BUG/MEDIUM: stconn: Update fsb date on partial sends
- MINOR: htx: Use a macro for overhead induced by HTX
- MINOR: channel: Add functions to get info on buffers and deal with HTX streams
- BUG/MINOR: stconn: Fix streamer detection for HTX streams
- BUG/MINOR: stconn: Use HTX-aware channel's functions to get info on buffer
- BUG/MINOR: stconn/applet: Report send activity only if there was output data
- BUG/MINOR: stconn: Report read activity on non-indep streams for partial sends
- BUG/MINOR: shctx: Remove old HA_SPIN_INIT
- REGTESTS: try to activate again the seamless reload test with the master CLI
- MINOR: proxy: Add "handshake" new timeout (frontend side)
- MEDIUM: quic: Add support for "handshake" timeout setting.
- MINOR: quic: Dump the expiration date of the idle timer task
- BUG/MINOR: quic: Malformed CONNECTION_CLOSE frame
- MEDIUM: session: handshake timeout (TCP)
- DOC: proxy: Add "handshake" timeout documentation.
- MINOR: quic: Rename "handshake" timeout to "client-hs"
- CLEANUP: haproxy: remove old comment from 1.1 from the file header
- BUG/MEDIUM: mux-h2: fail earlier on malloc in takeover()
- BUG/MEDIUM: mux-h1: fail earlier on malloc in takeover()
- BUG/MEDIUM: mux-fcgi: fail earlier on malloc in takeover()
- MINOR: rhttp: remove the unused outgoing connect() function
- MINOR: backend: without ->connect(), allow to pick another thread's connection
- BUG/MINOR: stream/cli: report correct stream age in "show sess"
- MINOR: stream/cli: add an optional "older" filter for "show sess"
- MINOR: stream/cli: add another filter "susp" to "show sess"
- MINOR: stktable: add stktable_deinit function
- BUG/MINOR: proxy/stktable: missing frees on proxy cleanup
- CLEANUP: backend: removing unused LB param
- MEDIUM: lbprm: store algo params on 32bits
- MEDIUM: log/balance: merge tcp/http algo with log ones
- Revert "MINOR: proxy: report a warning for max_ka_queue in proxy_cfg_ensure_no_http()"
- Revert "MINOR: tcp_rules: tcp-{request,response} requires TCP or HTTP mode"
- Revert "MINOR: stktable: "stick" requires TCP or HTTP mode"
- Revert "MINOR: cfgparse-listen: "http-send-name-header" requires TCP or HTTP mode"
- Revert "MINOR: cfgparse-listen: "dynamic-cookie-key" requires TCP or HTTP mode"
- Revert "MINOR: cfgparse-listen: "http-reuse" requires TCP or HTTP mode"
- Revert "MINOR: fcgi-app: "use-fcgi-app" requires TCP or HTTP mode"
- Revert "MINOR: http_htx/errors: prevent the use of some keywords when not in tcp/http mode"
- Revert "MINOR: flt_http_comp: "compression" requires TCP or HTTP mode"
- Revert "MINOR: filter: "filter" requires TCP or HTTP mode"
- MINOR: log/backend: ensure log exclusive params are not used in other modes
- MINOR: log/backend: prevent tcp-{request,response} use with LOG mode
- MINOR: log/backend: prevent stick table and stick rules with LOG mode
- MINOR: log/backend: prevent "http-send-name-header" use with LOG mode
- MINOR: log/backend: prevent "dynamic-cookie-key" use with LOG mode
- REGTESTS: http: add a test to validate chunked responses delivery
"log-balance" directive was recently introduced to configure the
balancing algorithm to use when in a log backend. However, it is
confusing and it causes issues when used in default section.
In this patch, we take another approach: first we remove the
"log-balance" directive, and instead we rely on existing "balance"
directive to configure log load balancing in log backend.
Some algorithms such as roundrobin can be used as-is in a log backend,
and for log-only algorithms, they are implemented as "log-$name" inside
the "backend" directive.
The documentation was updated accordingly.
This one reports streams considered as "suspicious", i.e. those with
no expiration dates or dates in the past, or those without a front
endpoint. More criteria could be added in the future.
It's often needed to be able to refine "show sess" when debugging, and
very often a first glance at old streams is performed, but that's a
difficult task in large dumps, and it takes lots of resources to dump
everything.
This commit adds "older <age>" to "show sess" in order to specify the
minimum age of streams that will be dumped. This should simplify the
identification of blocked ones.
Document the "handshake" timeout new setting available one frontend side.
This should at least be helpful for QUIC client connections to prevent
an attacker from refreshing plenty of connections without completing
the handshake step, leading haproxy to consume memory for nothing.
"disable-" prefix was missing. This param was correctly named in the list of
supported keywords in the global section, but not in the keyword
description.
No backport needed.
User @nwehrman reported in issue #2328 that the used of "proxy" instead
of "backend" in the argument of the "track" server keyword is confusing.
Admittedly, all other places in the doc use "backend/server" instead of
"proxy/server", so let's update it for the sake of consistency.
QUIC connections are pushed manually into a dedicated listener queue
when they are ready to be accepted. This happens after handshake
finalization or on 0-RTT packet reception. Listener is then woken up to
dequeue them with listener_accept().
This patch comptabilizes the number of connections currently stored in
the accept queue. If reaching a certain limit, INITIAL packets are
dropped on reception to prevent further QUIC connections allocation.
This should help to preserve system resources.
This limit is automatically derived from the listener backlog. Half of
its value is reserved for handshakes and the other half for accept
queues. By default, backlog is equal to maxconn which guarantee that
there can't be no more than maxconn connections in handshake or waiting
to be accepted.
Implement a limit per listener for concurrent number of QUIC
connections. When reached, INITIAL packets for new connections are
automatically dropped until the number of handshakes is reduced.
The limit value is automatically based on listener backlog, which itself
defaults to maxconn.
This feature is important to ensure CPU and memory resources are not
consume if too many handshakes attempt are started in parallel.
Special care is taken if a connection is released before handshake
completion. In this case, counter must be decremented. This forces to
ensure that member <qc.state> is set early in qc_new_conn() before any
quic_conn_release() invocation.
The documentation about -q seems wrong, it does not output messages
after the startup, it disables all messages. It was always quiet with
the stdio_quiet() function.
Must be backported in all stable versions.
Released version 2.9-dev9 with the following main changes :
- DOC: internal: filters: fix reference to entities.pdf
- BUG/MINOR: ssl: load correctly @system-ca when ca-base is define
- MINOR: lua: Add flags to configure logging behaviour
- MINOR: lua: change tune.lua.log.stderr default from 'on' to 'auto'
- BUG/MINOR: backend: fix wrong BUG_ON for avail conn
- BUG/MAJOR: backend: fix idle conn crash under low FD
- MINOR: backend: refactor insertion in avail conns tree
- DEBUG: mux-h2/flags: fix list of h2c flags used by the flags decoder
- BUG/MEDIUM: server/log: "mode log" after server keyword causes crash
- MINOR: connection: add conn_pr_mode_to_proto_mode() helper func
- BUG/MEDIUM: server: "proto" not working for dynamic servers
- MINOR: server: add helper function to detach server from proxy list
- DEBUG: add a tainted flag when ha_panic() is called
- DEBUG: lua: add tainted flags for stuck Lua contexts
- DEBUG: pools: detect that malloc_trim() is in progress
- BUG/MINOR: quic: do not consider idle timeout on CLOSING state
- MINOR: frontend: implement a dedicated actconn increment function
- BUG/MINOR: ssl: use a thread-safe sslconns increment
- MEDIUM: quic: count quic_conn instance for maxconn
- MEDIUM: quic: count quic_conn for global sslconns
- BUG/MINOR: ssl: suboptimal certificate selection with TLSv1.3 and dual ECDSA/RSA
- REGTESTS: ssl: update the filters test for TLSv1.3 and sigalgs
- BUG/MINOR: mux-quic: fix early close if unset client timeout
- BUG/MEDIUM: ssl: segfault when cipher is NULL
- BUG/MINOR: tcpcheck: Report hexstring instead of binary one on check failure
- MEDIUM: systemd: be more verbose about the reload
- MINOR: sample: Add fetcher for getting all cookie names
- BUG/MINOR: proto_reverse_connect: support SNI on active connect
- MINOR: proxy/stktable: add resolve_stick_rule helper function
- BUG/MINOR: stktable: missing free in parse_stick_table()
- BUG/MINOR: cfgparse/stktable: fix error message on stktable_init() failure
- MINOR: stktable: stktable_init() sets err_msg on error
- MINOR: stktable: check if a type should be used as-is
- MEDIUM: stktable/peers: "write-to" local table on peer updates
- CI: github: update wolfSSL to 5.6.4
- DOC: install: update the wolfSSL required version
- MINOR: server: Add parser support for set-proxy-v2-tlv-fmt
- MINOR: connection: Send out generic, user-defined server TLVs
- BUG/MEDIUM: pattern: don't trim pools under lock in pat_ref_purge_range()
- MINOR: mux-h2: always use h2_send() in h2_done_ff(), not h2_process()
- OPTIM: mux-h2: call h2_send() directly from h2_snd_buf()
- BUG/MINOR: server: remove some incorrect free() calls on null elements
To follow-up the implementation of the new set-proxy-v2-tlv-fmt
keyword in the server, the connection is updated to use the previously
allocated TLVs. If no value was specified, we send out an empty TLV.
As the feature is fully working with this commit, documentation and a
test for the server and default-server are added as well.
In this patch, we add the possibility to declare on a table definition
("table" in peer section, or "stick-table" in proxy section) that we
want the remote/peer updates on that table to be pushed on a local
haproxy table in addition to the source table.
Consider this example:
|peers mypeers
| peer local 127.0.0.1:3334
| peer clust 127.0.0.1:3333
| table t1.local type string size 10m store server_id,server_key expire 30s
| table t1.clust type string size 10m store server_id,server_key write-to mypeers/t1.local expire 30s
With this setup, we consider haproxy uses t1.local as cache/local table
for read and write operations, and that t1.clust is a remote table
containing datas processed from t1.local and similar tables from other
haproxy peers in a cluster setup. The t1.clust table will be used to
refresh the local/cache one via the "write-to" statement.
What will happen, is that every time haproxy will see entry updates for
the t1.clust table: it will overwrite t1.local table with fresh data and
will update the entry expiration timer. If t1.local entry doesn't exist
yet (key doesn't exist), it will automatically create it. Note that only
types that cannot be used for arithmetic ops will be handled, and this
to prevent processed values from the remote table from interfering with
computations based on values from the local table. (ie: prevent
cumulative counters from growing indefinitely).
"write-to" will only push supported types if they both exist in the source
and the target table. Be careful with server_id and server_key storage
because they are often declared implicitly when referencing a table in
sticking rules but it is required to declare them explicitly for them to
be pushed between a remote and a local table through "write-to" option.
Also note that the "write-to" target table should have the same type as
the source one, and that the key length should be strictly equal,
otherwise haproxy will raise an error due to the tables being
incompatibles. A table that is already being written to cannot be used
as a source table for a "write-to" target.
Thanks to this patch, it will now be possible to use sticking rules in
peer cluster context by using a local table as a local cache which
will be automatically refreshed by one or multiple remote table(s).
This commit depends on:
- "MINOR: stktable: stktable_init() sets err_msg on error"
- "MINOR: stktable: check if a type should be used as-is"
This new fetcher can be used to extract the list of cookie names from
Cookie request header or from Set-Cookie response header depending on
the stream direction. There is an optional argument that can be used
as the delimiter (which is assumed to be the first character of the
argument) between cookie names. The default delimiter is comma (,).
Note that we will treat the Cookie request header as a semi-colon
separated list of cookies and each Set-Cookie response header as
a single cookie and extract the cookie names accordingly.
After making it configurable in previous commit "MINOR: lua: Add flags
to configure logging behaviour", this patch changes the default value
of tune.lua.log.stderr from 'on' (unconditionally forward LUA logs to
stderr) to 'auto' (only forward LUA logs to stderr if logging via a
standard logger is disabled, or none is configured for the current context)
Since this is a change in behaviour, it shouldn't be backported
Until now, messages printed from LUA log functions were sent both to
the any logger configured for the current proxy, and additionally to
stderr (in most cases)
This introduces two flags to configure LUA log handling:
- tune.lua.log.loggers to use standard loggers or not
- tune.lua.log.stderr to use stderr, or not, or only conditionally
This addresses github feature request #2316
This can be backported to 2.8 as it doesn't change previous behaviour.
Released version 2.9-dev8 with the following main changes :
- MINOR: ssl: add an explicit error when 'ciphersuites' are not supported
- BUILD: ssl: enable 'ciphersuites' for WolfSSL
- BUILD: ssl: add 'ssl_c_r_dn' fetch for WolfSSL
- BUILD: ssl: add 'secure_memcmp' converter for WolfSSL and awslc
- BUILD: ssl: enable keylog for awslc
- CLEANUP: ssl: remove compat functions for openssl < 1.0.0
- BUILD: ssl: enable keylog for WolfSSL
- REGTESTS: pki: add a pki for SSL tests
- REGTESTS: ssl: update common.pem with the new pki
- REGTESTS: ssl: disable ssl_dh.vtc for WolfSSL
- REGTESTS: wolfssl: temporarly disable some failing reg-tests
- CI: ssl: add wolfssl to build-ssl.sh
- CI: ssl: add git id support for wolfssl download
- CI: github: add a wolfssl entry to the CI
- CI: github: update wolfssl to git revision d83f2fa
- CI: github: add awslc 1.16.0 to the push CI
- BUG/MINOR: quic: Avoid crashing with unsupported cryptographic algos
- REORG: quic: cleanup traces definition
- BUG/MINOR: quic: reject packet with no frame
- BUG/MEDIUM: mux-quic: fix RESET_STREAM on send-only stream
- BUG/MINOR: mux-quic: support initial 0 max-stream-data
- BUG/MINOR: h3: strengthen host/authority header parsing
- CLEANUP: connection: drop an uneeded leftover cast
- BUG/MAJOR: connection: make sure to always remove a connection from the tree
- BUG/MINOR: quic: fix qc.cids access on quic-conn fail alloc
- BUG/MINOR: quic: fix free on quic-conn fail alloc
- BUG/MINOR: mux-quic: fix free on qcs-new fail alloc
- BUG/MEDIUM: quic-conn: free unsent frames on retransmit to prevent crash
- MEDIUM: tree-wide: logsrv struct becomes logger
- MEDIUM: log: introduce log target
- DOC: config: log <address> becomes log <target> in "log" related doc
- MEDIUM: sink/log: stop relying on AF_UNSPEC for rings
- MINOR: log: support explicit log target as argument in __do_send_log()
- MINOR: log: remove the logger dependency in do_send_log()
- MEDIUM: log/sink: simplify log header handling
- MEDIUM: sink: inherit from caller fmt in ring_write() when rings didn't set one
- MINOR: sink: add sink_new_from_srv() function
- MAJOR: log: introduce log backends
- MINOR: log/balance: support for the "sticky" lb algorithm
- MINOR: log/balance: support for the "random" lb algorithm
- MINOR: lbprm: support for the "none" hash-type function
- MINOR: lbprm: compute the hash avalanche in gen_hash()
- MINOR: sample: add sample_process_cnv() function
- MEDIUM: log/balance: support for the "hash" lb algorithm
- REGTEST: add a test for log-backend used as a log target
- MINOR: server: introduce "log-bufsize" kw
- BUG/MEDIUM: stconn: Report a send activity everytime data were sent
- BUG/MEDIUM: applet: Report a send activity everytime data were sent
- BUG/MINOR: mux-h1: Send a 400-bad-request on shutdown before the first request
- MINOR: support for http-response set-timeout
- BUG/MINOR: mux-h2: make up other blocked streams upon removal from list
- DEBUG: pool: store the memprof bin on alloc() and update it on free()
- BUG/MEDIUM: quic_conn: let the scheduler kill the task when needed
- CLEANUP: hlua: Remove dead-code on error path in hlua_socket_new()
- BUG/MEDIUM: mux-h1: do not forget TLR/EOT even when no data is sent
- BUG/MINOR: htpp-ana/stats: Specify that HTX redirect messages have a C-L header
- BUG/MEDIUM: mux-h2: Don't report an error on shutr if a shutw is pending
- MEDIUM: stconn/channel: Move pipes used for the splicing in the SE descriptors
- MINOR: stconn: Start to introduce mux-to-mux fast-forwarding notion
- MINOR: stconn: Extend iobuf to handle a buffer in addition to a pipe
- MINOR: connection: Add new mux callbacks to perform data fast-forwarding
- MINOR: stconn: Temporarily remove kernel splicing support
- MINOR: mux-pt: Temporarily remove splicing support
- MINOR: mux-h1: Temporarily remove splicing support
- MINOR: connection: Remove mux callbacks about splicing
- MEDIUM: stconn: Add mux-to-mux fast-forward support
- MINOR: mux-h1: Use HTX extra field only for responses with known length
- MEDIUM: mux-h1: Properly handle state transitions of chunked outgoing messages
- MEDIUM: raw-sock: Specifiy amount of data to send via snd_pipe callback
- MINOR: mux-h1: Add function to add size of a chunk to an outgoind message
- MEDIUM: mux-h1: Simplify zero-copy on sending path
- MEDIUM: mux-h1: Simplify payload formatting based on HTX blocks on sending path
- MEDIUM: mux-h1: Add fast-forwarding support
- MINOR: h2: Set the BODYLESS_RESP flag on the HTX start-line if necessary
- MEDIUM: mux-h2: Add consumer-side fast-forwarding support
- MEDIUM: channel: don't look at iobuf to report an empty channel
- MINOR: tree-wide: Only rely on co_data() to check channel emptyness
- REGTESTS: Reenable HTTP tests about splicing
- CLEAN: mux-h1: Remove useless __maybe_unused attribute on h1_make_chunk()
- MEDIUM: mux-pt: Add fast-forwarding support
- MINOR: global: Add an option to disable the zero-copy forwarding
- BUILD: mux-h1: Fix build without kernel splicing support
- REORG: stconn/muxes: Rename init step in fast-forwarding
- MINOR: dgram: allow to set rcv/sndbuf for dgram sockets as well
- BUG/MINOR: mux-h2: fix http-request and http-keep-alive timeouts again
- BUG/MINOR: trace: fix trace parser error reporting
- BUG/MEDIUM: peers: Be sure to always refresh recconnect timer in sync task
- BUG/MEDIUM: peers: Fix synchro for huge number of tables
- MINOR: cfgparse: forbid mixing reverse and standard listeners
- MINOR: listener: add nbconn kw for reverse connect
- MINOR: server: convert @reverse to rev@ standard format
- MINOR: cfgparse: rename "rev@" prefix to "rhttp@"
- REGTESTS: remove maxconn from rhttp bind line
- MINOR: listener: forbid most keywords for reverse HTTP bind
- MINOR: sample: Added support for Arrays in sample_conv_json_query in sample.c
- MINOR: mux-h2/traces: explicitly show the error/refused stream states
- MINOR: mux-h2/traces: clarify the "rejected H2 request" event
- BUG/MINOR: mux-h2: commit the current stream ID even on reject
- BUG/MINOR: mux-h2: update tracked counters with req cnt/req err
Method now returns the content of Json Arrays, if it is specified in
Json Path as String. The start and end character is a square bracket. Any
complex object in the array is returned as Json, so that you might get Arrays
of Array or objects. Only recommended for Arrays of simple types (e.g.,
String or int) which will be returned as CSV String. Also updated
documentation and fixed issue with parenthesis and other changes from
comments.
This patch was discussed in issue #2281.
Signed-off-by: William Lallemand <wlallemand@haproxy.com>
Previously, maxconn keyword was reused for a specific usage on reverse
HTTP binds to specify the number of active connect to proceed. To avoid
confusion, introduce a new dedicated keyword 'nbconn' which is specific
to reverse HTTP bind.
This new keyword is forbidden for non-reverse listener. A fatal error is
emitted during config parsing if this rule is not respected. It's safe
because it's also forbidden to mix standard and reverse addresses on the
same bind line.
Internally, nbconn value will be reassigned to 'maxconn' member of
bind_conf structure. This ensures that listener layer will automatically
reenable the preconnect task each time a connection is closed.
tune.rcvbuf.client and tune.rcvbuf.server are not suitable for shared
dgram sockets because they're per connection so their units are not the
same. However, QUIC's listener and log servers are not connected and
take per-thread or per-process traffic where a socket log buffer might
be too small, causing undesirable packet losses and retransmits in the
case of QUIC. This essentially manifests in listener mode with new
connections taking a lot of time to set up under heavy traffic due to
the small queues causing delays. Let's add a few new settings allowing
to set these shared socket sizes on the frontend and backend side (which
reminds that these are per-front/back and not per client/server hence
not per connection).
The zero-copy forwarding or the mux-to-mux forwarding is a way to
fast-forward data without using the channels buffers. Data are transferred
from a mux to the other one. The kernel splicing is an optimization of the
zero-copy forwarding. But it can also use normal buffers (but not channels
ones). This way, it could be possible to fast-forward data with muxes not
supporting the kernel splicing (H2 and H3 muxes) but also with applets.
However, this mode can introduce regressions or bugs in future (just like
the kernel splicing). Thus, It could be usefull to disable this optim. To do
so, in configuration, the global tune settting
'tune.disable-zero-copy-forwarding' may be set in a global section or the
'-dZ' command line parameter may be used to start HAProxy. Of course, this
also disables the kernel splicing.
"log-bufsize" may now be used for a log server (in a log backend) to
configure the bufsize of implicit ring associated to the server (which
defaults to BUFSIZE).
hash lb algorithm can be configured with the "log-balance hash <cnv_list>"
directive. With this algorithm, the user specifies a converter list with
<cnv_list>.
The produced log message will be passed as-is to the provided converter
list, and the resulting hash will be used to select the log server that
will receive the log message.
Allow the use of the "none" hash-type function so that the key resulting
from the sample expression is directly used as the hash.
This can be useful to do the hashing manually using available hashing
converters, or even custom ones, and then inform haproxy that it can
directly rely on the sample expression result which is explictly handled
as an integer in this case.
In this patch we add basic support for the random algorithm:
random algorithm picks a random server using the result of the
statistical_prng() function as if it was a hash key to then compute the
related server ID.
There is no support for the <draw> parameter (which is implemented for
tcp/http load-balancing), because we don't have the required metrics to
evaluate server's load in log backends for the moment. Plus it would add
more complexity to the __do_send_log_backend() function so we'll keep it
this way for now but this might be needed in the future.
sticky algorithm always tries to send log messages to the first server in
the farm. The server will stay in front during queue and dequeue
operations (no other server can steal its place), unless it becomes
unavailable, in which case it will be replaced by another server from
the tree.
Using "mode log" in a backend section turns the proxy in a log backend
which can be used to log-balance logs between multiple log targets
(udp or tcp servers)
log backends can be used as regular log targets using the log directive
with "backend@be_name" prefix, like so:
| log backend@mybackend local0
A log backend will distribute log messages to servers according to the
log load-balancing algorithm that can be set using the "log-balance"
option from the log backend section. For now, only the roundrobin
algorithm is supported and set by default.
This is a follow up of the previous commit to emphasize that "log"
directive allows to provide a log target which may directly be a server
address but may also be a log transport facility such as rings. Thus we
use the term "target" instead of "address" to make it more generic.
Released version 2.9-dev7 with the following main changes :
- MINOR: support for http-request set-timeout client
- BUG/MINOR: mux-quic: remove full demux flag on ncbuf release
- CLEANUP: freq_ctr: make all freq_ctr readers take a const
- CLEANUP: stream: make the dump code not depend on the CLI appctx
- MINOR: stream: split stats_dump_full_strm_to_buffer() in two
- CLEANUP: stream: use const filters in the dump function
- CLEANUP: stream: make strm_dump_to_buffer() take a const stream
- MINOR: stream: make strm_dump_to_buffer() take an arbitrary buffer
- MINOR: stream: make strm_dump_to_buffer() show the list of filters
- MINOR: stream: make stream_dump() always multi-line
- MINOR: streams: add support for line prefixes to strm_dump_to_buffer()
- MEDIUM: stream: now provide full stream dumps in case of loops
- MINOR: debug: use the more detailed stream dump in panics
- CLEANUP: stream: remove the now unused stream_dump() function
- Revert "BUG/MEDIUM: quic: missing check of dcid for init pkt including a token"
- MINOR: stream: fix output alignment of stuck thread dumps
- BUG/MINOR: proto_reverse_connect: fix FD leak on connection error
- BUG/MINOR: tcp_act: fix attach-srv rule ACL parsing
- MINOR: connection: define error for reverse connect
- MINOR: connection: define mux flag for reverse support
- MINOR: tcp_act: remove limitation on protocol for attach-srv
- BUG/MINOR: proto_reverse_connect: fix FD leak upon connect
- BUG/MAJOR: plock: fix major bug in pl_take_w() introduced with EBO
- Revert "MEDIUM: sample: Small fix in function check_operator for eror reporting"
- DOC: sample: Add a comment in 'check_operator' to explain why 'vars_check_arg' should ignore the 'err' buffer
- DEV: sslkeylogger: handle file opening error
- MINOR: quic: define quic-socket bind setting
- MINOR: quic: handle perm error on bind during runtime
- MINOR: backend: refactor specific source address allocation
- MINOR: proto_reverse_connect: support source address setting
- BUILD: pool: Fix GCC error about potential null pointer dereference
- MINOR: hlua: Set context's appctx when the lua socket is created
- MINOR: hlua: Don't preform operations on a not connected socket
- MINOR: hlua: Save the lua socket's timeout in its context
- MINOR: hlua: Save the lua socket's server in its context
- MINOR: hlua: Test the hlua struct first when the lua socket is connecting
- BUG/MEDIUM: hlua: Initialize appctx used by a lua socket on connect only
- DEBUG: mux-h1: Fix event label from trace messages about payload formatting
- BUG/MINOR: mux-h1: Handle read0 in rcv_pipe() only when data receipt was tried
- BUG/MINOR: mux-h1: Ignore C-L when sending H1 messages if T-E is also set
- BUG/MEDIUM: h1: Ignore C-L value in the H1 parser if T-E is also set
- REGTESTS: filters: Don't set C-L header in the successful response to CONNECT
- MINOR: mux-h1: Add flags if outgoing msg contains a header about its payload
- MINOR: mux-h1: Rely on H1S_F_HAVE_CHNK to add T-E in outgoing messages
- BUG/MEDIUM: mux-h1: Add C-L header in outgoing message if it was removed
- BUG/MEDIUM: mux-h1; Ignore headers modifications about payload representation
- BUG/MINOR: h1-htx: Keep flags about C-L/T-E during HEAD response parsing
- MINOR: h1-htx: Declare successful tunnel establishment as bodyless
- BUILD: quic: allow USE_QUIC to work with AWSLC
- CI: github: add USE_QUIC=1 to aws-lc build
- BUG/MINOR: hq-interop: simplify parser requirement
- MEDIUM: cache: Add "Origin" header to secondary cache key
- MINOR: haproxy: permit to register features during boot
- MINOR: tcp_rules: tcp-{request,response} requires TCP or HTTP mode
- MINOR: stktable: "stick" requires TCP or HTTP mode
- MINOR: filter: "filter" requires TCP or HTTP mode
- MINOR: backend/balance: "balance" requires TCP or HTTP mode
- MINOR: flt_http_comp: "compression" requires TCP or HTTP mode
- MINOR: http_htx/errors: prevent the use of some keywords when not in tcp/http mode
- MINOR: fcgi-app: "use-fcgi-app" requires TCP or HTTP mode
- MINOR: cfgparse-listen: "http-send-name-header" requires TCP or HTTP mode
- MINOR: cfgparse-listen: "dynamic-cookie-key" requires TCP or HTTP mode
- MINOR: proxy: dynamic-cookie CLIs require TCP or HTTP mode
- MINOR: cfgparse-listen: "http-reuse" requires TCP or HTTP mode
- MINOR: proxy: report a warning for max_ka_queue in proxy_cfg_ensure_no_http()
- MINOR: cfgparse-listen: warn when use-server rules is used in wrong mode
- DOC: config: unify "log" directive doc
- MINOR: sink/log: fix some typos around postparsing logic
- MINOR: sink: remove useless check after sink creation
- MINOR: sink: don't rely on p->parent in sink appctx
- MINOR: sink: don't rely on forward_px to init sink forwarding
- MINOR: sink: refine forward_px usage
- MINOR: sink: function to add new sink servers
- BUG/MEDIUM: stconn: Fix comparison sign in sc_need_room()
- BUG/MEDIUM: actions: always apply a longest match on prefix lookup
"log" directive description was found 2 times in the configuration file:
First, in 3.1 in the "global parameters" chapter, and then in 4.2 in the
per-proxy keyword options.
Both descriptions are almost identical: having to maintain the "same"
documentation in 2 different places is error-prone. Due to this, some
precisions have been added in one of them, and were missing from
the other, and vice-versa, probably because one didn't see that the
"log" directive was also documented elsewhere.
To prevent the 2 descriptions from further diverging, and make it easier
to maintain, we merge them in the per-proxy "log" directive description
(in 4.2 chapter), and we add a pointer to it in the global "log" to
encourage the user to refer to the per-proxy "log" documentation for
usage details.
This patch add a hash of the Origin header to the cache's secondary key.
This enables to manage store responses that have a "Vary: Origin" header
in the cache when vary is enabled.
This cannot be considered as a means to manage CORS requests though, it
only processes the Origin header and hashes the presented value without
any form of URI normalization.
This need was expressed by Philipp Hossner in GitHub issue #251.
Co-Authored-by: Philipp Hossner <philipp.hossner@posteo.de>
Define a new bind option quic-socket :
quic-socket [ connection | listener ]
This new setting works in conjunction with the existing configuration
global tune.quic.socket-owner and reuse the same semantics.
The purpose of this setting is to allow to disable connection socket
usage on listener instances individually. This will notably be useful
when needing to deactivating it when encountered a fatal permission
error on bind() at runtime.
Added set-timeout for frontend side of session, so it can be used to set
custom per-client timeouts if needed. Added cur_client_timeout to fetch
client timeout samples.
Released version 2.9-dev6 with the following main changes :
- BUG/MINOR: quic: fdtab array underflow access
- DEBUG: pools: always record the caller for uncached allocs as well
- DEBUG: pools: pass the caller pointer to the check functions and macros
- DEBUG: pools: make pool_check_pattern() take a pointer to the pool
- DEBUG: pools: inspect pools on fatal error and dump information found
- BUG/MEDIUM: quic: quic_cc_conn ->cntrs counters unreachable
- DEBUG: pools: also print the item's pointer when crashing
- DEBUG: pools: also print the value of the tag when it doesn't match
- DEBUG: pools: print the contents surrounding the expected tag location
- MEDIUM: pools: refine pool size rounding
- BUG/MEDIUM: hlua: don't pass stale nargs argument to lua_resume()
- BUG/MINOR: hlua/init: coroutine may not resume itself
- BUG/MEDIUM: mux-fcgi: Don't swap trash and dbuf when handling STDERR records
- BUG/MINOR: promex: fix backend_agg_check_status
- BUG/MEDIUM: master/cli: Pin the master CLI on the first thread of the group 1
- MAJOR: import: update mt_list to support exponential back-off
- CLEANUP: pools: simplify the pool expression when no pool was matched in dump
- MINOR: samples: implement bytes_in and bytes_out samples
- DOC: configuration: add %[req.ver] sample to %HV
- BUG/MINOR: quic: Leak of frames to send.
- DOC: configuration: add %[query] to %HQ
- BUG/MINOR: freq_ctr: fix possible negative rate with the scaled API
- BUG/MAJOR: mux-h2: Report a protocol error for any DATA frame before headers
- BUILD: quic: fix build on centos 8 and USE_QUIC_OPENSSL_COMPAT
- Revert "MAJOR: import: update mt_list to support exponential back-off"
- BUG/MINOR: server: add missing free for server->rdr_pfx
- REGTESTS: ssl: skip OCSP test w/ WolfSSL
- REGTESTS: ssl: skip generate-certificates test w/ wolfSSL
- MINOR: logs: clarify the check of the log range
- MINOR: log: remove the unused curr_idx in struct smp_log_range
- CLEANUP: logs: rename a confusing local variable "curr_rg" to "smp_rg"
- MINOR: logs: use a single index to store the current range and index
- MEDIUM: logs: atomically check and update the log sample index
- CLEANUP: ring: rename the ring lock "RING_LOCK" instead of "LOGSRV_LOCK"
- BUG/MEDIUM: http-ana: Try to handle response before handling server abort
- MEDIUM: tools/ip: v4tov6() and v6tov4() rework
- MINOR: pattern/ip: offload ip conversion logic to helper functions
- MINOR: pattern: fix pat_{parse,match}_ip() function comments
- MINOR: pattern/ip: simplify pat_match_ip() function
- BUG/MEDIUM: server/cli: don't delete a dynamic server that has streams
- MINOR: hlua: Add support for the "http-after-res" action
- BUG/MINOR: proto_reverse_connect: fix preconnect with startup name resolution
- MINOR: proto_reverse_connect: prevent transparent server for pre-connect
- CI: cirrus-ci: display gdb bt if any
- MEDIUM: sample: Enhances converter "bytes" to take variable names as arguments
- MEDIUM: sample: Small fix in function check_operator for eror reporting
- MINOR: quic: handle external extra CIDs generator.
- BUG/MINOR: proto_reverse_connect: set default maxconn
- MINOR: proto_reverse_connect: refactor preconnect failure
- MINOR: proto_reverse_connect: remove unneeded wakeup
- MINOR: proto_reverse_connect: emit log for preconnect
Prior to this commit, converter "bytes" takes only integer values as
arguments. After this commit, it can take variable names as inputs.
This allows us to dynamically determine the offset/length and capture
them in variables. These variables can then be used with the converter.
Example use case: parsing a token present in a request header.
This commit introduces support for the "http-after-res" action in
hlua, enabling the invocation of a Lua function in a
"http-after-response" rule. With this enhancement, a Lua action can be
registered using the "http-after-res" action type:
core.register_action('myaction', {'http-after-res'}, myaction)
A new "lua.myaction" is created and can be invoked in a
"http-after-response" rule:
http-after-response lua.myaction
This addition provides greater flexibility and extensibility in
handling post-response actions using Lua.
This commit depends on:
- 4457783 ("MINOR: http_ana: position the FINAL flag for http_after_res execution")
Signed-off-by: Sébastien Gross <sgross@haproxy.com>
This reverts commit c618ed5ff4.
The list iterator is broken. As found by Fred, running QUIC single-
threaded shows that only the first connection is accepted because the
accepter relies on the element being initialized once detached (which
is expected and matches what MT_LIST_DELETE_SAFE() used to do before).
However while doing this in the quic_sock code seems to work, doing it
inside the macro show total breakage and the unit test doesn't work
anymore (random crashes). Thus it looks like the fix is not trivial,
let's roll this back for the time it will take to fix the loop.
The new mt_list code supports exponential back-off on conflict, which
is important for use cases where there is contention on a large number
of threads. The API evolved a little bit and required some updates:
- mt_list_for_each_entry_safe() is now in upper case to explicitly
show that it is a macro, and only uses the back element, doesn't
require a secondary pointer for deletes anymore.
- MT_LIST_DELETE_SAFE() doesn't exist anymore, instead one just has
to set the list iterator to NULL so that it is not re-inserted
into the list and the list is spliced there. One must be careful
because it was usually performed before freeing the element. Now
instead the element must be nulled before the continue/break.
- MT_LIST_LOCK_ELT() and MT_LIST_UNLOCK_ELT() have always been
unclear. They were replaced by mt_list_cut_around() and
mt_list_connect_elem() which more explicitly detach the element
and reconnect it into the list.
- MT_LIST_APPEND_LOCKED() was only in haproxy so it was left as-is
in list.h. It may however possibly benefit from being upstreamed.
This required tiny adaptations to event_hdl.c and quic_sock.c. The
test case was updated and the API doc added. Note that in order to
keep include files small, the struct mt_list definition remains in
list-t.h (par of the internal API) and was ifdef'd out in mt_list.h.
A test on QUIC with both quictls 1.1.1 and wolfssl 5.6.3 on ARM64 with
80 threads shows a drastic reduction of CPU usage thanks to this and
the refined memory barriers. Please note that the CPU usage on OpenSSL
3.0.9 is significantly higher due to the excessive use of atomic ops
by openssl, but 3.1 is only slightly above 1.1.1 though:
- before: 35 Gbps, 3.5 Mpps, 7800% CPU
- after: 41 Gbps, 4.2 Mpps, 2900% CPU
Released version 2.9-dev5 with the following main changes :
- BUG/MEDIUM: mux-h2: fix crash when checking for reverse connection after error
- BUILD: import: guard plock.h against multiple inclusion
- BUILD: pools: import plock.h to build even without thread support
- BUG/MINOR: ssl/cli: can't find ".crt" files when replacing a certificate
- BUG/MINOR: stream: protect stream_dump() against incomplete streams
- DOC: config: mention uid dependency on the tune.quic.socket-owner option
- MEDIUM: capabilities: enable support for Linux capabilities
- CLEANUP/MINOR: connection: Improve consistency of PPv2 related constants
- MEDIUM: connection: Generic, list-based allocation and look-up of PPv2 TLVs
- MEDIUM: sample: Add fetch for arbitrary TLVs
- MINOR: sample: Refactor fc_pp_authority by wrapping the generic TLV fetch
- MINOR: sample: Refactor fc_pp_unique_id by wrapping the generic TLV fetch
- MINOR: sample: Add common TLV types as constants for fc_pp_tlv
- MINOR: ssl_sock: avoid iterating realloc(+1) on stored context
- DOC: ssl: add some comments about the non-obvious session allocation stuff
- CLEANUP: ssl: keep a pointer to the server in ssl_sock_init()
- MEDIUM: ssl_sock: always use the SSL's server name, not the one from the tid
- MEDIUM: server/ssl: place an rwlock in the per-thread ssl server session
- MINOR: server/ssl: maintain an index of the last known valid SSL session
- MINOR: server/ssl: clear the shared good session index on failure
- MEDIUM: server/ssl: pick another thread's session when we have none yet
- MINOR: activity: report the current run queue size
- BUG/MINOR: checks: do not queue/wake a bounced check
- MINOR: checks: start the checks in sleeping state
- MINOR: checks: pin the check to its thread upon wakeup
- MINOR: check: remember when we migrate a check
- MINOR: check/activity: collect some per-thread check activity stats
- MINOR: checks: maintain counters of active checks per thread
- MINOR: check: also consider the random other thread's active checks
- MEDIUM: checks: search more aggressively for another thread on overload
- MEDIUM: checks: implement a queue in order to limit concurrent checks
- MINOR: checks: also consider the thread's queue for rebalancing
- DEBUG: applet: Properly report opposite SC expiration dates in traces
- BUG/MEDIUM: stconn: Update stream expiration date on blocked sends
- BUG/MINOR: stconn: Don't report blocked sends during connection establishment
- BUG/MEDIUM: stconn: Wake applets on sending path if there is a pending shutdown
- BUG/MEDIUM: stconn: Don't block sends if there is a pending shutdown
- BUG/MINOR: quic: Possible skipped RTT sampling
- MINOR: quic: Add a trace to quic_release_frm()
- BUG/MAJOR: quic: Really ignore malformed ACK frames.
- BUG/MINOR: quic: Unchecked pointer to packet number space dereferenced
- BUG/MEDIUM: connection: fix pool free regression with recent ppv2 TLV patches
- BUG/MEDIUM: h1-htx: Ensure chunked parsing with full output buffer
- BUG/MINOR: stream: further protect stream_dump() against incomplete sessions
- DOC: configuration: update examples for req.ver
- MINOR: properly mark the end of the CLI command in error messages
- BUILD: ssl: Build with new cryptographic library AWS-LC
- REGTESTS: ssl: skip ssl_dh test with AWS-LC
- BUILD: bug: make BUG_ON() void to avoid a rare warning
- BUILD: checks: shut up yet another stupid gcc warning
- MINOR: cpuset: add ha_cpuset_isset() to check for the presence of a CPU in a set
- MINOR: cpuset: add ha_cpuset_or() to bitwise-OR two CPU sets
- MINOR: cpuset: centralize a reliable bound cpu detection
- MEDIUM: threads: detect incomplete CPU bindings
- MEDIUM: threads: detect excessive thread counts vs cpu-map
- BUILD: quic: Compilation issue on 32-bits systems with quic_may_send_bytes()
- BUG/MINOR: quic: Unchecked pointer to Handshake packet number space
- MINOR: global: export the display_version() symbol
- MEDIUM: mworker: display a more accessible message when a worker crash
- MINOR: httpclient: allow to configure the retries
- MINOR: httpclient: allow to configure the timeout.connect
- BUG/MINOR: quic: Wrong RTT adjusments
- BUG/MINOR: quic: Wrong RTT computation (srtt and rrt_var)
- BUG/MINOR: stconn: Don't inhibit shutdown on connection on error
- BUG/MEDIUM: applet: Fix API for function to push new data in channels buffer
- BUG/MEDIUM: stconn: Report read activity when a stream is attached to front SC
- BUG/MEDIUM: applet: Report an error if applet request more room on aborted SC
- BUG/MEDIUM: stconn/stream: Forward shutdown on write timeout
- NUG/MEDIUM: stconn: Always update stream's expiration date after I/O
- BUG/MINOR: applet: Always expect data when CLI is waiting for a new command
- BUG/MINOR: ring/cli: Don't expect input data when showing events
- BUG/MINOR: quic: Dereferenced unchecked pointer to Handshke packet number space
- BUG/MINOR: hlua/action: incorrect message on E_YIELD error
- MINOR: http_ana: position the FINAL flag for http_after_res execution
- CI: scripts: add support to build-ssl.sh to download and build AWS-LC
- CI: add support to matrix.py to determine the latest AWS-LC release
- CI: Update matrix.py so all code is contained in functions.
- CI: github: Add a weekly CI run building with AWS-LC
- MINOR: ring: add a function to compute max ring payload
- BUG/MEDIUM: ring: adjust maxlen consistency check
- MINOR: sink: simplify post_sink_resolve function
- MINOR: log/sink: detect when log maxlen exceeds sink size
- MINOR: sink: inform the user when logs will be implicitly truncated
- MEDIUM: sink: don't perform implicit truncations when maxlen is not set
- MINOR: log: move log-forwarders cleanup in log.c
- MEDIUM: httpclient/logs: rely on per-proxy post-check instead of global one
- MINOR: log: add dup_logsrv() helper function
- MEDIUM: log/sink: make logsrv postparsing more generic
- MEDIUM: fcgi-app: properly postresolve logsrvs
- MEDIUM: spoe-agent: properly postresolve log rings
- MINOR: sink: add helper function to deallocate sink struct
- MEDIUM: sink/ring: introduce high level ring creation helper function
- MEDIUM: sink: add sink_finalize() function
- CLEANUP: log: remove unnecessary trim in __do_send_log
- MINOR: cache: Change hash function in default normalizer used in case of "vary"
- MINOR: tasks/stats: report the number of niced tasks in "show info"
- CI: Update to actions/checkout@v4
- MINOR: ssl: add support for 'curves' keyword on server lines
- BUG/MINOR: quic: Wrong cluster secret initialization
- CLEANUP: quic: Remove useless free_quic_tx_pkts() function.
- MEDIUM: init: initialize the trash earlier
- MINOR: tools: add function read_line_to_trash() to read a line of a file
- MINOR: cfgparse: use read_line_from_trash() to read from /sys
- MEDIUM: cfgparse: assign NUMA affinity to cpu-maps
- MINOR: cpuset: dynamically allocate cpu_map
- REORG: cpuset: move parse_cpu_set() and parse_cpumap() to cpuset.c
- CI: musl: highlight section if there are coredumps
- CI: musl: drop shopt in workflow invocation
More and more utility functions rely on the trash while most of the init
code doesn't have access to it because it's initialized very late (in
PRE_CHECK for the initial one). It's a pool, and it purposely supports
being reallocated, so let's initialize it in STG_POOL so that early
STG_INIT code can at least use it.
This patch implements the 'curves' keyword on server lines as well as
the 'ssl-default-server-curves' keyword in the global section.
It also add the keyword on the server line in the ssl_curves reg-test.
These keywords allow the configuration of the curves list for a server.
When using the httpclient, one could be bothered with it returning
after a very long time when failing. By default the httpclient has a
retries of 3 and a timeout connect of 5s, which can results in pause of
20s upon failure.
This patch allows the user to configure the "timeout connect" of the
httpclient so it could reduce the time to return an error.
This patch helps fixing part of the issue #2269.
Could be backported in 2.7 if needed.
When using the httpclient, one could be bothered with it returning after
a very long time when failing. By default the httpclient has a retries
of 3 and a timeout connect of 5s, which can results in pause of 20s
upon failure.
This patch allows the user to configure the retries of the httpclient so
it could reduce the time to return an error.
This patch helps fixing part of the issue #2269.
Could be backported in 2.7 if needed.
The progressive adoption of OpenSSL 3 and its abysmal handshake
performance has started to reveal situations where it simply isn't
possible anymore to succesfully run health checks on many servers,
because between the moment all the checks are started and the moment
the handshake finally completes, the timeout has expired!
This also has consequences on production traffic which gets
significantly delayed as well, all that for lots of checks. While it's
possible to increase the check delays, it doesn't solve everything as
checks still take a huge amount of time to converge in such conditions.
Here we take a different approach by permitting to enforce the maximum
concurrent checks per thread limitation and implementing an ordered
queue. Thanks to this, if a thread about to start a check has reached
its limit, it will add the check at the end of a queue and it will be
processed once another check is finished. This proves to be extremely
efficient, with all checks completing in a reasonable amount of time
and not being disturbed by the rest of the traffic from other checks.
They're just cycling slower, but at the speed the machine can handle.
One must understand however that if some complex checks perform multiple
exchanges, they will take a check slot for all the required duration.
This is why the limit is not enforced by default.
Tests on SSL show that a limit of 5-50 checks per thread on local
servers gives excellent results already, so that could be a good starting
point.
When the current check is overloaded (more running checks than the
configured limit), we'll try more aggressively to find another thread.
Instead of just opportunistically looking for one half as loaded, now if
the current thread has more than 1% more active checks than another one,
or has more than a configured limit of concurrent running checks, it will
search for a more suitable thread among 3 other random ones in order to
migrate the check there. The number of migrations remains very low (~1%)
and the checks load very fair across all threads (~1% as well). The new
parameter is called tune.max-checks-per-thread.
Based on the new, generic allocation infrastructure, a new sample
fetch fc_pp_tlv is introduced. It is an abstraction for existing
PPv2 TLV sample fetches. It takes any valid TLV ID as argument and
returns the value as a string, similar to fc_pp_authority and
fc_pp_unique_id.
For a while there has been the constraint of having to run as root for
transparent proxying, and we're starting to see some cases where QUIC is
not running in socket-per-connection mode due to the missing capability
that would be needed to bind a privileged port. It's not realistic to
ask all QUIC users on port 443 to run as root, so instead let's provide
a basic support for capabilities at least on linux. The ones currently
supported are cap_net_raw, cap_net_admin and cap_net_bind_service. The
mechanism was made OS-specific with a dedicated file because it really
is. It can be easily refined later for other OSes if needed.
A new keyword "setcaps" is added to the global section, to enumerate the
capabilities that must be kept when switching from root to non-root. This
is ignored in other situations though. HAProxy has to be built with
USE_LINUX_CAP=1 for this to be supported, which is enabled by default
for linux-glibc, linux-glibc-legacy and linux-musl.
A good way to test this is to start haproxy with such a config:
global
uid 1000
setcap cap_net_bind_service
frontend test
mode http
timeout client 3s
bind quic4@:443 ssl crt rsa+dh2048.pem allow-0rtt
and run it under "sudo strace -e trace=bind,setuid", then connecting
there from an H3 client. The bind() syscall must succeed despite the
user id having been switched.
This option defaults to "connection" but is also dependent on the user
being allowed to bind the specified port. Since QUIC can easily run on
non-privileged ports, usually this is not a problem, but if bound to port
443 it will usually fail. Let's mention this.
Released version 2.9-dev4 with the following main changes :
- DEV: flags/show-sess-to-flags: properly decode fd.state
- BUG/MINOR: stktable: allow sc-set-gpt(0) from tcp-request connection
- BUG/MINOR: stktable: allow sc-add-gpc from tcp-request connection
- DOC: typo: fix sc-set-gpt references
- SCRIPTS: git-show-backports: automatic ref and base detection with -m
- REGTESTS: Do not use REQUIRE_VERSION for HAProxy 2.5+ (3)
- DOC: jwt: Add explicit list of supported algorithms
- BUILD: Makefile: add the USE_QUIC option to make help
- BUILD: Makefile: add USE_QUIC_OPENSSL_COMPAT to make help
- BUILD: Makefile: realigned USE_* options in make help
- DEV: makefile: fix POSIX compatibility for "range" target
- IMPORT: plock: also support inlining the int code
- IMPORT: plock: always expose the inline version of the lock wait function
- IMPORT: lorw: support inlining the wait call
- MINOR: threads: inline the wait function for pthread_rwlock emulation
- MINOR: atomic: make sure to always relax after a failed CAS
- MINOR: pools: use EBO to wait for unlock during pool_flush()
- BUILD/IMPORT: fix compilation with PLOCK_DISABLE_EBO=1
- MINOR: quic+openssl_compat: Do not start without "limited-quic"
- MINOR: quic+openssl_compat: Emit an alert for "allow-0rtt" option
- BUG/MINOR: quic: allow-0rtt warning must only be emitted with quic bind
- BUG/MINOR: quic: ssl_quic_initial_ctx() uses error count not error code
- MINOR: pattern: do not needlessly lookup the LRU cache for empty lists
- IMPORT: xxhash: update xxHash to version 0.8.2
- MINOR: proxy: simplify parsing 'backend/server'
- MINOR: connection: centralize init/deinit of backend elements
- MEDIUM: connection: implement passive reverse
- MEDIUM: h2: reverse connection after SETTINGS reception
- MINOR: server: define reverse-connect server
- MINOR: backend: only allow reuse for reverse server
- MINOR: tcp-act: parse 'tcp-request attach-srv' session rule
- REGTESTS: provide a reverse-server test
- MINOR: tcp-act: define optional arg name for attach-srv
- MINOR: connection: use attach-srv name as SNI reuse parameter on reverse
- REGTESTS: provide a reverse-server test with name argument
- MINOR: proto: define dedicated protocol for active reverse connect
- MINOR: connection: extend conn_reverse() for active reverse
- MINOR: proto_reverse_connect: parse rev@ addresses for bind
- MINOR: connection: prepare init code paths for active reverse
- MEDIUM: proto_reverse_connect: bootstrap active reverse connection
- MINOR: proto_reverse_connect: handle early error before reversal
- MEDIUM: h2: implement active connection reversal
- MEDIUM: h2: prevent stream opening before connection reverse completed
- REGTESTS: write a full reverse regtest
- BUG/MINOR: h2: fix reverse if no timeout defined
- CI: fedora: fix "dnf" invocation syntax
- BUG/MINOR: hlua_fcn: potentially unsafe stktable_data_ptr usage
- DOC: lua: fix Sphinx warning from core.get_var()
- DOC: lua: fix core.register_action typo
- BUG/MINOR: ssl_sock: fix possible memory leak on OOM
- MEDIUM: map/acl: Improve pat_ref_set() efficiency (for "set-map", "add-acl" action perfs)
- MEDIUM: map/acl: Improve pat_ref_set_elt() efficiency (for "set-map", "add-acl"action perfs)
- MEDIUM: map/acl: Accelerate several functions using pat_ref_elt struct ->head list
- MEDIUM: map/acl: Replace map/acl spin lock by a read/write lock.
- DOC: map/acl: Remove the comments about map/acl performance issue
- DOC: Explanation of be_name and be_id fetches
- MINOR: connection: simplify removal of idle conns from their trees
- MINOR: server: move idle tree insert in a dedicated function
- MAJOR: connection: purge idle conn by last usage
The be_name and be_id fetches contain data related to the current
backend and can be used in frontend responses. Yet, in cases where
no backend is used due to a local response or backend selection
failure, these fetches retain details of the current frontend.
This patch enhances the clarity of the values provided by these
fetches.
Signed-off-by: Sbastien Gross <sgross@haproxy.com>
"converter" was used in place of "action" as a result of a copy-paste
error probably.
Also, rephrasing the "actions" keyword explanation to prevent confusion
between action name (which is the name of new action about to be created)
and action facilities where we want to expose the new action.
This could be backported to every stable versions.
Since f034139bc0 ("MINOR: lua: Allow reading "proc." scoped vars from
LUA core."), a new Sphinx warning is emitted when generating the lua doc:
"WARNING: Field list ends without a blank line; unexpected unindent."
This is due to a missing space after the line break continuation, sphinx
parser is very restrictive unfortunately!
Suppressing the warning and fixing the html output at the same time by
adding the missing space.
Implement parsing for "rev@" addresses on bind line. On config parsing,
server name is stored on the bind_conf.
Several new callbacks are defined on reverse_connect protocol to
complete parsing. listen callback is used to retrieve the server
instance from the bind_conf server name. If found, the server instance
is stored on the receiver. Checks are implemented to ensure HTTP/2
protocol only is used by the server.
Add an optional argument 'name' for attach-srv rule. This contains an
expression which will be used as an identifier inside the server idle
pool after reversal. To match this connection for a future transfer
through the server, the SNI server parameter must match this name. If no
name is defined, match will only occur with an empty SNI value.
For the moment, only the parsing step is implemented. An extra check is
added to ensure that the reverse server uses SSL with a SNI. Indeed, if
name is defined but server does not uses a SNI, connections will never
be selected on reused after reversal due to a hash mismatch.
Create a new tcp-request session rule 'attach-srv'.
The parsing handler is used to extract the server targetted with the
notation 'backend/server'. The server instance is stored in the act_rule
instance under the new union variant 'attach_srv'.
Extra checks are implemented in parsing to ensure attach-srv is only
used for proxy in HTTP mode and with listeners/server with no explicit
protocol reference or HTTP/2 only.
The action handler itself is really simple. It assigns the stored server
instance to the 'reverse' member of the connection instance. It will be
used in a future patch to implement passive reverse-connect.
Implement reverse-connect server. This server type cannot instantiate
its own connection on transfer. Instead, it can only reuse connection
from its idle pool. These connections will be populated using the future
'tcp-request session attach-srv' rule.
A reverse-connect has no address. Instead, it uses a new custom server
notation with '@' character prefix. For the moment, only '@reverse' is
defined. An extra check is implemented to ensure server is used in a
HTTP proxy.
Only sc-inc-gpc and sc-set-gpt do exist. The mix-up sc-inc-gpt crept in
in 71d189219 (DOC: config: Rework and uniformize how TCP/HTTP rules are
documented, 2021-10-14) and got copied in a92480462 (MINOR: http-rules:
Add missing actions in http-after-response ruleset, 2023-01-05).
Released version 2.9-dev3 with the following main changes :
- BUG/MINOR: ssl: OCSP callback only registered for first SSL_CTX
- BUG/MEDIUM: h3: Properly report a C-L header was found to the HTX start-line
- MINOR: sample: add pid sample
- MINOR: sample: implement act_conn sample fetch
- MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values
- MEDIUM: sample: implement us and ms variant of utime and ltime
- BUG/MINOR: sample: check alloc_trash_chunk() in conv_time_common()
- DOC: configuration: describe Td in Timing events
- MINOR: sample: implement the T* timer tags from the log-format as fetches
- DOC: configuration: add sample fetches for timing events
- BUG/MINOR: quic: Possible crash when acknowledging Initial v2 packets
- MINOR: quic: Export QUIC traces code from quic_conn.c
- MINOR: quic: Export QUIC CLI code from quic_conn.c
- MINOR: quic: Move TLS related code to quic_tls.c
- MINOR: quic: Add new "QUIC over SSL" C module.
- MINOR: quic: Add a new quic_ack.c C module for QUIC acknowledgements
- CLEANUP: quic: Defined but no more used function (quic_get_tls_enc_levels())
- MINOR: quic: Split QUIC connection code into three parts
- CLEANUP: quic: quic_conn struct cleanup
- MINOR: quic; Move the QUIC frame pool to its proper location
- BUG/MINOR: chunk: fix chunk_appendf() to not write a zero if buffer is full
- BUG/MEDIUM: h3: Be sure to handle fin bit on the last DATA frame
- DOC: configuration: rework the custom log format table
- BUG/MINOR: quic+openssl_compat: Non initialized TLS encryption levels
- CLEANUP: acl: remove cache_idx from acl struct
- REORG: cfgparse: extract curproxy as a global variable
- MINOR: acl: add acl() sample fetch
- BUILD: cfgparse: keep a single "curproxy"
- BUG/MEDIUM: bwlim: Reset analyse expiration date when then channel analyse ends
- MEDIUM: stream: Reset response analyse expiration date if there is no analyzer
- BUG/MINOR: htx/mux-h1: Properly handle bodyless responses when splicing is used
- BUG/MEDIUM: quic: consume contig space on requeue datagram
- BUG/MINOR: http-client: Don't forget to commit changes on HTX message
- CLEANUP: stconn: Move comment about sedesc fields on the field line
- REGTESTS: http: Create a dedicated script to test spliced bodyless responses
- REGTESTS: Test SPLICE feature is enabled to execute script about splicing
- BUG/MINOR: quic: reappend rxbuf buffer on fake dgram alloc error
- BUILD: quic: fix wrong potential NULL dereference
- MINOR: h3: abort request if not completed before full response
- BUG/MAJOR: http-ana: Get a fresh trash buffer for each header value replacement
- CLEANUP: quic: Remove quic_path_room().
- MINOR: quic: Amplification limit handling sanitization.
- MINOR: quic: Move some counters from [rt]x quic_conn anonymous struct
- MEDIUM: quic: Send CONNECTION_CLOSE packets from a dedicated buffer.
- MINOR: quic: Use a pool for the connection ID tree.
- MEDIUM: quic: Allow the quic_conn memory to be asap released.
- MINOR: quic: Release asap quic_conn memory (application level)
- MINOR: quic: Release asap quic_conn memory from ->close() xprt callback.
- MINOR: quic: Warning for OpenSSL wrapper QUIC bindings without "limited-quic"
- REORG: http: move has_forbidden_char() from h2.c to http.h
- BUG/MAJOR: h3: reject header values containing invalid chars
- MINOR: mux-h2/traces: also suggest invalid header upon parsing error
- MINOR: ist: add new function ist_find_range() to find a character range
- MINOR: http: add new function http_path_has_forbidden_char()
- MINOR: h2: pass accept-invalid-http-request down the request parser
- REGTESTS: http-rules: add accept-invalid-http-request for normalize-uri tests
- BUG/MINOR: h1: do not accept '#' as part of the URI component
- BUG/MINOR: h2: reject more chars from the :path pseudo header
- BUG/MINOR: h3: reject more chars from the :path pseudo header
- REGTESTS: http-rules: verify that we block '#' by default for normalize-uri
- DOC: clarify the handling of URL fragments in requests
- BUG/MAJOR: http: reject any empty content-length header value
- BUG/MINOR: http: skip leading zeroes in content-length values
- BUG/MEDIUM: mux-h1: fix incorrect state checking in h1_process_mux()
- BUG/MEDIUM: mux-h1: do not forget EOH even when no header is sent
- BUILD: mux-h1: shut a build warning on clang from previous commit
- DEV: makefile: add a new "range" target to iteratively build all commits
- CI: do not use "groupinstall" for Fedora Rawhide builds
- CI: get rid of travis-ci wrapper for Coverity scan
- BUG/MINOR: quic: mux started when releasing quic_conn
- BUG/MINOR: quic: Possible crash in quic_cc_conn_io_cb() traces.
- MINOR: quic: Add a trace for QUIC conn fd ready for receive
- BUG/MINOR: quic: Possible crash when issuing "show fd/sess" CLI commands
- BUG/MINOR: quic: Missing tasklet (quic_cc_conn_io_cb) memory release (leak)
- BUG/MEDIUM: quic: fix tasklet_wakeup loop on connection closing
- BUG/MINOR: hlua: fix invalid use of lua_pop on error paths
- MINOR: hlua: add hlua_stream_ctx_prepare helper function
- BUG/MEDIUM: hlua: streams don't support mixing lua-load with lua-load-per-thread
- MAJOR: threads/plock: update the embedded library again
- MINOR: stick-table: move the task_queue() call outside of the lock
- MINOR: stick-table: move the task_wakeup() call outside of the lock
- MEDIUM: stick-table: change the ref_cnt atomically
- MINOR: stick-table: better organize the struct stktable
- MEDIUM: peers: update ->commitupdate out of the lock using a CAS
- MEDIUM: peers: drop then re-acquire the wrlock in peer_send_teachmsgs()
- MEDIUM: peers: only read-lock peer_send_teachmsgs()
- MEDIUM: stick-table: use a distinct lock for the updates tree
- MEDIUM: stick-table: touch updates under an upgradable read lock
- MEDIUM: peers: drop the stick-table lock before entering peer_send_teachmsgs()
- MINOR: stick-table: move the update lock into its own cache line
- CLEANUP: stick-table: slightly reorder the stktable struct
- BUILD: defaults: use __WORDSIZE not LONGBITS for MAX_THREADS_PER_GROUP
- MINOR: tools: make ptr_hash() support 0-bit outputs
- MINOR: tools: improve ptr hash distribution on 64 bits
- OPTIM: tools: improve hash distribution using a better prime seed
- OPTIM: pools: use exponential back-off on shared pool allocation/release
- OPTIM: pools: make pool_get_from_os() / pool_put_to_os() not update ->allocated
- MINOR: pools: introduce the use of multiple buckets
- MEDIUM: pools: spread the allocated counter over a few buckets
- MEDIUM: pools: move the used counter over a few buckets
- MEDIUM: pools: move the needed_avg counter over a few buckets
- MINOR: pools: move the failed allocation counter over a few buckets
- MAJOR: pools: move the shared pool's free_list over multiple buckets
- MINOR: pools: make pool_evict_last_items() use pool_put_to_os_no_dec()
- BUILD: pools: fix build error on clang with inline vs forceinline
We indicate in path/pathq/url that they may contain '#' if the frontend
is configured with "option accept-invalid-http-request", and that option
mentions the fragment as well.
Implement 4 new fetches:
- ms_ltime
- ms_utime
- us_ltime
- us_utime
Which are the same as ltime and utime but with milliseconds and
microseconds input.
The converters also suports the %N conversion specifier like in date(1).
Unfortunately since %N is not supported by strftime, the format string
is parsed twice, once manually to replace %N, and once by strftime.
Implement %[accept_date] which returns the same as %Ts log-format tag.
Implement %[request_date] which is a timestamp for %tr.
accept_date and request_date take an faculative unit argument which can
be 's', 'ms' or 'us'.
The goal is to be able to convert these 2 timestamps to HAProxy date
format like its done with %T, %tr, %trg etc
Released version 2.9-dev2 with the following main changes :
- BUG/MINOR: quic: Possible leak when allocating an encryption level
- BUG/MINOR: quic: Missing QUIC connection path member initialization
- BUILD: quic: Compilation fixes for some gcc warnings with -O1
- DOC: ssl: Fix typo in 'ocsp-update' option
- DOC: ssl: Add ocsp-update troubleshooting clues and emphasize on crt-list only aspect
- BUG/MINOR: tcp_sample: bc_{dst,src} return IP not INT
- MEDIUM: acl/sample: unify sample conv parsing in a single function
- MINOR: sample: introduce c_pseudo() conv function
- MEDIUM: sample: add missing ADDR=>? compatibility matrix entries
- MINOR: sample: fix ipmask sample definition
- MEDIUM: tree-wide: fetches that may return IPV4+IPV6 now return ADDR
- MEDIUM: sample: introduce 'same' output type
- BUG/MINOR: quic: Possible crash in "show quic" dumping packet number spaces
- BUG/MINOR: cache: A 'max-age=0' cache-control directive can be overriden by a s-maxage
- BUG/MEDIUM: sink: invalid server list in sink_new_from_logsrv()
- BUG/MINOR: http_ext: unhandled ERR_ABORT in proxy_http_parse_7239()
- BUG/MINOR: sink: missing sft free in sink_deinit()
- BUG/MINOR: ring: size warning incorrectly reported as fatal error
- BUG/MINOR: ring: maxlen warning reported as alert
- BUG/MINOR: log: LF upsets maxlen for UDP targets
- MINOR: sink/api: pass explicit maxlen parameter to sink_write()
- BUG/MEDIUM: log: improper use of logsrv->maxlen for buffer targets
- BUG/MINOR: log: fix missing name error message in cfg_parse_log_forward()
- BUG/MINOR: log: fix multiple error paths in cfg_parse_log_forward()
- BUG/MINOR: log: free errmsg on error in cfg_parse_log_forward()
- BUG/MINOR: sink: invalid sft free in sink_deinit()
- BUG/MINOR: sink: fix errors handling in cfg_post_parse_ring()
- BUG/MINOR: server: set rid default value in new_server()
- MINOR: hlua_fcn/mailers: handle timeout mail from mailers section
- BUG/MINOR: sink/log: properly deinit srv in sink_new_from_logsrv()
- EXAMPLES: maintain haproxy 2.8 retrocompatibility for lua mailers script
- BUG/MINOR: hlua_fcn/queue: use atomic load to fetch queue size
- BUG/MINOR: config: Remove final '\n' in error messages
- BUG/MINOR: config: Lenient port configuration parsing
- BUG/MEDIUM: quic: token IV was not computed using a strong secret
- BUG/MINOR: quic: retry token remove one useless intermediate expand
- BUG/MEDIUM: quic: missing check of dcid for init pkt including a token
- BUG/MEDIUM: quic: timestamp shared in token was using internal time clock
- CLEANUP: quic: remove useless parameter 'key' from quic_packet_encrypt
- BUG/MINOR: hlua: hlua_yieldk ctx argument should support pointers
- BUG/MEDIUM: hlua_fcn/queue: bad pop_wait sequencing
- DOC: config: Fix fc_src description to state the source address is returned
- BUG/MINOR: sample: Fix wrong overflow detection in add/sub conveters
- BUG/MINOR: http: Return the right reason for 302
- MEDIUM: ssl: new sample fetch method to get curve name
- CI: add naming convention documentation
- CI: explicitely highlight VTest result section if there's something
- BUG/MINOR: quic: Unckecked encryption levels availability
- BUILD: quic: fix warning during compilation using gcc-6.5
- BUG/MINOR: hlua: add check for lua_newstate
- BUG/MINOR: h1-htx: Return the right reason for 302 FCGI responses
- MINOR: lua: Allow reading "proc." scoped vars from LUA core.
- MINOR: cpuset: add cpu_map_configured() to know if a cpu-map was found
- BUG/MINOR: config: do not detect NUMA topology when cpu-map is configured
- BUG/MINOR: cpuset: remove the bogus "proc" from the cpu_map struct
- BUG/MINOR: init: set process' affinity even in foreground
- CLEANUP: cpuset: remove the unused proc_t1 field in cpu_map
- CLEANUP: config: make parse_cpu_set() return documented values
- BUG/MINOR: server: Don't warn on server resolution failure with init-addr none
- MINOR: peers: add peers keyword registration
- MINOR: quic: Stop storing the TX encoded transport parameters
- MINOR: quic: Dynamic allocation for negotiated Initial TLS cipher context.
- MINOR: quic: Release asap the negotiated Initial TLS context.
- MINOR: quic: Add traces to qc_may_build_pkt()
- MEDIUM: quic: Packet building rework.
- CLEANUP: quic: Remove a useless TLS related variable from quic_conn_io_cb().
- MEDIUM: quic: Handshake I/O handler rework.
- MINOR: quic: Add traces for qc_frm_free()
- MINOR: quic: add trace about pktns packet/frames releasing
- BUG/MINOR: quic: Missing parentheses around PTO probe variable.
- MINOR: quic: Ping from Initial pktns before reaching anti-amplification limit
- BUG/MINOR: server-state: Ignore empty files
- BUG/MINOR: server-state: Avoid warning on 'file not found'
- BUG/MEDIUM: listener: Acquire proxy's lock in relax_listener() if necessary
- MINOR: quic: QUIC openssl wrapper implementation
- MINOR: quic: Include QUIC opensssl wrapper header from TLS stacks compatibility header
- MINOR: quic: Do not enable O-RTT with USE_QUIC_OPENSSL_COMPAT
- MINOR: quic: Set the QUIC connection as extra data before calling SSL_set_quic_method()
- MINOR: quic: Do not enable 0RTT with SSL_set_quic_early_data_enabled()
- MINOR: quic: Add a compilation option for the QUIC OpenSSL wrapper
- MINOR: quic: Export some KDF functions (QUIC-TLS)
- MINOR: quic: Make ->set_encryption_secrets() be callable two times
- MINOR: quic: Initialize TLS contexts for QUIC openssl wrapper
- MINOR: quic: Call the keylog callback for QUIC openssl wrapper from SSL_CTX_keylog()
- MINOR: quic: Add a quic_openssl_compat struct to quic_conn struct
- MINOR: quic: Useless call to SSL_CTX_set_quic_method()
- MINOR: quic: SSL context initialization with QUIC OpenSSL wrapper.
- MINOR: quic: Missing encoded transport parameters for QUIC OpenSSL wrapper
- MINOR: quic: Add "limited-quic" new tuning setting
- DOC: quic: Add "limited-quic" new tuning setting
- DOC: install: Document how to build a limited support for QUIC
Document "limited-quic" new tuning setting which must be used to
enable the QUIC listener bindings when haproxy is compiled against
a TLS/SSL stack without QUIC support.
This adds the "core.get_var()" method allow the reading
of "proc." scoped variables outside of TXN or HTTP/TCPApplet.
Fixes: #2212
Signed-off-by: Daan van Gorkum <djvg@djvg.net>
Adds a new sample fetch method to get the curve name used in the
key agreement to enable better observability. In OpenSSLv3, the function
`SSL_get_negotiated_group` returns the NID of the curve and from the NID,
we get the curve name by passing the NID to OBJ_nid2sn. This was not
available in v1.1.1. SSL_get_curve_name(), which returns the curve name
directly was merged into OpenSSL master branch last week but will be available
only in its next release.
A typo in the "fc_src" description was fixed. This sample returns the
original source IP address and not the destination one.
This patch should be backported as far as 2.6.
As the example/lua/mailers.lua script does its best to mimic the c-mailer
implementation, it should support the "timeout mail" directive as well.
This could be backported in 2.8.
The current limitation of the 'ocsp-update' option and the fact that it
can only be used in crt-lists was puzzling for some people so the doc
was amended to emphasize this specificity. A configuration extract was
added as well.
A few troubleshooting clues were added as well.
Must be backported in 2.8.
This patch fixes a misalignment in the 'ocsp-update' option description
and it splits the example log lines for readability.
Must be backported in 2.8.
Released version 2.9-dev1 with the following main changes :
- BUG/MINOR: stats: Fix Lua's `get_stats` function
- MINOR: stats: protect against future stats fields omissions
- BUG/MINOR: stream: do not use client-fin/server-fin with HTX
- BUG/MINOR: quic: Possible crash when SSL session init fails
- CONTRIB: Add vi file extensions to .gitignore
- BUG/MINOR: spoe: Only skip sending new frame after a receive attempt
- BUG/MINOR: peers: Improve detection of config errors in peers sections
- REG-TESTS: stickiness: Delay haproxys start to properly resolv variables
- DOC: quic: fix misspelled tune.quic.socket-owner
- DOC: config: fix jwt_verify() example using var()
- DOC: config: fix rfc7239 converter examples (again)
- BUG/MINOR: cfgparse-tcp: leak when re-declaring interface from bind line
- BUG/MINOR: proxy: add missing interface bind free in free_proxy
- BUG/MINOR: proxy/server: free default-server on deinit
- BUG/MEDIUM: hlua: Use front SC to detect EOI in HTTP applets' receive functions
- BUG/MINOR: ssl: log message non thread safe in SSL Hanshake failure
- BUG/MINOR: quic: Wrong encryption level flags checking
- BUG/MINOR: quic: Address inversion in "show quic full"
- BUG/MINOR: server: inherit from netns in srv_settings_cpy()
- BUG/MINOR: namespace: missing free in netns_sig_stop()
- BUG/MINOR: quic: Missing initialization (packet number space probing)
- BUG/MINOR: quic: Possible crash in quic_conn_prx_cntrs_update()
- BUG/MINOR: quic: Possible endless loop in quic_lstnr_dghdlr()
- MINOR: quic: Remove pool_zalloc() from qc_new_conn()
- MINOR: quic: Remove pool_zalloc() from qc_conn_alloc_ssl_ctx()
- MINOR: quic: Remove pool_zalloc() from quic_dgram_parse()
- BUG/MINOR: quic: Missing transport parameters initializations
- BUG/MEDIUM: mworker: increase maxsock with each new worker
- BUG/MINOR: quic: ticks comparison without ticks API use
- BUG/MINOR: quic: Missing TLS secret context initialization
- DOC: Add tune.h2.be.* and tune.h2.fe.* options to table of contents
- DOC: Add tune.h2.max-frame-size option to table of contents
- DOC: Attempt to fix dconv parsing error for tune.h2.fe.initial-window-size
- REGTESTS: h1_host_normalization : Add a barrier to not mix up log messages
- MEDIUM: mux-h1: Split h1_process_mux() to make code more readable
- REORG: mux-h1: Rename functions to emit chunk size/crlf in the output buffer
- MINOR: mux-h1: Add function to append the chunk size to the output buffer
- MINOR: mux-h1: Add function to prepend the chunk crlf to the output buffer
- MEDIUM: filters/htx: Don't rely on HTX extra field if payload is filtered
- MEDIIM: mux-h1: Add splicing support for chunked messages
- REGTESTS: Add a script to test the kernel splicing with chunked messages
- CLEANUP: mux-h1: Remove useless __maybe_unused statement
- BUG/MINOR: http_ext: fix if-none regression in forwardfor option
- REGTEST: add an extra testcase for ifnone-forwardfor
- BUG/MINOR: mworker: leak of a socketpair during startup failure
- BUG/MINOR: quic: Prevent deadlock with CID tree lock
- MEDIUM: ssl: handle the SSL_ERROR_ZERO_RETURN during the handshake
- BUG/MINOR: ssl: SSL_ERROR_ZERO_RETURN returns CO_ER_SSL_EMPTY
- BUILD: mux-h1: silence a harmless fallthrough warning
- BUG/MEDIUM: quic: error checking buffer large enought to receive the retry tag
- MINOR: ssl: allow to change the server signature algorithm on server lines
- MINOR: ssl: allow to change the client-sigalgs on server lines
- BUG/MINOR: config: fix stick table duplicate name check
- BUG/MINOR: quic: Missing random bits in Retry packet header
- BUG/MINOR: quic: Wrong Retry paquet version field endianess
- BUG/MINOR: quic: Wrong endianess for version field in Retry token
- IMPORT: slz: implement a synchronous flush() operation
- MINOR: compression/slz: add support for a pure flush of pending bytes
- MINOR: quic: Move QUIC TLS encryption level related code (quic_conn_enc_level_init())
- MINOR: quic: Move QUIC encryption level structure definition
- MINOR: quic: Implement a packet number space identification function
- MINOR: quic: Move packet number space related functions
- MEDIUM: quic: Dynamic allocations of packet number spaces
- CLEANUP: quic: Remove qc_list_all_rx_pkts() defined but not used
- MINOR: quic: Add a pool for the QUIC TLS encryption levels
- MEDIUM: quic: Dynamic allocations of QUIC TLS encryption levels
- MINOR: quic: Reduce the maximum length of TLS secrets
- CLEANUP: quic: Remove two useless pools a low QUIC connection level
- MEDIUM: quic: Handle the RX in one pass
- MINOR: quic: Remove call to qc_rm_hp_pkts() from I/O callback
- CLEANUP: quic: Remove server specific about Initial packet number space
- MEDIUM: quic: Release encryption levels and packet number spaces asap
- CLEANUP: quic: Remove a useless test about discarded pktns (qc_handle_crypto_frm())
- MINOR: quic: Move the packet number space status at quic_conn level
- MINOR: quic: Drop packet with type for discarded packet number space.
- BUILD: quic: Add a DISGUISE() to please some compiler to qc_prep_hpkts() 1st parameter
- BUILD: debug: avoid a build warning related to epoll_wait() in debug code
This patch introduces the "client-sigalgs" keyword for the server line,
which allows to configure the list of server signature algorithms
negociated during the handshake. Also available as
"ssl-default-server-client-sigalgs" in the global section.
This patch introduces the "sigalgs" keyword for the server line, which
allows to configure the list of server signature algorithms negociated
during the handshake. Also available as "ssl-default-server-sigalgs" in
the global section.
It appears that dconv dislikes the "see also" part being on the same line as
the regular paragraph. The beginning of the line does not show up in the
rendered version.
Attempt to fix this by inserting an additional newline which is consistent with
other options.
Complementary fix to ac456ab ("DOC: config: fix rfc7239 converter examples")
since somehow I managed to overlook one example..
This needs to be backported in 2.8 with ac456ab.
To prevent bogus matches, var() does not default to string type anymore
since 44c5ff6 ("MEDIUM: vars: make the var() sample fetch function really
return type ANY).
Thanks to the above fix, haproxy now returns an error if var() is used
within an ACL or IF condition and the matching type is not explicitly
set.
However, the documentation was not updated to reflect this change.
This partially fixes GH #2087 and must be backported up to 2.6.
Commit 511ddd5 introduced tune.quic.socket-owner parameter related to
QUIC socket behaviour. However it was misspelled in configuration.txt in
'bind' section as tune.quic.conn-owner.
Released version 2.8.0 with the following main changes :
- MINOR: compression: Improve the way Vary header is added
- BUILD: makefile: search for SSL_INC/wolfssl before SSL_INC
- MINOR: init: pre-allocate kernel data structures on init
- DOC: install: add details about WolfSSL
- BUG/MINOR: ssl_sock: add check for ha_meth
- BUG/MINOR: thread: add a check for pthread_create
- BUILD: init: print rlim_cur as regular integer
- DOC: install: specify the minimum openssl version recommended
- CLEANUP: mux-quic: remove unneeded fields in qcc
- MINOR: mux-quic: remove nb_streams from qcc
- MINOR: quic: fix stats naming for flow control BLOCKED frames
- BUG/MEDIUM: mux-quic: only set EOI on FIN
- BUG/MEDIUM: threads: fix a tiny race in thread_isolate()
- DOC: config: fix rfc7239 converter examples
- DOC: quic: remove experimental status for QUIC
- CLEANUP: mux-quic: rename functions for mux_ops
- CLEANUP: mux-quic: rename internal functions
- BUG/MINOR: mux-h2: refresh the idle_timer when the mux is empty
- DOC: config: Fix bind/server/peer documentation in the peers section
- BUILD: Makefile: use -pthread not -lpthread when threads are enabled
- CLEANUP: doc: remove 21 totally obsolete docs
- DOC: install: mention the common strict-aliasing warning on older compilers
- DOC: install: clarify a few points on the wolfSSL build method
- MINOR: quic: Add QUIC connection statistical counters values to "show quic"
- EXAMPLES: update the basic-config-edge file for 2.8
- MINOR: quic/cli: clarify the "show quic" help message
- MINOR: version: mention that it's LTS now.
These were docs for very old design thoughts or internal subsystems
which are now totally irrelevant and even misleading. Those with some
outdated ideas mixed with useful stuff were kept though.
Documentation about bind and server directives in the peers section was
retrieved from the proxy part but there are some limitations, especially for
the bind directive. And the same is true for the peer directive. It is
forbidden to have several listening addresses. Multiple addresses or port
range are not allowed.
Here, only the documentation is fixed. The configuration parsing will be
improved later to trigger errors on bad uses.
In addition, it is also specified that unix socket are supported.
This patch partially fixes the issue #2066. It should be backported to all
stable versions.
QUIC support can now be considered production-ready. As such, remove all
statements on the documentation concerning its experimental status.
Do not backport this one.
Some rfc7239 converter examples were not working and thus were misleading.
Fixing rfc7239_n2nn and rfc7239_n2np usage examples.
As both converters were introduced in 2.8, no backport needed.
The Linux kernel maintains data structures to track a processes' open file
descriptors, and it expands these structures as necessary when FD usage grows
(at every FD=2^X starting at 64). However when threading is in use, during
expansion the kernel will pause (observed up to 47ms) while it waits for thread
synchronization (see https://bugzilla.kernel.org/show_bug.cgi?id=217366).
This change addresses the issue and avoids the random pauses by opening the
maximum file descriptor during initialization, so that expansion will not occur
while processing traffic.
Released version 2.8-dev13 with the following main changes :
- DOC: add size format section to manual
- CLEANUP: mux-quic/h3: complete BUG_ON with comments
- MINOR: quic: remove return val of quic_aead_iv_build()
- MINOR: quic: use WARN_ON for encrypt failures
- BUG/MINOR: quic: handle Tx packet allocation failure properly
- MINOR: quic: fix alignment of oneline show quic
- MEDIUM: stconn/applet: Allow SF_SL_EOS flag alone
- MEDIUM: stconn: make the SE_FL_ERR_PENDING to ERROR transition systematic
- DOC: internal: add a bit of documentation for the stconn closing conditions
- DOC/MINOR: config: Fix typo in description for `ssl_bc` in configuration.txt
- BUILD: quic: re-enable chacha20_poly1305 for libressl
- MINOR: mux-quic: set both EOI EOS for stream fin
- MINOR: mux-quic: only set EOS on RESET_STREAM recv
- MINOR: mux-quic: report error on stream-endpoint earlier
- BUILD: makefile: fix build issue on GNU make < 3.82
- BUG/MINOR: mux-h2: Check H2_SF_BODY_TUNNEL on H2S flags and not demux frame ones
- MINOR: mux-h2: Set H2_SF_ES_RCVD flag when decoding the HEADERS frame
- MINOR: mux-h2: Add a function to propagate termination flags from h2s to SE
- BUG/MEDIUM: mux-h2: Propagate termination flags when frontend SC is created
- DEV: add a Lua helper script for SSL keys logging
- CLEANUP: makefile: don't display a dummy features list without a target
- BUILD: makefile: do not erase build options for some build options
- MINOR: quic: Add low level traces (addresses, DCID)
- BUG/MINOR: quic: Wrong token length check (quic_generate_retry_token())
- BUG/MINOR: quic: Missing Retry token length on receipt
- MINOR: quic: Align "show quic" command help information
- CLEANUP: quic: Indentation fix quic_rx_pkt_retrieve_conn()
- CLEANUP: quic: Useless tests in qc_rx_pkt_handle()
- MINOR: quic: Add some counters at QUIC connection level
- MINOR: quic: Add a counter for sent packets
- MINOR: hlua: hlua_smp2lua_str() may LJMP
- MINOR: hlua: hlua_smp2lua() may LJMP
- MINOR: hlua: hlua_arg2lua() may LJMP
- DOC: hlua: document hlua_lua2arg() function
- DOC: hlua: document hlua_lua2smp() function
- BUG/MINOR: hlua: unsafe hlua_lua2smp() usage
- BUILD: makefile: commit the tiny FreeBSD makefile stub
- BUILD: makefile: fix build options when building tools first
- BUILD: ist: do not put a cast in an array declaration
- BUILD: ist: use the literal declaration for ist_lc/ist_uc under TCC
- BUILD: compiler: systematically set USE_OBSOLETE_LINKER with TCC
- DOC: install: update reference to known supported versions
- SCRIPTS: publish-release: update the umask to keep group write access
Fix a minor typo in the description of the `ssl_bc` sample fetch method described under
Section `7.3.4. Fetching samples at Layer 5` in configuration.txt. Changed `other` to `to`.
The conditions where ERR, EOS and EOI are found are not always
crystal clear, and the fact that there's still a good bunch of
original ones dating from the early days and that seem to test for
non-existing cases doesn't help either.
After auditing the code base and projecting the 3 main muxes' stream
termination conditions, with Christopher and Amaury we could establish
the current flags matrix which indicates both what each combination
means for each mux and when it is set by each of them (or not set and
for what reason).
It should be sufficient to void doubts when adding code or when chasing
a bug.
It *must not* be backported because it is highly specific to the latest
2.8-dev.
Released version 2.8-dev12 with the following main changes :
- BUILD: mjson: Fix warning about unused variables
- MINOR: spoe: Don't stop disabled proxies
- BUG/MEDIUM: filters: Don't deinit filters for disabled proxies during startup
- BUG/MINOR: hlua_fcn/queue: fix broken pop_wait()
- BUG/MINOR: hlua_fcn/queue: fix reference leak
- CLEANUP: hlua_fcn/queue: make queue:push() easier to read
- BUG/MINOR: quic: Buggy acknowlegments of acknowlegments function
- DEBUG: list: add DEBUG_LIST to purposely corrupt list heads after delete
- MINOR: stats: report the total number of warnings issued
- MINOR: stats: report the number of times the global maxconn was reached
- BUG/MINOR: mux-quic: do not prevent shutw on error
- BUG/MINOR: mux-quic: do not free frame already released by quic-conn
- BUG/MINOR: mux-quic: no need to subscribe for detach streams
- MINOR: mux-quic: add traces for stream wake
- MINOR: mux-quic: do not send STREAM frames if already subscribe
- MINOR: mux-quic: factorize send subscribing
- MINOR: mux-quic: simplify return path of qc_send()
- MEDIUM: quic: streamline error notification
- MEDIUM: mux-quic: adjust transport layer error handling
- MINOR: stats: report the listener's protocol along with the address in stats
- BUG/MEDIUM: mux-fcgi: Never set SE_FL_EOS without SE_FL_EOI or SE_FL_ERROR
- BUG/MEDIUM: mux-fcgi: Don't request more room if mux is waiting for more data
- MINOR: stconn: Add a cross-reference between SE descriptor
- BUG/MINOR: proxy: missing free in free_proxy for redirect rules
- MINOR: proxy: add http_free_redirect_rule() function
- BUG/MINOR: http_rules: fix errors paths in http_parse_redirect_rule()
- CLEANUP: http_act: use http_free_redirect_rule() to clean redirect act
- MINOR: tree-wide: use free_acl_cond() where relevant
- CLEANUP: acl: discard prune_acl_cond() function
- BUG/MINOR: cli: don't complain about empty command on empty lines
- MINOR: cli: add an option to display the uptime in the CLI's prompt
- MINOR: master/cli: also implement the timed prompt on the master CLI
- MINOR: cli: make "show fd" identify QUIC connections and listeners
- MINOR: httpclient: allow to disable the DNS resolvers of the httpclient
- BUILD: debug: fix build issue on 32-bit platforms in "debug dev task"
- MINOR: ncbuf: missing malloc checks in standalone code
- DOC: lua: fix core.{proxies,frontends,backends} visibility
- EXAMPLES: fix race condition in lua mailers script
- BUG/MINOR: errors: handle malloc failure in usermsgs_put()
- BUG/MINOR: log: fix memory error handling in parse_logsrv()
- BUG/MINOR: quic: Wrong redispatch for external data on connection socket
- MINOR: htx: add function to set EOM reliably
- MINOR: mux-quic: remove dedicated function to handle standalone FIN
- BUG/MINOR: mux-quic: properly handle buf alloc failure
- BUG/MINOR: mux-quic: handle properly recv ncbuf alloc failure
- BUG/MINOR: quic: do not alloc buf count on alloc failure
- BUG/MINOR: mux-quic: differentiate failure on qc_stream_desc alloc
- BUG/MINOR: mux-quic: free task on qc_init() app ops failure
- MEDIUM: session/ssl: return the SSL error string during a SSL handshake error
- CI: enable monthly Fedora Rawhide clang builds
- MEDIUM: mworker/cli: does not disconnect the master CLI upon error
- MINOR: stconn: Remove useless test on sedesc on detach to release the xref
- MEDIUM: proxy: stop emitting logs for internal proxies when stopping
- MINOR: ssl: add new sample ssl_c_r_dn
- BUG/MEDIUM: mux-h2: make sure control frames do not refresh the idle timeout
- BUILD: ssl: ssl_c_r_dn fetches uses functiosn only available since 1.1.1
- BUG/MINOR: mux-quic: handle properly Tx buf exhaustion
- BUG/MINOR: h3: missing goto on buf alloc failure
- BUILD: ssl: get0_verified chain is available on libreSSL
- BUG/MINOR: makefile: use USE_LIBATOMIC instead of USE_ATOMIC
- MINOR: mux-quic: add trace to stream rcv_buf operation
- MINOR: mux-quic: properly report end-of-stream on recv
- MINOR: mux-quic: uninline qc_attach_sc()
- BUG/MEDIUM: mux-quic: fix EOI for request without payload
- MINOR: checks: make sure spread-checks is used also at boot time
- BUG/MINOR: tcp-rules: Don't shortened the inspect-delay when EOI is set
- REGTESTS: log: Reduce response inspect-delay for last_rule.vtc
- DOC: config: Clarify conditions to shorten the inspect-delay for TCP rules
- CLEANUP: server: remove useless tmptrash assigments in srv_update_status()
- BUG/MINOR: server: memory leak in _srv_update_status_op() on server DOWN
- CLEANUP: check; Remove some useless assignments to NULL
- CLEANUP: stats: update the trash chunk where it's used
- MINOR: clock: measure the total boot time
- MINOR: stats: report the boot time in "show info"
- BUG/MINOR: checks: postpone the startup of health checks by the boot time
- MINOR: clock: provide a function to automatically adjust now_offset
- BUG/MINOR: clock: automatically adjust the internal clock with the boot time
- CLEANUP: fcgi-app; Remove useless assignment to NULL
- REGTESTS: log: Reduce again response inspect-delay for last_rule.vtc
- CI: drop Fedora m32 pipeline in favour of cross matrix
- MEDIUM: checks: Stop scheduling healthchecks during stopping stage
- MEDIUM: resolvers: Stop scheduling resolution during stopping stage
- BUG/MINOR: hlua: SET_SAFE_LJMP misuse in hlua_event_runner()
- BUG/MINOR: debug: fix pointer check in debug_parse_cli_task()
This patch addresses #1514, adds the ability to fetch DN of the root
ca that was in the chain when client certificate was verified during SSL
handshake.
Despite the doc not mentionning it, core.{proxies,frontends,backends}
methods are also available from init context.
(through core.register_init() functions)
Updating the documentation to reflect this possibility.
httpclient.resolvers.disabled allow to disable completely the resolvers
of the httpclient, prevents the creation of the "default" resolvers
section, and does not insert the http do-resolve rule in the proxies.
This provides more consistency between the master and the worker. When
"prompt timed" is passed on the master, the timed mode is toggled. When
enabled, for a master it will show the master process' uptime, and for
a worker it will show this worker's uptime. Example:
master> prompt timed
[0:00:00:50] master> show proc
#<PID> <type> <reloads> <uptime> <version>
11940 master 1 [failed: 0] 0d00h02m10s 2.8-dev11-474c14-21
# workers
11955 worker 0 0d00h00m59s 2.8-dev11-474c14-21
# old workers
11942 worker 1 0d00h02m10s 2.8-dev11-474c14-21
# programs
[0:00:00:58] master> @!11955
[0:00:01:03] 11955> @!11942
[0:00:02:17] 11942> @
[0:00:01:10] master>
Entering "prompt timed" toggles reporting of the process' uptime in
the prompt, which will report days, hours, minutes and seconds since
it was started. As discussed with Tim in issue #2145, this can be
convenient to roughly estimate the time between two outputs, as well
as detecting that a process failed to be reloaded for example.
Released version 2.8-dev11 with the following main changes :
- BUILD: debug: do not check the isolated_thread variable in non-threaded builds
- BUILD: quic: fix build warning when threads are disabled
- CI: more granular failure on generating build matrix
- CLEANUP: quic: No more used q_buf structure
- CLEANUP: quic: Rename several <buf> variables in quic_frame.(c|h)
- CLEANUP: quic: Typo fix for quic_connection_id pool
- BUG/MINOR: quic: Wrong key update cipher context initialization for encryption
- BUG/MEDIUM: cache: Don't request more room than the max allowed
- MEDIUM: stconn: Be sure to always be able to unblock a SC that needs room
- EXAMPLES: fix IPV6 support for lua mailers script
- BUILD: ssl: buggy -Werror=dangling-pointer since gcc 13.0
- DOC: stconn: Update comments about ABRT/SHUT for stconn structure
- BUG/MEDIUM: stats: Require more room if buffer is almost full
- DOC: configuration: add info about ssl-engine for 2.6
- BUG/MINOR: mux-quic: fix transport VS app CONNECTION_CLOSE
- BUG/MEDIUM: mux-quic: wakeup tasklet to close on error
- DEV: flags: add a script to decode most flags in the "show sess all" output
- BUG/MINOR: quic: Possible crash when dumping version information
- BUG/MINOR: config: make compression work again in defaults section
- BUG/MEDIUM: stream: Forward shutdowns when unhandled errors are caught
- MEDIUM: stream: Resync analyzers at the end of process_stream() on change
- DEV: flags: add missing stream flags to show-sess-to-flags
- DEV: flags/show-sess-to-flags: only retrieve hex digits from hex fields
- DEV: flags/show-sess-to-flags: add support for color output
- CLEANUP: src/listener.c: remove redundant NULL check
In the announcement of 2.6 is mentioned that the openssl engine
is not enabled by default.
This patch add the information to the configuration.txt.
This is related to GitHub Issue #1752.
Should be back ported to 2.6
Released version 2.8-dev10 with the following main changes :
- BUG/MINOR: stats: fix typo in `TotalSplicedBytesOut` field name
- REGTESTS: add success test, "set server" via fqdn
- MINOR: ssl: disable CRL checks with WolfSSL when no CRL file
- BUG/MINOR: stream/cli: fix stream age calculation in "show sess"
- MINOR: debug: clarify "debug dev stream" help message
- DEBUG: cli: add "debug dev task" to show/wake/expire/kill tasks and tasklets
- BUG/MINOR: ssl/sample: x509_v_err_str converter output when not found
- REGTESTS: ssl: simplify X509_V code check in ssl_client_auth.vtc
- BUILD: cli: fix build on Windows due to isalnum() implemented as a macro
- MINOR: activity: use a single macro to iterate over all fields
- MINOR: activity: show the line header inside the SHOW_VAL macro
- MINOR: activity: iterate over all fields in a main loop for dumping
- MINOR: activity: allow "show activity" to restart dumping on any line
- MINOR: activity: allow "show activity" to restart in the middle of a line
- DEV: haring: automatically disable DEBUG_STRICT
- DEV: haring: update readme to suggest using the same build options for haring
- BUG/MINOR: debug: fix incorrect profiling status reporting in show threads
- MINOR: debug: permit the "debug dev loop" to run under isolation
- BUG/MEDIUM: mux-h2: Properly handle end of request to expect data from server
- BUG/MINOR: mux-quic: prevent quic_conn error code to be overwritten
- MINOR: mux-quic: add trace event for local error
- MINOR: mux-quic: wake up after recv only if avail data
- MINOR: mux-quic: adjust local error API
- MINOR: mux-quic: report local error on stream endpoint asap
- MINOR: mux-quic: close connection asap on local error
- BUG/MINOR: debug: do not emit empty lines in thread dumps
- BUG/MINOR: mux-h2: Also expect data when waiting for a tunnel establishment
- BUG/MINOR: time: fix NS_TO_TV macro
- MEDIUM: debug: simplify the thread dump mechanism
- MINOR: debug: write panic dump to stderr one thread at a time
- MINOR: debug: make "show threads" properly iterate over all threads
- CLEANUP: debug: remove the now unused ha_thread_dump_all_to_trash()
- MINOR: ssl: allow to change the server signature algorithm
- MINOR: ssl: allow to change the signature algorithm for client authentication
- MINOR: cli: Use applet API to write output message
- MINOR: stats: Use the applet API to write data
- MINOR: peers: Use the applet API to send message
- MINOR: stconn: Add a field to specify the room needed by the SC to progress
- MEDIUM: tree-wide: Change sc API to specify required free space to progress
- BUG/MEDIUM: stconn: Unblock SC from stream if there is enough room to progrees
- MEDIUM: applet: Check room needed to unblock opposite SC when data was consumed
- MEDIUM: stconn: Check room needed to unblock SC on fast-forward
- MEDIUM: stconn: Check room needed to unblock opposite SC when data was sent
- MINOR: hlua_fcn: fix Server.is_draining() return type
- MINOR: hlua_fcn: add Server.is_backup()
- MINOR: hlua_fcn: add Server.is_dynamic()
- MINOR: hlua_fcn: add Server.tracking()
- MINOR: hlua_fcn: add Server.get_trackers()
- MINOR: hlua_fcn: add Server.get_proxy()
- MINOR: hlua_fcn: add Server.get_pend_conn() and Server.get_cur_sess()
- MINOR: hlua_fcn: add Proxy.get_srv_act() and Proxy.get_srv_bck()
- DOC: lua/event: add ServerEvent class header
- MINOR: server/event_hdl: publish macro helper
- MINOR: server/event_hdl: add SERVER_STATE event
- OPTIM: server: publish UP/DOWN events from STATE change
- MINOR: hlua: expose SERVER_STATE event
- MINOR: server/event_hdl: add SERVER_ADMIN event
- MINOR: hlua: expose SERVER_ADMIN event
- MINOR: checks/event_hdl: SERVER_CHECK event
- MINOR: hlua/event_hdl: expose SERVER_CHECK event
- MINOR: mailers/hlua: disable email sending from lua
- MINOR: hlua: expose proxy mailers
- EXAMPLES: add lua mailers script to replace tcpcheck mailers
- BUG/MINOR: hlua: spinning loop in hlua_socket_handler()
- MINOR: server: fix message report when IDRAIN is set and MAINT is cleared
- CLEANUP: hlua: hlua_register_task() may longjmp
- REGTESTS: use lua mailer script for mailers tests
- MINOR: hlua: declare hlua_{ref,pushref,unref} functions
- MINOR: hlua: declare hlua_gethlua() function
- MINOR: hlua: declare hlua_yieldk() function
- MINOR: hlua_fcn: add Queue class
- EXAMPLES: mailqueue for lua mailers script
- MINOR: quic: add format argument for "show quic"
- MINOR: quic: implement oneline format for "show quic"
- MINOR: config: allow cpu-map to take commas in lists of ranges
- CLEANUP: fix a few reported typos in code comments
- DOC: fix a few reported typos in the config and install doc
The function that cpu-map uses to parse CPU sets, parse_cpu_set(), was
etended in 2.4 with commit a80823543 ("MINOR: cfgparse: support the
comma separator on parse_cpu_set") to support commas between ranges.
But since it was quite late in the development cycle, by then it was
decided not to add a last-minute surprise and not to magically support
commas in cpu-map, hence the "comma_allowed" argument.
Since then we know that it was not the best choice, because the comma
is silently ignored in the cpu-map syntax, causing all sorts of
surprises in field with threads running on a single node for example.
In addition it's quite common to copy-paste a taskset line and put it
directly into the haproxy configuration.
This commit relaxes this rule an finally allows cpu-map to support
commas between ranges. It simply consists in removing the comma_allowed
argument in the parse_cpu_set() function. The doc was updated to
reflect this.
Add a new output format "oneline" for "show quic" command. This prints
one connection per line with minimal information. The objective is to
have an equivalent of the netstat/ss tools with just enough information
to quickly find connection which are misbehaving.
A legend is printed on the first line to describe the field columns
starting with a dash character.
This should be backported up to 2.7.
Add an extra optional argument for "show quic" to specify desired output
format. Its objective is to control the verbosity per connections. For
the moment, only "full" is supported, which is the already implemented
output with maximum information.
This should be backported up to 2.7.
Adding a new lua class: Queue.
This class provides a generic FIFO storage mechanism that may be shared
between multiple lua contexts to easily pass data between them, as stock
Lua doesn't provide easy methods for passing data between multiple
coroutines.
New Queue object may be obtained using core.queue()
(it works like core.concat() for a concat Class)
Lua documentation was updated (including some usage examples)
Proxy mailers, which are configured using "email-alert" directives
in proxy sections from the configuration, are now being exposed
directly in lua thanks to the proxy:get_mailers() method which
returns a class containing the various mailers settings if email
alerts are configured for the given proxy (else nil is returned).
Both the class and the proxy method were marked as LEGACY since this
feature relies on mailers configuration, and the goal here is to provide
the last missing bits of information to lua scripts in order to make
them capable of sending email alerts instead of relying on the soon-to-
be deprecated mailers implementation based on checks (see src/mailers.c)
Lua documentation was updated accordingly.
Exposing a new hlua function, available from body or init contexts, that
forcefully disables the sending of email alerts even if the mailers are
defined in haproxy configuration.
This will help for sending email directly from lua.
(prevent legacy email sending from intefering with lua)
Exposing SERVER_CHECK event through the lua API.
New lua class named ServerEventCheck was added to provide additional
data for SERVER_CHECK event.
Lua documentation was updated accordingly.
