upstream/haproxy
1
0
Fork 0
mirror of https://github.com/haproxy/haproxy.git synced 2026-04-22 23:02:34 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions 15
haproxy/src/log.c

5955 lines
171 KiB
C
Raw Normal View History

[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-25 20:48:02 -04:00
/*
* General logging functions.
*
[MEDIUM] implement a monotonic internal clock If the system date is set backwards while haproxy is running, some scheduled events are delayed by the amount of time the clock went backwards. This is particularly problematic on systems where the date is set at boot, because it seldom happens that health-checks do not get sent for a few hours. Before switching to use clock_gettime() on systems which provide it, we can at least ensure that the clock is not going backwards and maintain two clocks : the "date" which represents what the user wants to see (mostly for logs), and an internal date stored in "now", used for scheduled events.
2008-06-22 11:18:02 -04:00
* Copyright 2000-2008 Willy Tarreau <w@1wt.eu>
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-25 20:48:02 -04:00
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version
* 2 of the License, or (at your option) any later version.
*
*/
MEDIUM: log-format: make the format parser more robust and more extensible The log-format parser reached a limit making it hard to add new features. It also suffers from a weak handling of certain incorrect corner cases, for example "%{foo}" is emitted as a litteral while syntactically it's an argument to no variable. Also the argument parser had to redo some of the job with some cases causing minor memory leaks (eg: ignored args). This work aims at improving the situation so that slightly better reporting is possible and that it becomes possible to extend the log format. The code has a few more states but looks significantly simpler. The parser is now capable of reporting ignored arguments and truncated lines.
2012-12-20 15:23:42 -05:00
#include <ctype.h>
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-25 20:48:02 -04:00
#include <stdarg.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <syslog.h>
#include <time.h>
#include <unistd.h>
[MEDIUM] add support for logging via a UNIX socket The code in haproxy-1.3.13.1 only supports syslogging to an internet address. The attached patch: - Adds support for syslogging to a UNIX domain socket (e.g., /dev/log). If the address field begins with '/' (absolute file path), then AF_UNIX is used to construct the socket. Otherwise, AF_INET is used. - Achieves clean single-source build on both Mac OS X and Linux (sockaddr_in.sin_len and sockaddr_un.sun_len field aren't always present). For handling sendto() failures in send_log(), it appears that the existing code is fine (no need to close/recreate socket) for both UDP and UNIX-domain syslog server. So I left things alone (did not close/recreate socket). Closing/recreating socket after each failure would also work, but would lead to increased amount of unnecessary socket creation/destruction if syslog is temporarily unavailable for some reason (especially for verbose loggers). Please consider this patch for inclusion into the upstream haproxy codebase.
2007-12-05 04:47:29 -05:00
#include <errno.h>
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-25 20:48:02 -04:00
#include <sys/time.h>
BUILD: log: iovec requires to include sys/uio.h on OpenBSD The following commit merged into 1.6-dev6 broke the build on OpenBSD : 609ac2a ("MEDIUM: log: replace sendto() with sendmsg() in __send_log()") Including sys/uio.h is enough to fix this. This fix needs to be backported to 1.6.
2016-08-10 12:30:56 -04:00
#include <sys/uio.h>
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-25 20:48:02 -04:00
REORG: include: update all files to use haproxy/api.h or api-t.h if needed All files that were including one of the following include files have been updated to only include haproxy/api.h or haproxy/api-t.h once instead: - common/config.h - common/compat.h - common/compiler.h - common/defaults.h - common/initcall.h - common/tools.h The choice is simple: if the file only requires type definitions, it includes api-t.h, otherwise it includes the full api.h. In addition, in these files, explicit includes for inttypes.h and limits.h were dropped since these are now covered by api.h and api-t.h. No other change was performed, given that this patch is large and affects 201 files. At least one (tools.h) was already freestanding and didn't get the new one added.
2020-05-27 06:58:42 -04:00
#include <haproxy/api.h>
CLEANUP: tree-wide: Remove any ref to stream-interfaces Stream-interfaces are gone. Corresponding files can be safely be removed. In addition, comments are updated accordingly.
2022-04-04 05:29:28 -04:00
#include <haproxy/applet.h>
MEDIUM: log: adds log forwarding section. Log forwarding: It is possible to declare one or multiple log forwarding section, haproxy will forward all received log messages to a log servers list. log-forward <name> Creates a new log forwarder proxy identified as <name>. bind <addr> [param*] Used to configure a log udp listener to receive messages to forward. Only udp listeners are allowed, address must be prefixed using 'udp@', 'udp4@' or 'udp6@'. This supports for all "bind" parameters found in 5.1 paragraph but most of them are irrelevant for udp/syslog case. log global log <address> [len <length>] [format <format>] [sample <ranges>:<smp_size>] <facility> [<level> [<minlevel>]] Used to configure target log servers. See more details on proxies documentation. If no format specified, haproxy tries to keep the incoming log format. Configured facility is ignored, except if incoming message does not present a facility but one is mandatory on the outgoing format. If there is no timestamp available in the input format, but the field exists in output format, haproxy will use the local date. Example: global log stderr format iso local7 ring myring description "My local buffer" format rfc5424 maxlen 1200 size 32764 timeout connect 5s timeout server 10s # syslog tcp server server mysyslogsrv 127.0.0.1:514 log-proto octet-count log-forward sylog-loadb bind udp4@127.0.0.1:1514 # all messages on stderr log global # all messages on local tcp syslog server log ring@myring local0 # load balance messages on 4 udp syslog servers log 127.0.0.1:10001 sample 1:4 local0 log 127.0.0.1:10002 sample 2:4 local0 log 127.0.0.1:10003 sample 3:4 local0 log 127.0.0.1:10004 sample 4:4 local0
2020-07-07 08:19:42 -04:00
#include <haproxy/cfgparse.h>
REORG: time: move time-keeping code and variables to clock.c There is currently a problem related to time keeping. We're mixing the functions to perform calculations with the os-dependent code needed to retrieve and adjust the local time. This patch extracts from time.{c,h} the parts that are solely dedicated to time keeping. These are the "now" or "before_poll" variables for example, as well as the various now_*() functions that make use of gettimeofday() and clock_gettime() to retrieve the current time. The "tv_*" functions moved there were also more appropriately renamed to "clock_*". Other parts used to compute stolen time are in other files, they will have to be picked next.
2021-10-08 03:33:24 -04:00
#include <haproxy/clock.h>
CLEANUP: include: tree-wide alphabetical sort of include files This patch fixes all the leftovers from the include cleanup campaign. There were not that many (~400 entries in ~150 files) but it was definitely worth doing it as it revealed a few duplicates.
2020-06-09 03:07:15 -04:00
#include <haproxy/fd.h>
REORG: include: move frontend.h to haproxy/frontend.h There was no type file for this one, it only contains frontend_accept().
2020-06-04 05:23:07 -04:00
#include <haproxy/frontend.h>
CLEANUP: include: tree-wide alphabetical sort of include files This patch fixes all the leftovers from the include cleanup campaign. There were not that many (~400 entries in ~150 files) but it was definitely worth doing it as it revealed a few duplicates.
2020-06-09 03:07:15 -04:00
#include <haproxy/global.h>
REORG: include: split common/http.h into haproxy/http{,-t}.h So the enums and structs were placed into http-t.h and the functions into http.h. This revealed that several files were dependeng on http.h but not including it, as it was silently inherited via other files.
2020-06-02 13:11:26 -04:00
#include <haproxy/http.h>
BUILD: tree-wide: add missing http_ana.h from many places At least 6 files make use of s->txn without including http_ana which defines it. They used to get it from other includes.
2021-10-06 12:56:42 -04:00
#include <haproxy/http_ana.h>
MEDIUM: log: adds log forwarding section. Log forwarding: It is possible to declare one or multiple log forwarding section, haproxy will forward all received log messages to a log servers list. log-forward <name> Creates a new log forwarder proxy identified as <name>. bind <addr> [param*] Used to configure a log udp listener to receive messages to forward. Only udp listeners are allowed, address must be prefixed using 'udp@', 'udp4@' or 'udp6@'. This supports for all "bind" parameters found in 5.1 paragraph but most of them are irrelevant for udp/syslog case. log global log <address> [len <length>] [format <format>] [sample <ranges>:<smp_size>] <facility> [<level> [<minlevel>]] Used to configure target log servers. See more details on proxies documentation. If no format specified, haproxy tries to keep the incoming log format. Configured facility is ignored, except if incoming message does not present a facility but one is mandatory on the outgoing format. If there is no timestamp available in the input format, but the field exists in output format, haproxy will use the local date. Example: global log stderr format iso local7 ring myring description "My local buffer" format rfc5424 maxlen 1200 size 32764 timeout connect 5s timeout server 10s # syslog tcp server server mysyslogsrv 127.0.0.1:514 log-proto octet-count log-forward sylog-loadb bind udp4@127.0.0.1:1514 # all messages on stderr log global # all messages on local tcp syslog server log ring@myring local0 # load balance messages on 4 udp syslog servers log 127.0.0.1:10001 sample 1:4 local0 log 127.0.0.1:10002 sample 2:4 local0 log 127.0.0.1:10003 sample 3:4 local0 log 127.0.0.1:10004 sample 4:4 local0
2020-07-07 08:19:42 -04:00
#include <haproxy/listener.h>
MEDIUM: log/balance: leverage lbprm api for log load-balancing log load-balancing implementation was not seamlessly integrated within lbprm API. The consequence is that it could become harder to maintain over time since it added some specific cases just for the log backend. Moreover, it resulted in some code duplication since balance algorithms that are common to logs and regular (tcp, http) backends were specifically rewritten for log backends. Thanks to the previous commit, we now have all the prerequisites to make log load-balancing fully leverage lbprm logic. Thus in this patch we make __do_send_log_backend() use existing lbprm algorithms, and we no longer require log-specific lbprm initialization in cfgparse.c and in postcheck_log_backend(). As a bonus, for log backends this allows weighed algorithms to properly support weights (ie: roundrobin, random and log-hash) since we now leverage the same lb algorithms that we use for tcp/http backends (doc was updated).
2024-03-28 10:42:37 -04:00
#include <haproxy/lb_chash.h>
#include <haproxy/lb_fwrr.h>
#include <haproxy/lb_map.h>
#include <haproxy/lb_ss.h>
REORG: include: move log.h to haproxy/log{,-t}.h The current state of the logging is a real mess. The main problem is that almost all files include log.h just in order to have access to the alert/warning functions like ha_alert() etc, and don't care about logs. But log.h also deals with real logging as well as log-format and depends on stream.h and various other things. As such it forces a few heavy files like stream.h to be loaded early and to hide missing dependencies depending where it's loaded. Among the missing ones is syslog.h which was often automatically included resulting in no less than 3 users missing it. Among 76 users, only 5 could be removed, and probably 70 don't need the full set of dependencies. A good approach would consist in splitting that file in 3 parts: - one for error output ("errors" ?). - one for log_format processing - and one for actual logging.
2020-06-04 16:01:04 -04:00
#include <haproxy/log.h>
MEDIUM: log: adds log forwarding section. Log forwarding: It is possible to declare one or multiple log forwarding section, haproxy will forward all received log messages to a log servers list. log-forward <name> Creates a new log forwarder proxy identified as <name>. bind <addr> [param*] Used to configure a log udp listener to receive messages to forward. Only udp listeners are allowed, address must be prefixed using 'udp@', 'udp4@' or 'udp6@'. This supports for all "bind" parameters found in 5.1 paragraph but most of them are irrelevant for udp/syslog case. log global log <address> [len <length>] [format <format>] [sample <ranges>:<smp_size>] <facility> [<level> [<minlevel>]] Used to configure target log servers. See more details on proxies documentation. If no format specified, haproxy tries to keep the incoming log format. Configured facility is ignored, except if incoming message does not present a facility but one is mandatory on the outgoing format. If there is no timestamp available in the input format, but the field exists in output format, haproxy will use the local date. Example: global log stderr format iso local7 ring myring description "My local buffer" format rfc5424 maxlen 1200 size 32764 timeout connect 5s timeout server 10s # syslog tcp server server mysyslogsrv 127.0.0.1:514 log-proto octet-count log-forward sylog-loadb bind udp4@127.0.0.1:1514 # all messages on stderr log global # all messages on local tcp syslog server log ring@myring local0 # load balance messages on 4 udp syslog servers log 127.0.0.1:10001 sample 1:4 local0 log 127.0.0.1:10002 sample 2:4 local0 log 127.0.0.1:10003 sample 3:4 local0 log 127.0.0.1:10004 sample 4:4 local0
2020-07-07 08:19:42 -04:00
#include <haproxy/proxy.h>
CLEANUP: include: tree-wide alphabetical sort of include files This patch fixes all the leftovers from the include cleanup campaign. There were not that many (~400 entries in ~150 files) but it was definitely worth doing it as it revealed a few duplicates.
2020-06-09 03:07:15 -04:00
#include <haproxy/sample.h>
REORG: rename cs_utils.h to sc_strm.h This file contains all the stream-connector functions that are specific to application layers of type stream. So let's name it accordingly so that it's easier to figure what's located there. The alphabetical ordering of include files was preserved.
2022-05-27 03:25:10 -04:00
#include <haproxy/sc_strm.h>
CLEANUP: include: tree-wide alphabetical sort of include files This patch fixes all the leftovers from the include cleanup campaign. There were not that many (~400 entries in ~150 files) but it was definitely worth doing it as it revealed a few duplicates.
2020-06-09 03:07:15 -04:00
#include <haproxy/sink.h>
REORG: include: move ssl_sock.h to haproxy/ssl_sock{,-t}.h Almost nothing changed, just moved a static inline at the end and moved an export from the types to the main file.
2020-06-04 14:30:20 -04:00
#include <haproxy/ssl_sock.h>
REORG: stconn: rename conn_stream.{c,h} to stconn.{c,h} There's no more reason for keepin the code and definitions in conn_stream, let's move all that to stconn. The alphabetical ordering of include files was adjusted.
2022-05-27 03:47:12 -04:00
#include <haproxy/stconn.h>
REORG: include: move stream.h to haproxy/stream{,-t}.h This one was not easy because it was embarking many includes with it, which other files would automatically find. At least global.h, arg.h and tools.h were identified. 93 total locations were identified, 8 additional includes had to be added. In the rare files where it was possible to finalize the sorting of includes by adjusting only one or two extra lines, it was done. But all files would need to be rechecked and cleaned up now. It was the last set of files in types/ and proto/ and these directories must not be reused anymore.
2020-06-04 17:46:14 -04:00
#include <haproxy/stream.h>
MINOR: log/backend: prevent tcp-{request,response} use with LOG mode We start implementing some postparsing compatibility checks for log backends. Here we report a warning if user tries to use tcp-{request,response} rules with log backend, and we properly ignore such rules when inherited from defaults section.
2023-11-16 04:48:34 -05:00
#include <haproxy/action.h>
REORG: include: move time.h from common/ to haproxy/ This one is included almost everywhere and used to rely on a few other .h that are not needed (unistd, stdlib, standard.h). It could possibly make sense to split it into multiple parts to distinguish operations performed on timers and the internal time accounting, but at this point it does not appear much important.
2020-06-01 05:05:15 -04:00
#include <haproxy/time.h>
MEDIUM: log/balance: support for the "hash" lb algorithm hash lb algorithm can be configured with the "log-balance hash <cnv_list>" directive. With this algorithm, the user specifies a converter list with <cnv_list>. The produced log message will be passed as-is to the provided converter list, and the resulting hash will be used to select the log server that will receive the log message.
2023-09-19 04:51:53 -04:00
#include <haproxy/hash.h>
CLEANUP: include: tree-wide alphabetical sort of include files This patch fixes all the leftovers from the include cleanup campaign. There were not that many (~400 entries in ~150 files) but it was definitely worth doing it as it revealed a few duplicates.
2020-06-09 03:07:15 -04:00
#include <haproxy/tools.h>
MEDIUM: ring: change the ring reader to use the new vector-based API now The code now looks cleaner and more easily shows what still needs to be addressed. There are not that many changes in practice, these are mostly mechanical, essentially hiding the buffer from the callers.
2024-02-27 01:58:26 -05:00
#include <haproxy/vecpair.h>
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-25 20:48:02 -04:00
MINOR: log: adds counters on received syslog messages. This patch adds a global counter of received syslog messages and this one is exported on CLI "show info" as "CumRecvLogs". This patch also updates internal conn counter and freq of the listener and the proxy for each received log message to prepare a further export on the "show stats".
2020-07-09 17:23:34 -04:00
/* global recv logs counter */
int cum_log_messages;
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-25 20:48:02 -04:00
MEDIUM: log: adds log forwarding section. Log forwarding: It is possible to declare one or multiple log forwarding section, haproxy will forward all received log messages to a log servers list. log-forward <name> Creates a new log forwarder proxy identified as <name>. bind <addr> [param*] Used to configure a log udp listener to receive messages to forward. Only udp listeners are allowed, address must be prefixed using 'udp@', 'udp4@' or 'udp6@'. This supports for all "bind" parameters found in 5.1 paragraph but most of them are irrelevant for udp/syslog case. log global log <address> [len <length>] [format <format>] [sample <ranges>:<smp_size>] <facility> [<level> [<minlevel>]] Used to configure target log servers. See more details on proxies documentation. If no format specified, haproxy tries to keep the incoming log format. Configured facility is ignored, except if incoming message does not present a facility but one is mandatory on the outgoing format. If there is no timestamp available in the input format, but the field exists in output format, haproxy will use the local date. Example: global log stderr format iso local7 ring myring description "My local buffer" format rfc5424 maxlen 1200 size 32764 timeout connect 5s timeout server 10s # syslog tcp server server mysyslogsrv 127.0.0.1:514 log-proto octet-count log-forward sylog-loadb bind udp4@127.0.0.1:1514 # all messages on stderr log global # all messages on local tcp syslog server log ring@myring local0 # load balance messages on 4 udp syslog servers log 127.0.0.1:10001 sample 1:4 local0 log 127.0.0.1:10002 sample 2:4 local0 log 127.0.0.1:10003 sample 3:4 local0 log 127.0.0.1:10004 sample 4:4 local0
2020-07-07 08:19:42 -04:00
/* log forward proxy list */
struct proxy *cfg_log_forward;
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
struct log_fmt_st {
BUG/MEDIUM: logs: segfault writing to log from Lua Michael Ezzell reported a bug causing haproxy to segfault during startup when trying to send syslog message from Lua. The function __send_log() can be called with *p that is NULL and/or when the configuration is not fully parsed, as is the case with Lua. This patch fixes this problem by using individual vectors instead of the pre-generated strings log_htp and log_htp_rfc5424. Also, this patch fixes a problem causing haproxy to write the wrong pid in the logs -- the log_htp(_rfc5424) strings were generated at the haproxy start, but "pid" value would be changed after haproxy is started in daemon/systemd mode.
2015-10-01 07:18:13 -04:00
char *name;
};
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
static const struct log_fmt_st log_formats[LOG_FORMATS] = {
BUG/MEDIUM: local log format regression. Since 2.3 default local log format always adds hostame field. This behavior change was due to log/sink re-work, because according to rfc3164 the hostname field is mandatory. This patch re-introduce a legacy "local" format which is analog to rfc3164 but with hostname stripped. This is the new default if logs are generated by haproxy. To stay compliant with previous configurations, the option "log-send-hostname" acts as if the default format is switched to rfc3164. This patch addresses the github issue #963 This patch should be backported in branches >= 2.3.
2020-11-27 10:24:34 -05:00
[LOG_FORMAT_LOCAL] = {
.name = "local",
},
BUG/MEDIUM: logs: segfault writing to log from Lua Michael Ezzell reported a bug causing haproxy to segfault during startup when trying to send syslog message from Lua. The function __send_log() can be called with *p that is NULL and/or when the configuration is not fully parsed, as is the case with Lua. This patch fixes this problem by using individual vectors instead of the pre-generated strings log_htp and log_htp_rfc5424. Also, this patch fixes a problem causing haproxy to write the wrong pid in the logs -- the log_htp(_rfc5424) strings were generated at the haproxy start, but "pid" value would be changed after haproxy is started in daemon/systemd mode.
2015-10-01 07:18:13 -04:00
[LOG_FORMAT_RFC3164] = {
.name = "rfc3164",
},
[LOG_FORMAT_RFC5424] = {
.name = "rfc5424",
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
},
[LOG_FORMAT_PRIO] = {
.name = "priority",
MEDIUM: log: support a new "short" format This format is meant to be used with local file descriptors. It emits messages only prefixed with a level, removing all the process name, system name, date and so on. It is similar to the printk() format used on Linux. It's suitable to be sent to a local logger compatible with systemd's output format. Note that the facility is still required but not used, hence it is suggested to use "daemon" to remind that it's a local logger. Example : log stdout format short daemon # send everything to stdout log stderr format short daemon notice # send important events to stderr
2018-11-12 02:45:00 -05:00
},
[LOG_FORMAT_SHORT] = {
.name = "short",
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
},
[LOG_FORMAT_TIMED] = {
.name = "timed",
},
[LOG_FORMAT_ISO] = {
.name = "iso",
MEDIUM: log: support a new "short" format This format is meant to be used with local file descriptors. It emits messages only prefixed with a level, removing all the process name, system name, date and so on. It is similar to the printk() format used on Linux. It's suitable to be sent to a local logger compatible with systemd's output format. Note that the facility is still required but not used, hence it is suggested to use "daemon" to remind that it's a local logger. Example : log stdout format short daemon # send everything to stdout log stderr format short daemon notice # send important events to stderr
2018-11-12 02:45:00 -05:00
},
MEDIUM: log: add a new "raw" format This format is pretty similar to the previous "short" format except that it also removes the severity level. Thus only the raw message is sent. This is suitable for use in containers, where only the raw information is expected and where the severity is supposed to come from the file descriptor used.
2018-11-12 05:57:56 -05:00
[LOG_FORMAT_RAW] = {
.name = "raw",
},
MEDIUM: logs: add support for RFC5424 header format per logger The function __send_log() iterates over senders and passes the header as the first vector to sendmsg(), thus it can send a logger-specific header in each message. A new logger arguments "format rfc5424" should be used in order to enable RFC5424 header format. For example: log 10.2.3.4:1234 len 2048 format rfc5424 local2 info
2015-09-22 10:05:32 -04:00
};
MINOR: log: add log_orig_to_str() function Get human readable string from log_orig enum members.
2024-05-06 08:33:16 -04:00
/* get human readable representation for log_orig enum members */
const char *log_orig_to_str(enum log_orig orig)
{
switch (orig) {
case LOG_ORIG_SESS_ERROR:
return "sess_error";
case LOG_ORIG_SESS_KILL:
return "sess_killed";
case LOG_ORIG_TXN_ACCEPT:
return "txn_accept";
case LOG_ORIG_TXN_REQUEST:
return "txn_request";
case LOG_ORIG_TXN_CONNECT:
return "txn_connect";
case LOG_ORIG_TXN_RESPONSE:
return "txn_response";
case LOG_ORIG_TXN_CLOSE:
return "txn_close";
default:
break;
}
return "unspec";
}
MEDIUM: log: add a new log format flag "E" The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It mostly makes sense to use it in the RFC5424 structured-data log formats. Example: log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 07:23:03 -05:00
/*
* This map is used with all the FD_* macros to check whether a particular bit
CLEANUP: assorted typo fixes in the code and comments This is 37th iteration of typo fixes
2023-11-21 13:54:16 -05:00
* is set or not. Each bit represents an ASCII code. ha_bit_set() sets those
MINOR: logs: use the new bitmap functions instead of fd_sets for encoding maps The fd_sets we've been using in the log encoding functions are not portable and were shown to break at least under Cygwin. This patch gets rid of them in favor of the new bitmap functions. It was verified with the config below that the log output was exactly the same before and after the change : defaults mode http option httplog log stdout local0 timeout client 1s timeout server 1s timeout connect 1s frontend foo bind :8001 capture request header chars len 255 backend bar option httpchk "GET" "/" "HTTP/1.0\r\nchars: \x01\x02\x03\x04\x05\x06\x07\x08\x09\x0b\x0c\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f\x20\x21\x22\x23\x24\x25\x26\x27\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f\x30\x31\x32\x33\x34\x35\x36\x37\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f\x40\x41\x42\x43\x44\x45\x46\x47\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f\x50\x51\x52\x53\x54\x55\x56\x57\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f\x60\x61\x62\x63\x64\x65\x66\x67\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f\x70\x71\x72\x73\x74\x75\x76\x77\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" server foo 127.0.0.1:8001 check
2019-06-07 05:10:07 -04:00
* bytes which should be escaped. When ha_bit_test() returns non-zero, it means
* that the byte should be escaped. Be careful to always pass bytes from 0 to
* 255 exclusively to the macros.
MEDIUM: log: add a new log format flag "E" The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It mostly makes sense to use it in the RFC5424 structured-data log formats. Example: log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 07:23:03 -05:00
*/
MINOR: log: add no_escape_map to bypass escape with _lf_encode_bytes() Providing no_escape_map as <map> argument to _lf_encode_bytes() function will make the function skip escaping since the map is empty. This is for convenience, as it might be useful to call lf_encode_chunk() to encoding binary data without escaping it.
2024-04-26 08:23:43 -04:00
long no_escape_map[(256/8) / sizeof(long)];
MINOR: logs: use the new bitmap functions instead of fd_sets for encoding maps The fd_sets we've been using in the log encoding functions are not portable and were shown to break at least under Cygwin. This patch gets rid of them in favor of the new bitmap functions. It was verified with the config below that the log output was exactly the same before and after the change : defaults mode http option httplog log stdout local0 timeout client 1s timeout server 1s timeout connect 1s frontend foo bind :8001 capture request header chars len 255 backend bar option httpchk "GET" "/" "HTTP/1.0\r\nchars: \x01\x02\x03\x04\x05\x06\x07\x08\x09\x0b\x0c\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f\x20\x21\x22\x23\x24\x25\x26\x27\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f\x30\x31\x32\x33\x34\x35\x36\x37\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f\x40\x41\x42\x43\x44\x45\x46\x47\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f\x50\x51\x52\x53\x54\x55\x56\x57\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f\x60\x61\x62\x63\x64\x65\x66\x67\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f\x70\x71\x72\x73\x74\x75\x76\x77\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" server foo 127.0.0.1:8001 check
2019-06-07 05:10:07 -04:00
long rfc5424_escape_map[(256/8) / sizeof(long)];
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
long json_escape_map[(256/8) / sizeof(long)];
MINOR: logs: use the new bitmap functions instead of fd_sets for encoding maps The fd_sets we've been using in the log encoding functions are not portable and were shown to break at least under Cygwin. This patch gets rid of them in favor of the new bitmap functions. It was verified with the config below that the log output was exactly the same before and after the change : defaults mode http option httplog log stdout local0 timeout client 1s timeout server 1s timeout connect 1s frontend foo bind :8001 capture request header chars len 255 backend bar option httpchk "GET" "/" "HTTP/1.0\r\nchars: \x01\x02\x03\x04\x05\x06\x07\x08\x09\x0b\x0c\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f\x20\x21\x22\x23\x24\x25\x26\x27\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f\x30\x31\x32\x33\x34\x35\x36\x37\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f\x40\x41\x42\x43\x44\x45\x46\x47\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f\x50\x51\x52\x53\x54\x55\x56\x57\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f\x60\x61\x62\x63\x64\x65\x66\x67\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f\x70\x71\x72\x73\x74\x75\x76\x77\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" server foo 127.0.0.1:8001 check
2019-06-07 05:10:07 -04:00
long hdr_encode_map[(256/8) / sizeof(long)];
long url_encode_map[(256/8) / sizeof(long)];
long http_encode_map[(256/8) / sizeof(long)];
MEDIUM: log: add a new log format flag "E" The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It mostly makes sense to use it in the RFC5424 structured-data log formats. Example: log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 07:23:03 -05:00
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-25 20:48:02 -04:00
const char *log_facilities[NB_LOG_FACILITIES] = {
"kern", "user", "mail", "daemon",
"auth", "syslog", "lpr", "news",
"uucp", "cron", "auth2", "ftp",
"ntp", "audit", "alert", "cron2",
"local0", "local1", "local2", "local3",
"local4", "local5", "local6", "local7"
};
const char *log_levels[NB_LOG_LEVELS] = {
"emerg", "alert", "crit", "err",
"warning", "notice", "info", "debug"
};
MINOR: log: add a new flag 'L' for locally processed requests People who use "option dontlog-normal" are bothered with redirects and stats being logged and reported as errors in the logs ("PR" = proxy blocked the request). This patch introduces a new flag 'L' for when a request is locally processed, that is not considered as an error by the log filters. That way we know a request was intercepted and processed by haproxy without logging the line when "option dontlog-normal" is in effect.
2013-06-10 10:42:09 -04:00
const char sess_term_cond[16] = "-LcCsSPRIDKUIIII"; /* normal, Local, CliTo, CliErr, SrvTo, SrvErr, PxErr, Resource, Internal, Down, Killed, Up, -- */
[MEDIUM] added the "reqtarpit" and "reqitarpit" features It is now possible to tarpit connections based on regex matches. The tarpit timeout is equal to the contimeout. A 500 server error response is faked, and the logs show the status flags as "PT" which indicate the connection has been tarpitted.
2006-09-03 03:56:00 -04:00
const char sess_fin_state[8] = "-RCHDLQT"; /* cliRequest, srvConnect, srvHeader, Data, Last, Queue, Tarpit */
OPTIM: log: declare empty buffer as global variable 'empty' buffer used in sess_build_logline() inside a loop, and since it is only being read from and not modified, until recently it ended up being cached most of the time and didn't cause overhead due to systematic push on the stack. However, due recent encoding work and new added variables on the stack, we're starting to reach a stack limit and declaring 'empty' buffer within the loop seems to cause non-negligible CPU overhead. Since the variable isn't modified during log generation, let's declare 'empty' buffer as a global variable outside from sess_build_logline() to prevent pushing it on the stack for each node evaluation.
2024-04-30 10:45:34 -04:00
const struct buffer empty = { };
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-25 20:48:02 -04:00
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
MEDIUM: log-format: backend source address %Bi %Bp %Bi return the backend source IP %Bp return the backend source port Add a function pointer in logformat_type to do additional configuration during the log-format variable parsing.
2012-03-02 08:35:21 -05:00
int prepare_addrsource(struct logformat_node *node, struct proxy *curproxy);
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
/* logformat alias types (internal use) */
enum logformat_alias_type {
MEDIUM: log: carry tag context in logformat node This is a pretty simple patch despite requiring to make some visible changes in the code: When parsing a logformat string, log tags (ie: '%tag', AKA log tags) are turned into logformat nodes with their type set to the type of the corresponding logformat_tag element which was matched by name. Thus, when "compiling" a logformat tag, we only keep a reference to the tag type from the original logformat_tag. For example, for "%B" log tag, we have the following logformat_tag element: { .name = "B", .type = LOG_FMT_BYTES, .mode = PR_MODE_TCP, .lw = LW_BYTES, .config_callback = NULL } When parsing "%B" string, we search for a matching logformat tag inside logformat_tags[] array using the provided name, once we find a matching element, we craft a logformat node whose type will be LOG_FMT_BYTES, but from the node itself, we no longer have access to other informations that are set in the logformat_tag struct element. Thus from a logformat_node resulting from a log tag, with current implementation, we cannot easily get back to matching logformat_tag struct element as it would require us to scan the whole logformat_tags array at runtime using node->type to find the matching element. Let's take a simpler path and consider all tag-specific LOG_FMT_* subtypes as being part of the same logformat node type: LOG_FMT_TAG. Thanks to that, we're now able to distinguish logformat nodes made from logformat tag from other logformat nodes, and link them to their corresponding logformat_tag element from logformat_tags[] array. All it costs is a simple indirection and an extra pointer in logformat_node struct. While at it, all LOG_FMT_* types related to logformat tags were moved inside log.c as they have no use outside of it since they are simply lookup indexes for sess_build_logline() and could even be replaced by function pointers some day...
2024-02-22 14:20:41 -05:00
LOG_FMT_GLOBAL,
MINOR: log: provide log origin in logformat expressions using '%OG' '%OG' logformat alias may be used to report the log origin (when/where) that triggered log generation using sess_build_logline(). Possible values are: - "sess_error": log was generated during session error handling - "sess_killed": log was generated during session abortion (killed embryonic session) - "txn_accept": log was generated right after frontend conn was accepted - "txn_request": log was generated after client request was received - "txn_connect": log was generated after backend connection establishment - "txn_response": log was generated during server response handling - "txn_close": log was generated at the final txn step, before closing - "unspec": unknown or not specified Documentation was updated.
2024-05-06 08:13:11 -04:00
LOG_FMT_ORIGIN,
MEDIUM: log: carry tag context in logformat node This is a pretty simple patch despite requiring to make some visible changes in the code: When parsing a logformat string, log tags (ie: '%tag', AKA log tags) are turned into logformat nodes with their type set to the type of the corresponding logformat_tag element which was matched by name. Thus, when "compiling" a logformat tag, we only keep a reference to the tag type from the original logformat_tag. For example, for "%B" log tag, we have the following logformat_tag element: { .name = "B", .type = LOG_FMT_BYTES, .mode = PR_MODE_TCP, .lw = LW_BYTES, .config_callback = NULL } When parsing "%B" string, we search for a matching logformat tag inside logformat_tags[] array using the provided name, once we find a matching element, we craft a logformat node whose type will be LOG_FMT_BYTES, but from the node itself, we no longer have access to other informations that are set in the logformat_tag struct element. Thus from a logformat_node resulting from a log tag, with current implementation, we cannot easily get back to matching logformat_tag struct element as it would require us to scan the whole logformat_tags array at runtime using node->type to find the matching element. Let's take a simpler path and consider all tag-specific LOG_FMT_* subtypes as being part of the same logformat node type: LOG_FMT_TAG. Thanks to that, we're now able to distinguish logformat nodes made from logformat tag from other logformat nodes, and link them to their corresponding logformat_tag element from logformat_tags[] array. All it costs is a simple indirection and an extra pointer in logformat_node struct. While at it, all LOG_FMT_* types related to logformat tags were moved inside log.c as they have no use outside of it since they are simply lookup indexes for sess_build_logline() and could even be replaced by function pointers some day...
2024-02-22 14:20:41 -05:00
LOG_FMT_CLIENTIP,
LOG_FMT_CLIENTPORT,
LOG_FMT_BACKENDIP,
LOG_FMT_BACKENDPORT,
LOG_FMT_FRONTENDIP,
LOG_FMT_FRONTENDPORT,
LOG_FMT_SERVERPORT,
LOG_FMT_SERVERIP,
LOG_FMT_COUNTER,
LOG_FMT_LOGCNT,
LOG_FMT_PID,
LOG_FMT_DATE,
LOG_FMT_DATEGMT,
LOG_FMT_DATELOCAL,
LOG_FMT_TS,
LOG_FMT_MS,
LOG_FMT_FRONTEND,
LOG_FMT_FRONTEND_XPRT,
LOG_FMT_BACKEND,
LOG_FMT_SERVER,
LOG_FMT_BYTES,
LOG_FMT_BYTES_UP,
LOG_FMT_Ta,
LOG_FMT_Th,
LOG_FMT_Ti,
LOG_FMT_TQ,
LOG_FMT_TW,
LOG_FMT_TC,
LOG_FMT_Tr,
LOG_FMT_tr,
LOG_FMT_trg,
LOG_FMT_trl,
LOG_FMT_TR,
LOG_FMT_TD,
LOG_FMT_TT,
LOG_FMT_TU,
LOG_FMT_STATUS,
LOG_FMT_CCLIENT,
LOG_FMT_CSERVER,
LOG_FMT_TERMSTATE,
LOG_FMT_TERMSTATE_CK,
LOG_FMT_ACTCONN,
LOG_FMT_FECONN,
LOG_FMT_BECONN,
LOG_FMT_SRVCONN,
LOG_FMT_RETRIES,
LOG_FMT_SRVQUEUE,
LOG_FMT_BCKQUEUE,
LOG_FMT_HDRREQUEST,
LOG_FMT_HDRRESPONS,
LOG_FMT_HDRREQUESTLIST,
LOG_FMT_HDRRESPONSLIST,
LOG_FMT_REQ,
LOG_FMT_HTTP_METHOD,
LOG_FMT_HTTP_URI,
LOG_FMT_HTTP_PATH,
LOG_FMT_HTTP_PATH_ONLY,
LOG_FMT_HTTP_QUERY,
LOG_FMT_HTTP_VERSION,
LOG_FMT_HOSTNAME,
LOG_FMT_UNIQUEID,
LOG_FMT_SSL_CIPHER,
LOG_FMT_SSL_VERSION,
};
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
/* log_format alias names */
static const struct logformat_alias logformat_aliases[] = {
MINOR: log: log-format: usable without httplog and tcplog Options httplog and tcplog aren't mandatory anymore for the log-format. The LW_ flags are now set during the log-format string parsing.
2012-04-02 10:22:10 -04:00
{ "o", LOG_FMT_GLOBAL, PR_MODE_TCP, 0, NULL }, /* global option */
MINOR: log: provide log origin in logformat expressions using '%OG' '%OG' logformat alias may be used to report the log origin (when/where) that triggered log generation using sess_build_logline(). Possible values are: - "sess_error": log was generated during session error handling - "sess_killed": log was generated during session abortion (killed embryonic session) - "txn_accept": log was generated right after frontend conn was accepted - "txn_request": log was generated after client request was received - "txn_connect": log was generated after backend connection establishment - "txn_response": log was generated during server response handling - "txn_close": log was generated at the final txn step, before closing - "unspec": unknown or not specified Documentation was updated.
2024-05-06 08:13:11 -04:00
{ "OG", LOG_FMT_ORIGIN, PR_MODE_TCP, 0, NULL }, /* human readable log origin */
MEDIUM: log: change a few log tokens to make them easier to remember Some log tokens have evolved in a way that is not completely logical. For example, frontend tokens sometimes begin with an 'f' and sometimes with an 'F'. Same for backend and server. So let's change a few cases without disrupting compatibility with existing setups : Bi => bi Bp => bp Ci => ci Cp => cp Fi => fi Fp => fp Si => si Sp => sp cc => CC cs => CS st => ST The old ones are still supported but deprecated and will be unsupported by the 1.5 release. However, a warning message is emitted when they're encounterd and it indicates what token should be used to replace them.
2012-12-20 11:22:52 -05:00
/* please keep these lines sorted ! */
{ "B", LOG_FMT_BYTES, PR_MODE_TCP, LW_BYTES, NULL }, /* bytes from server to client */
{ "CC", LOG_FMT_CCLIENT, PR_MODE_HTTP, LW_REQHDR, NULL }, /* client cookie */
{ "CS", LOG_FMT_CSERVER, PR_MODE_HTTP, LW_RSPHDR, NULL }, /* server cookie */
{ "H", LOG_FMT_HOSTNAME, PR_MODE_TCP, LW_INIT, NULL }, /* Hostname */
MEDIUM: proxy_protocol: Support sending unique IDs using PPv2 This patch adds the `unique-id` option to `proxy-v2-options`. If this option is set a unique ID will be generated based on the `unique-id-format` while sending the proxy protocol v2 header and stored as the unique id for the first stream of the connection. This feature is meant to be used in `tcp` mode. It works on HTTP mode, but might result in inconsistent unique IDs for the first request on a keep-alive connection, because the unique ID for the first stream is generated earlier than the others. Now that we can send unique IDs in `tcp` mode the `%ID` log variable is made available in TCP mode.
2020-03-13 07:34:24 -04:00
{ "ID", LOG_FMT_UNIQUEID, PR_MODE_TCP, LW_BYTES, NULL }, /* Unique ID */
MINOR: log: allow the HTTP status code to be logged even in TCP frontends Log format is defined in the frontend, and some frontends may be chained to an HTTP backend. Sometimes it's very convenient to be able to log the HTTP status code of these HTTP backends. This status is definitely present in the internal structures, it's just that we used to limit it to be used in HTTP frontends. So let's simply relax the check to allow it to be used in TCP frontends as well.
2014-06-13 06:21:40 -04:00
{ "ST", LOG_FMT_STATUS, PR_MODE_TCP, LW_RESP, NULL }, /* status code */
MINOR: log: log-format: usable without httplog and tcplog Options httplog and tcplog aren't mandatory anymore for the log-format. The LW_ flags are now set during the log-format string parsing.
2012-04-02 10:22:10 -04:00
{ "T", LOG_FMT_DATEGMT, PR_MODE_TCP, LW_INIT, NULL }, /* date GMT */
MEDIUM: log: Decompose %Tq in %Th %Ti %TR Tq is the time between the instant the connection is accepted and a complete valid request is received. This time includes the handshake (SSL / Proxy-Protocol), the idle when the browser does preconnect and the request reception. This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR which returns respectively the handshake time, the idle time and the duration of valid request reception. It also adds %Ta which reports the request's active time, which is the total time without %Th nor %Ti. It replaces %Tt as the total time, reporting accurate measurements for HTTP persistent connections. %Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only avalaible for HTTP connections. In addition to this, we have new timestamps %tr, %trg and %trl, which log the date of start of receipt of the request, respectively in the default format, in GMT time and in local time (by analogy with %t, %T and %Tl). All of them are obviously only available for HTTP. These values are more relevant as they more accurately represent the request date without being skewed by a browser's preconnect nor a keep-alive idle time. The HTTP log format and the CLF log format have been modified to use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This way the default log formats now produce the expected output for users who don't want to manually fiddle with the log-format directive. Example with the following log-format : log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r" The request was sent by hand using "openssl s_client -connect" : Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1" => 6 ms of SSL handshake, 2375 waiting before sending the first char (in fact the time to type the first line), 261 ms before the end of the request, no time spent in queue, 1 ms spend connecting to the server, immediate response, total active time for this request = 262ms. Total time from accept to close : 2643 ms. The timing now decomposes like this : first request 2nd request |<-------------------------------->|<-------------- ... t tr t tr ... ---|----|----|----|----|----|----|----|----|-- : Th Ti TR Tw Tc Tr Td : Ti ... :<---- Tq ---->: : :<-------------- Tt -------------->: :<--------- Ta --------->:
2016-07-28 11:19:45 -04:00
{ "Ta", LOG_FMT_Ta, PR_MODE_HTTP, LW_BYTES, NULL }, /* Time active (tr to end) */
MEDIUM: log: change a few log tokens to make them easier to remember Some log tokens have evolved in a way that is not completely logical. For example, frontend tokens sometimes begin with an 'f' and sometimes with an 'F'. Same for backend and server. So let's change a few cases without disrupting compatibility with existing setups : Bi => bi Bp => bp Ci => ci Cp => cp Fi => fi Fp => fp Si => si Sp => sp cc => CC cs => CS st => ST The old ones are still supported but deprecated and will be unsupported by the 1.5 release. However, a warning message is emitted when they're encounterd and it indicates what token should be used to replace them.
2012-12-20 11:22:52 -05:00
{ "Tc", LOG_FMT_TC, PR_MODE_TCP, LW_BYTES, NULL }, /* Tc */
MEDIUM: log: Decompose %Tq in %Th %Ti %TR Tq is the time between the instant the connection is accepted and a complete valid request is received. This time includes the handshake (SSL / Proxy-Protocol), the idle when the browser does preconnect and the request reception. This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR which returns respectively the handshake time, the idle time and the duration of valid request reception. It also adds %Ta which reports the request's active time, which is the total time without %Th nor %Ti. It replaces %Tt as the total time, reporting accurate measurements for HTTP persistent connections. %Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only avalaible for HTTP connections. In addition to this, we have new timestamps %tr, %trg and %trl, which log the date of start of receipt of the request, respectively in the default format, in GMT time and in local time (by analogy with %t, %T and %Tl). All of them are obviously only available for HTTP. These values are more relevant as they more accurately represent the request date without being skewed by a browser's preconnect nor a keep-alive idle time. The HTTP log format and the CLF log format have been modified to use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This way the default log formats now produce the expected output for users who don't want to manually fiddle with the log-format directive. Example with the following log-format : log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r" The request was sent by hand using "openssl s_client -connect" : Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1" => 6 ms of SSL handshake, 2375 waiting before sending the first char (in fact the time to type the first line), 261 ms before the end of the request, no time spent in queue, 1 ms spend connecting to the server, immediate response, total active time for this request = 262ms. Total time from accept to close : 2643 ms. The timing now decomposes like this : first request 2nd request |<-------------------------------->|<-------------- ... t tr t tr ... ---|----|----|----|----|----|----|----|----|-- : Th Ti TR Tw Tc Tr Td : Ti ... :<---- Tq ---->: : :<-------------- Tt -------------->: :<--------- Ta --------->:
2016-07-28 11:19:45 -04:00
{ "Th", LOG_FMT_Th, PR_MODE_TCP, LW_BYTES, NULL }, /* Time handshake */
{ "Ti", LOG_FMT_Ti, PR_MODE_HTTP, LW_BYTES, NULL }, /* Time idle */
{ "Tl", LOG_FMT_DATELOCAL, PR_MODE_TCP, LW_INIT, NULL }, /* date local timezone */
{ "Tq", LOG_FMT_TQ, PR_MODE_HTTP, LW_BYTES, NULL }, /* Tq=Th+Ti+TR */
{ "Tr", LOG_FMT_Tr, PR_MODE_HTTP, LW_BYTES, NULL }, /* Tr */
{ "TR", LOG_FMT_TR, PR_MODE_HTTP, LW_BYTES, NULL }, /* Time to receive a valid request */
MINOR: log: add the %Td log-format specifier As suggested by Pavlos, it's too bad that we didn't have a %Td log format tag given that there are a few mentions of Td corresponding to the data transmission time already in the doc, so this is now done. Just like the other specifiers, we report -1 if the connection failed before reaching the data transmission state.
2016-05-17 11:55:27 -04:00
{ "Td", LOG_FMT_TD, PR_MODE_TCP, LW_BYTES, NULL }, /* Td = Tt - (Tq + Tw + Tc + Tr) */
MEDIUM: log: change a few log tokens to make them easier to remember Some log tokens have evolved in a way that is not completely logical. For example, frontend tokens sometimes begin with an 'f' and sometimes with an 'F'. Same for backend and server. So let's change a few cases without disrupting compatibility with existing setups : Bi => bi Bp => bp Ci => ci Cp => cp Fi => fi Fp => fp Si => si Sp => sp cc => CC cs => CS st => ST The old ones are still supported but deprecated and will be unsupported by the 1.5 release. However, a warning message is emitted when they're encounterd and it indicates what token should be used to replace them.
2012-12-20 11:22:52 -05:00
{ "Ts", LOG_FMT_TS, PR_MODE_TCP, LW_INIT, NULL }, /* timestamp GMT */
MINOR: log: log-format: usable without httplog and tcplog Options httplog and tcplog aren't mandatory anymore for the log-format. The LW_ flags are now set during the log-format string parsing.
2012-04-02 10:22:10 -04:00
{ "Tt", LOG_FMT_TT, PR_MODE_TCP, LW_BYTES, NULL }, /* Tt */
MINOR: log: Add "Tu" timer It can be sometimes useful to measure total time of a request as seen from an end user, including TCP/TLS negotiation, server response time and transfer time. "Tt" currently provides something close to that, but it also takes client idle time into account, which is problematic for keep-alive requests as idle time can be very long. "Ta" is also not sufficient as it hides TCP/TLS negotiationtime. To improve that, introduce a "Tu" timer, without idle time and everything else. It roughly estimates time spent time spent from user point of view (without DNS resolution time), assuming network latency is the same in both directions.
2020-04-28 08:09:19 -04:00
{ "Tu", LOG_FMT_TU, PR_MODE_TCP, LW_BYTES, NULL }, /* Tu = Tt -Ti */
MEDIUM: log: change a few log tokens to make them easier to remember Some log tokens have evolved in a way that is not completely logical. For example, frontend tokens sometimes begin with an 'f' and sometimes with an 'F'. Same for backend and server. So let's change a few cases without disrupting compatibility with existing setups : Bi => bi Bp => bp Ci => ci Cp => cp Fi => fi Fp => fp Si => si Sp => sp cc => CC cs => CS st => ST The old ones are still supported but deprecated and will be unsupported by the 1.5 release. However, a warning message is emitted when they're encounterd and it indicates what token should be used to replace them.
2012-12-20 11:22:52 -05:00
{ "Tw", LOG_FMT_TW, PR_MODE_TCP, LW_BYTES, NULL }, /* Tw */
{ "U", LOG_FMT_BYTES_UP, PR_MODE_TCP, LW_BYTES, NULL }, /* bytes from client to server */
MINOR: log: log-format: usable without httplog and tcplog Options httplog and tcplog aren't mandatory anymore for the log-format. The LW_ flags are now set during the log-format string parsing.
2012-04-02 10:22:10 -04:00
{ "ac", LOG_FMT_ACTCONN, PR_MODE_TCP, LW_BYTES, NULL }, /* actconn */
MEDIUM: log: change a few log tokens to make them easier to remember Some log tokens have evolved in a way that is not completely logical. For example, frontend tokens sometimes begin with an 'f' and sometimes with an 'F'. Same for backend and server. So let's change a few cases without disrupting compatibility with existing setups : Bi => bi Bp => bp Ci => ci Cp => cp Fi => fi Fp => fp Si => si Sp => sp cc => CC cs => CS st => ST The old ones are still supported but deprecated and will be unsupported by the 1.5 release. However, a warning message is emitted when they're encounterd and it indicates what token should be used to replace them.
2012-12-20 11:22:52 -05:00
{ "b", LOG_FMT_BACKEND, PR_MODE_TCP, LW_INIT, NULL }, /* backend */
MINOR: log: log-format: usable without httplog and tcplog Options httplog and tcplog aren't mandatory anymore for the log-format. The LW_ flags are now set during the log-format string parsing.
2012-04-02 10:22:10 -04:00
{ "bc", LOG_FMT_BECONN, PR_MODE_TCP, LW_BYTES, NULL }, /* beconn */
MEDIUM: log: change a few log tokens to make them easier to remember Some log tokens have evolved in a way that is not completely logical. For example, frontend tokens sometimes begin with an 'f' and sometimes with an 'F'. Same for backend and server. So let's change a few cases without disrupting compatibility with existing setups : Bi => bi Bp => bp Ci => ci Cp => cp Fi => fi Fp => fp Si => si Sp => sp cc => CC cs => CS st => ST The old ones are still supported but deprecated and will be unsupported by the 1.5 release. However, a warning message is emitted when they're encounterd and it indicates what token should be used to replace them.
2012-12-20 11:22:52 -05:00
{ "bi", LOG_FMT_BACKENDIP, PR_MODE_TCP, LW_BCKIP, prepare_addrsource }, /* backend source ip */
{ "bp", LOG_FMT_BACKENDPORT, PR_MODE_TCP, LW_BCKIP, prepare_addrsource }, /* backend source port */
MINOR: log: log-format: usable without httplog and tcplog Options httplog and tcplog aren't mandatory anymore for the log-format. The LW_ flags are now set during the log-format string parsing.
2012-04-02 10:22:10 -04:00
{ "bq", LOG_FMT_BCKQUEUE, PR_MODE_TCP, LW_BYTES, NULL }, /* backend_queue */
BUG/MINOR: log: pin the front connection when front ip/ports are logged Mathias Weiersmueller reported an interesting issue with logs which Lukas diagnosed as dating back from commit 9b061e332 (1.5-dev9). When front connection information (ip, port) are logged in TCP mode and the log is emitted at the end of the connection (eg: because %B or any log tag requiring LW_BYTES is set), the log is emitted after the connection is closed, so the address and ports cannot be retrieved anymore. It could be argued that we'd make a special case of these to immediatly retrieve the source and destination addresses from the connection, but it seems cleaner to simply pin the front connection, marking it "tracked" by adding the LW_XPRT flag to mention that we'll need some of these elements at the last moment. Only LW_FRTIP and LW_CLIP are affected. Note that after this change, LW_FRTIP could simply be removed as it's not used anywhere. Note that the problem doesn't happen when using %[src] or %[dst] since all sample expressions set LW_XPRT. This must be backported to 1.7, 1.6 and 1.5.
2017-06-23 05:23:43 -04:00
{ "ci", LOG_FMT_CLIENTIP, PR_MODE_TCP, LW_CLIP | LW_XPRT, NULL }, /* client ip */
{ "cp", LOG_FMT_CLIENTPORT, PR_MODE_TCP, LW_CLIP | LW_XPRT, NULL }, /* client port */
MEDIUM: log: change a few log tokens to make them easier to remember Some log tokens have evolved in a way that is not completely logical. For example, frontend tokens sometimes begin with an 'f' and sometimes with an 'F'. Same for backend and server. So let's change a few cases without disrupting compatibility with existing setups : Bi => bi Bp => bp Ci => ci Cp => cp Fi => fi Fp => fp Si => si Sp => sp cc => CC cs => CS st => ST The old ones are still supported but deprecated and will be unsupported by the 1.5 release. However, a warning message is emitted when they're encounterd and it indicates what token should be used to replace them.
2012-12-20 11:22:52 -05:00
{ "f", LOG_FMT_FRONTEND, PR_MODE_TCP, LW_INIT, NULL }, /* frontend */
{ "fc", LOG_FMT_FECONN, PR_MODE_TCP, LW_BYTES, NULL }, /* feconn */
BUG/MINOR: log: pin the front connection when front ip/ports are logged Mathias Weiersmueller reported an interesting issue with logs which Lukas diagnosed as dating back from commit 9b061e332 (1.5-dev9). When front connection information (ip, port) are logged in TCP mode and the log is emitted at the end of the connection (eg: because %B or any log tag requiring LW_BYTES is set), the log is emitted after the connection is closed, so the address and ports cannot be retrieved anymore. It could be argued that we'd make a special case of these to immediatly retrieve the source and destination addresses from the connection, but it seems cleaner to simply pin the front connection, marking it "tracked" by adding the LW_XPRT flag to mention that we'll need some of these elements at the last moment. Only LW_FRTIP and LW_CLIP are affected. Note that after this change, LW_FRTIP could simply be removed as it's not used anywhere. Note that the problem doesn't happen when using %[src] or %[dst] since all sample expressions set LW_XPRT. This must be backported to 1.7, 1.6 and 1.5.
2017-06-23 05:23:43 -04:00
{ "fi", LOG_FMT_FRONTENDIP, PR_MODE_TCP, LW_FRTIP | LW_XPRT, NULL }, /* frontend ip */
{ "fp", LOG_FMT_FRONTENDPORT, PR_MODE_TCP, LW_FRTIP | LW_XPRT, NULL }, /* frontend port */
MEDIUM: log: change a few log tokens to make them easier to remember Some log tokens have evolved in a way that is not completely logical. For example, frontend tokens sometimes begin with an 'f' and sometimes with an 'F'. Same for backend and server. So let's change a few cases without disrupting compatibility with existing setups : Bi => bi Bp => bp Ci => ci Cp => cp Fi => fi Fp => fp Si => si Sp => sp cc => CC cs => CS st => ST The old ones are still supported but deprecated and will be unsupported by the 1.5 release. However, a warning message is emitted when they're encounterd and it indicates what token should be used to replace them.
2012-12-20 11:22:52 -05:00
{ "ft", LOG_FMT_FRONTEND_XPRT, PR_MODE_TCP, LW_INIT, NULL }, /* frontend with transport mode */
MINOR: logs: don't limit HTTP header captures to HTTP frontends Similar to previous patches, HTTP header captures are performed when a TCP frontend switches to an HTTP backend, but are not possible to report. So let's relax the check to explicitly allow them to be present in TCP frontends.
2014-06-13 06:23:06 -04:00
{ "hr", LOG_FMT_HDRREQUEST, PR_MODE_TCP, LW_REQHDR, NULL }, /* header request */
{ "hrl", LOG_FMT_HDRREQUESTLIST, PR_MODE_TCP, LW_REQHDR, NULL }, /* header request list */
{ "hs", LOG_FMT_HDRRESPONS, PR_MODE_TCP, LW_RSPHDR, NULL }, /* header response */
{ "hsl", LOG_FMT_HDRRESPONSLIST, PR_MODE_TCP, LW_RSPHDR, NULL }, /* header response list */
MEDIUM: logs: Add HTTP request-line log format directives This commit adds 4 new log format variables that parse the HTTP Request-Line for more specific logging than "%r" provides. For example, we can parse the following HTTP Request-Line with these new variables: "GET /foo?bar=baz HTTP/1.1" - %HM: HTTP Method ("GET") - %HV: HTTP Version ("HTTP/1.1") - %HU: HTTP Request-URI ("/foo?bar=baz") - %HP: HTTP Request-URI without query string ("/foo")
2015-04-27 17:37:03 -04:00
{ "HM", LOG_FMT_HTTP_METHOD, PR_MODE_HTTP, LW_REQ, NULL }, /* HTTP method */
MINOR: log: Logging HTTP path only with %HPO This patch adds a new logging variable '%HPO' for logging HTTP path only (without query string) from relative or absolute URI. For example: log-format "hpo=%HPO hp=%HP hu=%HU hq=%HQ" GET /r/1 HTTP/1.1 => hpo=/r/1 hp=/r/1 hu=/r/1 hq= GET /r/2?q=2 HTTP/1.1 => hpo=/r/2 hp=/r/2 hu=/r/2?q=2 hq=?q=2 GET http://host/r/3 HTTP/1.1 => hpo=/r/3 hp=http://host/r/3 hu=http://host/r/3 hq= GET http://host/r/4?q=4 HTTP/1.1 => hpo=/r/4 hp=http://host/r/4 hu=http://host/r/4?q=4 hq=?q=4
2020-11-30 13:27:47 -05:00
{ "HP", LOG_FMT_HTTP_PATH, PR_MODE_HTTP, LW_REQ, NULL }, /* HTTP relative or absolute path */
{ "HPO", LOG_FMT_HTTP_PATH_ONLY, PR_MODE_HTTP, LW_REQ, NULL }, /* HTTP path only (without host nor query string) */
MINOR: log: Add log-format variable %HQ, to log HTTP query strings Since sample fetches are not always available in the response phase, this patch implements %HQ such that: GET /foo?bar=baz HTTP/1.0 ...would be logged as: ?bar=baz
2015-07-31 12:14:16 -04:00
{ "HQ", LOG_FMT_HTTP_QUERY, PR_MODE_HTTP, LW_REQ, NULL }, /* HTTP query */
MEDIUM: logs: Add HTTP request-line log format directives This commit adds 4 new log format variables that parse the HTTP Request-Line for more specific logging than "%r" provides. For example, we can parse the following HTTP Request-Line with these new variables: "GET /foo?bar=baz HTTP/1.1" - %HM: HTTP Method ("GET") - %HV: HTTP Version ("HTTP/1.1") - %HU: HTTP Request-URI ("/foo?bar=baz") - %HP: HTTP Request-URI without query string ("/foo")
2015-04-27 17:37:03 -04:00
{ "HU", LOG_FMT_HTTP_URI, PR_MODE_HTTP, LW_REQ, NULL }, /* HTTP full URI */
{ "HV", LOG_FMT_HTTP_VERSION, PR_MODE_HTTP, LW_REQ, NULL }, /* HTTP version */
MINOR: log: add a new field "%lc" to implement a per-frontend log counter Sometimes it would be convenient to have a log counter so that from a log server we know whether some logs were lost or not. The frontend's log counter serves exactly this purpose. It's incremented each time a traffic log is produced. If a log is disabled using "http-request set-log-level silent", the counter will not be incremented. However, admin logs are not accounted for. Also, if logs are filtered out before being sent to the server because of a minimum level set on the log line, the counter will be increased anyway. The counter is 32-bit, so it will wrap, but that's not an issue considering that 4 billion logs are rarely in the same file, let alone close to each other.
2014-08-28 09:03:15 -04:00
{ "lc", LOG_FMT_LOGCNT, PR_MODE_TCP, LW_INIT, NULL }, /* log counter */
MEDIUM: log: change a few log tokens to make them easier to remember Some log tokens have evolved in a way that is not completely logical. For example, frontend tokens sometimes begin with an 'f' and sometimes with an 'F'. Same for backend and server. So let's change a few cases without disrupting compatibility with existing setups : Bi => bi Bp => bp Ci => ci Cp => cp Fi => fi Fp => fp Si => si Sp => sp cc => CC cs => CS st => ST The old ones are still supported but deprecated and will be unsupported by the 1.5 release. However, a warning message is emitted when they're encounterd and it indicates what token should be used to replace them.
2012-12-20 11:22:52 -05:00
{ "ms", LOG_FMT_MS, PR_MODE_TCP, LW_INIT, NULL }, /* accept date millisecond */
MINOR: log: log-format: usable without httplog and tcplog Options httplog and tcplog aren't mandatory anymore for the log-format. The LW_ flags are now set during the log-format string parsing.
2012-04-02 10:22:10 -04:00
{ "pid", LOG_FMT_PID, PR_MODE_TCP, LW_INIT, NULL }, /* log pid */
MEDIUM: log: change a few log tokens to make them easier to remember Some log tokens have evolved in a way that is not completely logical. For example, frontend tokens sometimes begin with an 'f' and sometimes with an 'F'. Same for backend and server. So let's change a few cases without disrupting compatibility with existing setups : Bi => bi Bp => bp Ci => ci Cp => cp Fi => fi Fp => fp Si => si Sp => sp cc => CC cs => CS st => ST The old ones are still supported but deprecated and will be unsupported by the 1.5 release. However, a warning message is emitted when they're encounterd and it indicates what token should be used to replace them.
2012-12-20 11:22:52 -05:00
{ "r", LOG_FMT_REQ, PR_MODE_HTTP, LW_REQ, NULL }, /* request */
{ "rc", LOG_FMT_RETRIES, PR_MODE_TCP, LW_BYTES, NULL }, /* retries */
BUG/MEDIUM: unique_id: HTTP request counter is not stable Patrick Hemmer reported that using unique_id_format and logs did not report the same unique ID counter since commit 9f09521 ("BUG/MEDIUM: unique_id: HTTP request counter must be unique!"). This is because the increment was done while producing the log message, so it was performed twice. A better solution consists in fetching a new value once per request and saving it in the request or session context for all of this request's life. It happens that sessions already have a unique ID field which is used for debugging and reporting errors, and which differs from the one sent in logs and unique_id header. So let's change this to reuse this field to have coherent IDs everywhere. As of now, a session gets a new unique ID once it is instanciated. This means that TCP sessions will also benefit from a unique ID that can be logged. And this ID is renewed for each extra HTTP request received on an existing session. Thus, all TCP sessions and HTTP requests will have distinct IDs that will be stable along all their life, and coherent between all places where they're used (logs, unique_id header, "show sess", "show errors"). This feature is 1.5-specific, no backport to 1.4 is needed.
2014-01-25 05:01:50 -05:00
{ "rt", LOG_FMT_COUNTER, PR_MODE_TCP, LW_REQ, NULL }, /* request counter (HTTP or TCP session) */
MEDIUM: log: change a few log tokens to make them easier to remember Some log tokens have evolved in a way that is not completely logical. For example, frontend tokens sometimes begin with an 'f' and sometimes with an 'F'. Same for backend and server. So let's change a few cases without disrupting compatibility with existing setups : Bi => bi Bp => bp Ci => ci Cp => cp Fi => fi Fp => fp Si => si Sp => sp cc => CC cs => CS st => ST The old ones are still supported but deprecated and will be unsupported by the 1.5 release. However, a warning message is emitted when they're encounterd and it indicates what token should be used to replace them.
2012-12-20 11:22:52 -05:00
{ "s", LOG_FMT_SERVER, PR_MODE_TCP, LW_SVID, NULL }, /* server */
{ "sc", LOG_FMT_SRVCONN, PR_MODE_TCP, LW_BYTES, NULL }, /* srv_conn */
{ "si", LOG_FMT_SERVERIP, PR_MODE_TCP, LW_SVIP, NULL }, /* server destination ip */
{ "sp", LOG_FMT_SERVERPORT, PR_MODE_TCP, LW_SVIP, NULL }, /* server destination port */
{ "sq", LOG_FMT_SRVQUEUE, PR_MODE_TCP, LW_BYTES, NULL }, /* srv_queue */
MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv These two new log-format tags report the SSL protocol version (%sslv) and the SSL ciphers (%sslc) used for the connection with the client. For instance, to append these information just after the client's IP/port address information on an HTTP log line, use the following configuration : log-format %Ci:%Cp\ %sslv:%sslc\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %st\ %B\ %cc\ \ %cs\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %{+Q}r It will report a line such as the following one : Oct 12 20:47:30 haproxy[9643]: 127.0.0.1:43602 TLSv1:AES-SHA [12/Oct/2012:20:47:30.303] stick2~ stick2/s1 7/0/12/0/19 200 145 - - ---- 0/0/0/0/0 0/0 "GET /?t=0 HTTP/1.0"
2012-10-12 14:17:54 -04:00
{ "sslc", LOG_FMT_SSL_CIPHER, PR_MODE_TCP, LW_XPRT, NULL }, /* client-side SSL ciphers */
{ "sslv", LOG_FMT_SSL_VERSION, PR_MODE_TCP, LW_XPRT, NULL }, /* client-side SSL protocol version */
MEDIUM: log: change a few log tokens to make them easier to remember Some log tokens have evolved in a way that is not completely logical. For example, frontend tokens sometimes begin with an 'f' and sometimes with an 'F'. Same for backend and server. So let's change a few cases without disrupting compatibility with existing setups : Bi => bi Bp => bp Ci => ci Cp => cp Fi => fi Fp => fp Si => si Sp => sp cc => CC cs => CS st => ST The old ones are still supported but deprecated and will be unsupported by the 1.5 release. However, a warning message is emitted when they're encounterd and it indicates what token should be used to replace them.
2012-12-20 11:22:52 -05:00
{ "t", LOG_FMT_DATE, PR_MODE_TCP, LW_INIT, NULL }, /* date */
MEDIUM: log: Decompose %Tq in %Th %Ti %TR Tq is the time between the instant the connection is accepted and a complete valid request is received. This time includes the handshake (SSL / Proxy-Protocol), the idle when the browser does preconnect and the request reception. This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR which returns respectively the handshake time, the idle time and the duration of valid request reception. It also adds %Ta which reports the request's active time, which is the total time without %Th nor %Ti. It replaces %Tt as the total time, reporting accurate measurements for HTTP persistent connections. %Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only avalaible for HTTP connections. In addition to this, we have new timestamps %tr, %trg and %trl, which log the date of start of receipt of the request, respectively in the default format, in GMT time and in local time (by analogy with %t, %T and %Tl). All of them are obviously only available for HTTP. These values are more relevant as they more accurately represent the request date without being skewed by a browser's preconnect nor a keep-alive idle time. The HTTP log format and the CLF log format have been modified to use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This way the default log formats now produce the expected output for users who don't want to manually fiddle with the log-format directive. Example with the following log-format : log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r" The request was sent by hand using "openssl s_client -connect" : Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1" => 6 ms of SSL handshake, 2375 waiting before sending the first char (in fact the time to type the first line), 261 ms before the end of the request, no time spent in queue, 1 ms spend connecting to the server, immediate response, total active time for this request = 262ms. Total time from accept to close : 2643 ms. The timing now decomposes like this : first request 2nd request |<-------------------------------->|<-------------- ... t tr t tr ... ---|----|----|----|----|----|----|----|----|-- : Th Ti TR Tw Tc Tr Td : Ti ... :<---- Tq ---->: : :<-------------- Tt -------------->: :<--------- Ta --------->:
2016-07-28 11:19:45 -04:00
{ "tr", LOG_FMT_tr, PR_MODE_HTTP, LW_INIT, NULL }, /* date of start of request */
{ "trg",LOG_FMT_trg, PR_MODE_HTTP, LW_INIT, NULL }, /* date of start of request, GMT */
{ "trl",LOG_FMT_trl, PR_MODE_HTTP, LW_INIT, NULL }, /* date of start of request, local */
MEDIUM: log: change a few log tokens to make them easier to remember Some log tokens have evolved in a way that is not completely logical. For example, frontend tokens sometimes begin with an 'f' and sometimes with an 'F'. Same for backend and server. So let's change a few cases without disrupting compatibility with existing setups : Bi => bi Bp => bp Ci => ci Cp => cp Fi => fi Fp => fp Si => si Sp => sp cc => CC cs => CS st => ST The old ones are still supported but deprecated and will be unsupported by the 1.5 release. However, a warning message is emitted when they're encounterd and it indicates what token should be used to replace them.
2012-12-20 11:22:52 -05:00
{ "ts", LOG_FMT_TERMSTATE, PR_MODE_TCP, LW_BYTES, NULL },/* termination state */
{ "tsc", LOG_FMT_TERMSTATE_CK, PR_MODE_TCP, LW_INIT, NULL },/* termination state */
MINOR: log: log-format: usable without httplog and tcplog Options httplog and tcplog aren't mandatory anymore for the log-format. The LW_ flags are now set during the log-format string parsing.
2012-04-02 10:22:10 -04:00
{ 0, 0, 0, 0, NULL }
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
};
BUG/MINOR: httpclient/log: free of invalid ptr with httpclient_log_format free_proxy() must check if the ptr is not httpclient_log_format before trying to free p->conf.logformat_string. No backport needed.
2022-12-22 09:37:01 -05:00
char httpclient_log_format[] = "%ci:%cp [%tr] %ft -/- %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r";
MEDIUM: log: Decompose %Tq in %Th %Ti %TR Tq is the time between the instant the connection is accepted and a complete valid request is received. This time includes the handshake (SSL / Proxy-Protocol), the idle when the browser does preconnect and the request reception. This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR which returns respectively the handshake time, the idle time and the duration of valid request reception. It also adds %Ta which reports the request's active time, which is the total time without %Th nor %Ti. It replaces %Tt as the total time, reporting accurate measurements for HTTP persistent connections. %Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only avalaible for HTTP connections. In addition to this, we have new timestamps %tr, %trg and %trl, which log the date of start of receipt of the request, respectively in the default format, in GMT time and in local time (by analogy with %t, %T and %Tl). All of them are obviously only available for HTTP. These values are more relevant as they more accurately represent the request date without being skewed by a browser's preconnect nor a keep-alive idle time. The HTTP log format and the CLF log format have been modified to use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This way the default log formats now produce the expected output for users who don't want to manually fiddle with the log-format directive. Example with the following log-format : log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r" The request was sent by hand using "openssl s_client -connect" : Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1" => 6 ms of SSL handshake, 2375 waiting before sending the first char (in fact the time to type the first line), 261 ms before the end of the request, no time spent in queue, 1 ms spend connecting to the server, immediate response, total active time for this request = 262ms. Total time from accept to close : 2643 ms. The timing now decomposes like this : first request 2nd request |<-------------------------------->|<-------------- ... t tr t tr ... ---|----|----|----|----|----|----|----|----|-- : Th Ti TR Tw Tc Tr Td : Ti ... :<---- Tq ---->: : :<-------------- Tt -------------->: :<--------- Ta --------->:
2016-07-28 11:19:45 -04:00
char default_http_log_format[] = "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"; // default format
MEDIUM: log: add the client's SNI to the default HTTPS log format During a troublehooting it came obvious that the SNI always ought to be logged on httpslog, as it explains errors caused by selection of the default certificate (or failure to do so in case of strict-sni). This expectation was also confirmed on the mailing list. Since the field may be empty it appeared important not to leave an empty string in the current format, so it was decided to place the field before a '/' preceding the SSL version and ciphers, so that in the worst case a missing field leads to a field looking like "/TLSv1.2/AES...", though usually a missing element still results in a "-" in logs. This will change the log format for users who already deployed the 2.5-dev versions (hence the medium level) but no released version was using this format yet so there's no harm for stable deployments. The reg-test was updated to check for "-" there since we don't send SNI in reg-tests. Link: https://www.mail-archive.com/haproxy@formilux.org/msg41410.html Cc: William Lallemand <wlallemand@haproxy.org>
2021-11-05 14:14:55 -04:00
char default_https_log_format[] = "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %[fc_err]/%[ssl_fc_err,hex]/%[ssl_c_err]/%[ssl_c_ca_err]/%[ssl_fc_is_resumed] %[ssl_fc_sni]/%sslv/%sslc";
MEDIUM: log: Decompose %Tq in %Th %Ti %TR Tq is the time between the instant the connection is accepted and a complete valid request is received. This time includes the handshake (SSL / Proxy-Protocol), the idle when the browser does preconnect and the request reception. This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR which returns respectively the handshake time, the idle time and the duration of valid request reception. It also adds %Ta which reports the request's active time, which is the total time without %Th nor %Ti. It replaces %Tt as the total time, reporting accurate measurements for HTTP persistent connections. %Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only avalaible for HTTP connections. In addition to this, we have new timestamps %tr, %trg and %trl, which log the date of start of receipt of the request, respectively in the default format, in GMT time and in local time (by analogy with %t, %T and %Tl). All of them are obviously only available for HTTP. These values are more relevant as they more accurately represent the request date without being skewed by a browser's preconnect nor a keep-alive idle time. The HTTP log format and the CLF log format have been modified to use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This way the default log formats now produce the expected output for users who don't want to manually fiddle with the log-format directive. Example with the following log-format : log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r" The request was sent by hand using "openssl s_client -connect" : Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1" => 6 ms of SSL handshake, 2375 waiting before sending the first char (in fact the time to type the first line), 261 ms before the end of the request, no time spent in queue, 1 ms spend connecting to the server, immediate response, total active time for this request = 262ms. Total time from accept to close : 2643 ms. The timing now decomposes like this : first request 2nd request |<-------------------------------->|<-------------- ... t tr t tr ... ---|----|----|----|----|----|----|----|----|-- : Th Ti TR Tw Tc Tr Td : Ti ... :<---- Tq ---->: : :<-------------- Tt -------------->: :<--------- Ta --------->:
2016-07-28 11:19:45 -04:00
char clf_http_log_format[] = "%{+Q}o %{-Q}ci - - [%trg] %r %ST %B \"\" \"\" %cp %ms %ft %b %s %TR %Tw %Tc %Tr %Ta %tsc %ac %fc %bc %sc %rc %sq %bq %CC %CS %hrl %hsl";
MEDIUM: log: change a few log tokens to make them easier to remember Some log tokens have evolved in a way that is not completely logical. For example, frontend tokens sometimes begin with an 'f' and sometimes with an 'F'. Same for backend and server. So let's change a few cases without disrupting compatibility with existing setups : Bi => bi Bp => bp Ci => ci Cp => cp Fi => fi Fp => fp Si => si Sp => sp cc => CC cs => CS st => ST The old ones are still supported but deprecated and will be unsupported by the 1.5 release. However, a warning message is emitted when they're encounterd and it indicates what token should be used to replace them.
2012-12-20 11:22:52 -05:00
char default_tcp_log_format[] = "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq";
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
char *log_format = NULL;
MEDIUM: logs: add a new RFC5424 log-format for the structured-data This patch adds a new RFC5424-specific log-format for the structured-data that is automatically send by __send_log() when the sender is in RFC5424 mode. A new statement "log-format-sd" should be used in order to set log-format for the structured-data part in RFC5424 formatted syslog messages. Example: log-format-sd [exampleSDID@1234\ bytes=\"%B\"\ status=\"%ST\"]
2015-09-25 13:17:44 -04:00
/* Default string used for structured-data part in RFC5424 formatted
* syslog messages.
*/
char default_rfc5424_sd_log_format[] = "- ";
MEDIUM: logs: add support for RFC5424 header format per logger The function __send_log() iterates over senders and passes the header as the first vector to sendmsg(), thus it can send a logger-specific header in each message. A new logger arguments "format rfc5424" should be used in order to enable RFC5424 header format. For example: log 10.2.3.4:1234 len 2048 format rfc5424 local2 info
2015-09-22 10:05:32 -04:00
MEDIUM: proxy/log: leverage lf_expr API for logformat preparsing Currently, the way proxy-oriented logformat directives are handled is way too complicated. Indeed, "log-format", "log-format-error", "log-format-sd" and "unique-id-format" all rely on preparsing hints stored inside proxy->conf member struct. Those preparsing hints include the original string that should be compiled once the proxy parameters are known plus the config file and line number where the string was found to generate precise error messages in case of failure during the compiling process that happens within check_config_validity(). Now that lf_expr API permits to compile a lf_expr struct that was previously prepared (with original string and config hints), let's leverage lf_expr_compile() from check_config_validity() and instead of relying on individual proxy->conf hints for each logformat expression, store string and config hints in the lf_expr struct directly and use lf_expr helpers funcs to handle them when relevant (ie: original logformat string freeing is now done at a central place inside lf_expr_deinit(), which allows for some simplifications) Doing so allows us to greatly simplify the preparsing logic for those 4 proxy directives, and to finally save some space in the proxy struct. Also, since httpclient proxy has its "logformat" automatically compiled in check_config_validity(), we now use the file hint from the logformat expression struct to set an explicit name that will be reported in case of error ("parsing [httpclient:0] : ...") and remove the extraneous check in httpclient_precheck() (logformat was parsed twice previously..)
2024-03-05 09:44:43 -05:00
/* returns true if the input logformat string is one of the default ones declared
* above
*/
static inline int logformat_str_isdefault(const char *str)
{
return str == httpclient_log_format ||
str == default_http_log_format ||
str == default_https_log_format ||
str == clf_http_log_format ||
str == default_tcp_log_format ||
str == default_rfc5424_sd_log_format;
}
/* free logformat str if it is not a default (static) one */
static inline void logformat_str_free(char **str)
{
if (!logformat_str_isdefault(*str))
ha_free(str);
}
/* duplicate and return logformat str if it is not a default (static)
* one, else return the original one
*/
static inline char *logformat_str_dup(char *str)
{
if (logformat_str_isdefault(str))
return str;
return strdup(str);
}
MINOR: log: report the number of dropped logs in the stats It's easy to detect when logs on some paths are lost as sendmsg() will return EAGAIN. This is particularly true when sending to /dev/log, which often doesn't support a big logging capacity. Let's keep track of these and report the total number of dropped messages in "show info".
2018-11-12 01:25:28 -05:00
/* total number of dropped logs */
unsigned int dropped_logs = 0;
MEDIUM: log: use a separate buffer for the header and for the message Make sendmsg() use two vectors, one for the message header that is updated by update_log_hdr() and one for the message buffer.
2015-09-19 16:09:02 -04:00
/* This is a global syslog message buffer, common to all outgoing
* messages. It contains only the data part.
BUG/CRITICAL: log: fix risk of crash in development snapshot Commit a1cc38 introduced a regression which was easy to trigger till ad4cd58 (snapshots 20120222 to 20120311 included). The bug was still present after that but harder to trigger. The bug is caused by the use of two distinct log buffers due to intermediary changes. The issue happens when an HTTP request is logged just after a TCP request during the same second and the HTTP request is too large for the buffer. In this case, it happens that the HTTP request is logged into the TCP buffer instead and that length controls can't detect anything. Starting with bddd4f, the issue is still possible when logging too large an HTTP request just after a send_log() call (typically a server status change). We owe a big thanks to Sander Klein for testing several snapshots and more specifically for taking significant risks in production by letting the buggy version crash several times in order to provide an exploitable core ! The bug could not have been found without this precious help. Thank you Sander ! This fix does not need to be backported, it did not affect any released version.
2012-03-19 11:51:53 -04:00
*/
MEDIUM: threads/logs: Make logs thread-safe log buffers and static variables used in log functions are now thread-local. So there is no need to lock anything to log messages. Moreover, per-thread init/deinit functions are now used to initialize these buffers.
2017-06-02 10:20:16 -04:00
THREAD_LOCAL char *logline = NULL;
BUG/CRITICAL: log: fix risk of crash in development snapshot Commit a1cc38 introduced a regression which was easy to trigger till ad4cd58 (snapshots 20120222 to 20120311 included). The bug was still present after that but harder to trigger. The bug is caused by the use of two distinct log buffers due to intermediary changes. The issue happens when an HTTP request is logged just after a TCP request during the same second and the HTTP request is too large for the buffer. In this case, it happens that the HTTP request is logged into the TCP buffer instead and that length controls can't detect anything. Starting with bddd4f, the issue is still possible when logging too large an HTTP request just after a send_log() call (typically a server status change). We owe a big thanks to Sander Klein for testing several snapshots and more specifically for taking significant risks in production by letting the buggy version crash several times in order to provide an exploitable core ! The bug could not have been found without this precious help. Thank you Sander ! This fix does not need to be backported, it did not affect any released version.
2012-03-19 11:51:53 -04:00
MEDIUM: logs: add a new RFC5424 log-format for the structured-data This patch adds a new RFC5424-specific log-format for the structured-data that is automatically send by __send_log() when the sender is in RFC5424 mode. A new statement "log-format-sd" should be used in order to set log-format for the structured-data part in RFC5424 formatted syslog messages. Example: log-format-sd [exampleSDID@1234\ bytes=\"%B\"\ status=\"%ST\"]
2015-09-25 13:17:44 -04:00
/* A global syslog message buffer, common to all RFC5424 syslog messages.
* Currently, it is used for generating the structured-data part.
*/
MEDIUM: threads/logs: Make logs thread-safe log buffers and static variables used in log functions are now thread-local. So there is no need to lock anything to log messages. Moreover, per-thread init/deinit functions are now used to initialize these buffers.
2017-06-02 10:20:16 -04:00
THREAD_LOCAL char *logline_rfc5424 = NULL;
MEDIUM: logs: add a new RFC5424 log-format for the structured-data This patch adds a new RFC5424-specific log-format for the structured-data that is automatically send by __send_log() when the sender is in RFC5424 mode. A new statement "log-format-sd" should be used in order to set log-format for the structured-data part in RFC5424 formatted syslog messages. Example: log-format-sd [exampleSDID@1234\ bytes=\"%B\"\ status=\"%ST\"]
2015-09-25 13:17:44 -04:00
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
struct logformat_node_args {
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
char *name;
int mask;
};
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
struct logformat_node_args node_args_list[] = {
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
// global
{ "M", LOG_OPT_MANDATORY },
{ "Q", LOG_OPT_QUOTE },
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
{ "X", LOG_OPT_HEXA },
MEDIUM: log: add a new log format flag "E" The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It mostly makes sense to use it in the RFC5424 structured-data log formats. Example: log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 07:23:03 -05:00
{ "E", LOG_OPT_ESC },
MINOR: log: add +bin logformat node option Support '+bin' option argument on logformat nodes to try to preserve binary output type with binary sample expressions. For this, we rely on the log/sink API which is capable of conveying binary data since all related functions don't search for a terminating NULL byte in provided log payload as they take a string pointer and a string length as argument. Example: log-format "%{+bin}o %[bin(00AABB)]" Will produce: 00aabb (output was piped to `hexdump -ve '1/1 "%.2x"'` to dump raw bytes as HEX characters) This should be used carefully, because many syslog endpoints don't expect binary data (especially NULL bytes). This is mainly intended for use with set-var-fmt actions or with ring/udp log endpoints that know how to deal with such binary payloads. Also, this option is only supported globally (for use with '%o'), it will not have any effect when set on an individual node. (it makes no sense to have binary data in the middle of log payload that was started without binary data option)
2024-04-25 10:29:01 -04:00
{ "bin", LOG_OPT_BIN },
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
{ "json", LOG_OPT_ENCODE_JSON },
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
{ "cbor", LOG_OPT_ENCODE_CBOR },
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
{ 0, 0 }
};
MEDIUM: log-format: backend source address %Bi %Bp %Bi return the backend source IP %Bp return the backend source port Add a function pointer in logformat_type to do additional configuration during the log-format variable parsing.
2012-03-02 08:35:21 -05:00
/*
* callback used to configure addr source retrieval
*/
int prepare_addrsource(struct logformat_node *node, struct proxy *curproxy)
{
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
if ((curproxy->flags & PR_FL_CHECKED))
return 0;
MEDIUM: log-format: backend source address %Bi %Bp %Bi return the backend source IP %Bp return the backend source port Add a function pointer in logformat_type to do additional configuration during the log-format variable parsing.
2012-03-02 08:35:21 -05:00
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
curproxy->options2 |= PR_O2_SRC_ADDR;
return 1;
MEDIUM: log-format: backend source address %Bi %Bp %Bi return the backend source IP %Bp return the backend source port Add a function pointer in logformat_type to do additional configuration during the log-format variable parsing.
2012-03-02 08:35:21 -05:00
}
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
/*
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
* Parse args in a logformat_node. Returns 0 in error
CLEANUP: log-format: fix return code of function parse_logformat_var_args() This patch replace the successful return code from 0 to 1. The error code is replaced from 1 to 0. The return code of this function is actually unused, so this patch cannot modify the behaviour.
2016-11-22 17:13:04 -05:00
* case, otherwise, it returns 1.
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
*/
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
int parse_logformat_node_args(char *args, struct logformat_node *node, char **err)
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
{
int i = 0;
int end = 0;
int flags = 0; // 1 = + 2 = -
char *sp = NULL; // start pointer
MEDIUM: log-format: Use standard HAProxy log system to report errors The function log format emit its own error message using Alert(). This patch replaces this behavior and uses the standard HAProxy error system (with memprintf). The benefits are: - cleaning the log system - the logformat can ignore the caller (actually the caller must set a flag designing the caller function). - Make the usage of the logformat function easy for future components.
2016-11-22 18:41:28 -05:00
if (args == NULL) {
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
memprintf(err, "internal error: parse_logformat_node_args() expects non null 'args'");
CLEANUP: log-format: fix return code of function parse_logformat_var_args() This patch replace the successful return code from 0 to 1. The error code is replaced from 1 to 0. The return code of this function is actually unused, so this patch cannot modify the behaviour.
2016-11-22 17:13:04 -05:00
return 0;
MEDIUM: log-format: Use standard HAProxy log system to report errors The function log format emit its own error message using Alert(). This patch replaces this behavior and uses the standard HAProxy error system (with memprintf). The benefits are: - cleaning the log system - the logformat can ignore the caller (actually the caller must set a flag designing the caller function). - Make the usage of the logformat function easy for future components.
2016-11-22 18:41:28 -05:00
}
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
while (1) {
if (*args == '\0')
end = 1;
if (*args == '+') {
// add flag
sp = args + 1;
flags = 1;
}
if (*args == '-') {
// delete flag
sp = args + 1;
flags = 2;
}
if (*args == '\0' || *args == ',') {
*args = '\0';
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
for (i = 0; sp && node_args_list[i].name; i++) {
if (strcmp(sp, node_args_list[i].name) == 0) {
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
if (flags == 1) {
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
/* Ensure we don't mix encoding types, existing
* encoding type prevails over new ones
*/
if (node->options & LOG_OPT_ENCODE)
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
node->options |= (node_args_list[i].mask & ~LOG_OPT_ENCODE);
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
else
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
node->options |= node_args_list[i].mask;
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
break;
} else if (flags == 2) {
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
node->options &= ~node_args_list[i].mask;
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
break;
}
}
}
sp = NULL;
if (end)
break;
}
BUG/MEDIUM: log: fix possible segfault during config parsing When log format arguments are specified within braces with no '+' nor '-' prefix, the NULL string is compared with known keywords causing a crash. This only happens during parsing so it does not affect runtime processing.
2012-12-20 12:19:26 -05:00
args++;
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
}
CLEANUP: log-format: fix return code of function parse_logformat_var_args() This patch replace the successful return code from 0 to 1. The error code is replaced from 1 to 0. The return code of this function is actually unused, so this patch cannot modify the behaviour.
2016-11-22 17:13:04 -05:00
return 1;
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
}
/*
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
* Parse an alias '%aliasname' or '%{args}aliasname' in log-format. The caller
MEDIUM: log-format: make the format parser more robust and more extensible The log-format parser reached a limit making it hard to add new features. It also suffers from a weak handling of certain incorrect corner cases, for example "%{foo}" is emitted as a litteral while syntactically it's an argument to no variable. Also the argument parser had to redo some of the job with some cases causing minor memory leaks (eg: ignored args). This work aims at improving the situation so that slightly better reporting is possible and that it becomes possible to extend the log format. The code has a few more states but looks significantly simpler. The parser is now capable of reporting ignored arguments and truncated lines.
2012-12-20 15:23:42 -05:00
* must pass the args part in the <arg> pointer with its length in <arg_len>,
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
* and aliasname with its length in <alias> and <alias_len> respectively. <arg>
* is ignored when arg_len is 0. Neither <alias> nor <alias_len> may be null.
CLEANUP: log-format: fix return code of the function parse_logformat_var() This patch replaces the successful return code from 0 to 1. The error code is replaced from -1 to 0. The return code of this function is actually unused, so this patch cannot modify the behaviour.
2016-11-22 16:06:04 -05:00
* Returns false in error case and err is filled, otherwise returns true.
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
*/
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
static int parse_logformat_alias(char *arg, int arg_len, char *name, int name_len, int typecast,
char *alias, int alias_len, struct lf_expr *lf_expr,
int *defoptions, char **err)
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
{
MEDIUM: log-format: make the format parser more robust and more extensible The log-format parser reached a limit making it hard to add new features. It also suffers from a weak handling of certain incorrect corner cases, for example "%{foo}" is emitted as a litteral while syntactically it's an argument to no variable. Also the argument parser had to redo some of the job with some cases causing minor memory leaks (eg: ignored args). This work aims at improving the situation so that slightly better reporting is possible and that it becomes possible to extend the log format. The code has a few more states but looks significantly simpler. The parser is now capable of reporting ignored arguments and truncated lines.
2012-12-20 15:23:42 -05:00
int j;
MINOR: log: store lf_expr nodes inside substruct Add another struct level inside lf_expr struct to allow new information to be stored alongside lf_expr nodes.
2024-03-25 06:29:58 -04:00
struct list *list_format= &lf_expr->nodes.list;
BUG/MINOR: log: properly free memory on logformat parse error and deinit() This patch may be backported to all supported versions.
2019-04-29 18:40:02 -04:00
struct logformat_node *node = NULL;
MEDIUM: log-format: make the format parser more robust and more extensible The log-format parser reached a limit making it hard to add new features. It also suffers from a weak handling of certain incorrect corner cases, for example "%{foo}" is emitted as a litteral while syntactically it's an argument to no variable. Also the argument parser had to redo some of the job with some cases causing minor memory leaks (eg: ignored args). This work aims at improving the situation so that slightly better reporting is possible and that it becomes possible to extend the log format. The code has a few more states but looks significantly simpler. The parser is now capable of reporting ignored arguments and truncated lines.
2012-12-20 15:23:42 -05:00
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
for (j = 0; logformat_aliases[j].name; j++) { // search a log type
if (strlen(logformat_aliases[j].name) == alias_len &&
strncmp(alias, logformat_aliases[j].name, alias_len) == 0) {
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
node = calloc(1, sizeof(*node));
if (!node) {
memprintf(err, "out of memory error");
goto error_free;
}
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
node->type = LOG_FMT_ALIAS;
node->alias = &logformat_aliases[j];
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
node->typecast = typecast;
MINOR: log: skip custom logformat_node name if empty Reminder: Since 3.0-dev4, we can optionally give a name to logformat nodes: log-format "%(custom_name1)B %(custom_name2)[str(value)]" But we may also optionally set the expected node type by appending ':type' after the name, type being either sint,str or bool, like this: log-format "%(string_as_int:sint)[str(14)]" However, it is currently not possible to provide a type without providing a name that is a least 1 char long. But it could be useful to provide a type without setting a name, like this, for typecasting purposes only: log-format "%(:sint)[bool(true)]" Thus in order to allow this usage, don't set node->name if node name is not at least 1 character long. By doing so, node->name will remain NULL and will not be considered, but the typecast setting will.
2024-04-25 03:50:14 -04:00
if (name && name_len)
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
node->name = my_strndup(name, name_len);
BUG/MINOR: log: fix global lf_expr node options behavior In 507223d5 ("MINOR: log: global lf_expr node options"), a mistake was made because it was assumed that only the last occurence of %o (LOG_FMT_GLOBAL) should be kept as global node options. However, although not documented, it is possible to have multiple %o within a single logformat expression to change the global settings on the fly. For instance, consider this example: log-format "%{+X}o test1=%ms %{-X}o test2=%ms %{+X}o test3=%ms" Prior to 3f2e8d0ed ("MEDIUM: log: lf_* build helpers now take a ctx argument"), this would output something like this: test1=18B test2=395 test3=18B This is because global options is properly updated as the lf_expr string is parsed. But now due to 507223d5 and 3f2e8d0ed, only the last %o occurence is considered. With the above example, this gives: test1=18B test2=18B test3=18B To restore historical behavior, let's partially revert 507223d5: to compute global node options, we now start with all options enabled and then for each configurable node in lf_expr_postcheck(), we keep options common to the current node and previous nodes using AND masking, this way we really end up with options common to all nodes. No backport needed.
2024-04-29 05:55:27 -04:00
node->options = *defoptions;
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
if (arg_len) {
node->arg = my_strndup(arg, arg_len);
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
if (!parse_logformat_node_args(node->arg, node, err))
BUG/MINOR: log: properly free memory on logformat parse error and deinit() This patch may be backported to all supported versions.
2019-04-29 18:40:02 -04:00
goto error_free;
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
}
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
if (node->alias->type == LOG_FMT_GLOBAL) {
BUG/MINOR: log: fix global lf_expr node options behavior In 507223d5 ("MINOR: log: global lf_expr node options"), a mistake was made because it was assumed that only the last occurence of %o (LOG_FMT_GLOBAL) should be kept as global node options. However, although not documented, it is possible to have multiple %o within a single logformat expression to change the global settings on the fly. For instance, consider this example: log-format "%{+X}o test1=%ms %{-X}o test2=%ms %{+X}o test3=%ms" Prior to 3f2e8d0ed ("MEDIUM: log: lf_* build helpers now take a ctx argument"), this would output something like this: test1=18B test2=395 test3=18B This is because global options is properly updated as the lf_expr string is parsed. But now due to 507223d5 and 3f2e8d0ed, only the last %o occurence is considered. With the above example, this gives: test1=18B test2=18B test3=18B To restore historical behavior, let's partially revert 507223d5: to compute global node options, we now start with all options enabled and then for each configurable node in lf_expr_postcheck(), we keep options common to the current node and previous nodes using AND masking, this way we really end up with options common to all nodes. No backport needed.
2024-04-29 05:55:27 -04:00
*defoptions = node->options;
BUG/MINOR: log: fix global lf_expr node options behavior (2nd try) In 98b44e8 ("BUG/MINOR: log: fix global lf_expr node options behavior"), I properly restored global node options behavior for when encoding is not used, however the fix is not optimal when encoding is involved: Indeed, encoding logic in sess_build_logline() relies on global node options to know if encoding must be handled expression-wide or individually. However, because of the above fix, if an expression is made of 1 or multiple nodes that all set an encoding option manually (without '%o'), we consider that the option was set globally, but that's probably not what the user intended. Instead we should only evaluate global options from '%o', so that it remains possible to skip global encoding when needed. No backport needed.
2024-04-29 09:58:36 -04:00
if (lf_expr->nodes.options == LOG_OPT_NONE)
lf_expr->nodes.options = node->options;
else {
/* global options were previously set and were
* overwritten for nodes that appear after the
* current one.
*
* However, for lf_expr->nodes.options we must
* keep a track of options common to ALL nodes,
* thus we take previous global options into
* account to compute the new logformat
* expression wide (global) node options.
*/
lf_expr->nodes.options &= node->options;
}
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
free_logformat_node(node);
MEDIUM: log-format: make the format parser more robust and more extensible The log-format parser reached a limit making it hard to add new features. It also suffers from a weak handling of certain incorrect corner cases, for example "%{foo}" is emitted as a litteral while syntactically it's an argument to no variable. Also the argument parser had to redo some of the job with some cases causing minor memory leaks (eg: ignored args). This work aims at improving the situation so that slightly better reporting is possible and that it becomes possible to extend the log format. The code has a few more states but looks significantly simpler. The parser is now capable of reporting ignored arguments and truncated lines.
2012-12-20 15:23:42 -05:00
} else {
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
LIST_APPEND(list_format, &node->list);
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
}
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
return 1;
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
}
}
MEDIUM: log-format: make the format parser more robust and more extensible The log-format parser reached a limit making it hard to add new features. It also suffers from a weak handling of certain incorrect corner cases, for example "%{foo}" is emitted as a litteral while syntactically it's an argument to no variable. Also the argument parser had to redo some of the job with some cases causing minor memory leaks (eg: ignored args). This work aims at improving the situation so that slightly better reporting is possible and that it becomes possible to extend the log format. The code has a few more states but looks significantly simpler. The parser is now capable of reporting ignored arguments and truncated lines.
2012-12-20 15:23:42 -05:00
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
j = alias[alias_len];
alias[alias_len] = 0;
memprintf(err, "no such format alias '%s'. If you wanted to emit the '%%' character verbatim, you need to use '%%%%'", alias);
alias[alias_len] = j;
BUG/MINOR: log: properly free memory on logformat parse error and deinit() This patch may be backported to all supported versions.
2019-04-29 18:40:02 -04:00
error_free:
BUG/MINOR: log: fix potential lf->name memory leak Recent commit 2ed6068 ("MINOR: log: custom name for logformat node") introduced a potential memory leak because when custom name is provided, lf->name value is allocated using strdup(), thus is expected to be freed alongside the node when the node is released. However lf->name was only freed in some common places within log.c cleanups and helpers func, but in reality there are still cases where lf nodes are manually freed without making use of freeing helpers. So this is what this patch does, it makes sure all lf freeing places now leverage the free_logformat_node() helper function that takes care of freeing all known allocated elements within the node, including custom name. This commit depends on: - "MINOR: log: add free_logformat_node() helper function" No backport needed unless 2ed6068 gets backported.
2024-02-22 09:14:21 -05:00
free_logformat_node(node);
CLEANUP: log-format: fix return code of the function parse_logformat_var() This patch replaces the successful return code from 0 to 1. The error code is replaced from -1 to 0. The return code of this function is actually unused, so this patch cannot modify the behaviour.
2016-11-22 16:06:04 -05:00
return 0;
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
}
/*
* push to the logformat linked list
*
* start: start pointer
* end: end text pointer
* type: string type
MEDIUM: tree-wide: add logformat expressions wrapper log format expressions are broadly used within the code: once they are parsed from input string, they are converted to a linked list of logformat nodes. We're starting to face some limitations because we're simply storing the converted expression as a generic logformat_node list. The first issue we're facing is that storing logformat expressions that way doesn't allow us to add metadata alongside the list, which is part of the prerequites for implementing log-profiles. Another issue with storing logformat expressions as generic lists of logformat_node elements is that it's starting to become really hard to tell when we rely on logformat expressions or not in the code given that there isn't always a comment near the list declaration or manipulation to indicate that it's relying on logformat expressions under the hood, so this adds some complexity for code maintenance. This patch looks quite impressive due to changes in a lot of header and source files (since logformat expressions are broadly used), but it does a simple thing: it defines the lf_expr structure which itself holds a generic list of logformat nodes, and then declares some helpers to manipulate lf_expr elements and fixes the code so that we now exclusively manipulate logformat_node lists as lf_expr elements outside of log.c. For now, lf_expr struct only contains the list of logformat nodes (no additional metadata), but now that we have dedicated type and helpers, doing so in the future won't be problematic at all and won't require extensive code changes.
2024-02-23 09:57:21 -05:00
* lf_expr: destination logformat expr (list of fmt nodes)
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
*
* LOG_TEXT: copy chars from start to end excluding end.
*
*/
MEDIUM: tree-wide: add logformat expressions wrapper log format expressions are broadly used within the code: once they are parsed from input string, they are converted to a linked list of logformat nodes. We're starting to face some limitations because we're simply storing the converted expression as a generic logformat_node list. The first issue we're facing is that storing logformat expressions that way doesn't allow us to add metadata alongside the list, which is part of the prerequites for implementing log-profiles. Another issue with storing logformat expressions as generic lists of logformat_node elements is that it's starting to become really hard to tell when we rely on logformat expressions or not in the code given that there isn't always a comment near the list declaration or manipulation to indicate that it's relying on logformat expressions under the hood, so this adds some complexity for code maintenance. This patch looks quite impressive due to changes in a lot of header and source files (since logformat expressions are broadly used), but it does a simple thing: it defines the lf_expr structure which itself holds a generic list of logformat nodes, and then declares some helpers to manipulate lf_expr elements and fixes the code so that we now exclusively manipulate logformat_node lists as lf_expr elements outside of log.c. For now, lf_expr struct only contains the list of logformat nodes (no additional metadata), but now that we have dedicated type and helpers, doing so in the future won't be problematic at all and won't require extensive code changes.
2024-02-23 09:57:21 -05:00
int add_to_logformat_list(char *start, char *end, int type, struct lf_expr *lf_expr, char **err)
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
{
MINOR: log: store lf_expr nodes inside substruct Add another struct level inside lf_expr struct to allow new information to be stored alongside lf_expr nodes.
2024-03-25 06:29:58 -04:00
struct list *list_format = &lf_expr->nodes.list;
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
char *str;
BUG/MINOR: log: add_to_logformat_list() used the wrong constants The <type> argument was checked against LOG_FMT_* but it was passed as LF_* which are two independant enums. It happens that the 3 first entries in these enums do match, but this broke some experimental changes which required another state, so let's fix this now.
2012-12-20 15:59:12 -05:00
if (type == LF_TEXT) { /* type text */
CLEANUP: uniformize last argument of malloc/calloc Instead of repeating the type of the LHS argument (sizeof(struct ...)) in calls to malloc/calloc, we directly use the pointer name (sizeof(*...)). The following Coccinelle patch was used: @@ type T; T *x; @@ x = malloc( - sizeof(T) + sizeof(*x) ) @@ type T; T *x; @@ x = calloc(1, - sizeof(T) + sizeof(*x) ) When the LHS is not just a variable name, no change is made. Moreover, the following patch was used to ensure that "1" is consistently used as a first argument of calloc, not the last one: @@ @@ calloc( + 1, ... - ,1 )
2016-04-03 07:48:43 -04:00
struct logformat_node *node = calloc(1, sizeof(*node));
BUG/MINOR: log-format: uncatched memory allocation functions Some return code of memory allocation functions are not tested. This patch fix theses checks.
2016-11-22 17:24:10 -05:00
if (!node) {
MEDIUM: log-format: Use standard HAProxy log system to report errors The function log format emit its own error message using Alert(). This patch replaces this behavior and uses the standard HAProxy error system (with memprintf). The benefits are: - cleaning the log system - the logformat can ignore the caller (actually the caller must set a flag designing the caller function). - Make the usage of the logformat function easy for future components.
2016-11-22 18:41:28 -05:00
memprintf(err, "out of memory error");
BUG/MINOR: log-format: uncatched memory allocation functions Some return code of memory allocation functions are not tested. This patch fix theses checks.
2016-11-22 17:24:10 -05:00
return 0;
}
CLEANUP: uniformize last argument of malloc/calloc Instead of repeating the type of the LHS argument (sizeof(struct ...)) in calls to malloc/calloc, we directly use the pointer name (sizeof(*...)). The following Coccinelle patch was used: @@ type T; T *x; @@ x = malloc( - sizeof(T) + sizeof(*x) ) @@ type T; T *x; @@ x = calloc(1, - sizeof(T) + sizeof(*x) ) When the LHS is not just a variable name, no change is made. Moreover, the following patch was used to ensure that "1" is consistently used as a first argument of calloc, not the last one: @@ @@ calloc( + 1, ... - ,1 )
2016-04-03 07:48:43 -04:00
str = calloc(1, end - start + 1);
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
strncpy(str, start, end - start);
str[end - start] = '\0';
node->arg = str;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
node->type = LOG_FMT_TEXT; // type string
CLEANUP: lists/tree-wide: rename some list operations to avoid some confusion The current "ADD" vs "ADDQ" is confusing because when thinking in terms of appending at the end of a list, "ADD" naturally comes to mind, but here it does the opposite, it inserts. Several times already it's been incorrectly used where ADDQ was expected, the latest of which was a fortunate accident explained in 6fa922562 ("CLEANUP: stream: explain why we queue the stream at the head of the server list"). Let's use more explicit (but slightly longer) names now: LIST_ADD -> LIST_INSERT LIST_ADDQ -> LIST_APPEND LIST_ADDED -> LIST_INLIST LIST_DEL -> LIST_DELETE The same is true for MT_LISTs, including their "TRY" variant. LIST_DEL_INIT keeps its short name to encourage to use it instead of the lazier LIST_DELETE which is often less safe. The change is large (~674 non-comment entries) but is mechanical enough to remain safe. No permutation was performed, so any out-of-tree code can easily map older names to new ones. The list doc was updated.
2021-04-21 01:32:39 -04:00
LIST_APPEND(list_format, &node->list);
BUG/MINOR: log: add_to_logformat_list() used the wrong constants The <type> argument was checked against LOG_FMT_* but it was passed as LF_* which are two independant enums. It happens that the 3 first entries in these enums do match, but this broke some experimental changes which required another state, so let's fix this now.
2012-12-20 15:59:12 -05:00
} else if (type == LF_SEPARATOR) {
CLEANUP: uniformize last argument of malloc/calloc Instead of repeating the type of the LHS argument (sizeof(struct ...)) in calls to malloc/calloc, we directly use the pointer name (sizeof(*...)). The following Coccinelle patch was used: @@ type T; T *x; @@ x = malloc( - sizeof(T) + sizeof(*x) ) @@ type T; T *x; @@ x = calloc(1, - sizeof(T) + sizeof(*x) ) When the LHS is not just a variable name, no change is made. Moreover, the following patch was used to ensure that "1" is consistently used as a first argument of calloc, not the last one: @@ @@ calloc( + 1, ... - ,1 )
2016-04-03 07:48:43 -04:00
struct logformat_node *node = calloc(1, sizeof(*node));
BUG/MINOR: log-format: uncatched memory allocation functions Some return code of memory allocation functions are not tested. This patch fix theses checks.
2016-11-22 17:24:10 -05:00
if (!node) {
MEDIUM: log-format: Use standard HAProxy log system to report errors The function log format emit its own error message using Alert(). This patch replaces this behavior and uses the standard HAProxy error system (with memprintf). The benefits are: - cleaning the log system - the logformat can ignore the caller (actually the caller must set a flag designing the caller function). - Make the usage of the logformat function easy for future components.
2016-11-22 18:41:28 -05:00
memprintf(err, "out of memory error");
BUG/MINOR: log-format: uncatched memory allocation functions Some return code of memory allocation functions are not tested. This patch fix theses checks.
2016-11-22 17:24:10 -05:00
return 0;
}
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
node->type = LOG_FMT_SEPARATOR;
CLEANUP: lists/tree-wide: rename some list operations to avoid some confusion The current "ADD" vs "ADDQ" is confusing because when thinking in terms of appending at the end of a list, "ADD" naturally comes to mind, but here it does the opposite, it inserts. Several times already it's been incorrectly used where ADDQ was expected, the latest of which was a fortunate accident explained in 6fa922562 ("CLEANUP: stream: explain why we queue the stream at the head of the server list"). Let's use more explicit (but slightly longer) names now: LIST_ADD -> LIST_INSERT LIST_ADDQ -> LIST_APPEND LIST_ADDED -> LIST_INLIST LIST_DEL -> LIST_DELETE The same is true for MT_LISTs, including their "TRY" variant. LIST_DEL_INIT keeps its short name to encourage to use it instead of the lazier LIST_DELETE which is often less safe. The change is large (~674 non-comment entries) but is mechanical enough to remain safe. No permutation was performed, so any out-of-tree code can easily map older names to new ones. The list doc was updated.
2021-04-21 01:32:39 -04:00
LIST_APPEND(list_format, &node->list);
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
}
MEDIUM: log-format: strict parsing and enable fail Until now, the function parse_logformat_string() never fails. It send warnings when it parses bad format, and returns expression in best effort. This patch replaces warnings by alert and returns a fail code. Maybe the warning mode is designed for a compatibility with old configuration versions. If it is the case, now this compatibility is broken. [wt: no, the reason is that an alert must cause a startup failure, but this will be OK with next patch]
2016-11-22 17:11:21 -05:00
return 1;
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
}
MEDIUM: log: add the ability to include samples in logs Using %[expression] it becomes possible to make the log engine fetch some samples from the request or the response and provide them in the logs. Note that this feature is still limited, it does not yet allow to apply converters, to limit the output length, nor to specify the direction which should be fetched when a fetch function works in both directions. However it's quite convenient to log SSL information or to include some information that are used in stick tables. It is worth noting that this has been done in the generic log format handler, which means that the same information may be used to build the unique-id header and to pass the information to a backend server.
2012-12-20 18:09:23 -05:00
/*
MEDIUM: tree-wide: add logformat expressions wrapper log format expressions are broadly used within the code: once they are parsed from input string, they are converted to a linked list of logformat nodes. We're starting to face some limitations because we're simply storing the converted expression as a generic logformat_node list. The first issue we're facing is that storing logformat expressions that way doesn't allow us to add metadata alongside the list, which is part of the prerequites for implementing log-profiles. Another issue with storing logformat expressions as generic lists of logformat_node elements is that it's starting to become really hard to tell when we rely on logformat expressions or not in the code given that there isn't always a comment near the list declaration or manipulation to indicate that it's relying on logformat expressions under the hood, so this adds some complexity for code maintenance. This patch looks quite impressive due to changes in a lot of header and source files (since logformat expressions are broadly used), but it does a simple thing: it defines the lf_expr structure which itself holds a generic list of logformat nodes, and then declares some helpers to manipulate lf_expr elements and fixes the code so that we now exclusively manipulate logformat_node lists as lf_expr elements outside of log.c. For now, lf_expr struct only contains the list of logformat nodes (no additional metadata), but now that we have dedicated type and helpers, doing so in the future won't be problematic at all and won't require extensive code changes.
2024-02-23 09:57:21 -05:00
* Parse the sample fetch expression <text> and add a node to <lf_expr> upon
CLEANUP: log: fix obsolete comment for add_sample_to_logformat_list() Since 833cc794 ("MEDIUM: sample: handle comma-delimited converter list") logformat expressions now support having a comma-delimited converter list right after the fetch. Let's remove a leftover comment from the initial implementation that says otherwise.
2024-02-22 10:13:13 -05:00
* success. The curpx->conf.args.ctx must be set by the caller. If an end pointer
MEDIUM: log-format: make the LF parser aware of sample expressions' end For a very long time it used to be impossible to pass a closing square bracket as a valid character in argument to a sample fetch function or to a converter because the LF parser used to stop on the first such character found and to pass what was between the first '[' and the first ']' to sample_parse_expr(). This patch addresses this by passing the whole string to sample_parse_expr() which is the only one authoritative to indicate the first character that does not belong to the expression. The LF parser then verifies it matches a ']' or fails. As a result it is finally possible to write rules such as the following, which is totally valid an unambigous : http-request redirect location %[url,regsub([.:/?-],!,g)] |-----| | | arg1 | `---> arg3 `-----> arg2 |-----------------| converter |---------------------| sample expression |------------------------| log-format tag
2020-02-14 11:33:06 -05:00
* is passed in <endptr>, it will be updated with the pointer to the first character
* not part of the sample expression.
MEDIUM: log-format: strict parsing and enable fail Until now, the function parse_logformat_string() never fails. It send warnings when it parses bad format, and returns expression in best effort. This patch replaces warnings by alert and returns a fail code. Maybe the warning mode is designed for a compatibility with old configuration versions. If it is the case, now this compatibility is broken. [wt: no, the reason is that an alert must cause a startup failure, but this will be OK with next patch]
2016-11-22 17:11:21 -05:00
*
* In error case, the function returns 0, otherwise it returns 1.
MEDIUM: log: add the ability to include samples in logs Using %[expression] it becomes possible to make the log engine fetch some samples from the request or the response and provide them in the logs. Note that this feature is still limited, it does not yet allow to apply converters, to limit the output length, nor to specify the direction which should be fetched when a fetch function works in both directions. However it's quite convenient to log SSL information or to include some information that are used in stick tables. It is worth noting that this has been done in the generic log format handler, which means that the same information may be used to build the unique-id header and to pass the information to a backend server.
2012-12-20 18:09:23 -05:00
*/
MEDIUM: log: add compiling logic to logformat expressions split parse_logformat_string() into two functions: parse_logformat_string() sticks to the same behavior, but now becomes an helper for lf_expr_compile() which uses explicit arguments so that it becomes possible to use lf_expr_compile() without a proxy, but also compile an expression which was previously prepared for compiling (set string and config hints within the logformat expression to avoid manually storing string and config context if the compiling step happens later). lf_expr_dup() may be used to duplicate an expression before it is compiled, lf_expr_xfer() now makes sure that the input logformat is already compiled. This is some prerequisite works for log-profiles implementation, no functional change should be expected.
2024-02-29 08:54:43 -05:00
static int add_sample_to_logformat_list(char *text, char *name, int name_len, int typecast,
char *arg, int arg_len, struct lf_expr *lf_expr,
BUG/MINOR: log: fix global lf_expr node options behavior In 507223d5 ("MINOR: log: global lf_expr node options"), a mistake was made because it was assumed that only the last occurence of %o (LOG_FMT_GLOBAL) should be kept as global node options. However, although not documented, it is possible to have multiple %o within a single logformat expression to change the global settings on the fly. For instance, consider this example: log-format "%{+X}o test1=%ms %{-X}o test2=%ms %{+X}o test3=%ms" Prior to 3f2e8d0ed ("MEDIUM: log: lf_* build helpers now take a ctx argument"), this would output something like this: test1=18B test2=395 test3=18B This is because global options is properly updated as the lf_expr string is parsed. But now due to 507223d5 and 3f2e8d0ed, only the last %o occurence is considered. With the above example, this gives: test1=18B test2=18B test3=18B To restore historical behavior, let's partially revert 507223d5: to compute global node options, we now start with all options enabled and then for each configurable node in lf_expr_postcheck(), we keep options common to the current node and previous nodes using AND masking, this way we really end up with options common to all nodes. No backport needed.
2024-04-29 05:55:27 -04:00
struct arg_list *al, int options, int cap, char **err, char **endptr)
MEDIUM: log: add the ability to include samples in logs Using %[expression] it becomes possible to make the log engine fetch some samples from the request or the response and provide them in the logs. Note that this feature is still limited, it does not yet allow to apply converters, to limit the output length, nor to specify the direction which should be fetched when a fetch function works in both directions. However it's quite convenient to log SSL information or to include some information that are used in stick tables. It is worth noting that this has been done in the generic log format handler, which means that the same information may be used to build the unique-id header and to pass the information to a backend server.
2012-12-20 18:09:23 -05:00
{
char *cmd[2];
MINOR: log: store lf_expr nodes inside substruct Add another struct level inside lf_expr struct to allow new information to be stored alongside lf_expr nodes.
2024-03-25 06:29:58 -04:00
struct list *list_format = &lf_expr->nodes.list;
BUG/MINOR: log: properly free memory on logformat parse error and deinit() This patch may be backported to all supported versions.
2019-04-29 18:40:02 -04:00
struct sample_expr *expr = NULL;
struct logformat_node *node = NULL;
MEDIUM: log: add the ability to include samples in logs Using %[expression] it becomes possible to make the log engine fetch some samples from the request or the response and provide them in the logs. Note that this feature is still limited, it does not yet allow to apply converters, to limit the output length, nor to specify the direction which should be fetched when a fetch function works in both directions. However it's quite convenient to log SSL information or to include some information that are used in stick tables. It is worth noting that this has been done in the generic log format handler, which means that the same information may be used to build the unique-id header and to pass the information to a backend server.
2012-12-20 18:09:23 -05:00
int cmd_arg;
cmd[0] = text;
cmd[1] = "";
cmd_arg = 0;
MEDIUM: log: add compiling logic to logformat expressions split parse_logformat_string() into two functions: parse_logformat_string() sticks to the same behavior, but now becomes an helper for lf_expr_compile() which uses explicit arguments so that it becomes possible to use lf_expr_compile() without a proxy, but also compile an expression which was previously prepared for compiling (set string and config hints within the logformat expression to avoid manually storing string and config context if the compiling step happens later). lf_expr_dup() may be used to duplicate an expression before it is compiled, lf_expr_xfer() now makes sure that the input logformat is already compiled. This is some prerequisite works for log-profiles implementation, no functional change should be expected.
2024-02-29 08:54:43 -05:00
expr = sample_parse_expr(cmd, &cmd_arg, lf_expr->conf.file, lf_expr->conf.line, err,
al, endptr);
MEDIUM: log: add the ability to include samples in logs Using %[expression] it becomes possible to make the log engine fetch some samples from the request or the response and provide them in the logs. Note that this feature is still limited, it does not yet allow to apply converters, to limit the output length, nor to specify the direction which should be fetched when a fetch function works in both directions. However it's quite convenient to log SSL information or to include some information that are used in stick tables. It is worth noting that this has been done in the generic log format handler, which means that the same information may be used to build the unique-id header and to pass the information to a backend server.
2012-12-20 18:09:23 -05:00
if (!expr) {
MEDIUM: log-format: Use standard HAProxy log system to report errors The function log format emit its own error message using Alert(). This patch replaces this behavior and uses the standard HAProxy error system (with memprintf). The benefits are: - cleaning the log system - the logformat can ignore the caller (actually the caller must set a flag designing the caller function). - Make the usage of the logformat function easy for future components.
2016-11-22 18:41:28 -05:00
memprintf(err, "failed to parse sample expression <%s> : %s", text, *err);
BUG/MINOR: log: properly free memory on logformat parse error and deinit() This patch may be backported to all supported versions.
2019-04-29 18:40:02 -04:00
goto error_free;
MEDIUM: log: add the ability to include samples in logs Using %[expression] it becomes possible to make the log engine fetch some samples from the request or the response and provide them in the logs. Note that this feature is still limited, it does not yet allow to apply converters, to limit the output length, nor to specify the direction which should be fetched when a fetch function works in both directions. However it's quite convenient to log SSL information or to include some information that are used in stick tables. It is worth noting that this has been done in the generic log format handler, which means that the same information may be used to build the unique-id header and to pass the information to a backend server.
2012-12-20 18:09:23 -05:00
}
CLEANUP: uniformize last argument of malloc/calloc Instead of repeating the type of the LHS argument (sizeof(struct ...)) in calls to malloc/calloc, we directly use the pointer name (sizeof(*...)). The following Coccinelle patch was used: @@ type T; T *x; @@ x = malloc( - sizeof(T) + sizeof(*x) ) @@ type T; T *x; @@ x = calloc(1, - sizeof(T) + sizeof(*x) ) When the LHS is not just a variable name, no change is made. Moreover, the following patch was used to ensure that "1" is consistently used as a first argument of calloc, not the last one: @@ @@ calloc( + 1, ... - ,1 )
2016-04-03 07:48:43 -04:00
node = calloc(1, sizeof(*node));
BUG/MINOR: log-format: uncatched memory allocation functions Some return code of memory allocation functions are not tested. This patch fix theses checks.
2016-11-22 17:24:10 -05:00
if (!node) {
BUG/MINOR: log: fix leak in add_sample_to_logformat_list() error path If add_sample_to_logformat_list() fails to allocate new logformat_node, then we directly jump to error_free label to cleanup the node using free_logformat_node() before returning an error. However if the node failed to allocate, then the sample expression that was allocated just before (not yet assigned) isn't released (free_logformat_node() is a no-op when NULL is provided). Thus if expr wasn't assigned to the node during early failure, then it must be manually released. This bug was introduced by 2462e5bcc ("BUG/MINOR: log: fix potential lf->name memory leak") which wasn't marked for backports. It only affects 3.0.
2024-05-13 10:24:10 -04:00
release_sample_expr(expr);
MEDIUM: log-format: Use standard HAProxy log system to report errors The function log format emit its own error message using Alert(). This patch replaces this behavior and uses the standard HAProxy error system (with memprintf). The benefits are: - cleaning the log system - the logformat can ignore the caller (actually the caller must set a flag designing the caller function). - Make the usage of the logformat function easy for future components.
2016-11-22 18:41:28 -05:00
memprintf(err, "out of memory error");
BUG/MINOR: log: properly free memory on logformat parse error and deinit() This patch may be backported to all supported versions.
2019-04-29 18:40:02 -04:00
goto error_free;
BUG/MINOR: log-format: uncatched memory allocation functions Some return code of memory allocation functions are not tested. This patch fix theses checks.
2016-11-22 17:24:10 -05:00
}
MINOR: log: skip custom logformat_node name if empty Reminder: Since 3.0-dev4, we can optionally give a name to logformat nodes: log-format "%(custom_name1)B %(custom_name2)[str(value)]" But we may also optionally set the expected node type by appending ':type' after the name, type being either sint,str or bool, like this: log-format "%(string_as_int:sint)[str(14)]" However, it is currently not possible to provide a type without providing a name that is a least 1 char long. But it could be useful to provide a type without setting a name, like this, for typecasting purposes only: log-format "%(:sint)[bool(true)]" Thus in order to allow this usage, don't set node->name if node name is not at least 1 character long. By doing so, node->name will remain NULL and will not be considered, but the typecast setting will.
2024-04-25 03:50:14 -04:00
if (name && name_len)
MINOR: log: custom name for logformat node Add the ability to specify custom name (will be used for representation in verbose output types such as json) to logformat nodes. For now, a custom name should be composed by characters [a-zA-Z0-9-_]*
2024-01-08 12:38:29 -05:00
node->name = my_strndup(name, name_len);
MEDIUM: log: add the ability to include samples in logs Using %[expression] it becomes possible to make the log engine fetch some samples from the request or the response and provide them in the logs. Note that this feature is still limited, it does not yet allow to apply converters, to limit the output length, nor to specify the direction which should be fetched when a fetch function works in both directions. However it's quite convenient to log SSL information or to include some information that are used in stick tables. It is worth noting that this has been done in the generic log format handler, which means that the same information may be used to build the unique-id header and to pass the information to a backend server.
2012-12-20 18:09:23 -05:00
node->type = LOG_FMT_EXPR;
MINOR: log: explicit typecasting for logformat nodes Add the ability to manually specify desired output type after a custom field name for logformat nodes. Forcing the type can be useful to ensure value is stored with the proper type representation. (i.e.: forcing numerical to string to work around the limited resolution of JS number types) By default, type is set to SMP_T_SAME, which means the original type will be preserved. Currently supported types are: bool, str, sint
2024-02-20 04:29:49 -05:00
node->typecast = typecast;
MEDIUM: log: add the ability to include samples in logs Using %[expression] it becomes possible to make the log engine fetch some samples from the request or the response and provide them in the logs. Note that this feature is still limited, it does not yet allow to apply converters, to limit the output length, nor to specify the direction which should be fetched when a fetch function works in both directions. However it's quite convenient to log SSL information or to include some information that are used in stick tables. It is worth noting that this has been done in the generic log format handler, which means that the same information may be used to build the unique-id header and to pass the information to a backend server.
2012-12-20 18:09:23 -05:00
node->expr = expr;
BUG/MINOR: log: fix global lf_expr node options behavior In 507223d5 ("MINOR: log: global lf_expr node options"), a mistake was made because it was assumed that only the last occurence of %o (LOG_FMT_GLOBAL) should be kept as global node options. However, although not documented, it is possible to have multiple %o within a single logformat expression to change the global settings on the fly. For instance, consider this example: log-format "%{+X}o test1=%ms %{-X}o test2=%ms %{+X}o test3=%ms" Prior to 3f2e8d0ed ("MEDIUM: log: lf_* build helpers now take a ctx argument"), this would output something like this: test1=18B test2=395 test3=18B This is because global options is properly updated as the lf_expr string is parsed. But now due to 507223d5 and 3f2e8d0ed, only the last %o occurence is considered. With the above example, this gives: test1=18B test2=18B test3=18B To restore historical behavior, let's partially revert 507223d5: to compute global node options, we now start with all options enabled and then for each configurable node in lf_expr_postcheck(), we keep options common to the current node and previous nodes using AND masking, this way we really end up with options common to all nodes. No backport needed.
2024-04-29 05:55:27 -04:00
node->options = options;
MEDIUM: log: add the ability to include samples in logs Using %[expression] it becomes possible to make the log engine fetch some samples from the request or the response and provide them in the logs. Note that this feature is still limited, it does not yet allow to apply converters, to limit the output length, nor to specify the direction which should be fetched when a fetch function works in both directions. However it's quite convenient to log SSL information or to include some information that are used in stick tables. It is worth noting that this has been done in the generic log format handler, which means that the same information may be used to build the unique-id header and to pass the information to a backend server.
2012-12-20 18:09:23 -05:00
if (arg_len) {
node->arg = my_strndup(arg, arg_len);
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
if (!parse_logformat_node_args(node->arg, node, err))
BUG/MINOR: log: properly free memory on logformat parse error and deinit() This patch may be backported to all supported versions.
2019-04-29 18:40:02 -04:00
goto error_free;
MEDIUM: log: add the ability to include samples in logs Using %[expression] it becomes possible to make the log engine fetch some samples from the request or the response and provide them in the logs. Note that this feature is still limited, it does not yet allow to apply converters, to limit the output length, nor to specify the direction which should be fetched when a fetch function works in both directions. However it's quite convenient to log SSL information or to include some information that are used in stick tables. It is worth noting that this has been done in the generic log format handler, which means that the same information may be used to build the unique-id header and to pass the information to a backend server.
2012-12-20 18:09:23 -05:00
}
MINOR: log: indicate it when some unreliable sample fetches are logged If a log-format involves some sample fetches that may not be present at the logging instant, we can now report a warning. Note that this is done both for log-format and for add-header and carefully respects the original fetch keyword's capabilities.
2013-01-07 19:10:24 -05:00
if (expr->fetch->val & cap & SMP_VAL_REQUEST)
MEDIUM: log: add the ability to include samples in logs Using %[expression] it becomes possible to make the log engine fetch some samples from the request or the response and provide them in the logs. Note that this feature is still limited, it does not yet allow to apply converters, to limit the output length, nor to specify the direction which should be fetched when a fetch function works in both directions. However it's quite convenient to log SSL information or to include some information that are used in stick tables. It is worth noting that this has been done in the generic log format handler, which means that the same information may be used to build the unique-id header and to pass the information to a backend server.
2012-12-20 18:09:23 -05:00
node->options |= LOG_OPT_REQ_CAP; /* fetch method is request-compatible */
MINOR: log: indicate it when some unreliable sample fetches are logged If a log-format involves some sample fetches that may not be present at the logging instant, we can now report a warning. Note that this is done both for log-format and for add-header and carefully respects the original fetch keyword's capabilities.
2013-01-07 19:10:24 -05:00
if (expr->fetch->val & cap & SMP_VAL_RESPONSE)
MEDIUM: log: add the ability to include samples in logs Using %[expression] it becomes possible to make the log engine fetch some samples from the request or the response and provide them in the logs. Note that this feature is still limited, it does not yet allow to apply converters, to limit the output length, nor to specify the direction which should be fetched when a fetch function works in both directions. However it's quite convenient to log SSL information or to include some information that are used in stick tables. It is worth noting that this has been done in the generic log format handler, which means that the same information may be used to build the unique-id header and to pass the information to a backend server.
2012-12-20 18:09:23 -05:00
node->options |= LOG_OPT_RES_CAP; /* fetch method is response-compatible */
MEDIUM: log-format: strict parsing and enable fail Until now, the function parse_logformat_string() never fails. It send warnings when it parses bad format, and returns expression in best effort. This patch replaces warnings by alert and returns a fail code. Maybe the warning mode is designed for a compatibility with old configuration versions. If it is the case, now this compatibility is broken. [wt: no, the reason is that an alert must cause a startup failure, but this will be OK with next patch]
2016-11-22 17:11:21 -05:00
if (!(expr->fetch->val & cap)) {
MEDIUM: log-format: Use standard HAProxy log system to report errors The function log format emit its own error message using Alert(). This patch replaces this behavior and uses the standard HAProxy error system (with memprintf). The benefits are: - cleaning the log system - the logformat can ignore the caller (actually the caller must set a flag designing the caller function). - Make the usage of the logformat function easy for future components.
2016-11-22 18:41:28 -05:00
memprintf(err, "sample fetch <%s> may not be reliably used here because it needs '%s' which is not available here",
text, sample_src_names(expr->fetch->use));
BUG/MINOR: log: properly free memory on logformat parse error and deinit() This patch may be backported to all supported versions.
2019-04-29 18:40:02 -04:00
goto error_free;
MEDIUM: log-format: strict parsing and enable fail Until now, the function parse_logformat_string() never fails. It send warnings when it parses bad format, and returns expression in best effort. This patch replaces warnings by alert and returns a fail code. Maybe the warning mode is designed for a compatibility with old configuration versions. If it is the case, now this compatibility is broken. [wt: no, the reason is that an alert must cause a startup failure, but this will be OK with next patch]
2016-11-22 17:11:21 -05:00
}
MINOR: log: indicate it when some unreliable sample fetches are logged If a log-format involves some sample fetches that may not be present at the logging instant, we can now report a warning. Note that this is done both for log-format and for add-header and carefully respects the original fetch keyword's capabilities.
2013-01-07 19:10:24 -05:00
BUG/MINOR: log: fix global lf_expr node options behavior In 507223d5 ("MINOR: log: global lf_expr node options"), a mistake was made because it was assumed that only the last occurence of %o (LOG_FMT_GLOBAL) should be kept as global node options. However, although not documented, it is possible to have multiple %o within a single logformat expression to change the global settings on the fly. For instance, consider this example: log-format "%{+X}o test1=%ms %{-X}o test2=%ms %{+X}o test3=%ms" Prior to 3f2e8d0ed ("MEDIUM: log: lf_* build helpers now take a ctx argument"), this would output something like this: test1=18B test2=395 test3=18B This is because global options is properly updated as the lf_expr string is parsed. But now due to 507223d5 and 3f2e8d0ed, only the last %o occurence is considered. With the above example, this gives: test1=18B test2=18B test3=18B To restore historical behavior, let's partially revert 507223d5: to compute global node options, we now start with all options enabled and then for each configurable node in lf_expr_postcheck(), we keep options common to the current node and previous nodes using AND masking, this way we really end up with options common to all nodes. No backport needed.
2024-04-29 05:55:27 -04:00
if ((options & LOG_OPT_HTTP) && (expr->fetch->use & (SMP_USE_L6REQ|SMP_USE_L6RES))) {
MINOR: payload/config: Warn if a L6 sample fetch is used from an HTTP proxy L6 sample fetches are now ignored when called from an HTTP proxy. Thus, a warning is emitted during the startup if such usage is detected. It is true for most ACLs and for log-format strings. Unfortunately, it is a bit painful to do so for sample expressions. This patch relies on the commit "MINOR: action: Use a generic function to check validity of an action rule list".
2021-03-26 05:02:46 -04:00
ha_warning("parsing [%s:%d] : L6 sample fetch <%s> ignored in HTTP log-format string.\n",
MEDIUM: log: add compiling logic to logformat expressions split parse_logformat_string() into two functions: parse_logformat_string() sticks to the same behavior, but now becomes an helper for lf_expr_compile() which uses explicit arguments so that it becomes possible to use lf_expr_compile() without a proxy, but also compile an expression which was previously prepared for compiling (set string and config hints within the logformat expression to avoid manually storing string and config context if the compiling step happens later). lf_expr_dup() may be used to duplicate an expression before it is compiled, lf_expr_xfer() now makes sure that the input logformat is already compiled. This is some prerequisite works for log-profiles implementation, no functional change should be expected.
2024-02-29 08:54:43 -05:00
lf_expr->conf.file, lf_expr->conf.line, text);
MINOR: payload/config: Warn if a L6 sample fetch is used from an HTTP proxy L6 sample fetches are now ignored when called from an HTTP proxy. Thus, a warning is emitted during the startup if such usage is detected. It is true for most ACLs and for log-format strings. Unfortunately, it is a bit painful to do so for sample expressions. This patch relies on the commit "MINOR: action: Use a generic function to check validity of an action rule list".
2021-03-26 05:02:46 -04:00
}
CLEANUP: lists/tree-wide: rename some list operations to avoid some confusion The current "ADD" vs "ADDQ" is confusing because when thinking in terms of appending at the end of a list, "ADD" naturally comes to mind, but here it does the opposite, it inserts. Several times already it's been incorrectly used where ADDQ was expected, the latest of which was a fortunate accident explained in 6fa922562 ("CLEANUP: stream: explain why we queue the stream at the head of the server list"). Let's use more explicit (but slightly longer) names now: LIST_ADD -> LIST_INSERT LIST_ADDQ -> LIST_APPEND LIST_ADDED -> LIST_INLIST LIST_DEL -> LIST_DELETE The same is true for MT_LISTs, including their "TRY" variant. LIST_DEL_INIT keeps its short name to encourage to use it instead of the lazier LIST_DELETE which is often less safe. The change is large (~674 non-comment entries) but is mechanical enough to remain safe. No permutation was performed, so any out-of-tree code can easily map older names to new ones. The list doc was updated.
2021-04-21 01:32:39 -04:00
LIST_APPEND(list_format, &node->list);
MEDIUM: log-format: strict parsing and enable fail Until now, the function parse_logformat_string() never fails. It send warnings when it parses bad format, and returns expression in best effort. This patch replaces warnings by alert and returns a fail code. Maybe the warning mode is designed for a compatibility with old configuration versions. If it is the case, now this compatibility is broken. [wt: no, the reason is that an alert must cause a startup failure, but this will be OK with next patch]
2016-11-22 17:11:21 -05:00
return 1;
BUG/MINOR: log: properly free memory on logformat parse error and deinit() This patch may be backported to all supported versions.
2019-04-29 18:40:02 -04:00
error_free:
BUG/MINOR: log: fix potential lf->name memory leak Recent commit 2ed6068 ("MINOR: log: custom name for logformat node") introduced a potential memory leak because when custom name is provided, lf->name value is allocated using strdup(), thus is expected to be freed alongside the node when the node is released. However lf->name was only freed in some common places within log.c cleanups and helpers func, but in reality there are still cases where lf nodes are manually freed without making use of freeing helpers. So this is what this patch does, it makes sure all lf freeing places now leverage the free_logformat_node() helper function that takes care of freeing all known allocated elements within the node, including custom name. This commit depends on: - "MINOR: log: add free_logformat_node() helper function" No backport needed unless 2ed6068 gets backported.
2024-02-22 09:14:21 -05:00
free_logformat_node(node);
BUG/MINOR: log: properly free memory on logformat parse error and deinit() This patch may be backported to all supported versions.
2019-04-29 18:40:02 -04:00
return 0;
MEDIUM: log: add the ability to include samples in logs Using %[expression] it becomes possible to make the log engine fetch some samples from the request or the response and provide them in the logs. Note that this feature is still limited, it does not yet allow to apply converters, to limit the output length, nor to specify the direction which should be fetched when a fetch function works in both directions. However it's quite convenient to log SSL information or to include some information that are used in stick tables. It is worth noting that this has been done in the generic log format handler, which means that the same information may be used to build the unique-id header and to pass the information to a backend server.
2012-12-20 18:09:23 -05:00
}
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
/*
MEDIUM: log: add compiling logic to logformat expressions split parse_logformat_string() into two functions: parse_logformat_string() sticks to the same behavior, but now becomes an helper for lf_expr_compile() which uses explicit arguments so that it becomes possible to use lf_expr_compile() without a proxy, but also compile an expression which was previously prepared for compiling (set string and config hints within the logformat expression to avoid manually storing string and config context if the compiling step happens later). lf_expr_dup() may be used to duplicate an expression before it is compiled, lf_expr_xfer() now makes sure that the input logformat is already compiled. This is some prerequisite works for log-profiles implementation, no functional change should be expected.
2024-02-29 08:54:43 -05:00
* Compile logformat expression (from string to list of logformat nodes)
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
*
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
* Aliases are preceded by % and composed by characters [a-zA-Z0-9]* : %aliasname
* Expressions are preceded by % and enclosed in square brackets: %[expr]
* You can set arguments using { } : %{many arguments}aliasname
* %{many arguments}[expr]
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
*
MEDIUM: tree-wide: add logformat expressions wrapper log format expressions are broadly used within the code: once they are parsed from input string, they are converted to a linked list of logformat nodes. We're starting to face some limitations because we're simply storing the converted expression as a generic logformat_node list. The first issue we're facing is that storing logformat expressions that way doesn't allow us to add metadata alongside the list, which is part of the prerequites for implementing log-profiles. Another issue with storing logformat expressions as generic lists of logformat_node elements is that it's starting to become really hard to tell when we rely on logformat expressions or not in the code given that there isn't always a comment near the list declaration or manipulation to indicate that it's relying on logformat expressions under the hood, so this adds some complexity for code maintenance. This patch looks quite impressive due to changes in a lot of header and source files (since logformat expressions are broadly used), but it does a simple thing: it defines the lf_expr structure which itself holds a generic list of logformat nodes, and then declares some helpers to manipulate lf_expr elements and fixes the code so that we now exclusively manipulate logformat_node lists as lf_expr elements outside of log.c. For now, lf_expr struct only contains the list of logformat nodes (no additional metadata), but now that we have dedicated type and helpers, doing so in the future won't be problematic at all and won't require extensive code changes.
2024-02-23 09:57:21 -05:00
* lf_expr: the destination logformat expression (logformat_node list)
MEDIUM: log: add compiling logic to logformat expressions split parse_logformat_string() into two functions: parse_logformat_string() sticks to the same behavior, but now becomes an helper for lf_expr_compile() which uses explicit arguments so that it becomes possible to use lf_expr_compile() without a proxy, but also compile an expression which was previously prepared for compiling (set string and config hints within the logformat expression to avoid manually storing string and config context if the compiling step happens later). lf_expr_dup() may be used to duplicate an expression before it is compiled, lf_expr_xfer() now makes sure that the input logformat is already compiled. This is some prerequisite works for log-profiles implementation, no functional change should be expected.
2024-02-29 08:54:43 -05:00
* which is supposed to be configured (str and conf set) but
CLEANUP: assorted typo fixes in the code and comments This is 41st iteration of typo fixes
2024-04-14 03:23:52 -04:00
* shouldn't be compiled (shouldn't contain any nodes)
MEDIUM: log: add compiling logic to logformat expressions split parse_logformat_string() into two functions: parse_logformat_string() sticks to the same behavior, but now becomes an helper for lf_expr_compile() which uses explicit arguments so that it becomes possible to use lf_expr_compile() without a proxy, but also compile an expression which was previously prepared for compiling (set string and config hints within the logformat expression to avoid manually storing string and config context if the compiling step happens later). lf_expr_dup() may be used to duplicate an expression before it is compiled, lf_expr_xfer() now makes sure that the input logformat is already compiled. This is some prerequisite works for log-profiles implementation, no functional change should be expected.
2024-02-29 08:54:43 -05:00
* al: arg list where sample expr should store arg dependency (if the logformat
* expression involves sample expressions), may be NULL
BUG/MEDIUM: log: emit '-' for empty fields again Commit 2b0108ad accidently got rid of the ability to emit a "-" for empty log fields. This can happen for captured request and response cookies, as well as for fetches. Since we don't want to have this done for headers however, we set the default log method when parsing the format. It is still possible to force the desired mode using +M/-M.
2013-02-05 12:52:25 -05:00
* options: LOG_OPT_* to force on every node
MINOR: log: indicate it when some unreliable sample fetches are logged If a log-format involves some sample fetches that may not be present at the logging instant, we can now report a warning. Note that this is done both for log-format and for add-header and carefully respects the original fetch keyword's capabilities.
2013-01-07 19:10:24 -05:00
* cap: all SMP_VAL_* flags supported by the consumer
MEDIUM: log-format: strict parsing and enable fail Until now, the function parse_logformat_string() never fails. It send warnings when it parses bad format, and returns expression in best effort. This patch replaces warnings by alert and returns a fail code. Maybe the warning mode is designed for a compatibility with old configuration versions. If it is the case, now this compatibility is broken. [wt: no, the reason is that an alert must cause a startup failure, but this will be OK with next patch]
2016-11-22 17:11:21 -05:00
*
MEDIUM: log-format: Use standard HAProxy log system to report errors The function log format emit its own error message using Alert(). This patch replaces this behavior and uses the standard HAProxy error system (with memprintf). The benefits are: - cleaning the log system - the logformat can ignore the caller (actually the caller must set a flag designing the caller function). - Make the usage of the logformat function easy for future components.
2016-11-22 18:41:28 -05:00
* The function returns 1 in success case, otherwise, it returns 0 and err is filled.
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
*/
MEDIUM: log: add compiling logic to logformat expressions split parse_logformat_string() into two functions: parse_logformat_string() sticks to the same behavior, but now becomes an helper for lf_expr_compile() which uses explicit arguments so that it becomes possible to use lf_expr_compile() without a proxy, but also compile an expression which was previously prepared for compiling (set string and config hints within the logformat expression to avoid manually storing string and config context if the compiling step happens later). lf_expr_dup() may be used to duplicate an expression before it is compiled, lf_expr_xfer() now makes sure that the input logformat is already compiled. This is some prerequisite works for log-profiles implementation, no functional change should be expected.
2024-02-29 08:54:43 -05:00
int lf_expr_compile(struct lf_expr *lf_expr,
struct arg_list *al, int options, int cap, char **err)
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
{
MEDIUM: proxy/log: leverage lf_expr API for logformat preparsing Currently, the way proxy-oriented logformat directives are handled is way too complicated. Indeed, "log-format", "log-format-error", "log-format-sd" and "unique-id-format" all rely on preparsing hints stored inside proxy->conf member struct. Those preparsing hints include the original string that should be compiled once the proxy parameters are known plus the config file and line number where the string was found to generate precise error messages in case of failure during the compiling process that happens within check_config_validity(). Now that lf_expr API permits to compile a lf_expr struct that was previously prepared (with original string and config hints), let's leverage lf_expr_compile() from check_config_validity() and instead of relying on individual proxy->conf hints for each logformat expression, store string and config hints in the lf_expr struct directly and use lf_expr helpers funcs to handle them when relevant (ie: original logformat string freeing is now done at a central place inside lf_expr_deinit(), which allows for some simplifications) Doing so allows us to greatly simplify the preparsing logic for those 4 proxy directives, and to finally save some space in the proxy struct. Also, since httpclient proxy has its "logformat" automatically compiled in check_config_validity(), we now use the file hint from the logformat expression struct to set an explicit name that will be reported in case of error ("parsing [httpclient:0] : ...") and remove the extraneous check in httpclient_precheck() (logformat was parsed twice previously..)
2024-03-05 09:44:43 -05:00
char *fmt = lf_expr->str; /* will be freed unless default */
MINOR: log: make parse_logformat_string() take a const char * Sometimes we can't pass a char *, and there is no need for this since we strdup() it.
2012-12-24 06:36:33 -05:00
char *sp, *str, *backfmt; /* start pointer for text parts */
MEDIUM: log-format: make the format parser more robust and more extensible The log-format parser reached a limit making it hard to add new features. It also suffers from a weak handling of certain incorrect corner cases, for example "%{foo}" is emitted as a litteral while syntactically it's an argument to no variable. Also the argument parser had to redo some of the job with some cases causing minor memory leaks (eg: ignored args). This work aims at improving the situation so that slightly better reporting is possible and that it becomes possible to extend the log format. The code has a few more states but looks significantly simpler. The parser is now capable of reporting ignored arguments and truncated lines.
2012-12-20 15:23:42 -05:00
char *arg = NULL; /* start pointer for args */
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
char *alias = NULL; /* start pointer for aliases */
MINOR: log: custom name for logformat node Add the ability to specify custom name (will be used for representation in verbose output types such as json) to logformat nodes. For now, a custom name should be composed by characters [a-zA-Z0-9-_]*
2024-01-08 12:38:29 -05:00
char *name = NULL; /* token name (optional) */
MINOR: log: explicit typecasting for logformat nodes Add the ability to manually specify desired output type after a custom field name for logformat nodes. Forcing the type can be useful to ensure value is stored with the proper type representation. (i.e.: forcing numerical to string to work around the limited resolution of JS number types) By default, type is set to SMP_T_SAME, which means the original type will be preserved. Currently supported types are: bool, str, sint
2024-02-20 04:29:49 -05:00
char *typecast_str = NULL; /* token output type (if custom name is set) */
MEDIUM: log-format: make the format parser more robust and more extensible The log-format parser reached a limit making it hard to add new features. It also suffers from a weak handling of certain incorrect corner cases, for example "%{foo}" is emitted as a litteral while syntactically it's an argument to no variable. Also the argument parser had to redo some of the job with some cases causing minor memory leaks (eg: ignored args). This work aims at improving the situation so that slightly better reporting is possible and that it becomes possible to extend the log format. The code has a few more states but looks significantly simpler. The parser is now capable of reporting ignored arguments and truncated lines.
2012-12-20 15:23:42 -05:00
int arg_len = 0;
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
int alias_len = 0;
MINOR: log: custom name for logformat node Add the ability to specify custom name (will be used for representation in verbose output types such as json) to logformat nodes. For now, a custom name should be composed by characters [a-zA-Z0-9-_]*
2024-01-08 12:38:29 -05:00
int name_len = 0;
MINOR: log: explicit typecasting for logformat nodes Add the ability to manually specify desired output type after a custom field name for logformat nodes. Forcing the type can be useful to ensure value is stored with the proper type representation. (i.e.: forcing numerical to string to work around the limited resolution of JS number types) By default, type is set to SMP_T_SAME, which means the original type will be preserved. Currently supported types are: bool, str, sint
2024-02-20 04:29:49 -05:00
int typecast = SMP_T_SAME; /* relaxed by default */
MEDIUM: log-format: make the format parser more robust and more extensible The log-format parser reached a limit making it hard to add new features. It also suffers from a weak handling of certain incorrect corner cases, for example "%{foo}" is emitted as a litteral while syntactically it's an argument to no variable. Also the argument parser had to redo some of the job with some cases causing minor memory leaks (eg: ignored args). This work aims at improving the situation so that slightly better reporting is possible and that it becomes possible to extend the log format. The code has a few more states but looks significantly simpler. The parser is now capable of reporting ignored arguments and truncated lines.
2012-12-20 15:23:42 -05:00
int cformat; /* current token format */
int pformat; /* previous token format */
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
MEDIUM: log: add compiling logic to logformat expressions split parse_logformat_string() into two functions: parse_logformat_string() sticks to the same behavior, but now becomes an helper for lf_expr_compile() which uses explicit arguments so that it becomes possible to use lf_expr_compile() without a proxy, but also compile an expression which was previously prepared for compiling (set string and config hints within the logformat expression to avoid manually storing string and config context if the compiling step happens later). lf_expr_dup() may be used to duplicate an expression before it is compiled, lf_expr_xfer() now makes sure that the input logformat is already compiled. This is some prerequisite works for log-profiles implementation, no functional change should be expected.
2024-02-29 08:54:43 -05:00
BUG_ON((lf_expr->flags & LF_FL_COMPILED));
if (!fmt)
return 1; // nothing to do
MINOR: log: make parse_logformat_string() take a const char * Sometimes we can't pass a char *, and there is no need for this since we strdup() it.
2012-12-24 06:36:33 -05:00
sp = str = backfmt = strdup(fmt);
BUG/MINOR: log-format: uncatched memory allocation functions Some return code of memory allocation functions are not tested. This patch fix theses checks.
2016-11-22 17:24:10 -05:00
if (!str) {
MEDIUM: log-format: Use standard HAProxy log system to report errors The function log format emit its own error message using Alert(). This patch replaces this behavior and uses the standard HAProxy error system (with memprintf). The benefits are: - cleaning the log system - the logformat can ignore the caller (actually the caller must set a flag designing the caller function). - Make the usage of the logformat function easy for future components.
2016-11-22 18:41:28 -05:00
memprintf(err, "out of memory error");
BUG/MINOR: log-format: uncatched memory allocation functions Some return code of memory allocation functions are not tested. This patch fix theses checks.
2016-11-22 17:24:10 -05:00
return 0;
}
MINOR: log: log-format: usable without httplog and tcplog Options httplog and tcplog aren't mandatory anymore for the log-format. The LW_ flags are now set during the log-format string parsing.
2012-04-02 10:22:10 -04:00
MEDIUM: log: add compiling logic to logformat expressions split parse_logformat_string() into two functions: parse_logformat_string() sticks to the same behavior, but now becomes an helper for lf_expr_compile() which uses explicit arguments so that it becomes possible to use lf_expr_compile() without a proxy, but also compile an expression which was previously prepared for compiling (set string and config hints within the logformat expression to avoid manually storing string and config context if the compiling step happens later). lf_expr_dup() may be used to duplicate an expression before it is compiled, lf_expr_xfer() now makes sure that the input logformat is already compiled. This is some prerequisite works for log-profiles implementation, no functional change should be expected.
2024-02-29 08:54:43 -05:00
/* Prepare lf_expr nodes, past this lf_expr doesn't know about ->str
* anymore as ->str and ->nodes are part of the same union. ->str has
* been saved as local 'fmt' string pointer, so we must free it before
* returning.
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
*/
MINOR: log: store lf_expr nodes inside substruct Add another struct level inside lf_expr struct to allow new information to be stored alongside lf_expr nodes.
2024-03-25 06:29:58 -04:00
LIST_INIT(&lf_expr->nodes.list);
BUG/MINOR: log: fix global lf_expr node options behavior (2nd try) In 98b44e8 ("BUG/MINOR: log: fix global lf_expr node options behavior"), I properly restored global node options behavior for when encoding is not used, however the fix is not optimal when encoding is involved: Indeed, encoding logic in sess_build_logline() relies on global node options to know if encoding must be handled expression-wide or individually. However, because of the above fix, if an expression is made of 1 or multiple nodes that all set an encoding option manually (without '%o'), we consider that the option was set globally, but that's probably not what the user intended. Instead we should only evaluate global options from '%o', so that it remains possible to skip global encoding when needed. No backport needed.
2024-04-29 09:58:36 -04:00
lf_expr->nodes.options = LOG_OPT_NONE;
MEDIUM: log: add compiling logic to logformat expressions split parse_logformat_string() into two functions: parse_logformat_string() sticks to the same behavior, but now becomes an helper for lf_expr_compile() which uses explicit arguments so that it becomes possible to use lf_expr_compile() without a proxy, but also compile an expression which was previously prepared for compiling (set string and config hints within the logformat expression to avoid manually storing string and config context if the compiling step happens later). lf_expr_dup() may be used to duplicate an expression before it is compiled, lf_expr_xfer() now makes sure that the input logformat is already compiled. This is some prerequisite works for log-profiles implementation, no functional change should be expected.
2024-02-29 08:54:43 -05:00
/* we must set the compiled flag now for proper deinit in case of failure */
lf_expr->flags |= LF_FL_COMPILED;
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
MEDIUM: log-format: make the format parser more robust and more extensible The log-format parser reached a limit making it hard to add new features. It also suffers from a weak handling of certain incorrect corner cases, for example "%{foo}" is emitted as a litteral while syntactically it's an argument to no variable. Also the argument parser had to redo some of the job with some cases causing minor memory leaks (eg: ignored args). This work aims at improving the situation so that slightly better reporting is possible and that it becomes possible to extend the log format. The code has a few more states but looks significantly simpler. The parser is now capable of reporting ignored arguments and truncated lines.
2012-12-20 15:23:42 -05:00
for (cformat = LF_INIT; cformat != LF_END; str++) {
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
pformat = cformat;
MEDIUM: log-format: make the format parser more robust and more extensible The log-format parser reached a limit making it hard to add new features. It also suffers from a weak handling of certain incorrect corner cases, for example "%{foo}" is emitted as a litteral while syntactically it's an argument to no variable. Also the argument parser had to redo some of the job with some cases causing minor memory leaks (eg: ignored args). This work aims at improving the situation so that slightly better reporting is possible and that it becomes possible to extend the log format. The code has a few more states but looks significantly simpler. The parser is now capable of reporting ignored arguments and truncated lines.
2012-12-20 15:23:42 -05:00
if (!*str)
cformat = LF_END; // preset it to save all states from doing this
CLEANUP: Fix typos in the log subsystem Fix some misspells in the code comments of the log subsystem.
2018-11-15 15:10:04 -05:00
/* The principle of the two-step state machine below is to first detect a change, and
MEDIUM: log-format: make the format parser more robust and more extensible The log-format parser reached a limit making it hard to add new features. It also suffers from a weak handling of certain incorrect corner cases, for example "%{foo}" is emitted as a litteral while syntactically it's an argument to no variable. Also the argument parser had to redo some of the job with some cases causing minor memory leaks (eg: ignored args). This work aims at improving the situation so that slightly better reporting is possible and that it becomes possible to extend the log format. The code has a few more states but looks significantly simpler. The parser is now capable of reporting ignored arguments and truncated lines.
2012-12-20 15:23:42 -05:00
* second have all common paths processed at one place. The common paths are the ones
* encountered in text areas (LF_INIT, LF_TEXT, LF_SEPARATOR) and at the end (LF_END).
* We use the common LF_INIT state to dispatch to the different final states.
*/
switch (pformat) {
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
case LF_STARTALIAS: // text immediately following a '%'
arg = NULL; alias = NULL;
MINOR: log: custom name for logformat node Add the ability to specify custom name (will be used for representation in verbose output types such as json) to logformat nodes. For now, a custom name should be composed by characters [a-zA-Z0-9-_]*
2024-01-08 12:38:29 -05:00
name = NULL;
name_len = 0;
MINOR: log: explicit typecasting for logformat nodes Add the ability to manually specify desired output type after a custom field name for logformat nodes. Forcing the type can be useful to ensure value is stored with the proper type representation. (i.e.: forcing numerical to string to work around the limited resolution of JS number types) By default, type is set to SMP_T_SAME, which means the original type will be preserved. Currently supported types are: bool, str, sint
2024-02-20 04:29:49 -05:00
typecast = SMP_T_SAME;
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
arg_len = alias_len = 0;
MINOR: log: custom name for logformat node Add the ability to specify custom name (will be used for representation in verbose output types such as json) to logformat nodes. For now, a custom name should be composed by characters [a-zA-Z0-9-_]*
2024-01-08 12:38:29 -05:00
if (*str == '(') { // custom output name
cformat = LF_STONAME;
name = str + 1;
}
else
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
goto startalias;
MINOR: log: custom name for logformat node Add the ability to specify custom name (will be used for representation in verbose output types such as json) to logformat nodes. For now, a custom name should be composed by characters [a-zA-Z0-9-_]*
2024-01-08 12:38:29 -05:00
break;
case LF_STONAME: // text immediately following '%('
MINOR: log: explicit typecasting for logformat nodes Add the ability to manually specify desired output type after a custom field name for logformat nodes. Forcing the type can be useful to ensure value is stored with the proper type representation. (i.e.: forcing numerical to string to work around the limited resolution of JS number types) By default, type is set to SMP_T_SAME, which means the original type will be preserved. Currently supported types are: bool, str, sint
2024-02-20 04:29:49 -05:00
case LF_STOTYPE:
if (cformat == LF_STONAME && *str == ':') { // start custom output type
cformat = LF_STOTYPE;
name_len = str -name;
typecast_str = str + 1;
}
else if (*str == ')') { // end of custom output name
if (cformat == LF_STONAME)
name_len = str - name;
else {
/* custom type */
*str = 0; // so that typecast_str is 0 terminated
typecast = type_to_smp(typecast_str);
if (typecast != SMP_T_STR && typecast != SMP_T_SINT &&
typecast != SMP_T_BOOL) {
memprintf(err, "unexpected output type '%.*s' at position %d line : '%s'. Supported types are: str, sint, bool", (int)(str - typecast_str), typecast_str, (int)(typecast_str - backfmt), fmt);
goto fail;
}
}
MINOR: log: custom name for logformat node Add the ability to specify custom name (will be used for representation in verbose output types such as json) to logformat nodes. For now, a custom name should be composed by characters [a-zA-Z0-9-_]*
2024-01-08 12:38:29 -05:00
cformat = LF_EDONAME;
} else if (!isalnum((unsigned char)*str) && *str != '_' && *str != '-') {
memprintf(err, "invalid character in custom name near '%c' at position %d line : '%s'",
*str, (int)(str - backfmt), fmt);
goto fail;
}
break;
case LF_EDONAME: // text immediately following %(name)
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
startalias:
MEDIUM: log-format: make the format parser more robust and more extensible The log-format parser reached a limit making it hard to add new features. It also suffers from a weak handling of certain incorrect corner cases, for example "%{foo}" is emitted as a litteral while syntactically it's an argument to no variable. Also the argument parser had to redo some of the job with some cases causing minor memory leaks (eg: ignored args). This work aims at improving the situation so that slightly better reporting is possible and that it becomes possible to extend the log format. The code has a few more states but looks significantly simpler. The parser is now capable of reporting ignored arguments and truncated lines.
2012-12-20 15:23:42 -05:00
if (*str == '{') { // optional argument
cformat = LF_STARG;
arg = str + 1;
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
}
MEDIUM: log: add the ability to include samples in logs Using %[expression] it becomes possible to make the log engine fetch some samples from the request or the response and provide them in the logs. Note that this feature is still limited, it does not yet allow to apply converters, to limit the output length, nor to specify the direction which should be fetched when a fetch function works in both directions. However it's quite convenient to log SSL information or to include some information that are used in stick tables. It is worth noting that this has been done in the generic log format handler, which means that the same information may be used to build the unique-id header and to pass the information to a backend server.
2012-12-20 18:09:23 -05:00
else if (*str == '[') {
cformat = LF_STEXPR;
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
alias = str + 1; // store expr in alias name
MEDIUM: log: add the ability to include samples in logs Using %[expression] it becomes possible to make the log engine fetch some samples from the request or the response and provide them in the logs. Note that this feature is still limited, it does not yet allow to apply converters, to limit the output length, nor to specify the direction which should be fetched when a fetch function works in both directions. However it's quite convenient to log SSL information or to include some information that are used in stick tables. It is worth noting that this has been done in the generic log format handler, which means that the same information may be used to build the unique-id header and to pass the information to a backend server.
2012-12-20 18:09:23 -05:00
}
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
else if (isalpha((unsigned char)*str)) { // alias name
cformat = LF_ALIAS;
alias = str;
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
}
MEDIUM: log-format: make the format parser more robust and more extensible The log-format parser reached a limit making it hard to add new features. It also suffers from a weak handling of certain incorrect corner cases, for example "%{foo}" is emitted as a litteral while syntactically it's an argument to no variable. Also the argument parser had to redo some of the job with some cases causing minor memory leaks (eg: ignored args). This work aims at improving the situation so that slightly better reporting is possible and that it becomes possible to extend the log format. The code has a few more states but looks significantly simpler. The parser is now capable of reporting ignored arguments and truncated lines.
2012-12-20 15:23:42 -05:00
else if (*str == '%')
CLEANUP: assorted typo fixes in the code and comments This is 8th iteration of typo fixes
2020-04-16 14:51:34 -04:00
cformat = LF_TEXT; // convert this character to a literal (useful for '%')
BUILD: log: fix build warning on Solaris The is* macros must not use a char on Solaris. Unsigned char is OK. Casting char to int is wrong as well since we get a negative value. src/log.c: In function `parse_logformat_string': src/log.c:454: warning: subscript has type `char'
2013-12-15 19:38:33 -05:00
else if (isdigit((unsigned char)*str) || *str == ' ' || *str == '\t') {
MEDIUM: log-format: relax parsing of '%' followed by unsupported characters At the moment when a '%' character is followed by any unhandled character, it is considered as a variable name, and if it cannot be resolved, a warning is emitted and the configuration goes on. When we start using log-format for redirect rules, it may happen that some people accidently use '%' instead of '%%' without understanding the cause of the issue. Thus we do two things here : - if a single '%' is followed by a blank or a digit, we fix it and emit a warning explaining how this should be done ; this ensures that existing configs continue to work ; - if a single '%' is followed by an unknown variable name, we report it and explain how to emit a verbatim '%' in case this is what the user desired.
2013-12-02 11:45:48 -05:00
/* single '%' followed by blank or digit, send them both */
cformat = LF_TEXT;
pformat = LF_TEXT; /* finally we include the previous char as well */
sp = str - 1; /* send both the '%' and the current char */
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
memprintf(err, "unexpected alias name near '%c' at position %d line : '%s'. Maybe you want to write a single '%%', use the syntax '%%%%'",
MEDIUM: log-format: Use standard HAProxy log system to report errors The function log format emit its own error message using Alert(). This patch replaces this behavior and uses the standard HAProxy error system (with memprintf). The benefits are: - cleaning the log system - the logformat can ignore the caller (actually the caller must set a flag designing the caller function). - Make the usage of the logformat function easy for future components.
2016-11-22 18:41:28 -05:00
*str, (int)(str - backfmt), fmt);
BUG/MINOR: log: fix minor resource leaks on logformat error path As reported by Ilya in issue #392, Coverity found that we're leaking allocated strings on error paths in parse_logformat(). Let's use a proper exit label for failures instead of seeding return 0 everywhere. This should be backported to all supported versions.
2019-12-11 06:05:39 -05:00
goto fail;
MEDIUM: log-format: relax parsing of '%' followed by unsupported characters At the moment when a '%' character is followed by any unhandled character, it is considered as a variable name, and if it cannot be resolved, a warning is emitted and the configuration goes on. When we start using log-format for redirect rules, it may happen that some people accidently use '%' instead of '%%' without understanding the cause of the issue. Thus we do two things here : - if a single '%' is followed by a blank or a digit, we fix it and emit a warning explaining how this should be done ; this ensures that existing configs continue to work ; - if a single '%' is followed by an unknown variable name, we report it and explain how to emit a verbatim '%' in case this is what the user desired.
2013-12-02 11:45:48 -05:00
}
MEDIUM: log-format: make the format parser more robust and more extensible The log-format parser reached a limit making it hard to add new features. It also suffers from a weak handling of certain incorrect corner cases, for example "%{foo}" is emitted as a litteral while syntactically it's an argument to no variable. Also the argument parser had to redo some of the job with some cases causing minor memory leaks (eg: ignored args). This work aims at improving the situation so that slightly better reporting is possible and that it becomes possible to extend the log format. The code has a few more states but looks significantly simpler. The parser is now capable of reporting ignored arguments and truncated lines.
2012-12-20 15:23:42 -05:00
else
CLEANUP: assorted typo fixes in the code and comments This is 8th iteration of typo fixes
2020-04-16 14:51:34 -04:00
cformat = LF_INIT; // handle other cases of literals
MEDIUM: log-format: make the format parser more robust and more extensible The log-format parser reached a limit making it hard to add new features. It also suffers from a weak handling of certain incorrect corner cases, for example "%{foo}" is emitted as a litteral while syntactically it's an argument to no variable. Also the argument parser had to redo some of the job with some cases causing minor memory leaks (eg: ignored args). This work aims at improving the situation so that slightly better reporting is possible and that it becomes possible to extend the log format. The code has a few more states but looks significantly simpler. The parser is now capable of reporting ignored arguments and truncated lines.
2012-12-20 15:23:42 -05:00
break;
case LF_STARG: // text immediately following '%{'
if (*str == '}') { // end of arg
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
cformat = LF_EDARG;
MEDIUM: log-format: make the format parser more robust and more extensible The log-format parser reached a limit making it hard to add new features. It also suffers from a weak handling of certain incorrect corner cases, for example "%{foo}" is emitted as a litteral while syntactically it's an argument to no variable. Also the argument parser had to redo some of the job with some cases causing minor memory leaks (eg: ignored args). This work aims at improving the situation so that slightly better reporting is possible and that it becomes possible to extend the log format. The code has a few more states but looks significantly simpler. The parser is now capable of reporting ignored arguments and truncated lines.
2012-12-20 15:23:42 -05:00
arg_len = str - arg;
*str = 0; // used for reporting errors
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
}
MEDIUM: log-format: make the format parser more robust and more extensible The log-format parser reached a limit making it hard to add new features. It also suffers from a weak handling of certain incorrect corner cases, for example "%{foo}" is emitted as a litteral while syntactically it's an argument to no variable. Also the argument parser had to redo some of the job with some cases causing minor memory leaks (eg: ignored args). This work aims at improving the situation so that slightly better reporting is possible and that it becomes possible to extend the log format. The code has a few more states but looks significantly simpler. The parser is now capable of reporting ignored arguments and truncated lines.
2012-12-20 15:23:42 -05:00
break;
case LF_EDARG: // text immediately following '%{arg}'
MEDIUM: log: add the ability to include samples in logs Using %[expression] it becomes possible to make the log engine fetch some samples from the request or the response and provide them in the logs. Note that this feature is still limited, it does not yet allow to apply converters, to limit the output length, nor to specify the direction which should be fetched when a fetch function works in both directions. However it's quite convenient to log SSL information or to include some information that are used in stick tables. It is worth noting that this has been done in the generic log format handler, which means that the same information may be used to build the unique-id header and to pass the information to a backend server.
2012-12-20 18:09:23 -05:00
if (*str == '[') {
cformat = LF_STEXPR;
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
alias = str + 1; // store expr in alias name
MEDIUM: log: add the ability to include samples in logs Using %[expression] it becomes possible to make the log engine fetch some samples from the request or the response and provide them in the logs. Note that this feature is still limited, it does not yet allow to apply converters, to limit the output length, nor to specify the direction which should be fetched when a fetch function works in both directions. However it's quite convenient to log SSL information or to include some information that are used in stick tables. It is worth noting that this has been done in the generic log format handler, which means that the same information may be used to build the unique-id header and to pass the information to a backend server.
2012-12-20 18:09:23 -05:00
break;
}
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
else if (isalnum((unsigned char)*str)) { // alias name
cformat = LF_ALIAS;
alias = str;
MEDIUM: log-format: make the format parser more robust and more extensible The log-format parser reached a limit making it hard to add new features. It also suffers from a weak handling of certain incorrect corner cases, for example "%{foo}" is emitted as a litteral while syntactically it's an argument to no variable. Also the argument parser had to redo some of the job with some cases causing minor memory leaks (eg: ignored args). This work aims at improving the situation so that slightly better reporting is possible and that it becomes possible to extend the log format. The code has a few more states but looks significantly simpler. The parser is now capable of reporting ignored arguments and truncated lines.
2012-12-20 15:23:42 -05:00
break;
}
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
memprintf(err, "parse argument modifier without alias name near '%%{%s}'", arg);
BUG/MINOR: log: fix minor resource leaks on logformat error path As reported by Ilya in issue #392, Coverity found that we're leaking allocated strings on error paths in parse_logformat(). Let's use a proper exit label for failures instead of seeding return 0 everywhere. This should be backported to all supported versions.
2019-12-11 06:05:39 -05:00
goto fail;
MEDIUM: log-format: make the format parser more robust and more extensible The log-format parser reached a limit making it hard to add new features. It also suffers from a weak handling of certain incorrect corner cases, for example "%{foo}" is emitted as a litteral while syntactically it's an argument to no variable. Also the argument parser had to redo some of the job with some cases causing minor memory leaks (eg: ignored args). This work aims at improving the situation so that slightly better reporting is possible and that it becomes possible to extend the log format. The code has a few more states but looks significantly simpler. The parser is now capable of reporting ignored arguments and truncated lines.
2012-12-20 15:23:42 -05:00
MEDIUM: log: add the ability to include samples in logs Using %[expression] it becomes possible to make the log engine fetch some samples from the request or the response and provide them in the logs. Note that this feature is still limited, it does not yet allow to apply converters, to limit the output length, nor to specify the direction which should be fetched when a fetch function works in both directions. However it's quite convenient to log SSL information or to include some information that are used in stick tables. It is worth noting that this has been done in the generic log format handler, which means that the same information may be used to build the unique-id header and to pass the information to a backend server.
2012-12-20 18:09:23 -05:00
case LF_STEXPR: // text immediately following '%['
MEDIUM: log-format: make the LF parser aware of sample expressions' end For a very long time it used to be impossible to pass a closing square bracket as a valid character in argument to a sample fetch function or to a converter because the LF parser used to stop on the first such character found and to pass what was between the first '[' and the first ']' to sample_parse_expr(). This patch addresses this by passing the whole string to sample_parse_expr() which is the only one authoritative to indicate the first character that does not belong to the expression. The LF parser then verifies it matches a ']' or fails. As a result it is finally possible to write rules such as the following, which is totally valid an unambigous : http-request redirect location %[url,regsub([.:/?-],!,g)] |-----| | | arg1 | `---> arg3 `-----> arg2 |-----------------| converter |---------------------| sample expression |------------------------| log-format tag
2020-02-14 11:33:06 -05:00
/* the whole sample expression is parsed at once,
* returning the pointer to the first character not
* part of the expression, which MUST be the trailing
* angle bracket.
*/
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
if (!add_sample_to_logformat_list(alias, name, name_len, typecast, arg, arg_len, lf_expr, al, options, cap, err, &str))
MEDIUM: log-format: make the LF parser aware of sample expressions' end For a very long time it used to be impossible to pass a closing square bracket as a valid character in argument to a sample fetch function or to a converter because the LF parser used to stop on the first such character found and to pass what was between the first '[' and the first ']' to sample_parse_expr(). This patch addresses this by passing the whole string to sample_parse_expr() which is the only one authoritative to indicate the first character that does not belong to the expression. The LF parser then verifies it matches a ']' or fails. As a result it is finally possible to write rules such as the following, which is totally valid an unambigous : http-request redirect location %[url,regsub([.:/?-],!,g)] |-----| | | arg1 | `---> arg3 `-----> arg2 |-----------------| converter |---------------------| sample expression |------------------------| log-format tag
2020-02-14 11:33:06 -05:00
goto fail;
if (*str == ']') {
// end of arg, go on with next state
cformat = pformat = LF_EDEXPR;
sp = str;
}
else {
char c = *str;
*str = 0;
BUILD: general: always pass unsigned chars to is* functions The isalnum(), isalpha(), isdigit() etc functions from ctype.h are supposed to take an int in argument which must either reflect an unsigned char or EOF. In practice on some platforms they're implemented as macros referencing an array, and when passed a char, they either cause a warning "array subscript has type 'char'" when lucky, or cause random segfaults when unlucky. It's quite unconvenient by the way since none of them may return true for negative values. The recent introduction of cygwin to the list of regularly tested build platforms revealed a lot of breakage there due to the same issues again. So this patch addresses the problem all over the code at once. It adds unsigned char casts to every valid use case, and also drops the unneeded double cast to int that was sometimes added on top of it. It may be backported by dropping irrelevant changes if that helps better support uncommon platforms. It's unlikely to fix bugs on platforms which would already not emit any warning though.
2020-02-25 02:16:33 -05:00
if (isprint((unsigned char)c))
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
memprintf(err, "expected ']' after '%s', but found '%c'", alias, c);
MEDIUM: log-format: make the LF parser aware of sample expressions' end For a very long time it used to be impossible to pass a closing square bracket as a valid character in argument to a sample fetch function or to a converter because the LF parser used to stop on the first such character found and to pass what was between the first '[' and the first ']' to sample_parse_expr(). This patch addresses this by passing the whole string to sample_parse_expr() which is the only one authoritative to indicate the first character that does not belong to the expression. The LF parser then verifies it matches a ']' or fails. As a result it is finally possible to write rules such as the following, which is totally valid an unambigous : http-request redirect location %[url,regsub([.:/?-],!,g)] |-----| | | arg1 | `---> arg3 `-----> arg2 |-----------------| converter |---------------------| sample expression |------------------------| log-format tag
2020-02-14 11:33:06 -05:00
else
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
memprintf(err, "missing ']' after '%s'", alias);
BUG/MEDIUM: log-format: fix possible endless loop in parse_logformat_string() This patch adds a missing break to end the loop in case when '%[' is not properly closed with ']'. The issue has been introduced with commit cd0d2ed ("MEDIUM: log-format: make the LF parser aware of sample expressions' end").
2020-06-30 15:16:43 -04:00
goto fail;
MEDIUM: log: add the ability to include samples in logs Using %[expression] it becomes possible to make the log engine fetch some samples from the request or the response and provide them in the logs. Note that this feature is still limited, it does not yet allow to apply converters, to limit the output length, nor to specify the direction which should be fetched when a fetch function works in both directions. However it's quite convenient to log SSL information or to include some information that are used in stick tables. It is worth noting that this has been done in the generic log format handler, which means that the same information may be used to build the unique-id header and to pass the information to a backend server.
2012-12-20 18:09:23 -05:00
}
break;
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
case LF_ALIAS: // text part of a alias name
alias_len = str - alias;
BUILD: log: fix build warning on Solaris The is* macros must not use a char on Solaris. Unsigned char is OK. Casting char to int is wrong as well since we get a negative value. src/log.c: In function `parse_logformat_string': src/log.c:454: warning: subscript has type `char'
2013-12-15 19:38:33 -05:00
if (!isalnum((unsigned char)*str))
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
cformat = LF_INIT; // not alias name anymore
MEDIUM: log-format: make the format parser more robust and more extensible The log-format parser reached a limit making it hard to add new features. It also suffers from a weak handling of certain incorrect corner cases, for example "%{foo}" is emitted as a litteral while syntactically it's an argument to no variable. Also the argument parser had to redo some of the job with some cases causing minor memory leaks (eg: ignored args). This work aims at improving the situation so that slightly better reporting is possible and that it becomes possible to extend the log format. The code has a few more states but looks significantly simpler. The parser is now capable of reporting ignored arguments and truncated lines.
2012-12-20 15:23:42 -05:00
break;
MEDIUM: log: add the ability to include samples in logs Using %[expression] it becomes possible to make the log engine fetch some samples from the request or the response and provide them in the logs. Note that this feature is still limited, it does not yet allow to apply converters, to limit the output length, nor to specify the direction which should be fetched when a fetch function works in both directions. However it's quite convenient to log SSL information or to include some information that are used in stick tables. It is worth noting that this has been done in the generic log format handler, which means that the same information may be used to build the unique-id header and to pass the information to a backend server.
2012-12-20 18:09:23 -05:00
default: // LF_INIT, LF_TEXT, LF_SEPARATOR, LF_END, LF_EDEXPR
MEDIUM: log-format: make the format parser more robust and more extensible The log-format parser reached a limit making it hard to add new features. It also suffers from a weak handling of certain incorrect corner cases, for example "%{foo}" is emitted as a litteral while syntactically it's an argument to no variable. Also the argument parser had to redo some of the job with some cases causing minor memory leaks (eg: ignored args). This work aims at improving the situation so that slightly better reporting is possible and that it becomes possible to extend the log format. The code has a few more states but looks significantly simpler. The parser is now capable of reporting ignored arguments and truncated lines.
2012-12-20 15:23:42 -05:00
cformat = LF_INIT;
}
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
if (cformat == LF_INIT) { /* resynchronize state to text/sep/startalias */
MEDIUM: log-format: make the format parser more robust and more extensible The log-format parser reached a limit making it hard to add new features. It also suffers from a weak handling of certain incorrect corner cases, for example "%{foo}" is emitted as a litteral while syntactically it's an argument to no variable. Also the argument parser had to redo some of the job with some cases causing minor memory leaks (eg: ignored args). This work aims at improving the situation so that slightly better reporting is possible and that it becomes possible to extend the log format. The code has a few more states but looks significantly simpler. The parser is now capable of reporting ignored arguments and truncated lines.
2012-12-20 15:23:42 -05:00
switch (*str) {
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
case '%': cformat = LF_STARTALIAS; break;
MEDIUM: log-format: make the format parser more robust and more extensible The log-format parser reached a limit making it hard to add new features. It also suffers from a weak handling of certain incorrect corner cases, for example "%{foo}" is emitted as a litteral while syntactically it's an argument to no variable. Also the argument parser had to redo some of the job with some cases causing minor memory leaks (eg: ignored args). This work aims at improving the situation so that slightly better reporting is possible and that it becomes possible to extend the log format. The code has a few more states but looks significantly simpler. The parser is now capable of reporting ignored arguments and truncated lines.
2012-12-20 15:23:42 -05:00
case 0 : cformat = LF_END; break;
MINOR: log-format: allow to preserve spacing in log format strings Now it's possible to preserve spacing everywhere except in "log-format", "log-format-sd" and "unique-id-format" directives, where spaces are delimiters and are merged. That may be useful when the response payload is specified as a log format string by "lf-file" or "lf-string", or even for headers or anything else. In order to merge spaces, a new option LOG_OPT_MERGE_SPACES is applied exclusively on options passed to function parse_logformat_string(). This patch fixes an issue #701 ("http-request return log-format file evaluation altering spacing of ASCII output/art").
2020-06-23 12:16:44 -04:00
case ' ':
if (options & LOG_OPT_MERGE_SPACES) {
cformat = LF_SEPARATOR;
break;
}
BUILD: logs: use __fallthrough in build_log_header() This avoids 4 build warnings when preprocessing happens before compiling with gcc >= 7.
2022-11-14 01:22:57 -05:00
__fallthrough;
MEDIUM: log-format: make the format parser more robust and more extensible The log-format parser reached a limit making it hard to add new features. It also suffers from a weak handling of certain incorrect corner cases, for example "%{foo}" is emitted as a litteral while syntactically it's an argument to no variable. Also the argument parser had to redo some of the job with some cases causing minor memory leaks (eg: ignored args). This work aims at improving the situation so that slightly better reporting is possible and that it becomes possible to extend the log format. The code has a few more states but looks significantly simpler. The parser is now capable of reporting ignored arguments and truncated lines.
2012-12-20 15:23:42 -05:00
default : cformat = LF_TEXT; break;
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
}
}
MEDIUM: log-format: make the format parser more robust and more extensible The log-format parser reached a limit making it hard to add new features. It also suffers from a weak handling of certain incorrect corner cases, for example "%{foo}" is emitted as a litteral while syntactically it's an argument to no variable. Also the argument parser had to redo some of the job with some cases causing minor memory leaks (eg: ignored args). This work aims at improving the situation so that slightly better reporting is possible and that it becomes possible to extend the log format. The code has a few more states but looks significantly simpler. The parser is now capable of reporting ignored arguments and truncated lines.
2012-12-20 15:23:42 -05:00
if (cformat != pformat || pformat == LF_SEPARATOR) {
switch (pformat) {
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
case LF_ALIAS:
if (!parse_logformat_alias(arg, arg_len, name, name_len, typecast, alias, alias_len, lf_expr, &options, err))
BUG/MINOR: log: fix minor resource leaks on logformat error path As reported by Ilya in issue #392, Coverity found that we're leaking allocated strings on error paths in parse_logformat(). Let's use a proper exit label for failures instead of seeding return 0 everywhere. This should be backported to all supported versions.
2019-12-11 06:05:39 -05:00
goto fail;
MEDIUM: log-format: make the format parser more robust and more extensible The log-format parser reached a limit making it hard to add new features. It also suffers from a weak handling of certain incorrect corner cases, for example "%{foo}" is emitted as a litteral while syntactically it's an argument to no variable. Also the argument parser had to redo some of the job with some cases causing minor memory leaks (eg: ignored args). This work aims at improving the situation so that slightly better reporting is possible and that it becomes possible to extend the log format. The code has a few more states but looks significantly simpler. The parser is now capable of reporting ignored arguments and truncated lines.
2012-12-20 15:23:42 -05:00
break;
case LF_TEXT:
case LF_SEPARATOR:
MEDIUM: tree-wide: add logformat expressions wrapper log format expressions are broadly used within the code: once they are parsed from input string, they are converted to a linked list of logformat nodes. We're starting to face some limitations because we're simply storing the converted expression as a generic logformat_node list. The first issue we're facing is that storing logformat expressions that way doesn't allow us to add metadata alongside the list, which is part of the prerequites for implementing log-profiles. Another issue with storing logformat expressions as generic lists of logformat_node elements is that it's starting to become really hard to tell when we rely on logformat expressions or not in the code given that there isn't always a comment near the list declaration or manipulation to indicate that it's relying on logformat expressions under the hood, so this adds some complexity for code maintenance. This patch looks quite impressive due to changes in a lot of header and source files (since logformat expressions are broadly used), but it does a simple thing: it defines the lf_expr structure which itself holds a generic list of logformat nodes, and then declares some helpers to manipulate lf_expr elements and fixes the code so that we now exclusively manipulate logformat_node lists as lf_expr elements outside of log.c. For now, lf_expr struct only contains the list of logformat nodes (no additional metadata), but now that we have dedicated type and helpers, doing so in the future won't be problematic at all and won't require extensive code changes.
2024-02-23 09:57:21 -05:00
if (!add_to_logformat_list(sp, str, pformat, lf_expr, err))
BUG/MINOR: log: fix minor resource leaks on logformat error path As reported by Ilya in issue #392, Coverity found that we're leaking allocated strings on error paths in parse_logformat(). Let's use a proper exit label for failures instead of seeding return 0 everywhere. This should be backported to all supported versions.
2019-12-11 06:05:39 -05:00
goto fail;
MEDIUM: log-format: make the format parser more robust and more extensible The log-format parser reached a limit making it hard to add new features. It also suffers from a weak handling of certain incorrect corner cases, for example "%{foo}" is emitted as a litteral while syntactically it's an argument to no variable. Also the argument parser had to redo some of the job with some cases causing minor memory leaks (eg: ignored args). This work aims at improving the situation so that slightly better reporting is possible and that it becomes possible to extend the log format. The code has a few more states but looks significantly simpler. The parser is now capable of reporting ignored arguments and truncated lines.
2012-12-20 15:23:42 -05:00
break;
}
sp = str; /* new start of text at every state switch and at every separator */
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
}
}
MEDIUM: log-format: make the format parser more robust and more extensible The log-format parser reached a limit making it hard to add new features. It also suffers from a weak handling of certain incorrect corner cases, for example "%{foo}" is emitted as a litteral while syntactically it's an argument to no variable. Also the argument parser had to redo some of the job with some cases causing minor memory leaks (eg: ignored args). This work aims at improving the situation so that slightly better reporting is possible and that it becomes possible to extend the log format. The code has a few more states but looks significantly simpler. The parser is now capable of reporting ignored arguments and truncated lines.
2012-12-20 15:23:42 -05:00
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
if (pformat == LF_STARTALIAS || pformat == LF_STARG || pformat == LF_STEXPR || pformat == LF_STONAME || pformat == LF_STOTYPE || pformat == LF_EDONAME) {
memprintf(err, "truncated line after '%s'", alias ? alias : arg ? arg : "%");
BUG/MINOR: log: fix minor resource leaks on logformat error path As reported by Ilya in issue #392, Coverity found that we're leaking allocated strings on error paths in parse_logformat(). Let's use a proper exit label for failures instead of seeding return 0 everywhere. This should be backported to all supported versions.
2019-12-11 06:05:39 -05:00
goto fail;
MEDIUM: log-format: strict parsing and enable fail Until now, the function parse_logformat_string() never fails. It send warnings when it parses bad format, and returns expression in best effort. This patch replaces warnings by alert and returns a fail code. Maybe the warning mode is designed for a compatibility with old configuration versions. If it is the case, now this compatibility is broken. [wt: no, the reason is that an alert must cause a startup failure, but this will be OK with next patch]
2016-11-22 17:11:21 -05:00
}
MEDIUM: proxy/log: leverage lf_expr API for logformat preparsing Currently, the way proxy-oriented logformat directives are handled is way too complicated. Indeed, "log-format", "log-format-error", "log-format-sd" and "unique-id-format" all rely on preparsing hints stored inside proxy->conf member struct. Those preparsing hints include the original string that should be compiled once the proxy parameters are known plus the config file and line number where the string was found to generate precise error messages in case of failure during the compiling process that happens within check_config_validity(). Now that lf_expr API permits to compile a lf_expr struct that was previously prepared (with original string and config hints), let's leverage lf_expr_compile() from check_config_validity() and instead of relying on individual proxy->conf hints for each logformat expression, store string and config hints in the lf_expr struct directly and use lf_expr helpers funcs to handle them when relevant (ie: original logformat string freeing is now done at a central place inside lf_expr_deinit(), which allows for some simplifications) Doing so allows us to greatly simplify the preparsing logic for those 4 proxy directives, and to finally save some space in the proxy struct. Also, since httpclient proxy has its "logformat" automatically compiled in check_config_validity(), we now use the file hint from the logformat expression struct to set an explicit name that will be reported in case of error ("parsing [httpclient:0] : ...") and remove the extraneous check in httpclient_precheck() (logformat was parsed twice previously..)
2024-03-05 09:44:43 -05:00
logformat_str_free(&fmt);
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
ha_free(&backfmt);
MEDIUM: log: add compiling logic to logformat expressions split parse_logformat_string() into two functions: parse_logformat_string() sticks to the same behavior, but now becomes an helper for lf_expr_compile() which uses explicit arguments so that it becomes possible to use lf_expr_compile() without a proxy, but also compile an expression which was previously prepared for compiling (set string and config hints within the logformat expression to avoid manually storing string and config context if the compiling step happens later). lf_expr_dup() may be used to duplicate an expression before it is compiled, lf_expr_xfer() now makes sure that the input logformat is already compiled. This is some prerequisite works for log-profiles implementation, no functional change should be expected.
2024-02-29 08:54:43 -05:00
return 1;
fail:
MEDIUM: proxy/log: leverage lf_expr API for logformat preparsing Currently, the way proxy-oriented logformat directives are handled is way too complicated. Indeed, "log-format", "log-format-error", "log-format-sd" and "unique-id-format" all rely on preparsing hints stored inside proxy->conf member struct. Those preparsing hints include the original string that should be compiled once the proxy parameters are known plus the config file and line number where the string was found to generate precise error messages in case of failure during the compiling process that happens within check_config_validity(). Now that lf_expr API permits to compile a lf_expr struct that was previously prepared (with original string and config hints), let's leverage lf_expr_compile() from check_config_validity() and instead of relying on individual proxy->conf hints for each logformat expression, store string and config hints in the lf_expr struct directly and use lf_expr helpers funcs to handle them when relevant (ie: original logformat string freeing is now done at a central place inside lf_expr_deinit(), which allows for some simplifications) Doing so allows us to greatly simplify the preparsing logic for those 4 proxy directives, and to finally save some space in the proxy struct. Also, since httpclient proxy has its "logformat" automatically compiled in check_config_validity(), we now use the file hint from the logformat expression struct to set an explicit name that will be reported in case of error ("parsing [httpclient:0] : ...") and remove the extraneous check in httpclient_precheck() (logformat was parsed twice previously..)
2024-03-05 09:44:43 -05:00
logformat_str_free(&fmt);
MEDIUM: log: add compiling logic to logformat expressions split parse_logformat_string() into two functions: parse_logformat_string() sticks to the same behavior, but now becomes an helper for lf_expr_compile() which uses explicit arguments so that it becomes possible to use lf_expr_compile() without a proxy, but also compile an expression which was previously prepared for compiling (set string and config hints within the logformat expression to avoid manually storing string and config context if the compiling step happens later). lf_expr_dup() may be used to duplicate an expression before it is compiled, lf_expr_xfer() now makes sure that the input logformat is already compiled. This is some prerequisite works for log-profiles implementation, no functional change should be expected.
2024-02-29 08:54:43 -05:00
ha_free(&backfmt);
return 0;
}
/* lf_expr_compile() helper: uses <curproxy> to deduce settings and
* simplify function usage, mostly for legacy purpose
*
* curproxy->conf.args.ctx must be set by the caller.
*
* The logformat expression will be scheduled for postcheck on the proxy unless
* the proxy was already checked, in which case all checks will be performed right
* away.
*
* Returns 1 on success and 0 on failure. On failure: <lf_expr> will be cleaned
* up and <err> will be set.
*/
int parse_logformat_string(const char *fmt, struct proxy *curproxy,
struct lf_expr *lf_expr,
int options, int cap, char **err)
{
int ret;
/* reinit lf_expr (if previously set) */
lf_expr_deinit(lf_expr);
lf_expr->str = strdup(fmt);
if (!lf_expr->str) {
memprintf(err, "out of memory error");
goto fail;
}
/* Save some parsing infos to raise relevant error messages during
* postparsing if needed
*/
if (curproxy->conf.args.file) {
lf_expr->conf.file = strdup(curproxy->conf.args.file);
lf_expr->conf.line = curproxy->conf.args.line;
}
ret = lf_expr_compile(lf_expr, &curproxy->conf.args, options, cap, err);
if (!ret)
goto fail;
BUG/MEDIUM: log: fix lf_expr_postcheck() behavior with default section Since 7a21c3a4ef ("MAJOR: log: implement proper postparsing for logformat expressions"), logformat expressions stored in a default section are not postchecked anymore. This is because the REGISTER_POST_PROXY_CHECK() only evaluates regular proxies. Because of this, proxy options which are automatically enabled on the proxy depending on the logformat expression features in use are not set on the default proxy, which means such options are not passed to the regular proxies that inherit from it (proxies that and will actually be running the logformat expression during runtime). Because of that, a logformat expression stored inside a default section and executed by a regular proxy may not behave properly. Also, since 03ca16f38b ("OPTIM: log: resolve logformat options during postparsing"), it's even worse because logformat node options postresoving is also skipped, which may also alter logformat expression encoding feature. To fix the issue, let's add a special case for default proxies in parse_logformat_string() and lf_expr_postcheck() so that default proxies are postchecked on the fly during parsing time in a "relaxed" way as we cannot assume that the features involved in the logformat expression won't be compatible with the proxy actually running it since we may have different types of proxies inheriting from the same default section. This bug was discovered while trying to address GH #2597. It should be backported to 3.0 with 7a21c3a4ef and 03ca16f38b.
2024-06-11 04:15:36 -04:00
if (!(curproxy->cap & PR_CAP_DEF) &&
!(curproxy->flags & PR_FL_CHECKED)) {
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
/* add the lf_expr to the proxy checks to delay postparsing
* since config-related proxy properties are not stable yet
*/
LIST_APPEND(&curproxy->conf.lf_checks, &lf_expr->list);
}
else {
BUG/MEDIUM: log: fix lf_expr_postcheck() behavior with default section Since 7a21c3a4ef ("MAJOR: log: implement proper postparsing for logformat expressions"), logformat expressions stored in a default section are not postchecked anymore. This is because the REGISTER_POST_PROXY_CHECK() only evaluates regular proxies. Because of this, proxy options which are automatically enabled on the proxy depending on the logformat expression features in use are not set on the default proxy, which means such options are not passed to the regular proxies that inherit from it (proxies that and will actually be running the logformat expression during runtime). Because of that, a logformat expression stored inside a default section and executed by a regular proxy may not behave properly. Also, since 03ca16f38b ("OPTIM: log: resolve logformat options during postparsing"), it's even worse because logformat node options postresoving is also skipped, which may also alter logformat expression encoding feature. To fix the issue, let's add a special case for default proxies in parse_logformat_string() and lf_expr_postcheck() so that default proxies are postchecked on the fly during parsing time in a "relaxed" way as we cannot assume that the features involved in the logformat expression won't be compatible with the proxy actually running it since we may have different types of proxies inheriting from the same default section. This bug was discovered while trying to address GH #2597. It should be backported to 3.0 with 7a21c3a4ef and 03ca16f38b.
2024-06-11 04:15:36 -04:00
/* default proxy, or regular proxy and probably called during
* runtime or with proxy already checked, perform the postcheck
* right away
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
*/
if (!lf_expr_postcheck(lf_expr, curproxy, err))
goto fail;
}
return 1;
MEDIUM: log: add compiling logic to logformat expressions split parse_logformat_string() into two functions: parse_logformat_string() sticks to the same behavior, but now becomes an helper for lf_expr_compile() which uses explicit arguments so that it becomes possible to use lf_expr_compile() without a proxy, but also compile an expression which was previously prepared for compiling (set string and config hints within the logformat expression to avoid manually storing string and config context if the compiling step happens later). lf_expr_dup() may be used to duplicate an expression before it is compiled, lf_expr_xfer() now makes sure that the input logformat is already compiled. This is some prerequisite works for log-profiles implementation, no functional change should be expected.
2024-02-29 08:54:43 -05:00
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
fail:
MEDIUM: log: add compiling logic to logformat expressions split parse_logformat_string() into two functions: parse_logformat_string() sticks to the same behavior, but now becomes an helper for lf_expr_compile() which uses explicit arguments so that it becomes possible to use lf_expr_compile() without a proxy, but also compile an expression which was previously prepared for compiling (set string and config hints within the logformat expression to avoid manually storing string and config context if the compiling step happens later). lf_expr_dup() may be used to duplicate an expression before it is compiled, lf_expr_xfer() now makes sure that the input logformat is already compiled. This is some prerequisite works for log-profiles implementation, no functional change should be expected.
2024-02-29 08:54:43 -05:00
lf_expr_deinit(lf_expr);
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
return 0;
}
OPTIM: log: resolve logformat options during postparsing In lf_buildctx_prepare(), we perform costly bitwise operations for every nodes to resolve node options and check for incompatibilities with global options. In fact, all this logic may safely be performed during postparsing. This is what we're doing in this commit. Doing so saves us from unnecessary runtime checks and could help speedup sess_build_logline(). Since checks are not as costly as before (due to them being performed during postparsing and not on log building path anymore), an complementary check for OPT_HTTP vs OPT_ENCODE incompatibity was added: encoding is ignored if HTTP option is set, unless HTTP option wasn't set globally and encoding was set globally, which means encoding takes the precedence Thanks to this patch, lf_buildctx_prepare() now only takes care of assigning proper typecast and options settings depending if it's used from global or per-node context, and prepares CBOR-specific structure members when CBOR encode option is set.
2024-04-30 11:42:00 -04:00
/* automatically resolves incompatible LOG_OPT options by taking into
* account current options and global options
*/
static inline void _lf_expr_postcheck_node_opt(int *options, int g_options)
{
/* encoding is incompatible with HTTP option, so it is ignored
* if HTTP option is set, unless HTTP option wasn't set globally
* and encoding was set globally, which means encoding takes the
* precedence>
*/
if (*options & LOG_OPT_HTTP) {
if ((g_options & (LOG_OPT_HTTP | LOG_OPT_ENCODE)) == LOG_OPT_ENCODE) {
/* global encoding enabled and http enabled individually */
*options &= ~LOG_OPT_HTTP;
}
else
*options &= ~LOG_OPT_ENCODE;
}
if (*options & LOG_OPT_ENCODE) {
/* when encoding is set, ignore +E option */
*options &= ~LOG_OPT_ESC;
}
}
/* Performs LOG_OPT postparsing check on logformat node <node> belonging to a
* given logformat expression <lf_expr>
*
* It returns 1 on success and 0 on error, <err> will be set in case of error
*/
static int lf_expr_postcheck_node_opt(struct lf_expr *lf_expr, struct logformat_node *node, char **err)
{
/* per-node encoding options cannot be disabled if already
* enabled globally
*
* Also, ensure we don't mix encoding types, global setting
* prevails over per-node one.
*
* Finally, ignore LOG_OPT_BIN since it is a global-only option
*/
if (lf_expr->nodes.options & LOG_OPT_ENCODE) {
node->options &= ~(LOG_OPT_BIN | LOG_OPT_ENCODE);
node->options |= (lf_expr->nodes.options & LOG_OPT_ENCODE);
}
else
node->options &= ~LOG_OPT_BIN;
_lf_expr_postcheck_node_opt(&node->options, lf_expr->nodes.options);
return 1;
}
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
/* Performs a postparsing check on logformat expression <expr> for a given <px>
* proxy. The function will behave differently depending on the proxy state
* (during parsing we will try to adapt proxy configuration to make it
* compatible with logformat expression, but once the proxy is checked, we fail
* as soon as we face incompatibilities)
*
BUG/MEDIUM: log: fix lf_expr_postcheck() behavior with default section Since 7a21c3a4ef ("MAJOR: log: implement proper postparsing for logformat expressions"), logformat expressions stored in a default section are not postchecked anymore. This is because the REGISTER_POST_PROXY_CHECK() only evaluates regular proxies. Because of this, proxy options which are automatically enabled on the proxy depending on the logformat expression features in use are not set on the default proxy, which means such options are not passed to the regular proxies that inherit from it (proxies that and will actually be running the logformat expression during runtime). Because of that, a logformat expression stored inside a default section and executed by a regular proxy may not behave properly. Also, since 03ca16f38b ("OPTIM: log: resolve logformat options during postparsing"), it's even worse because logformat node options postresoving is also skipped, which may also alter logformat expression encoding feature. To fix the issue, let's add a special case for default proxies in parse_logformat_string() and lf_expr_postcheck() so that default proxies are postchecked on the fly during parsing time in a "relaxed" way as we cannot assume that the features involved in the logformat expression won't be compatible with the proxy actually running it since we may have different types of proxies inheriting from the same default section. This bug was discovered while trying to address GH #2597. It should be backported to 3.0 with 7a21c3a4ef and 03ca16f38b.
2024-06-11 04:15:36 -04:00
* If the proxy is a default section, then allow the postcheck to succeed:
* the logformat expression may or may not work properly depending on the
* actual proxy that effectively runs it during runtime, but we have to stay
* permissive since we cannot assume it won't work.
*
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
* It returns 1 on success and 0 on error, <err> will be set in case of error.
*/
int lf_expr_postcheck(struct lf_expr *lf_expr, struct proxy *px, char **err)
{
struct logformat_node *lf;
BUG/MEDIUM: log: fix lf_expr_postcheck() behavior with default section Since 7a21c3a4ef ("MAJOR: log: implement proper postparsing for logformat expressions"), logformat expressions stored in a default section are not postchecked anymore. This is because the REGISTER_POST_PROXY_CHECK() only evaluates regular proxies. Because of this, proxy options which are automatically enabled on the proxy depending on the logformat expression features in use are not set on the default proxy, which means such options are not passed to the regular proxies that inherit from it (proxies that and will actually be running the logformat expression during runtime). Because of that, a logformat expression stored inside a default section and executed by a regular proxy may not behave properly. Also, since 03ca16f38b ("OPTIM: log: resolve logformat options during postparsing"), it's even worse because logformat node options postresoving is also skipped, which may also alter logformat expression encoding feature. To fix the issue, let's add a special case for default proxies in parse_logformat_string() and lf_expr_postcheck() so that default proxies are postchecked on the fly during parsing time in a "relaxed" way as we cannot assume that the features involved in the logformat expression won't be compatible with the proxy actually running it since we may have different types of proxies inheriting from the same default section. This bug was discovered while trying to address GH #2597. It should be backported to 3.0 with 7a21c3a4ef and 03ca16f38b.
2024-06-11 04:15:36 -04:00
int default_px = (px->cap & PR_CAP_DEF);
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
if (!(px->flags & PR_FL_CHECKED))
px->to_log |= LW_INIT;
OPTIM: log: resolve logformat options during postparsing In lf_buildctx_prepare(), we perform costly bitwise operations for every nodes to resolve node options and check for incompatibilities with global options. In fact, all this logic may safely be performed during postparsing. This is what we're doing in this commit. Doing so saves us from unnecessary runtime checks and could help speedup sess_build_logline(). Since checks are not as costly as before (due to them being performed during postparsing and not on log building path anymore), an complementary check for OPT_HTTP vs OPT_ENCODE incompatibity was added: encoding is ignored if HTTP option is set, unless HTTP option wasn't set globally and encoding was set globally, which means encoding takes the precedence Thanks to this patch, lf_buildctx_prepare() now only takes care of assigning proper typecast and options settings depending if it's used from global or per-node context, and prepares CBOR-specific structure members when CBOR encode option is set.
2024-04-30 11:42:00 -04:00
/* postcheck global node options */
_lf_expr_postcheck_node_opt(&lf_expr->nodes.options, LOG_OPT_NONE);
MINOR: log: store lf_expr nodes inside substruct Add another struct level inside lf_expr struct to allow new information to be stored alongside lf_expr nodes.
2024-03-25 06:29:58 -04:00
list_for_each_entry(lf, &lf_expr->nodes.list, list) {
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
if (lf->type == LOG_FMT_EXPR) {
struct sample_expr *expr = lf->expr;
uint8_t http_needed = !!(expr->fetch->use & SMP_USE_HTTP_ANY);
if ((px->flags & PR_FL_CHECKED)) {
/* fail as soon as proxy properties are not compatible */
if (http_needed && !px->http_needed) {
memprintf(err, "sample fetch '%s' requires HTTP enabled proxy which is not available here",
expr->fetch->kw);
goto fail;
}
OPTIM: log: resolve logformat options during postparsing In lf_buildctx_prepare(), we perform costly bitwise operations for every nodes to resolve node options and check for incompatibilities with global options. In fact, all this logic may safely be performed during postparsing. This is what we're doing in this commit. Doing so saves us from unnecessary runtime checks and could help speedup sess_build_logline(). Since checks are not as costly as before (due to them being performed during postparsing and not on log building path anymore), an complementary check for OPT_HTTP vs OPT_ENCODE incompatibity was added: encoding is ignored if HTTP option is set, unless HTTP option wasn't set globally and encoding was set globally, which means encoding takes the precedence Thanks to this patch, lf_buildctx_prepare() now only takes care of assigning proper typecast and options settings depending if it's used from global or per-node context, and prepares CBOR-specific structure members when CBOR encode option is set.
2024-04-30 11:42:00 -04:00
goto next_node;
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
}
/* check if we need to allocate an http_txn struct for HTTP parsing */
/* Note, we may also need to set curpx->to_log with certain fetches */
px->http_needed |= http_needed;
/* FIXME: temporary workaround for missing LW_XPRT and LW_REQ flags
* needed with some sample fetches (eg: ssl*). We always set it for
* now on, but this will leave with sample capabilities soon.
*/
px->to_log |= LW_XPRT;
if (px->http_needed)
px->to_log |= LW_REQ;
}
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
else if (lf->type == LOG_FMT_ALIAS) {
BUG/MEDIUM: log: fix lf_expr_postcheck() behavior with default section Since 7a21c3a4ef ("MAJOR: log: implement proper postparsing for logformat expressions"), logformat expressions stored in a default section are not postchecked anymore. This is because the REGISTER_POST_PROXY_CHECK() only evaluates regular proxies. Because of this, proxy options which are automatically enabled on the proxy depending on the logformat expression features in use are not set on the default proxy, which means such options are not passed to the regular proxies that inherit from it (proxies that and will actually be running the logformat expression during runtime). Because of that, a logformat expression stored inside a default section and executed by a regular proxy may not behave properly. Also, since 03ca16f38b ("OPTIM: log: resolve logformat options during postparsing"), it's even worse because logformat node options postresoving is also skipped, which may also alter logformat expression encoding feature. To fix the issue, let's add a special case for default proxies in parse_logformat_string() and lf_expr_postcheck() so that default proxies are postchecked on the fly during parsing time in a "relaxed" way as we cannot assume that the features involved in the logformat expression won't be compatible with the proxy actually running it since we may have different types of proxies inheriting from the same default section. This bug was discovered while trying to address GH #2597. It should be backported to 3.0 with 7a21c3a4ef and 03ca16f38b.
2024-06-11 04:15:36 -04:00
if (lf->alias->mode == PR_MODE_HTTP &&
!default_px && px->mode != PR_MODE_HTTP) {
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
memprintf(err, "format alias '%s' is reserved for HTTP mode",
lf->alias->name);
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
goto fail;
}
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
if (lf->alias->config_callback &&
!lf->alias->config_callback(lf, px)) {
memprintf(err, "cannot configure format alias '%s' in this context",
lf->alias->name);
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
goto fail;
}
if (!(px->flags & PR_FL_CHECKED))
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
px->to_log |= lf->alias->lw;
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
}
OPTIM: log: resolve logformat options during postparsing In lf_buildctx_prepare(), we perform costly bitwise operations for every nodes to resolve node options and check for incompatibilities with global options. In fact, all this logic may safely be performed during postparsing. This is what we're doing in this commit. Doing so saves us from unnecessary runtime checks and could help speedup sess_build_logline(). Since checks are not as costly as before (due to them being performed during postparsing and not on log building path anymore), an complementary check for OPT_HTTP vs OPT_ENCODE incompatibity was added: encoding is ignored if HTTP option is set, unless HTTP option wasn't set globally and encoding was set globally, which means encoding takes the precedence Thanks to this patch, lf_buildctx_prepare() now only takes care of assigning proper typecast and options settings depending if it's used from global or per-node context, and prepares CBOR-specific structure members when CBOR encode option is set.
2024-04-30 11:42:00 -04:00
next_node:
/* postcheck individual node's options */
if (!lf_expr_postcheck_node_opt(lf_expr, lf, err))
goto fail;
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
}
BUG/MEDIUM: log: fix lf_expr_postcheck() behavior with default section Since 7a21c3a4ef ("MAJOR: log: implement proper postparsing for logformat expressions"), logformat expressions stored in a default section are not postchecked anymore. This is because the REGISTER_POST_PROXY_CHECK() only evaluates regular proxies. Because of this, proxy options which are automatically enabled on the proxy depending on the logformat expression features in use are not set on the default proxy, which means such options are not passed to the regular proxies that inherit from it (proxies that and will actually be running the logformat expression during runtime). Because of that, a logformat expression stored inside a default section and executed by a regular proxy may not behave properly. Also, since 03ca16f38b ("OPTIM: log: resolve logformat options during postparsing"), it's even worse because logformat node options postresoving is also skipped, which may also alter logformat expression encoding feature. To fix the issue, let's add a special case for default proxies in parse_logformat_string() and lf_expr_postcheck() so that default proxies are postchecked on the fly during parsing time in a "relaxed" way as we cannot assume that the features involved in the logformat expression won't be compatible with the proxy actually running it since we may have different types of proxies inheriting from the same default section. This bug was discovered while trying to address GH #2597. It should be backported to 3.0 with 7a21c3a4ef and 03ca16f38b.
2024-06-11 04:15:36 -04:00
if (!default_px && (px->to_log & (LW_REQ | LW_RESP)) &&
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
(px->mode != PR_MODE_HTTP && !(px->options & PR_O_HTTP_UPG))) {
MINOR: log: change wording in lf_expr_postcheck() error message logformat_node was referenced as "node" in the error message reported to the user, but in fact it is referred to as "item" in user documentation. Using "item" in the error message to better comply with the doc. Error message was introduced with 7a21c3a4ef ("MAJOR: log: implement proper postparsing for logformat expressions")
2024-06-11 03:31:26 -04:00
memprintf(err, "logformat expression not usable here (at least one item depends on HTTP mode)");
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
goto fail;
}
MEDIUM: log-format: strict parsing and enable fail Until now, the function parse_logformat_string() never fails. It send warnings when it parses bad format, and returns expression in best effort. This patch replaces warnings by alert and returns a fail code. Maybe the warning mode is designed for a compatibility with old configuration versions. If it is the case, now this compatibility is broken. [wt: no, the reason is that an alert must cause a startup failure, but this will be OK with next patch]
2016-11-22 17:11:21 -05:00
return 1;
BUG/MINOR: log: fix minor resource leaks on logformat error path As reported by Ilya in issue #392, Coverity found that we're leaking allocated strings on error paths in parse_logformat(). Let's use a proper exit label for failures instead of seeding return 0 everywhere. This should be backported to all supported versions.
2019-12-11 06:05:39 -05:00
fail:
return 0;
MINOR: config: Parse the string of the log-format config keyword parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list
2012-02-08 10:37:49 -05:00
}
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
/* postparse logformats defined at <px> level */
static int postcheck_logformat_proxy(struct proxy *px)
{
char *err = NULL;
struct lf_expr *lf_expr, *back_lf;
int err_code = ERR_NONE;
list_for_each_entry_safe(lf_expr, back_lf, &px->conf.lf_checks, list) {
MEDIUM: log: add compiling logic to logformat expressions split parse_logformat_string() into two functions: parse_logformat_string() sticks to the same behavior, but now becomes an helper for lf_expr_compile() which uses explicit arguments so that it becomes possible to use lf_expr_compile() without a proxy, but also compile an expression which was previously prepared for compiling (set string and config hints within the logformat expression to avoid manually storing string and config context if the compiling step happens later). lf_expr_dup() may be used to duplicate an expression before it is compiled, lf_expr_xfer() now makes sure that the input logformat is already compiled. This is some prerequisite works for log-profiles implementation, no functional change should be expected.
2024-02-29 08:54:43 -05:00
BUG_ON(!(lf_expr->flags & LF_FL_COMPILED));
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
if (!lf_expr_postcheck(lf_expr, px, &err))
err_code |= ERR_FATAL | ERR_ALERT;
/* check performed, ensure it doesn't get checked twice */
LIST_DEL_INIT(&lf_expr->list);
if (err_code & ERR_CODE)
break;
}
if (err) {
memprintf(&err, "error detected while postparsing logformat expression used by %s '%s' : %s", proxy_type_str(px), px->id, err);
if (lf_expr->conf.file)
memprintf(&err, "parsing [%s:%d] : %s.\n", lf_expr->conf.file, lf_expr->conf.line, err);
ha_alert("%s", err);
ha_free(&err);
}
return err_code;
}
MINOR: log: Add "sample" new keyword to "log" lines. This patch implements the parsing of "sample" new optional keyword for "log" lines to be able to sample and balance the load of log messages between serveral log destinations declared by "log" lines. This keyword must be followed by a list of comma seperated ranges of indexes numbered from 1 to define the samples to be used to balance the load of logs to send. This "sample" keyword must be used on "log" lines obviously before the remaining optional ones without keyword. The list of ranges must be followed by a colon character to separate it from the log sampling size. With such following configuration declarations: log stderr local0 log 127.0.0.1:10001 sample 2-3,8-11:11 local0 log 127.0.0.2:10002 sample 5:5 local0 in addition to being sent to stderr, about the second "log" line, every 11 logs the logs #2 up to #3 would be sent to 127.0.0.1:10001, then #8 up tp #11 four logs would be sent to the same log server and so on periodically. Logs would be sent to 127.0.0.2:100002 every 5 logs. It is also possible to define the size of the sample with a value different of the maximum of the high limits of the ranges, for instance as follows: log 127.0.0.1:10001 sample 2-3,8-11:15 local0 as before the two logs #2 and #3 would be sent to 127.0.0.1:10001, then #8 up tp #11 logs, but in this case here, this would be done periodically every 15 messages. Also note that the ranges must not overlap each others. This is to ease the way the logs are periodically sent.
2019-04-24 10:14:33 -04:00
/*
CLEANUP: assorted typo fixes in the code and comments This is 8th iteration of typo fixes
2020-04-16 14:51:34 -04:00
* Parse the first range of indexes from a string made of a list of comma separated
MINOR: log: Add "sample" new keyword to "log" lines. This patch implements the parsing of "sample" new optional keyword for "log" lines to be able to sample and balance the load of log messages between serveral log destinations declared by "log" lines. This keyword must be followed by a list of comma seperated ranges of indexes numbered from 1 to define the samples to be used to balance the load of logs to send. This "sample" keyword must be used on "log" lines obviously before the remaining optional ones without keyword. The list of ranges must be followed by a colon character to separate it from the log sampling size. With such following configuration declarations: log stderr local0 log 127.0.0.1:10001 sample 2-3,8-11:11 local0 log 127.0.0.2:10002 sample 5:5 local0 in addition to being sent to stderr, about the second "log" line, every 11 logs the logs #2 up to #3 would be sent to 127.0.0.1:10001, then #8 up tp #11 four logs would be sent to the same log server and so on periodically. Logs would be sent to 127.0.0.2:100002 every 5 logs. It is also possible to define the size of the sample with a value different of the maximum of the high limits of the ranges, for instance as follows: log 127.0.0.1:10001 sample 2-3,8-11:15 local0 as before the two logs #2 and #3 would be sent to 127.0.0.1:10001, then #8 up tp #11 logs, but in this case here, this would be done periodically every 15 messages. Also note that the ranges must not overlap each others. This is to ease the way the logs are periodically sent.
2019-04-24 10:14:33 -04:00
* ranges of indexes. Note that an index may be considered as a particular range
* with a high limit to the low limit.
*/
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
int get_logger_smp_range(unsigned int *low, unsigned int *high, char **arg, char **err)
MINOR: log: Add "sample" new keyword to "log" lines. This patch implements the parsing of "sample" new optional keyword for "log" lines to be able to sample and balance the load of log messages between serveral log destinations declared by "log" lines. This keyword must be followed by a list of comma seperated ranges of indexes numbered from 1 to define the samples to be used to balance the load of logs to send. This "sample" keyword must be used on "log" lines obviously before the remaining optional ones without keyword. The list of ranges must be followed by a colon character to separate it from the log sampling size. With such following configuration declarations: log stderr local0 log 127.0.0.1:10001 sample 2-3,8-11:11 local0 log 127.0.0.2:10002 sample 5:5 local0 in addition to being sent to stderr, about the second "log" line, every 11 logs the logs #2 up to #3 would be sent to 127.0.0.1:10001, then #8 up tp #11 four logs would be sent to the same log server and so on periodically. Logs would be sent to 127.0.0.2:100002 every 5 logs. It is also possible to define the size of the sample with a value different of the maximum of the high limits of the ranges, for instance as follows: log 127.0.0.1:10001 sample 2-3,8-11:15 local0 as before the two logs #2 and #3 would be sent to 127.0.0.1:10001, then #8 up tp #11 logs, but in this case here, this would be done periodically every 15 messages. Also note that the ranges must not overlap each others. This is to ease the way the logs are periodically sent.
2019-04-24 10:14:33 -04:00
{
char *end, *p;
*low = *high = 0;
p = *arg;
end = strchr(p, ',');
if (!end)
end = p + strlen(p);
*high = *low = read_uint((const char **)&p, end);
if (!*low || (p != end && *p != '-'))
goto err;
if (p == end)
goto done;
p++;
*high = read_uint((const char **)&p, end);
if (!*high || *high <= *low || p != end)
goto err;
done:
if (*end == ',')
end++;
*arg = end;
return 1;
err:
memprintf(err, "wrong sample range '%s'", *arg);
return 0;
}
/*
* Returns 1 if the range defined by <low> and <high> overlaps
* one of them in <rgs> array of ranges with <sz> the size of this
* array, 0 if not.
*/
int smp_log_ranges_overlap(struct smp_log_range *rgs, size_t sz,
unsigned int low, unsigned int high, char **err)
{
size_t i;
for (i = 0; i < sz; i++) {
if ((low >= rgs[i].low && low <= rgs[i].high) ||
(high >= rgs[i].low && high <= rgs[i].high)) {
memprintf(err, "ranges are overlapping");
return 1;
}
}
return 0;
}
int smp_log_range_cmp(const void *a, const void *b)
{
const struct smp_log_range *rg_a = a;
const struct smp_log_range *rg_b = b;
if (rg_a->high < rg_b->low)
return -1;
else if (rg_a->low > rg_b->high)
return 1;
return 0;
}
MEDIUM: log: introduce log target log targets were immediately embedded in logger struct (previously named logsrv) and could not be used outside of this context. In this patch, we're introducing log_target type with the associated helper functions so that it becomes possible to declare and use log targets outside of loggers scope.
2023-09-11 10:10:37 -04:00
/* helper func */
static inline void init_log_target(struct log_target *target)
{
target->type = 0;
target->flags = LOG_TARGET_FL_NONE;
target->addr = NULL;
MAJOR: log: introduce log backends Using "mode log" in a backend section turns the proxy in a log backend which can be used to log-balance logs between multiple log targets (udp or tcp servers) log backends can be used as regular log targets using the log directive with "backend@be_name" prefix, like so: | log backend@mybackend local0 A log backend will distribute log messages to servers according to the log load-balancing algorithm that can be set using the "log-balance" option from the log backend section. For now, only the roundrobin algorithm is supported and set by default.
2023-09-13 05:52:31 -04:00
target->resolv_name = NULL;
MEDIUM: log: introduce log target log targets were immediately embedded in logger struct (previously named logsrv) and could not be used outside of this context. In this patch, we're introducing log_target type with the associated helper functions so that it becomes possible to declare and use log targets outside of loggers scope.
2023-09-11 10:10:37 -04:00
}
MEDIUM: log/balance: support FQDN for UDP log servers In previous log backend implementation, we created a pseudo log target for each declared log server, and we made the log target's address point to the actual server address to save some time and prevent unecessary copies. But this was done without knowing that when FQDN is involved (more broadly when dns/resolution is involved), the "port" part of server addr should not be relied upon, and we should explicitly use ->svc_port for that purpose. With that in mind and thanks to the previous commit, some changes were required: we allocate a dedicated addr within the log target when target is in DGRAM mode. The addr is first initialized with known values and it is then updated automatically by _srv_set_inetaddr() during runtime. (the change is atomic so readers don't need to worry about it) addr from server "log target" (INET/DGRAM mode) is made of the combination of server's address (lacking the port part) and server's svc_port.
2023-11-09 10:57:01 -05:00
void deinit_log_target(struct log_target *target)
MEDIUM: log: introduce log target log targets were immediately embedded in logger struct (previously named logsrv) and could not be used outside of this context. In this patch, we're introducing log_target type with the associated helper functions so that it becomes possible to declare and use log targets outside of loggers scope.
2023-09-11 10:10:37 -04:00
{
ha_free(&target->addr);
if (!(target->flags & LOG_TARGET_FL_RESOLVED))
MAJOR: log: introduce log backends Using "mode log" in a backend section turns the proxy in a log backend which can be used to log-balance logs between multiple log targets (udp or tcp servers) log backends can be used as regular log targets using the log directive with "backend@be_name" prefix, like so: | log backend@mybackend local0 A log backend will distribute log messages to servers according to the log load-balancing algorithm that can be set using the "log-balance" option from the log backend section. For now, only the roundrobin algorithm is supported and set by default.
2023-09-13 05:52:31 -04:00
ha_free(&target->resolv_name);
MEDIUM: log: introduce log target log targets were immediately embedded in logger struct (previously named logsrv) and could not be used outside of this context. In this patch, we're introducing log_target type with the associated helper functions so that it becomes possible to declare and use log targets outside of loggers scope.
2023-09-11 10:10:37 -04:00
}
/* returns 0 on failure and positive value on success */
static int dup_log_target(struct log_target *def, struct log_target *cpy)
{
BUG_ON((def->flags & LOG_TARGET_FL_RESOLVED)); /* postparsing already done, invalid use */
init_log_target(cpy);
MEDIUM: sink/log: stop relying on AF_UNSPEC for rings Since a5b325f92 ("MINOR: protocol: add a real family for existing FDs"), we don't rely anymore on AF_UNSPEC for buffer rings in do_send_log. But we kept it as a parsing hint to differentiate between implicit and named rings during ring buffer postparsing. However it is still a bit confusing and forces us to systematically rely on target->addr, even for named buffer rings where it doesn't make much sense anymore. Now that target->addr was made a pointer in a recent commit, we can choose not to initialize it when not needed (i.e.: named rings) and use this as a hint to distinguish implicit rings during init since they rely on the addr struct to temporarily store the ring's address until the ring is actually created during postparsing step.
2023-09-13 13:28:34 -04:00
if (def->addr) {
cpy->addr = malloc(sizeof(*cpy->addr));
if (!cpy->addr)
goto error;
*cpy->addr = *def->addr;
}
MAJOR: log: introduce log backends Using "mode log" in a backend section turns the proxy in a log backend which can be used to log-balance logs between multiple log targets (udp or tcp servers) log backends can be used as regular log targets using the log directive with "backend@be_name" prefix, like so: | log backend@mybackend local0 A log backend will distribute log messages to servers according to the log load-balancing algorithm that can be set using the "log-balance" option from the log backend section. For now, only the roundrobin algorithm is supported and set by default.
2023-09-13 05:52:31 -04:00
if (def->resolv_name) {
cpy->resolv_name = strdup(def->resolv_name);
if (!cpy->resolv_name)
MEDIUM: log: introduce log target log targets were immediately embedded in logger struct (previously named logsrv) and could not be used outside of this context. In this patch, we're introducing log_target type with the associated helper functions so that it becomes possible to declare and use log targets outside of loggers scope.
2023-09-11 10:10:37 -04:00
goto error;
}
cpy->type = def->type;
return 1;
error:
deinit_log_target(cpy);
return 0;
}
MINOR: log/backend: prevent tcp-{request,response} use with LOG mode We start implementing some postparsing compatibility checks for log backends. Here we report a warning if user tries to use tcp-{request,response} rules with log backend, and we properly ignore such rules when inherited from defaults section.
2023-11-16 04:48:34 -05:00
/* check that current configuration is compatible with "mode log" */
static int _postcheck_log_backend_compat(struct proxy *be)
{
int err_code = ERR_NONE;
BUG/MINOR: log/balance: detect if user tries to use unsupported algo b61147fd ("MEDIUM: log/balance: merge tcp/http algo with log ones") introduced some ambiguities, because while it shares some algos with the ones from mode {tcp,http}, we forgot report an error when the user tries to use an algorithm that is not available in this mode (as per the doc). Because of that, haproxy would silently drop log messages during runtime. To fix that, we ensure that algo is one of the supported ones during log backend postparsing. If the algo is not supported, we raise an error. This should be backported in 2.9 with b61147fd
2024-03-28 11:06:58 -04:00
int balance_algo = (be->lbprm.algo & BE_LB_ALGO);
MINOR: log/backend: prevent tcp-{request,response} use with LOG mode We start implementing some postparsing compatibility checks for log backends. Here we report a warning if user tries to use tcp-{request,response} rules with log backend, and we properly ignore such rules when inherited from defaults section.
2023-11-16 04:48:34 -05:00
if (!LIST_ISEMPTY(&be->tcp_req.inspect_rules) ||
!LIST_ISEMPTY(&be->tcp_req.l4_rules) ||
!LIST_ISEMPTY(&be->tcp_req.l5_rules)) {
ha_warning("Cannot use tcp-request rules with 'mode log' in %s '%s'. They will be ignored.\n",
proxy_type_str(be), be->id);
err_code |= ERR_WARN;
free_act_rules(&be->tcp_req.inspect_rules);
free_act_rules(&be->tcp_req.l4_rules);
free_act_rules(&be->tcp_req.l5_rules);
}
if (!LIST_ISEMPTY(&be->tcp_rep.inspect_rules)) {
ha_warning("Cannot use tcp-response rules with 'mode log' in %s '%s'. They will be ignored.\n",
proxy_type_str(be), be->id);
err_code |= ERR_WARN;
free_act_rules(&be->tcp_rep.inspect_rules);
}
MINOR: log/backend: prevent stick table and stick rules with LOG mode Report a warning and prevent errors if user tries to declare a stick table or use stick rules within a log backend.
2023-11-16 05:29:58 -05:00
if (be->table) {
ha_warning("Cannot use stick table with 'mode log' in %s '%s'. It will be ignored.\n",
proxy_type_str(be), be->id);
err_code |= ERR_WARN;
stktable_deinit(be->table);
ha_free(&be->table);
}
if (!LIST_ISEMPTY(&be->storersp_rules) ||
!LIST_ISEMPTY(&be->sticking_rules)) {
ha_warning("Cannot use sticking rules with 'mode log' in %s '%s'. They will be ignored.\n",
proxy_type_str(be), be->id);
err_code |= ERR_WARN;
free_stick_rules(&be->storersp_rules);
free_stick_rules(&be->sticking_rules);
}
MINOR: log/backend: prevent "http-send-name-header" use with LOG mode It doesn't make sense to use the "http-send-name-header" directive inside a log backend so we report a warning in with case and reset the setting.
2023-11-16 11:11:37 -05:00
if (isttest(be->server_id_hdr_name)) {
MINOR: log: fix "http-send-name-header" ignore warning message Warning message to indicate that the "http-send-name-header" option is ignored for backend in "mode log" was referenced using its internal struct wording instead of public name (as seen in the documentation). Let's fix that. It may be backported with c7783fb ("MINOR: log/backend: prevent "http-send-name-header" use with LOG mode") in 2.9.
2024-06-10 12:11:49 -04:00
ha_warning("Cannot set \"http-send-name-header\" with 'mode log' in %s '%s'. It will be ignored.\n",
MINOR: log/backend: prevent "http-send-name-header" use with LOG mode It doesn't make sense to use the "http-send-name-header" directive inside a log backend so we report a warning in with case and reset the setting.
2023-11-16 11:11:37 -05:00
proxy_type_str(be), be->id);
err_code |= ERR_WARN;
istfree(&be->server_id_hdr_name);
}
MINOR: log/backend: prevent "dynamic-cookie-key" use with LOG mode It doesn't make sense to set "dynamic-cookie-key" inside a log backend, thus we report a warning to the user and reset the setting.
2023-11-16 11:16:43 -05:00
if (be->dyncookie_key) {
ha_warning("Cannot set \"dynamic-cookie-key\" with 'mode log' in %s '%s'. It will be ignored.\n",
proxy_type_str(be), be->id);
err_code |= ERR_WARN;
ha_free(&be->dyncookie_key);
}
MINOR: log/backend: prevent "use-server" rules use with LOG mode server_rules declared using "use-server" keyword within a proxy are not supported inside a log backend (with "mode log" set), so we report a warning to the user and reset the setting.
2023-11-23 10:40:19 -05:00
if (!LIST_ISEMPTY(&be->server_rules)) {
ha_warning("Cannot use \"use-server\" rules with 'mode log' in %s '%s'. They will be ignored.\n",
proxy_type_str(be), be->id);
err_code |= ERR_WARN;
free_server_rules(&be->server_rules);
}
BUG/MINOR: log/balance: detect if user tries to use unsupported algo b61147fd ("MEDIUM: log/balance: merge tcp/http algo with log ones") introduced some ambiguities, because while it shares some algos with the ones from mode {tcp,http}, we forgot report an error when the user tries to use an algorithm that is not available in this mode (as per the doc). Because of that, haproxy would silently drop log messages during runtime. To fix that, we ensure that algo is one of the supported ones during log backend postparsing. If the algo is not supported, we raise an error. This should be backported in 2.9 with b61147fd
2024-03-28 11:06:58 -04:00
if (balance_algo != BE_LB_ALGO_RR &&
balance_algo != BE_LB_ALGO_RND &&
MINOR: lbprm: implement true "sticky" balance algo As previously mentioned in cd352c0db ("MINOR: log/balance: rename "log-sticky" to "sticky""), let's define a sticky algorithm that may be used from any protocol. Sticky algorithm sticks on the same server as long as it remains available. The documentation was updated accordingly.
2024-03-28 12:24:53 -04:00
balance_algo != BE_LB_ALGO_SS &&
BUG/MINOR: log/balance: detect if user tries to use unsupported algo b61147fd ("MEDIUM: log/balance: merge tcp/http algo with log ones") introduced some ambiguities, because while it shares some algos with the ones from mode {tcp,http}, we forgot report an error when the user tries to use an algorithm that is not available in this mode (as per the doc). Because of that, haproxy would silently drop log messages during runtime. To fix that, we ensure that algo is one of the supported ones during log backend postparsing. If the algo is not supported, we raise an error. This should be backported in 2.9 with b61147fd
2024-03-28 11:06:58 -04:00
balance_algo != BE_LB_ALGO_LH) {
MEDIUM: log/balance: leverage lbprm api for log load-balancing log load-balancing implementation was not seamlessly integrated within lbprm API. The consequence is that it could become harder to maintain over time since it added some specific cases just for the log backend. Moreover, it resulted in some code duplication since balance algorithms that are common to logs and regular (tcp, http) backends were specifically rewritten for log backends. Thanks to the previous commit, we now have all the prerequisites to make log load-balancing fully leverage lbprm logic. Thus in this patch we make __do_send_log_backend() use existing lbprm algorithms, and we no longer require log-specific lbprm initialization in cfgparse.c and in postcheck_log_backend(). As a bonus, for log backends this allows weighed algorithms to properly support weights (ie: roundrobin, random and log-hash) since we now leverage the same lb algorithms that we use for tcp/http backends (doc was updated).
2024-03-28 10:42:37 -04:00
/* cannot correct the error since lbprm init was already performed
* in cfgparse.c, so fail loudly
*/
BUG/MINOR: log/balance: detect if user tries to use unsupported algo b61147fd ("MEDIUM: log/balance: merge tcp/http algo with log ones") introduced some ambiguities, because while it shares some algos with the ones from mode {tcp,http}, we forgot report an error when the user tries to use an algorithm that is not available in this mode (as per the doc). Because of that, haproxy would silently drop log messages during runtime. To fix that, we ensure that algo is one of the supported ones during log backend postparsing. If the algo is not supported, we raise an error. This should be backported in 2.9 with b61147fd
2024-03-28 11:06:58 -04:00
ha_alert("in %s '%s': \"balance\" only supports 'roundrobin', 'random', 'sticky' and 'log-hash'.\n", proxy_type_str(be), be->id);
err_code |= ERR_ALERT | ERR_FATAL;
}
MINOR: log/backend: prevent tcp-{request,response} use with LOG mode We start implementing some postparsing compatibility checks for log backends. Here we report a warning if user tries to use tcp-{request,response} rules with log backend, and we properly ignore such rules when inherited from defaults section.
2023-11-16 04:48:34 -05:00
return err_code;
}
MAJOR: log: introduce log backends Using "mode log" in a backend section turns the proxy in a log backend which can be used to log-balance logs between multiple log targets (udp or tcp servers) log backends can be used as regular log targets using the log directive with "backend@be_name" prefix, like so: | log backend@mybackend local0 A log backend will distribute log messages to servers according to the log load-balancing algorithm that can be set using the "log-balance" option from the log backend section. For now, only the roundrobin algorithm is supported and set by default.
2023-09-13 05:52:31 -04:00
static int postcheck_log_backend(struct proxy *be)
{
char *msg = NULL;
struct server *srv;
int err_code = ERR_NONE;
int target_type = -1; // -1 is unused in log_tgt enum
if (be->mode != PR_MODE_SYSLOG ||
(be->flags & (PR_FL_DISABLED|PR_FL_STOPPED)))
return ERR_NONE; /* nothing to do */
MINOR: log/backend: prevent tcp-{request,response} use with LOG mode We start implementing some postparsing compatibility checks for log backends. Here we report a warning if user tries to use tcp-{request,response} rules with log backend, and we properly ignore such rules when inherited from defaults section.
2023-11-16 04:48:34 -05:00
err_code |= _postcheck_log_backend_compat(be);
if (err_code & ERR_CODE)
return err_code;
MEDIUM: log/balance: support for the "hash" lb algorithm hash lb algorithm can be configured with the "log-balance hash <cnv_list>" directive. With this algorithm, the user specifies a converter list with <cnv_list>. The produced log message will be passed as-is to the provided converter list, and the resulting hash will be used to select the log server that will receive the log message.
2023-09-19 04:51:53 -04:00
/* "log-balance hash" needs to compile its expression */
MEDIUM: log/balance: merge tcp/http algo with log ones "log-balance" directive was recently introduced to configure the balancing algorithm to use when in a log backend. However, it is confusing and it causes issues when used in default section. In this patch, we take another approach: first we remove the "log-balance" directive, and instead we rely on existing "balance" directive to configure log load balancing in log backend. Some algorithms such as roundrobin can be used as-is in a log backend, and for log-only algorithms, they are implemented as "log-$name" inside the "backend" directive. The documentation was updated accordingly.
2023-11-15 05:15:50 -05:00
if ((be->lbprm.algo & BE_LB_ALGO) == BE_LB_ALGO_LH) {
MEDIUM: log/balance: support for the "hash" lb algorithm hash lb algorithm can be configured with the "log-balance hash <cnv_list>" directive. With this algorithm, the user specifies a converter list with <cnv_list>. The produced log message will be passed as-is to the provided converter list, and the resulting hash will be used to select the log server that will receive the log message.
2023-09-19 04:51:53 -04:00
struct sample_expr *expr;
char *expr_str = NULL;
char *err_str = NULL;
int idx = 0;
/* only map-based hash method is supported for now */
if ((be->lbprm.algo & BE_LB_HASH_TYPE) != BE_LB_HASH_MAP) {
memprintf(&msg, "unsupported hash method (from \"hash-type\")");
err_code |= ERR_ALERT | ERR_FATAL;
goto end;
}
/* a little bit of explanation about what we're going to do here:
* as the user gave us a list of converters, instead of the fetch+conv list
* tuple as we're used to, we need to insert a dummy fetch at the start of
* the converter list so that sample_parse_expr() is able to properly parse
* the expr. We're explicitly using str() as dummy fetch, since the input
* sample that will be passed to the converter list at runtime will be a
* string (the log message about to be sent). Doing so allows sample_parse_expr()
* to ensure that the provided converters will be compatible with string type.
*/
memprintf(&expr_str, "str(dummy),%s", be->lbprm.arg_str);
if (!expr_str) {
memprintf(&msg, "memory error during converter list argument parsing (from \"log-balance hash\")");
err_code |= ERR_ALERT | ERR_FATAL;
goto end;
}
expr = sample_parse_expr((char*[]){expr_str, NULL}, &idx,
be->conf.file,
be->conf.line,
&err_str, NULL, NULL);
if (!expr) {
memprintf(&msg, "%s (from converter list argument in \"log-balance hash\")", err_str);
ha_free(&err_str);
err_code |= ERR_ALERT | ERR_FATAL;
ha_free(&expr_str);
goto end;
}
/* We expect the log_message->conv_list expr to resolve as a binary-compatible
* value because its output will be passed to gen_hash() to compute the hash.
*
* So we check the last converter's output type to ensure that it can be
* converted into the expected type. Invalid output type will result in an
* error to prevent unexpected results during runtime.
*/
if (sample_casts[smp_expr_output_type(expr)][SMP_T_BIN] == NULL) {
memprintf(&msg, "invalid output type at the end of converter list for \"log-balance hash\" directive");
err_code |= ERR_ALERT | ERR_FATAL;
release_sample_expr(expr);
ha_free(&expr_str);
goto end;
}
ha_free(&expr_str);
be->lbprm.expr = expr;
}
MAJOR: log: introduce log backends Using "mode log" in a backend section turns the proxy in a log backend which can be used to log-balance logs between multiple log targets (udp or tcp servers) log backends can be used as regular log targets using the log directive with "backend@be_name" prefix, like so: | log backend@mybackend local0 A log backend will distribute log messages to servers according to the log load-balancing algorithm that can be set using the "log-balance" option from the log backend section. For now, only the roundrobin algorithm is supported and set by default.
2023-09-13 05:52:31 -04:00
/* finish the initialization of proxy's servers */
srv = be->srv;
while (srv) {
BUG/MEDIUM: server: invalid address (post)parsing checks This bug was introduced with 29b76ca ("BUG/MEDIUM: server/log: "mode log" after server keyword causes crash ") Indeed, we cannot safely rely on addr_proto being set when str2sa_range() returns in parse_server() (even if SRV_PARSE_PARSE_ADDR is set), because proto lookup might be bypassed when FQDN addresses are involved. Unfortunately, the above patch wrongly assumed that proto would always be set when SRV_PARSE_PARSE_ADDR was passed to parse_server() (so when str2sa_range() was called), resulting in invalid postparsing checks being performed, which could as well lead to crashes with log backends ("mode log" set) because some postparsing init was skipped as a result of proto not being set and this wasn't expected later in the init code. To fix this, we now make use of the previous patch to perform server's address compatibility checks on hints that are always set when str2sa_range() succesfully returns. For log backend, we're also adding a complementary test to check if the address family is of expected type, else we report an error, plus we're moving the postinit logic in log api since _srv_check_proxy_mode() is only meant to check proxy mode compatibility and we were abusing it. This patch depends on: - "MINOR: tools: make str2sa_range() directly return type hints" No backport required unless 29b76ca gets backported.
2023-11-09 07:45:03 -05:00
BUG_ON(srv->log_target);
BUG_ON(srv->addr_type.proto_type != PROTO_TYPE_DGRAM &&
srv->addr_type.proto_type != PROTO_TYPE_STREAM);
srv->log_target = malloc(sizeof(*srv->log_target));
if (!srv->log_target) {
memprintf(&msg, "memory error when allocating log server '%s'\n", srv->id);
err_code |= ERR_ALERT | ERR_FATAL;
goto end;
}
MEDIUM: log/balance: support FQDN for UDP log servers In previous log backend implementation, we created a pseudo log target for each declared log server, and we made the log target's address point to the actual server address to save some time and prevent unecessary copies. But this was done without knowing that when FQDN is involved (more broadly when dns/resolution is involved), the "port" part of server addr should not be relied upon, and we should explicitly use ->svc_port for that purpose. With that in mind and thanks to the previous commit, some changes were required: we allocate a dedicated addr within the log target when target is in DGRAM mode. The addr is first initialized with known values and it is then updated automatically by _srv_set_inetaddr() during runtime. (the change is atomic so readers don't need to worry about it) addr from server "log target" (INET/DGRAM mode) is made of the combination of server's address (lacking the port part) and server's svc_port.
2023-11-09 10:57:01 -05:00
init_log_target(srv->log_target);
if (srv->addr_type.proto_type == PROTO_TYPE_DGRAM) {
BUG/MEDIUM: server: invalid address (post)parsing checks This bug was introduced with 29b76ca ("BUG/MEDIUM: server/log: "mode log" after server keyword causes crash ") Indeed, we cannot safely rely on addr_proto being set when str2sa_range() returns in parse_server() (even if SRV_PARSE_PARSE_ADDR is set), because proto lookup might be bypassed when FQDN addresses are involved. Unfortunately, the above patch wrongly assumed that proto would always be set when SRV_PARSE_PARSE_ADDR was passed to parse_server() (so when str2sa_range() was called), resulting in invalid postparsing checks being performed, which could as well lead to crashes with log backends ("mode log" set) because some postparsing init was skipped as a result of proto not being set and this wasn't expected later in the init code. To fix this, we now make use of the previous patch to perform server's address compatibility checks on hints that are always set when str2sa_range() succesfully returns. For log backend, we're also adding a complementary test to check if the address family is of expected type, else we report an error, plus we're moving the postinit logic in log api since _srv_check_proxy_mode() is only meant to check proxy mode compatibility and we were abusing it. This patch depends on: - "MINOR: tools: make str2sa_range() directly return type hints" No backport required unless 29b76ca gets backported.
2023-11-09 07:45:03 -05:00
srv->log_target->type = LOG_TARGET_DGRAM;
MEDIUM: log/balance: support FQDN for UDP log servers In previous log backend implementation, we created a pseudo log target for each declared log server, and we made the log target's address point to the actual server address to save some time and prevent unecessary copies. But this was done without knowing that when FQDN is involved (more broadly when dns/resolution is involved), the "port" part of server addr should not be relied upon, and we should explicitly use ->svc_port for that purpose. With that in mind and thanks to the previous commit, some changes were required: we allocate a dedicated addr within the log target when target is in DGRAM mode. The addr is first initialized with known values and it is then updated automatically by _srv_set_inetaddr() during runtime. (the change is atomic so readers don't need to worry about it) addr from server "log target" (INET/DGRAM mode) is made of the combination of server's address (lacking the port part) and server's svc_port.
2023-11-09 10:57:01 -05:00
/* Try to allocate log target addr (only used in DGRAM mode) */
srv->log_target->addr = calloc(1, sizeof(*srv->log_target->addr));
if (!srv->log_target->addr) {
memprintf(&msg, "memory error when allocating log server '%s'\n", srv->id);
err_code |= ERR_ALERT | ERR_FATAL;
goto end;
}
/* We must initialize it with known addr:svc_port, it will then
* be updated automatically by the server API for runtime changes
*/
ipcpy(&srv->addr, srv->log_target->addr);
set_host_port(srv->log_target->addr, srv->svc_port);
}
BUG/MEDIUM: server: invalid address (post)parsing checks This bug was introduced with 29b76ca ("BUG/MEDIUM: server/log: "mode log" after server keyword causes crash ") Indeed, we cannot safely rely on addr_proto being set when str2sa_range() returns in parse_server() (even if SRV_PARSE_PARSE_ADDR is set), because proto lookup might be bypassed when FQDN addresses are involved. Unfortunately, the above patch wrongly assumed that proto would always be set when SRV_PARSE_PARSE_ADDR was passed to parse_server() (so when str2sa_range() was called), resulting in invalid postparsing checks being performed, which could as well lead to crashes with log backends ("mode log" set) because some postparsing init was skipped as a result of proto not being set and this wasn't expected later in the init code. To fix this, we now make use of the previous patch to perform server's address compatibility checks on hints that are always set when str2sa_range() succesfully returns. For log backend, we're also adding a complementary test to check if the address family is of expected type, else we report an error, plus we're moving the postinit logic in log api since _srv_check_proxy_mode() is only meant to check proxy mode compatibility and we were abusing it. This patch depends on: - "MINOR: tools: make str2sa_range() directly return type hints" No backport required unless 29b76ca gets backported.
2023-11-09 07:45:03 -05:00
else {
/* for now BUFFER type only supports TCP server to it's almost
MEDIUM: log/balance: support FQDN for UDP log servers In previous log backend implementation, we created a pseudo log target for each declared log server, and we made the log target's address point to the actual server address to save some time and prevent unecessary copies. But this was done without knowing that when FQDN is involved (more broadly when dns/resolution is involved), the "port" part of server addr should not be relied upon, and we should explicitly use ->svc_port for that purpose. With that in mind and thanks to the previous commit, some changes were required: we allocate a dedicated addr within the log target when target is in DGRAM mode. The addr is first initialized with known values and it is then updated automatically by _srv_set_inetaddr() during runtime. (the change is atomic so readers don't need to worry about it) addr from server "log target" (INET/DGRAM mode) is made of the combination of server's address (lacking the port part) and server's svc_port.
2023-11-09 10:57:01 -05:00
* explicit
BUG/MEDIUM: server: invalid address (post)parsing checks This bug was introduced with 29b76ca ("BUG/MEDIUM: server/log: "mode log" after server keyword causes crash ") Indeed, we cannot safely rely on addr_proto being set when str2sa_range() returns in parse_server() (even if SRV_PARSE_PARSE_ADDR is set), because proto lookup might be bypassed when FQDN addresses are involved. Unfortunately, the above patch wrongly assumed that proto would always be set when SRV_PARSE_PARSE_ADDR was passed to parse_server() (so when str2sa_range() was called), resulting in invalid postparsing checks being performed, which could as well lead to crashes with log backends ("mode log" set) because some postparsing init was skipped as a result of proto not being set and this wasn't expected later in the init code. To fix this, we now make use of the previous patch to perform server's address compatibility checks on hints that are always set when str2sa_range() succesfully returns. For log backend, we're also adding a complementary test to check if the address family is of expected type, else we report an error, plus we're moving the postinit logic in log api since _srv_check_proxy_mode() is only meant to check proxy mode compatibility and we were abusing it. This patch depends on: - "MINOR: tools: make str2sa_range() directly return type hints" No backport required unless 29b76ca gets backported.
2023-11-09 07:45:03 -05:00
*/
srv->log_target->type = LOG_TARGET_BUFFER;
MEDIUM: log/balance: support FQDN for UDP log servers In previous log backend implementation, we created a pseudo log target for each declared log server, and we made the log target's address point to the actual server address to save some time and prevent unecessary copies. But this was done without knowing that when FQDN is involved (more broadly when dns/resolution is involved), the "port" part of server addr should not be relied upon, and we should explicitly use ->svc_port for that purpose. With that in mind and thanks to the previous commit, some changes were required: we allocate a dedicated addr within the log target when target is in DGRAM mode. The addr is first initialized with known values and it is then updated automatically by _srv_set_inetaddr() during runtime. (the change is atomic so readers don't need to worry about it) addr from server "log target" (INET/DGRAM mode) is made of the combination of server's address (lacking the port part) and server's svc_port.
2023-11-09 10:57:01 -05:00
srv->log_target->sink = sink_new_from_srv(srv, "log backend");
if (!srv->log_target->sink) {
memprintf(&msg, "error when creating sink from '%s' log server", srv->id);
err_code |= ERR_ALERT | ERR_FATAL;
goto end;
}
BUG/MEDIUM: server: invalid address (post)parsing checks This bug was introduced with 29b76ca ("BUG/MEDIUM: server/log: "mode log" after server keyword causes crash ") Indeed, we cannot safely rely on addr_proto being set when str2sa_range() returns in parse_server() (even if SRV_PARSE_PARSE_ADDR is set), because proto lookup might be bypassed when FQDN addresses are involved. Unfortunately, the above patch wrongly assumed that proto would always be set when SRV_PARSE_PARSE_ADDR was passed to parse_server() (so when str2sa_range() was called), resulting in invalid postparsing checks being performed, which could as well lead to crashes with log backends ("mode log" set) because some postparsing init was skipped as a result of proto not being set and this wasn't expected later in the init code. To fix this, we now make use of the previous patch to perform server's address compatibility checks on hints that are always set when str2sa_range() succesfully returns. For log backend, we're also adding a complementary test to check if the address family is of expected type, else we report an error, plus we're moving the postinit logic in log api since _srv_check_proxy_mode() is only meant to check proxy mode compatibility and we were abusing it. This patch depends on: - "MINOR: tools: make str2sa_range() directly return type hints" No backport required unless 29b76ca gets backported.
2023-11-09 07:45:03 -05:00
}
MAJOR: log: introduce log backends Using "mode log" in a backend section turns the proxy in a log backend which can be used to log-balance logs between multiple log targets (udp or tcp servers) log backends can be used as regular log targets using the log directive with "backend@be_name" prefix, like so: | log backend@mybackend local0 A log backend will distribute log messages to servers according to the log load-balancing algorithm that can be set using the "log-balance" option from the log backend section. For now, only the roundrobin algorithm is supported and set by default.
2023-09-13 05:52:31 -04:00
if (target_type == -1)
target_type = srv->log_target->type;
BUG/MEDIUM: server: invalid address (post)parsing checks This bug was introduced with 29b76ca ("BUG/MEDIUM: server/log: "mode log" after server keyword causes crash ") Indeed, we cannot safely rely on addr_proto being set when str2sa_range() returns in parse_server() (even if SRV_PARSE_PARSE_ADDR is set), because proto lookup might be bypassed when FQDN addresses are involved. Unfortunately, the above patch wrongly assumed that proto would always be set when SRV_PARSE_PARSE_ADDR was passed to parse_server() (so when str2sa_range() was called), resulting in invalid postparsing checks being performed, which could as well lead to crashes with log backends ("mode log" set) because some postparsing init was skipped as a result of proto not being set and this wasn't expected later in the init code. To fix this, we now make use of the previous patch to perform server's address compatibility checks on hints that are always set when str2sa_range() succesfully returns. For log backend, we're also adding a complementary test to check if the address family is of expected type, else we report an error, plus we're moving the postinit logic in log api since _srv_check_proxy_mode() is only meant to check proxy mode compatibility and we were abusing it. This patch depends on: - "MINOR: tools: make str2sa_range() directly return type hints" No backport required unless 29b76ca gets backported.
2023-11-09 07:45:03 -05:00
MAJOR: log: introduce log backends Using "mode log" in a backend section turns the proxy in a log backend which can be used to log-balance logs between multiple log targets (udp or tcp servers) log backends can be used as regular log targets using the log directive with "backend@be_name" prefix, like so: | log backend@mybackend local0 A log backend will distribute log messages to servers according to the log load-balancing algorithm that can be set using the "log-balance" option from the log backend section. For now, only the roundrobin algorithm is supported and set by default.
2023-09-13 05:52:31 -04:00
if (target_type != srv->log_target->type) {
memprintf(&msg, "cannot mix server types within a log backend, '%s' srv's network type differs from previous server", srv->id);
err_code |= ERR_ALERT | ERR_FATAL;
goto end;
}
MEDIUM: log/balance: support FQDN for UDP log servers In previous log backend implementation, we created a pseudo log target for each declared log server, and we made the log target's address point to the actual server address to save some time and prevent unecessary copies. But this was done without knowing that when FQDN is involved (more broadly when dns/resolution is involved), the "port" part of server addr should not be relied upon, and we should explicitly use ->svc_port for that purpose. With that in mind and thanks to the previous commit, some changes were required: we allocate a dedicated addr within the log target when target is in DGRAM mode. The addr is first initialized with known values and it is then updated automatically by _srv_set_inetaddr() during runtime. (the change is atomic so readers don't need to worry about it) addr from server "log target" (INET/DGRAM mode) is made of the combination of server's address (lacking the port part) and server's svc_port.
2023-11-09 10:57:01 -05:00
srv->log_target->flags |= LOG_TARGET_FL_RESOLVED;
MAJOR: log: introduce log backends Using "mode log" in a backend section turns the proxy in a log backend which can be used to log-balance logs between multiple log targets (udp or tcp servers) log backends can be used as regular log targets using the log directive with "backend@be_name" prefix, like so: | log backend@mybackend local0 A log backend will distribute log messages to servers according to the log load-balancing algorithm that can be set using the "log-balance" option from the log backend section. For now, only the roundrobin algorithm is supported and set by default.
2023-09-13 05:52:31 -04:00
srv = srv->next;
}
end:
if (err_code & ERR_CODE) {
ha_alert("log backend '%s': failed to initialize: %s.\n", be->id, msg);
ha_free(&msg);
}
return err_code;
}
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
/* resolves a single logger entry (it is expected to be called
MEDIUM: log/sink: make logsrv postparsing more generic We previously had postparsing logic but only for logsrv sinks, but now we need to make this operation on logsrv directly instead of sinks to prepare for additional postparsing logic that is not sink-specific. To do this, we migrated post_sink_resolve() and sink_postresolve_logsrvs() to their postresolve_logsrvs() and postresolve_logsrv_list() equivalents. Then, we split postresolve_logsrv_list() so that the sink-only logic stays in sink.c (sink_resolve_logsrv_buffer() function), and the "generic" target part stays in log.c as resolve_logsrv(). Error messages formatting was preserved as far as possible but some slight variations are to be expected. As for the functional aspect, no change should be expected.
2023-08-17 11:38:30 -04:00
* at postparsing stage)
*
* Returns err_code which defaults to ERR_NONE and can be set to a combination
* of ERR_WARN, ERR_ALERT, ERR_FATAL and ERR_ABORT in case of errors.
* <msg> could be set at any time (it will usually be set on error, but
CLEANUP: assorted typo fixes in the code and comments This is 37th iteration of typo fixes
2023-11-21 13:54:16 -05:00
* could also be set when no error occurred to report a diag warning), thus is
MEDIUM: log/sink: make logsrv postparsing more generic We previously had postparsing logic but only for logsrv sinks, but now we need to make this operation on logsrv directly instead of sinks to prepare for additional postparsing logic that is not sink-specific. To do this, we migrated post_sink_resolve() and sink_postresolve_logsrvs() to their postresolve_logsrvs() and postresolve_logsrv_list() equivalents. Then, we split postresolve_logsrv_list() so that the sink-only logic stays in sink.c (sink_resolve_logsrv_buffer() function), and the "generic" target part stays in log.c as resolve_logsrv(). Error messages formatting was preserved as far as possible but some slight variations are to be expected. As for the functional aspect, no change should be expected.
2023-08-17 11:38:30 -04:00
* up to the caller to check it and to free it.
*/
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
int resolve_logger(struct logger *logger, char **msg)
MEDIUM: log/sink: make logsrv postparsing more generic We previously had postparsing logic but only for logsrv sinks, but now we need to make this operation on logsrv directly instead of sinks to prepare for additional postparsing logic that is not sink-specific. To do this, we migrated post_sink_resolve() and sink_postresolve_logsrvs() to their postresolve_logsrvs() and postresolve_logsrv_list() equivalents. Then, we split postresolve_logsrv_list() so that the sink-only logic stays in sink.c (sink_resolve_logsrv_buffer() function), and the "generic" target part stays in log.c as resolve_logsrv(). Error messages formatting was preserved as far as possible but some slight variations are to be expected. As for the functional aspect, no change should be expected.
2023-08-17 11:38:30 -04:00
{
MEDIUM: log: introduce log target log targets were immediately embedded in logger struct (previously named logsrv) and could not be used outside of this context. In this patch, we're introducing log_target type with the associated helper functions so that it becomes possible to declare and use log targets outside of loggers scope.
2023-09-11 10:10:37 -04:00
struct log_target *target = &logger->target;
MEDIUM: log/sink: make logsrv postparsing more generic We previously had postparsing logic but only for logsrv sinks, but now we need to make this operation on logsrv directly instead of sinks to prepare for additional postparsing logic that is not sink-specific. To do this, we migrated post_sink_resolve() and sink_postresolve_logsrvs() to their postresolve_logsrvs() and postresolve_logsrv_list() equivalents. Then, we split postresolve_logsrv_list() so that the sink-only logic stays in sink.c (sink_resolve_logsrv_buffer() function), and the "generic" target part stays in log.c as resolve_logsrv(). Error messages formatting was preserved as far as possible but some slight variations are to be expected. As for the functional aspect, no change should be expected.
2023-08-17 11:38:30 -04:00
int err_code = ERR_NONE;
MEDIUM: log: introduce log target log targets were immediately embedded in logger struct (previously named logsrv) and could not be used outside of this context. In this patch, we're introducing log_target type with the associated helper functions so that it becomes possible to declare and use log targets outside of loggers scope.
2023-09-11 10:10:37 -04:00
if (target->type == LOG_TARGET_BUFFER)
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
err_code = sink_resolve_logger_buffer(logger, msg);
MAJOR: log: introduce log backends Using "mode log" in a backend section turns the proxy in a log backend which can be used to log-balance logs between multiple log targets (udp or tcp servers) log backends can be used as regular log targets using the log directive with "backend@be_name" prefix, like so: | log backend@mybackend local0 A log backend will distribute log messages to servers according to the log load-balancing algorithm that can be set using the "log-balance" option from the log backend section. For now, only the roundrobin algorithm is supported and set by default.
2023-09-13 05:52:31 -04:00
else if (target->type == LOG_TARGET_BACKEND) {
struct proxy *be;
MEDIUM: log: introduce log target log targets were immediately embedded in logger struct (previously named logsrv) and could not be used outside of this context. In this patch, we're introducing log_target type with the associated helper functions so that it becomes possible to declare and use log targets outside of loggers scope.
2023-09-11 10:10:37 -04:00
MAJOR: log: introduce log backends Using "mode log" in a backend section turns the proxy in a log backend which can be used to log-balance logs between multiple log targets (udp or tcp servers) log backends can be used as regular log targets using the log directive with "backend@be_name" prefix, like so: | log backend@mybackend local0 A log backend will distribute log messages to servers according to the log load-balancing algorithm that can be set using the "log-balance" option from the log backend section. For now, only the roundrobin algorithm is supported and set by default.
2023-09-13 05:52:31 -04:00
/* special case */
be = proxy_find_by_name(target->be_name, PR_CAP_BE, 0);
if (!be) {
memprintf(msg, "uses unknown log backend '%s'", target->be_name);
err_code |= ERR_ALERT | ERR_FATAL;
}
else if (be->mode != PR_MODE_SYSLOG) {
memprintf(msg, "uses incompatible log backend '%s'", target->be_name);
err_code |= ERR_ALERT | ERR_FATAL;
}
ha_free(&target->be_name); /* backend is resolved and will replace name hint */
target->be = be;
}
MEDIUM: log: introduce log target log targets were immediately embedded in logger struct (previously named logsrv) and could not be used outside of this context. In this patch, we're introducing log_target type with the associated helper functions so that it becomes possible to declare and use log targets outside of loggers scope.
2023-09-11 10:10:37 -04:00
target->flags |= LOG_TARGET_FL_RESOLVED;
MEDIUM: log/sink: make logsrv postparsing more generic We previously had postparsing logic but only for logsrv sinks, but now we need to make this operation on logsrv directly instead of sinks to prepare for additional postparsing logic that is not sink-specific. To do this, we migrated post_sink_resolve() and sink_postresolve_logsrvs() to their postresolve_logsrvs() and postresolve_logsrv_list() equivalents. Then, we split postresolve_logsrv_list() so that the sink-only logic stays in sink.c (sink_resolve_logsrv_buffer() function), and the "generic" target part stays in log.c as resolve_logsrv(). Error messages formatting was preserved as far as possible but some slight variations are to be expected. As for the functional aspect, no change should be expected.
2023-08-17 11:38:30 -04:00
return err_code;
}
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
/* tries to duplicate <def> logger
MINOR: log: add dup_logsrv() helper function ease code maintenance by introducing dup_logsrv() helper function to properly duplicate an existing logsrv struct.
2023-07-05 09:52:19 -04:00
*
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
* Returns the newly allocated and duplicated logger or NULL
MINOR: log: add dup_logsrv() helper function ease code maintenance by introducing dup_logsrv() helper function to properly duplicate an existing logsrv struct.
2023-07-05 09:52:19 -04:00
* in case of error.
*/
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
struct logger *dup_logger(struct logger *def)
MINOR: log: add dup_logsrv() helper function ease code maintenance by introducing dup_logsrv() helper function to properly duplicate an existing logsrv struct.
2023-07-05 09:52:19 -04:00
{
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
struct logger *cpy = malloc(sizeof(*cpy));
MINOR: log: add dup_logsrv() helper function ease code maintenance by introducing dup_logsrv() helper function to properly duplicate an existing logsrv struct.
2023-07-05 09:52:19 -04:00
/* copy everything that can be easily copied */
memcpy(cpy, def, sizeof(*cpy));
/* default values */
cpy->conf.file = NULL;
BUG/MINOR: log: smp_rgs array issues with inherited global log directives When a log directive is defined in the global section, each time we use "log global" in a proxy section, the global log directives are duplicated for the current proxy. This works by creating a new proxy logger struct and duplicating every members for each global one. However, smp_rgs logger member is a special pointer member that is allocated when "range" is used on a log directive. Currently, we simply copy the array pointer (from the global one), instead of creating our own copy. Because of that, range log sampling may not work properly in some situations prior to 3f1284560 ("MINOR: log: remove the unused curr_idx in struct smp_log_range") when used in global log directives, for instance: global log 127.0.0.1:5114 format raw sample 1-2,3:4 local0 info # should receive 75% of all proxy logs log 127.0.0.1:5115 format raw sample 4:4 local0 info # should receive 25% of all proxy logs listen proxy1 log global listen proxy2 log global May not work as expected, because curr_idx was stored within smp_rgs array member prior to 3f1284560, and due to this bug, it happens to be shared between every log directive inherited from a "global" one. The result is that curr_idx counter will not behave properly because the index will be increased globally instead of per-log directive, and it could even suffer from concurrent thread accesses under load since we don't own the global log directive's lock when manipulating it. Another issue that was revealed because of this bug is that the smp_rgs array allocated during config parsing is never freed in free_logger(), resulting in small memory leak during clean exit. To fix these issues all at once, let's properly duplicate smp_rgs logger struct member in dup_logger() like we already do for other special members so that every log directive have its own sms_rgs copy, and then systematically free it in free_logger(). While this bug affects all stable versions (including 2.4), it's probably best to not backport this beyond 2.6 because of 211ea252d ("BUG/MINOR: logs: fix logsrv leaks on clean exit") prerequisite that first appears in 2.6. [ada: for versions prior to 2.9, 969e212 ("MINOR: log: add dup_logsrv() helper function") and 76acde91 ("BUG/MINOR: log: keep the ref in dup_logger()") must be backported first. Note: Some ctx adjustments should be performed because 'logger' struct used to be named 'logsrv' in the past and 2.9 introduced logger target struct member. Thus it's probably easier to manually apply 76acde91 and the current bugfix by hand directly on top of 969e212. ]
2024-05-14 04:22:19 -04:00
cpy->lb.smp_rgs = NULL;
MINOR: log: add dup_logsrv() helper function ease code maintenance by introducing dup_logsrv() helper function to properly duplicate an existing logsrv struct.
2023-07-05 09:52:19 -04:00
LIST_INIT(&cpy->list);
/* special members */
MEDIUM: log: introduce log target log targets were immediately embedded in logger struct (previously named logsrv) and could not be used outside of this context. In this patch, we're introducing log_target type with the associated helper functions so that it becomes possible to declare and use log targets outside of loggers scope.
2023-09-11 10:10:37 -04:00
if (dup_log_target(&def->target, &cpy->target) == 0)
goto error;
MINOR: log: add dup_logsrv() helper function ease code maintenance by introducing dup_logsrv() helper function to properly duplicate an existing logsrv struct.
2023-07-05 09:52:19 -04:00
if (def->conf.file) {
cpy->conf.file = strdup(def->conf.file);
if (!cpy->conf.file)
goto error;
}
BUG/MINOR: log: smp_rgs array issues with inherited global log directives When a log directive is defined in the global section, each time we use "log global" in a proxy section, the global log directives are duplicated for the current proxy. This works by creating a new proxy logger struct and duplicating every members for each global one. However, smp_rgs logger member is a special pointer member that is allocated when "range" is used on a log directive. Currently, we simply copy the array pointer (from the global one), instead of creating our own copy. Because of that, range log sampling may not work properly in some situations prior to 3f1284560 ("MINOR: log: remove the unused curr_idx in struct smp_log_range") when used in global log directives, for instance: global log 127.0.0.1:5114 format raw sample 1-2,3:4 local0 info # should receive 75% of all proxy logs log 127.0.0.1:5115 format raw sample 4:4 local0 info # should receive 25% of all proxy logs listen proxy1 log global listen proxy2 log global May not work as expected, because curr_idx was stored within smp_rgs array member prior to 3f1284560, and due to this bug, it happens to be shared between every log directive inherited from a "global" one. The result is that curr_idx counter will not behave properly because the index will be increased globally instead of per-log directive, and it could even suffer from concurrent thread accesses under load since we don't own the global log directive's lock when manipulating it. Another issue that was revealed because of this bug is that the smp_rgs array allocated during config parsing is never freed in free_logger(), resulting in small memory leak during clean exit. To fix these issues all at once, let's properly duplicate smp_rgs logger struct member in dup_logger() like we already do for other special members so that every log directive have its own sms_rgs copy, and then systematically free it in free_logger(). While this bug affects all stable versions (including 2.4), it's probably best to not backport this beyond 2.6 because of 211ea252d ("BUG/MINOR: logs: fix logsrv leaks on clean exit") prerequisite that first appears in 2.6. [ada: for versions prior to 2.9, 969e212 ("MINOR: log: add dup_logsrv() helper function") and 76acde91 ("BUG/MINOR: log: keep the ref in dup_logger()") must be backported first. Note: Some ctx adjustments should be performed because 'logger' struct used to be named 'logsrv' in the past and 2.9 introduced logger target struct member. Thus it's probably easier to manually apply 76acde91 and the current bugfix by hand directly on top of 969e212. ]
2024-05-14 04:22:19 -04:00
if (def->lb.smp_rgs) {
cpy->lb.smp_rgs = malloc(sizeof(*cpy->lb.smp_rgs) * def->lb.smp_rgs_sz);
if (!cpy->lb.smp_rgs)
goto error;
memcpy(cpy->lb.smp_rgs, def->lb.smp_rgs,
sizeof(*cpy->lb.smp_rgs) * def->lb.smp_rgs_sz);
}
BUG/MINOR: log: keep the ref in dup_logger() This bug was introduced with 969e212 ("MINOR: log: add dup_logsrv() helper function") When duplicating an existing log entry, we must take care to inherit from its original ->ref if it is set, because not doing so would make 28ac0999 ("MINOR: log: Keep the ref when a log server is copied to avoid duplicate entries") ineffective given that global log directives will lose their original reference when duplicated resursively (at least twice), which is what happens when global log directives are first inherited to defaults which are then inherited to a regular proxy at the end of the chain. This can be easily reproduced using the following configuration: |global | log stdout format raw local0 | |defaults | log global | |frontend test | log global | ... Logs from "test" proxy will be duplicated because test incorrectly inherited from global "log" directives twice, which 28ac0999 would normally detect and prevent. No backport needed unless 969e212 gets backported.
2023-11-13 04:19:12 -05:00
/* inherit from original reference if set */
cpy->ref = (def->ref) ? def->ref : def;
MINOR: log: add dup_logsrv() helper function ease code maintenance by introducing dup_logsrv() helper function to properly duplicate an existing logsrv struct.
2023-07-05 09:52:19 -04:00
return cpy;
error:
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
free_logger(cpy);
MINOR: log: add dup_logsrv() helper function ease code maintenance by introducing dup_logsrv() helper function to properly duplicate an existing logsrv struct.
2023-07-05 09:52:19 -04:00
return NULL;
}
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
/* frees <logger> after freeing all of its allocated fields. The
BUG/MINOR: logs: fix logsrv leaks on clean exit Log servers are a real mess because: - entries are duplicated using memcpy() without their strings being reallocated, which results in these ones not being freeable every time. - a new field, ring_name, was added in 2.2 by commit 99c453df9 ("MEDIUM: ring: new section ring to declare custom ring buffers.") but it's never initialized during copies, causing the same issue - no attempt is made at freeing all that. Of course, running "haproxy -c" under ASAN quickly notices that and dumps a core. This patch adds the missing strdup() and initialization where required, adds a new free_logsrv() function to cleanly free() such a structure, calls it from the proxy when iterating over logsrvs instead of silently leaking their file names and ring names, and adds the same logsrv loop to the proxy_free_defaults() function so that we don't leak defaults sections on exit. It looks a bit entangled, but it comes as a whole because all this stuff is inter-dependent and was missing. It's probably preferable not to backport this in the foreseable future as it may reveal other jokes if some obscure parts continue to memcpy() the logsrv struct.
2022-03-17 14:47:33 -04:00
* server must not belong to a list anymore. Logsrv may be NULL, which is
* silently ignored.
*/
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
void free_logger(struct logger *logger)
BUG/MINOR: logs: fix logsrv leaks on clean exit Log servers are a real mess because: - entries are duplicated using memcpy() without their strings being reallocated, which results in these ones not being freeable every time. - a new field, ring_name, was added in 2.2 by commit 99c453df9 ("MEDIUM: ring: new section ring to declare custom ring buffers.") but it's never initialized during copies, causing the same issue - no attempt is made at freeing all that. Of course, running "haproxy -c" under ASAN quickly notices that and dumps a core. This patch adds the missing strdup() and initialization where required, adds a new free_logsrv() function to cleanly free() such a structure, calls it from the proxy when iterating over logsrvs instead of silently leaking their file names and ring names, and adds the same logsrv loop to the proxy_free_defaults() function so that we don't leak defaults sections on exit. It looks a bit entangled, but it comes as a whole because all this stuff is inter-dependent and was missing. It's probably preferable not to backport this in the foreseable future as it may reveal other jokes if some obscure parts continue to memcpy() the logsrv struct.
2022-03-17 14:47:33 -04:00
{
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
if (!logger)
BUG/MINOR: logs: fix logsrv leaks on clean exit Log servers are a real mess because: - entries are duplicated using memcpy() without their strings being reallocated, which results in these ones not being freeable every time. - a new field, ring_name, was added in 2.2 by commit 99c453df9 ("MEDIUM: ring: new section ring to declare custom ring buffers.") but it's never initialized during copies, causing the same issue - no attempt is made at freeing all that. Of course, running "haproxy -c" under ASAN quickly notices that and dumps a core. This patch adds the missing strdup() and initialization where required, adds a new free_logsrv() function to cleanly free() such a structure, calls it from the proxy when iterating over logsrvs instead of silently leaking their file names and ring names, and adds the same logsrv loop to the proxy_free_defaults() function so that we don't leak defaults sections on exit. It looks a bit entangled, but it comes as a whole because all this stuff is inter-dependent and was missing. It's probably preferable not to backport this in the foreseable future as it may reveal other jokes if some obscure parts continue to memcpy() the logsrv struct.
2022-03-17 14:47:33 -04:00
return;
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
BUG_ON(LIST_INLIST(&logger->list));
ha_free(&logger->conf.file);
MEDIUM: log: introduce log target log targets were immediately embedded in logger struct (previously named logsrv) and could not be used outside of this context. In this patch, we're introducing log_target type with the associated helper functions so that it becomes possible to declare and use log targets outside of loggers scope.
2023-09-11 10:10:37 -04:00
deinit_log_target(&logger->target);
BUG/MINOR: log: smp_rgs array issues with inherited global log directives When a log directive is defined in the global section, each time we use "log global" in a proxy section, the global log directives are duplicated for the current proxy. This works by creating a new proxy logger struct and duplicating every members for each global one. However, smp_rgs logger member is a special pointer member that is allocated when "range" is used on a log directive. Currently, we simply copy the array pointer (from the global one), instead of creating our own copy. Because of that, range log sampling may not work properly in some situations prior to 3f1284560 ("MINOR: log: remove the unused curr_idx in struct smp_log_range") when used in global log directives, for instance: global log 127.0.0.1:5114 format raw sample 1-2,3:4 local0 info # should receive 75% of all proxy logs log 127.0.0.1:5115 format raw sample 4:4 local0 info # should receive 25% of all proxy logs listen proxy1 log global listen proxy2 log global May not work as expected, because curr_idx was stored within smp_rgs array member prior to 3f1284560, and due to this bug, it happens to be shared between every log directive inherited from a "global" one. The result is that curr_idx counter will not behave properly because the index will be increased globally instead of per-log directive, and it could even suffer from concurrent thread accesses under load since we don't own the global log directive's lock when manipulating it. Another issue that was revealed because of this bug is that the smp_rgs array allocated during config parsing is never freed in free_logger(), resulting in small memory leak during clean exit. To fix these issues all at once, let's properly duplicate smp_rgs logger struct member in dup_logger() like we already do for other special members so that every log directive have its own sms_rgs copy, and then systematically free it in free_logger(). While this bug affects all stable versions (including 2.4), it's probably best to not backport this beyond 2.6 because of 211ea252d ("BUG/MINOR: logs: fix logsrv leaks on clean exit") prerequisite that first appears in 2.6. [ada: for versions prior to 2.9, 969e212 ("MINOR: log: add dup_logsrv() helper function") and 76acde91 ("BUG/MINOR: log: keep the ref in dup_logger()") must be backported first. Note: Some ctx adjustments should be performed because 'logger' struct used to be named 'logsrv' in the past and 2.9 introduced logger target struct member. Thus it's probably easier to manually apply 76acde91 and the current bugfix by hand directly on top of 969e212. ]
2024-05-14 04:22:19 -04:00
free(logger->lb.smp_rgs);
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
free(logger);
BUG/MINOR: logs: fix logsrv leaks on clean exit Log servers are a real mess because: - entries are duplicated using memcpy() without their strings being reallocated, which results in these ones not being freeable every time. - a new field, ring_name, was added in 2.2 by commit 99c453df9 ("MEDIUM: ring: new section ring to declare custom ring buffers.") but it's never initialized during copies, causing the same issue - no attempt is made at freeing all that. Of course, running "haproxy -c" under ASAN quickly notices that and dumps a core. This patch adds the missing strdup() and initialization where required, adds a new free_logsrv() function to cleanly free() such a structure, calls it from the proxy when iterating over logsrvs instead of silently leaking their file names and ring names, and adds the same logsrv loop to the proxy_free_defaults() function so that we don't leak defaults sections on exit. It looks a bit entangled, but it comes as a whole because all this stuff is inter-dependent and was missing. It's probably preferable not to backport this in the foreseable future as it may reveal other jokes if some obscure parts continue to memcpy() the logsrv struct.
2022-03-17 14:47:33 -04:00
}
MEDIUM: log: introduce log target log targets were immediately embedded in logger struct (previously named logsrv) and could not be used outside of this context. In this patch, we're introducing log_target type with the associated helper functions so that it becomes possible to declare and use log targets outside of loggers scope.
2023-09-11 10:10:37 -04:00
/* Parse single log target
* Returns 0 on failure and positive value on success
*/
static int parse_log_target(char *raw, struct log_target *target, char **err)
{
int port1, port2, fd;
struct protocol *proto;
struct sockaddr_storage *sk;
init_log_target(target);
MEDIUM: sink/log: stop relying on AF_UNSPEC for rings Since a5b325f92 ("MINOR: protocol: add a real family for existing FDs"), we don't rely anymore on AF_UNSPEC for buffer rings in do_send_log. But we kept it as a parsing hint to differentiate between implicit and named rings during ring buffer postparsing. However it is still a bit confusing and forces us to systematically rely on target->addr, even for named buffer rings where it doesn't make much sense anymore. Now that target->addr was made a pointer in a recent commit, we can choose not to initialize it when not needed (i.e.: named rings) and use this as a hint to distinguish implicit rings during init since they rely on the addr struct to temporarily store the ring's address until the ring is actually created during postparsing step.
2023-09-13 13:28:34 -04:00
// target addr is NULL at this point
if (strncmp(raw, "ring@", 5) == 0) {
target->type = LOG_TARGET_BUFFER;
target->ring_name = strdup(raw + 5);
goto done;
}
MAJOR: log: introduce log backends Using "mode log" in a backend section turns the proxy in a log backend which can be used to log-balance logs between multiple log targets (udp or tcp servers) log backends can be used as regular log targets using the log directive with "backend@be_name" prefix, like so: | log backend@mybackend local0 A log backend will distribute log messages to servers according to the log load-balancing algorithm that can be set using the "log-balance" option from the log backend section. For now, only the roundrobin algorithm is supported and set by default.
2023-09-13 05:52:31 -04:00
else if (strncmp(raw, "backend@", 8) == 0) {
target->type = LOG_TARGET_BACKEND;
target->be_name = strdup(raw + 8);
goto done;
}
MEDIUM: log: introduce log target log targets were immediately embedded in logger struct (previously named logsrv) and could not be used outside of this context. In this patch, we're introducing log_target type with the associated helper functions so that it becomes possible to declare and use log targets outside of loggers scope.
2023-09-11 10:10:37 -04:00
MEDIUM: sink/log: stop relying on AF_UNSPEC for rings Since a5b325f92 ("MINOR: protocol: add a real family for existing FDs"), we don't rely anymore on AF_UNSPEC for buffer rings in do_send_log. But we kept it as a parsing hint to differentiate between implicit and named rings during ring buffer postparsing. However it is still a bit confusing and forces us to systematically rely on target->addr, even for named buffer rings where it doesn't make much sense anymore. Now that target->addr was made a pointer in a recent commit, we can choose not to initialize it when not needed (i.e.: named rings) and use this as a hint to distinguish implicit rings during init since they rely on the addr struct to temporarily store the ring's address until the ring is actually created during postparsing step.
2023-09-13 13:28:34 -04:00
/* try to allocate log target addr */
MEDIUM: log: introduce log target log targets were immediately embedded in logger struct (previously named logsrv) and could not be used outside of this context. In this patch, we're introducing log_target type with the associated helper functions so that it becomes possible to declare and use log targets outside of loggers scope.
2023-09-11 10:10:37 -04:00
target->addr = malloc(sizeof(*target->addr));
if (!target->addr) {
memprintf(err, "memory error");
goto error;
}
MEDIUM: sink/log: stop relying on AF_UNSPEC for rings Since a5b325f92 ("MINOR: protocol: add a real family for existing FDs"), we don't rely anymore on AF_UNSPEC for buffer rings in do_send_log. But we kept it as a parsing hint to differentiate between implicit and named rings during ring buffer postparsing. However it is still a bit confusing and forces us to systematically rely on target->addr, even for named buffer rings where it doesn't make much sense anymore. Now that target->addr was made a pointer in a recent commit, we can choose not to initialize it when not needed (i.e.: named rings) and use this as a hint to distinguish implicit rings during init since they rely on the addr struct to temporarily store the ring's address until the ring is actually created during postparsing step.
2023-09-13 13:28:34 -04:00
target->type = LOG_TARGET_DGRAM; // default type
MEDIUM: log: introduce log target log targets were immediately embedded in logger struct (previously named logsrv) and could not be used outside of this context. In this patch, we're introducing log_target type with the associated helper functions so that it becomes possible to declare and use log targets outside of loggers scope.
2023-09-11 10:10:37 -04:00
/* parse the target address */
MINOR: tools: make str2sa_range() directly return type hints str2sa_range() already allows the caller to provide <proto> in order to get a pointer on the protocol matching with the string input thanks to 5fc9328a ("MINOR: tools: make str2sa_range() directly return the protocol") However, as stated into the commit message, there is a trick: "we can fail to return a protocol in case the caller accepts an fqdn for use later. This is what servers do and in this case it is valid to return no protocol" In this case, we're unable to return protocol because the protocol lookup depends on both the [proto type + xprt type] and the [family type] to be known. While family type might not be directly resolved when fqdn is involved (because family type might be discovered using DNS queries), proto type and xprt type are already known. As such, the caller might be interested in knowing those address related hints even if the address family type is not yet resolved and thus the matching protocol cannot be looked up. Thus in this patch we add the optional net_addr_type (custom type) argument to str2sa_range to enable the caller to check the protocol type and transport type when the function succeeds.
2023-11-09 05:19:24 -05:00
sk = str2sa_range(raw, NULL, &port1, &port2, &fd, &proto, NULL,
MEDIUM: log: introduce log target log targets were immediately embedded in logger struct (previously named logsrv) and could not be used outside of this context. In this patch, we're introducing log_target type with the associated helper functions so that it becomes possible to declare and use log targets outside of loggers scope.
2023-09-11 10:10:37 -04:00
err, NULL, NULL,
PA_O_RESOLVE | PA_O_PORT_OK | PA_O_RAW_FD | PA_O_DGRAM | PA_O_STREAM | PA_O_DEFAULT_DGRAM);
if (!sk)
goto error;
if (fd != -1)
target->type = LOG_TARGET_FD;
*target->addr = *sk;
if (sk->ss_family == AF_INET || sk->ss_family == AF_INET6) {
if (!port1)
set_host_port(target->addr, SYSLOG_PORT);
}
if (proto && proto->xprt_type == PROTO_TYPE_STREAM) {
static unsigned long ring_ids;
MEDIUM: sink/log: stop relying on AF_UNSPEC for rings Since a5b325f92 ("MINOR: protocol: add a real family for existing FDs"), we don't rely anymore on AF_UNSPEC for buffer rings in do_send_log. But we kept it as a parsing hint to differentiate between implicit and named rings during ring buffer postparsing. However it is still a bit confusing and forces us to systematically rely on target->addr, even for named buffer rings where it doesn't make much sense anymore. Now that target->addr was made a pointer in a recent commit, we can choose not to initialize it when not needed (i.e.: named rings) and use this as a hint to distinguish implicit rings during init since they rely on the addr struct to temporarily store the ring's address until the ring is actually created during postparsing step.
2023-09-13 13:28:34 -04:00
/* Implicit sink buffer will be initialized in post_check
* (target->addr is set in this case)
MEDIUM: log: introduce log target log targets were immediately embedded in logger struct (previously named logsrv) and could not be used outside of this context. In this patch, we're introducing log_target type with the associated helper functions so that it becomes possible to declare and use log targets outside of loggers scope.
2023-09-11 10:10:37 -04:00
*/
target->type = LOG_TARGET_BUFFER;
/* compute unique name for the ring */
memprintf(&target->ring_name, "ring#%lu", ++ring_ids);
}
done:
return 1;
error:
deinit_log_target(target);
return 0;
}
MINOR: log: move 'log' keyword parsing in dedicated function Now, the function parse_logsrv should be used to parse a "log" line. This function will update the list of loggers passed in argument. It can release all log servers when "no log" line was parsed (by the caller) or it can parse "log global" or "log <address> ... " lines. It takes care of checking the caller context (global or not) to prohibit "log global" usage in the global section.
2018-03-26 09:54:32 -04:00
/*
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
* Parse "log" keyword and update <loggers> list accordingly.
MINOR: log: move 'log' keyword parsing in dedicated function Now, the function parse_logsrv should be used to parse a "log" line. This function will update the list of loggers passed in argument. It can release all log servers when "no log" line was parsed (by the caller) or it can parse "log global" or "log <address> ... " lines. It takes care of checking the caller context (global or not) to prohibit "log global" usage in the global section.
2018-03-26 09:54:32 -04:00
*
* When <do_del> is set, it means the "no log" line was parsed, so all log
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
* servers in <loggers> are released.
MINOR: log: move 'log' keyword parsing in dedicated function Now, the function parse_logsrv should be used to parse a "log" line. This function will update the list of loggers passed in argument. It can release all log servers when "no log" line was parsed (by the caller) or it can parse "log global" or "log <address> ... " lines. It takes care of checking the caller context (global or not) to prohibit "log global" usage in the global section.
2018-03-26 09:54:32 -04:00
*
* Otherwise, we try to parse the "log" line. First of all, when the list is not
* the global one, we look for the parameter "global". If we find it,
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
* global.loggers is copied. Else we parse each arguments.
MINOR: log: move 'log' keyword parsing in dedicated function Now, the function parse_logsrv should be used to parse a "log" line. This function will update the list of loggers passed in argument. It can release all log servers when "no log" line was parsed (by the caller) or it can parse "log global" or "log <address> ... " lines. It takes care of checking the caller context (global or not) to prohibit "log global" usage in the global section.
2018-03-26 09:54:32 -04:00
*
* The function returns 1 in success case, otherwise, it returns 0 and err is
* filled.
*/
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
int parse_logger(char **args, struct list *loggers, int do_del, const char *file, int linenum, char **err)
MINOR: log: move 'log' keyword parsing in dedicated function Now, the function parse_logsrv should be used to parse a "log" line. This function will update the list of loggers passed in argument. It can release all log servers when "no log" line was parsed (by the caller) or it can parse "log global" or "log <address> ... " lines. It takes care of checking the caller context (global or not) to prohibit "log global" usage in the global section.
2018-03-26 09:54:32 -04:00
{
BUG/MINOR: log: fix memory leak on logsrv parse error In case of parsing error on logsrv, we can leave parse_logsrv() without releasing logsrv->ring_name or smp_rgs. Let's free them on the error path. This should fix issue #926 detected by Coverity. The impact is only a tiny leak just before reporting a fatal error, so it will essentially annoy valgrind. This can be backported to 2.0 (just drop the ring part).
2020-10-27 04:51:37 -04:00
struct smp_log_range *smp_rgs = NULL;
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
struct logger *logger = NULL;
MINOR: log: move 'log' keyword parsing in dedicated function Now, the function parse_logsrv should be used to parse a "log" line. This function will update the list of loggers passed in argument. It can release all log servers when "no log" line was parsed (by the caller) or it can parse "log global" or "log <address> ... " lines. It takes care of checking the caller context (global or not) to prohibit "log global" usage in the global section.
2018-03-26 09:54:32 -04:00
int cur_arg;
/*
* "no log": delete previous herited or defined syslog
* servers.
*/
if (do_del) {
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
struct logger *back;
MINOR: log: move 'log' keyword parsing in dedicated function Now, the function parse_logsrv should be used to parse a "log" line. This function will update the list of loggers passed in argument. It can release all log servers when "no log" line was parsed (by the caller) or it can parse "log global" or "log <address> ... " lines. It takes care of checking the caller context (global or not) to prohibit "log global" usage in the global section.
2018-03-26 09:54:32 -04:00
if (*(args[1]) != 0) {
memprintf(err, "'no log' does not expect arguments");
goto error;
}
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
list_for_each_entry_safe(logger, back, loggers, list) {
LIST_DEL_INIT(&logger->list);
free_logger(logger);
MINOR: log: move 'log' keyword parsing in dedicated function Now, the function parse_logsrv should be used to parse a "log" line. This function will update the list of loggers passed in argument. It can release all log servers when "no log" line was parsed (by the caller) or it can parse "log global" or "log <address> ... " lines. It takes care of checking the caller context (global or not) to prohibit "log global" usage in the global section.
2018-03-26 09:54:32 -04:00
}
return 1;
}
/*
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
* "log global": copy global.loggers linked list to the end of loggers
* list. But first, we check (loggers != global.loggers).
MINOR: log: move 'log' keyword parsing in dedicated function Now, the function parse_logsrv should be used to parse a "log" line. This function will update the list of loggers passed in argument. It can release all log servers when "no log" line was parsed (by the caller) or it can parse "log global" or "log <address> ... " lines. It takes care of checking the caller context (global or not) to prohibit "log global" usage in the global section.
2018-03-26 09:54:32 -04:00
*/
CLEANUP: Compare the return value of `XXXcmp()` functions with zero According to coding-style.txt it is recommended to use: `strcmp(a, b) == 0` instead of `!strcmp(a, b)` So let's do this. The change was performed by running the following (very long) coccinelle patch on src/: @@ statement S; expression E; expression F; @@ if ( ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) != 0 ) ( S | { ... } ) @@ statement S; expression E; expression F; @@ if ( - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 ) ( S | { ... } ) @@ expression E; expression F; expression G; @@ ( G && ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) != 0 ) @@ expression E; expression F; expression G; @@ ( G || ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) != 0 ) @@ expression E; expression F; expression G; @@ ( ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) != 0 && G ) @@ expression E; expression F; expression G; @@ ( ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) != 0 || G ) @@ expression E; expression F; expression G; @@ ( G && - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 ) @@ expression E; expression F; expression G; @@ ( G || - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 ) @@ expression E; expression F; expression G; @@ ( - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 && G ) @@ expression E; expression F; expression G; @@ ( - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 || G ) @@ expression E; expression F; expression G; @@ ( - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 )
2021-01-02 16:31:53 -05:00
if (*(args[1]) && *(args[2]) == 0 && strcmp(args[1], "global") == 0) {
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
if (loggers == &global.loggers) {
MINOR: log: move 'log' keyword parsing in dedicated function Now, the function parse_logsrv should be used to parse a "log" line. This function will update the list of loggers passed in argument. It can release all log servers when "no log" line was parsed (by the caller) or it can parse "log global" or "log <address> ... " lines. It takes care of checking the caller context (global or not) to prohibit "log global" usage in the global section.
2018-03-26 09:54:32 -04:00
memprintf(err, "'global' is not supported for a global syslog server");
goto error;
}
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
list_for_each_entry(logger, &global.loggers, list) {
struct logger *node;
MINOR: log: Keep the ref when a log server is copied to avoid duplicate entries With "log global" line, the global list of loggers are copied into the proxy's struct. The list coming from the default section is also copied when a frontend or a backend section is parsed. So it is possible to have duplicate entries in the proxy's list. For instance, with this following config, all messages will be logged twice: global log 127.0.0.1 local0 debug daemon defaults mode http log global option httplog frontend front-http log global bind *:8888 default_backend back-http backend back-http server www 127.0.0.1:8000
2018-03-26 10:09:19 -04:00
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
list_for_each_entry(node, loggers, list) {
if (node->ref == logger)
goto skip_logger;
MINOR: log: Keep the ref when a log server is copied to avoid duplicate entries With "log global" line, the global list of loggers are copied into the proxy's struct. The list coming from the default section is also copied when a frontend or a backend section is parsed. So it is possible to have duplicate entries in the proxy's list. For instance, with this following config, all messages will be logged twice: global log 127.0.0.1 local0 debug daemon defaults mode http log global option httplog frontend front-http log global bind *:8888 default_backend back-http backend back-http server www 127.0.0.1:8000
2018-03-26 10:09:19 -04:00
}
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
/* duplicate logger from global */
node = dup_logger(logger);
BUG/MINOR: log: fix memory error handling in parse_logsrv() A check was missing in parse_logsrv() to make sure that malloc-dependent variable is checked for non-NULL before using it. If malloc fails, the function raises an error and stops, like it's already done at a few other places within the function. This partially fixes GH #2130. It should be backported to every stable versions.
2023-05-11 12:57:23 -04:00
if (!node) {
memprintf(err, "out of memory error");
goto error;
}
MINOR: log: add dup_logsrv() helper function ease code maintenance by introducing dup_logsrv() helper function to properly duplicate an existing logsrv struct.
2023-07-05 09:52:19 -04:00
/* manually override some values */
ha_free(&node->conf.file);
MINOR: log: register config file and line number on log servers. This patch registers the parsed file and the line where a log server is declared to make those information available in configuration post check. Those new informations were added on error messages probed resolving ring names on post configuration check.
2021-04-02 04:13:43 -04:00
node->conf.file = strdup(file);
node->conf.line = linenum;
MINOR: log: Keep the ref when a log server is copied to avoid duplicate entries With "log global" line, the global list of loggers are copied into the proxy's struct. The list coming from the default section is also copied when a frontend or a backend section is parsed. So it is possible to have duplicate entries in the proxy's list. For instance, with this following config, all messages will be logged twice: global log 127.0.0.1 local0 debug daemon defaults mode http log global option httplog frontend front-http log global bind *:8888 default_backend back-http backend back-http server www 127.0.0.1:8000
2018-03-26 10:09:19 -04:00
MINOR: log: add dup_logsrv() helper function ease code maintenance by introducing dup_logsrv() helper function to properly duplicate an existing logsrv struct.
2023-07-05 09:52:19 -04:00
/* add to list */
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
LIST_APPEND(loggers, &node->list);
MINOR: log: add dup_logsrv() helper function ease code maintenance by introducing dup_logsrv() helper function to properly duplicate an existing logsrv struct.
2023-07-05 09:52:19 -04:00
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
skip_logger:
MINOR: log: Keep the ref when a log server is copied to avoid duplicate entries With "log global" line, the global list of loggers are copied into the proxy's struct. The list coming from the default section is also copied when a frontend or a backend section is parsed. So it is possible to have duplicate entries in the proxy's list. For instance, with this following config, all messages will be logged twice: global log 127.0.0.1 local0 debug daemon defaults mode http log global option httplog frontend front-http log global bind *:8888 default_backend back-http backend back-http server www 127.0.0.1:8000
2018-03-26 10:09:19 -04:00
continue;
MINOR: log: move 'log' keyword parsing in dedicated function Now, the function parse_logsrv should be used to parse a "log" line. This function will update the list of loggers passed in argument. It can release all log servers when "no log" line was parsed (by the caller) or it can parse "log global" or "log <address> ... " lines. It takes care of checking the caller context (global or not) to prohibit "log global" usage in the global section.
2018-03-26 09:54:32 -04:00
}
return 1;
}
/*
* "log <address> ...: parse a syslog server line
*/
if (*(args[1]) == 0 || *(args[2]) == 0) {
memprintf(err, "expects <address> and <facility> %s as arguments",
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
((loggers == &global.loggers) ? "" : "or global"));
MINOR: log: move 'log' keyword parsing in dedicated function Now, the function parse_logsrv should be used to parse a "log" line. This function will update the list of loggers passed in argument. It can release all log servers when "no log" line was parsed (by the caller) or it can parse "log global" or "log <address> ... " lines. It takes care of checking the caller context (global or not) to prohibit "log global" usage in the global section.
2018-03-26 09:54:32 -04:00
goto error;
}
MEDIUM: log: add support for logging to existing file descriptors In certain situations it would be desirable to log to an existing file descriptor, the most common case being a pipe between containers or processes. The main issue with pipes is that using write() on them will randomly truncate messages. But there is a trick. By using writev(), we can atomically deliver or drop a message, which perfectly fits the purpose. The only caveat is that large messages (4096 bytes on modern operating systems) may be interleaved with messages from other processes if using nbproc for example. In practice such messages are rare and most of the time when users need such type of logging, the load is low enough for a single process to be running so this is not really a problem. This logging method thus uses unbuffered writev() calls and is uses more CPU than if it used its own buffer with large writes at once, though this is not a problem for moderate loads. Logging to a file descriptor attached to a file also works with the side effect that the process is significantly slowed down during disk accesses and that it's not possible to rotate the file without restarting the process. For this reason this option is not offered as a configuration option, since it would confuse most users, but one could decide to redirect haproxy's output to a file during debugging sessions. Two aliases "stdout" and "stderr" are provided, but keep in mind that these are closed by default in daemon mode. When logging to a pipe or socket at a high enough rate, some logs will be dropped and the number of dropped messages is reported in "show info".
2018-11-12 01:34:59 -05:00
/* take care of "stdout" and "stderr" as regular aliases for fd@1 / fd@2 */
if (strcmp(args[1], "stdout") == 0)
args[1] = "fd@1";
else if (strcmp(args[1], "stderr") == 0)
args[1] = "fd@2";
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
logger = calloc(1, sizeof(*logger));
if (!logger) {
MINOR: log: move 'log' keyword parsing in dedicated function Now, the function parse_logsrv should be used to parse a "log" line. This function will update the list of loggers passed in argument. It can release all log servers when "no log" line was parsed (by the caller) or it can parse "log global" or "log <address> ... " lines. It takes care of checking the caller context (global or not) to prohibit "log global" usage in the global section.
2018-03-26 09:54:32 -04:00
memprintf(err, "out of memory");
goto error;
}
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
LIST_INIT(&logger->list);
logger->conf.file = strdup(file);
logger->conf.line = linenum;
MINOR: log: register config file and line number on log servers. This patch registers the parsed file and the line where a log server is declared to make those information available in configuration post check. Those new informations were added on error messages probed resolving ring names on post configuration check.
2021-04-02 04:13:43 -04:00
MINOR: log: move 'log' keyword parsing in dedicated function Now, the function parse_logsrv should be used to parse a "log" line. This function will update the list of loggers passed in argument. It can release all log servers when "no log" line was parsed (by the caller) or it can parse "log global" or "log <address> ... " lines. It takes care of checking the caller context (global or not) to prohibit "log global" usage in the global section.
2018-03-26 09:54:32 -04:00
/* skip address for now, it will be parsed at the end */
cur_arg = 2;
/* just after the address, a length may be specified */
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
logger->maxlen = MAX_SYSLOG_LEN;
MINOR: log: move 'log' keyword parsing in dedicated function Now, the function parse_logsrv should be used to parse a "log" line. This function will update the list of loggers passed in argument. It can release all log servers when "no log" line was parsed (by the caller) or it can parse "log global" or "log <address> ... " lines. It takes care of checking the caller context (global or not) to prohibit "log global" usage in the global section.
2018-03-26 09:54:32 -04:00
if (strcmp(args[cur_arg], "len") == 0) {
int len = atoi(args[cur_arg+1]);
if (len < 80 || len > 65535) {
memprintf(err, "invalid log length '%s', must be between 80 and 65535",
args[cur_arg+1]);
goto error;
}
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
logger->maxlen = len;
MINOR: log: move 'log' keyword parsing in dedicated function Now, the function parse_logsrv should be used to parse a "log" line. This function will update the list of loggers passed in argument. It can release all log servers when "no log" line was parsed (by the caller) or it can parse "log global" or "log <address> ... " lines. It takes care of checking the caller context (global or not) to prohibit "log global" usage in the global section.
2018-03-26 09:54:32 -04:00
cur_arg += 2;
}
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
if (logger->maxlen > global.max_syslog_len)
global.max_syslog_len = logger->maxlen;
MINOR: log: move 'log' keyword parsing in dedicated function Now, the function parse_logsrv should be used to parse a "log" line. This function will update the list of loggers passed in argument. It can release all log servers when "no log" line was parsed (by the caller) or it can parse "log global" or "log <address> ... " lines. It takes care of checking the caller context (global or not) to prohibit "log global" usage in the global section.
2018-03-26 09:54:32 -04:00
/* after the length, a format may be specified */
if (strcmp(args[cur_arg], "format") == 0) {
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
logger->format = get_log_format(args[cur_arg+1]);
if (logger->format == LOG_FORMAT_UNSPEC) {
MINOR: log: move 'log' keyword parsing in dedicated function Now, the function parse_logsrv should be used to parse a "log" line. This function will update the list of loggers passed in argument. It can release all log servers when "no log" line was parsed (by the caller) or it can parse "log global" or "log <address> ... " lines. It takes care of checking the caller context (global or not) to prohibit "log global" usage in the global section.
2018-03-26 09:54:32 -04:00
memprintf(err, "unknown log format '%s'", args[cur_arg+1]);
goto error;
}
cur_arg += 2;
}
MINOR: log: Add "sample" new keyword to "log" lines. This patch implements the parsing of "sample" new optional keyword for "log" lines to be able to sample and balance the load of log messages between serveral log destinations declared by "log" lines. This keyword must be followed by a list of comma seperated ranges of indexes numbered from 1 to define the samples to be used to balance the load of logs to send. This "sample" keyword must be used on "log" lines obviously before the remaining optional ones without keyword. The list of ranges must be followed by a colon character to separate it from the log sampling size. With such following configuration declarations: log stderr local0 log 127.0.0.1:10001 sample 2-3,8-11:11 local0 log 127.0.0.2:10002 sample 5:5 local0 in addition to being sent to stderr, about the second "log" line, every 11 logs the logs #2 up to #3 would be sent to 127.0.0.1:10001, then #8 up tp #11 four logs would be sent to the same log server and so on periodically. Logs would be sent to 127.0.0.2:100002 every 5 logs. It is also possible to define the size of the sample with a value different of the maximum of the high limits of the ranges, for instance as follows: log 127.0.0.1:10001 sample 2-3,8-11:15 local0 as before the two logs #2 and #3 would be sent to 127.0.0.1:10001, then #8 up tp #11 logs, but in this case here, this would be done periodically every 15 messages. Also note that the ranges must not overlap each others. This is to ease the way the logs are periodically sent.
2019-04-24 10:14:33 -04:00
if (strcmp(args[cur_arg], "sample") == 0) {
unsigned low, high;
char *p, *beg, *end, *smp_sz_str;
size_t smp_rgs_sz = 0, smp_sz = 0, new_smp_sz;
p = args[cur_arg+1];
smp_sz_str = strchr(p, ':');
if (!smp_sz_str) {
memprintf(err, "Missing sample size");
goto error;
}
*smp_sz_str++ = '\0';
end = p + strlen(p);
while (p != end) {
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
if (!get_logger_smp_range(&low, &high, &p, err))
MINOR: log: Add "sample" new keyword to "log" lines. This patch implements the parsing of "sample" new optional keyword for "log" lines to be able to sample and balance the load of log messages between serveral log destinations declared by "log" lines. This keyword must be followed by a list of comma seperated ranges of indexes numbered from 1 to define the samples to be used to balance the load of logs to send. This "sample" keyword must be used on "log" lines obviously before the remaining optional ones without keyword. The list of ranges must be followed by a colon character to separate it from the log sampling size. With such following configuration declarations: log stderr local0 log 127.0.0.1:10001 sample 2-3,8-11:11 local0 log 127.0.0.2:10002 sample 5:5 local0 in addition to being sent to stderr, about the second "log" line, every 11 logs the logs #2 up to #3 would be sent to 127.0.0.1:10001, then #8 up tp #11 four logs would be sent to the same log server and so on periodically. Logs would be sent to 127.0.0.2:100002 every 5 logs. It is also possible to define the size of the sample with a value different of the maximum of the high limits of the ranges, for instance as follows: log 127.0.0.1:10001 sample 2-3,8-11:15 local0 as before the two logs #2 and #3 would be sent to 127.0.0.1:10001, then #8 up tp #11 logs, but in this case here, this would be done periodically every 15 messages. Also note that the ranges must not overlap each others. This is to ease the way the logs are periodically sent.
2019-04-24 10:14:33 -04:00
goto error;
if (smp_rgs && smp_log_ranges_overlap(smp_rgs, smp_rgs_sz, low, high, err))
goto error;
smp_rgs = my_realloc2(smp_rgs, (smp_rgs_sz + 1) * sizeof *smp_rgs);
if (!smp_rgs) {
memprintf(err, "out of memory error");
goto error;
}
smp_rgs[smp_rgs_sz].low = low;
smp_rgs[smp_rgs_sz].high = high;
smp_rgs[smp_rgs_sz].sz = high - low + 1;
if (smp_rgs[smp_rgs_sz].high > smp_sz)
smp_sz = smp_rgs[smp_rgs_sz].high;
smp_rgs_sz++;
}
BUG/MINOR: log: Detect missing sampling ranges in config Consider a config like: global log 127.0.0.1:10001 sample :10 local0 No sampling ranges are given here, leading to NULL being passed as the first argument to qsort. This configuration does not make sense anyway, a log without ranges would never log. Thus output an error if no ranges are given. This bug was introduced in d95ea2897eb951c72fd169f36b6a79905f2ed999. This fix must be backported to HAProxy 2.0.
2019-06-23 16:10:10 -04:00
if (smp_rgs == NULL) {
memprintf(err, "no sampling ranges given");
goto error;
}
MINOR: log: Add "sample" new keyword to "log" lines. This patch implements the parsing of "sample" new optional keyword for "log" lines to be able to sample and balance the load of log messages between serveral log destinations declared by "log" lines. This keyword must be followed by a list of comma seperated ranges of indexes numbered from 1 to define the samples to be used to balance the load of logs to send. This "sample" keyword must be used on "log" lines obviously before the remaining optional ones without keyword. The list of ranges must be followed by a colon character to separate it from the log sampling size. With such following configuration declarations: log stderr local0 log 127.0.0.1:10001 sample 2-3,8-11:11 local0 log 127.0.0.2:10002 sample 5:5 local0 in addition to being sent to stderr, about the second "log" line, every 11 logs the logs #2 up to #3 would be sent to 127.0.0.1:10001, then #8 up tp #11 four logs would be sent to the same log server and so on periodically. Logs would be sent to 127.0.0.2:100002 every 5 logs. It is also possible to define the size of the sample with a value different of the maximum of the high limits of the ranges, for instance as follows: log 127.0.0.1:10001 sample 2-3,8-11:15 local0 as before the two logs #2 and #3 would be sent to 127.0.0.1:10001, then #8 up tp #11 logs, but in this case here, this would be done periodically every 15 messages. Also note that the ranges must not overlap each others. This is to ease the way the logs are periodically sent.
2019-04-24 10:14:33 -04:00
beg = smp_sz_str;
end = beg + strlen(beg);
new_smp_sz = read_uint((const char **)&beg, end);
if (!new_smp_sz || beg != end) {
memprintf(err, "wrong sample size '%s' for sample range '%s'",
smp_sz_str, args[cur_arg+1]);
goto error;
}
if (new_smp_sz < smp_sz) {
memprintf(err, "sample size %zu should be greater or equal to "
"%zu the maximum of the high ranges limits",
new_smp_sz, smp_sz);
goto error;
}
smp_sz = new_smp_sz;
/* Let's order <smp_rgs> array. */
qsort(smp_rgs, smp_rgs_sz, sizeof(struct smp_log_range), smp_log_range_cmp);
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
logger->lb.smp_rgs = smp_rgs;
logger->lb.smp_rgs_sz = smp_rgs_sz;
logger->lb.smp_sz = smp_sz;
MINOR: log: Add "sample" new keyword to "log" lines. This patch implements the parsing of "sample" new optional keyword for "log" lines to be able to sample and balance the load of log messages between serveral log destinations declared by "log" lines. This keyword must be followed by a list of comma seperated ranges of indexes numbered from 1 to define the samples to be used to balance the load of logs to send. This "sample" keyword must be used on "log" lines obviously before the remaining optional ones without keyword. The list of ranges must be followed by a colon character to separate it from the log sampling size. With such following configuration declarations: log stderr local0 log 127.0.0.1:10001 sample 2-3,8-11:11 local0 log 127.0.0.2:10002 sample 5:5 local0 in addition to being sent to stderr, about the second "log" line, every 11 logs the logs #2 up to #3 would be sent to 127.0.0.1:10001, then #8 up tp #11 four logs would be sent to the same log server and so on periodically. Logs would be sent to 127.0.0.2:100002 every 5 logs. It is also possible to define the size of the sample with a value different of the maximum of the high limits of the ranges, for instance as follows: log 127.0.0.1:10001 sample 2-3,8-11:15 local0 as before the two logs #2 and #3 would be sent to 127.0.0.1:10001, then #8 up tp #11 logs, but in this case here, this would be done periodically every 15 messages. Also note that the ranges must not overlap each others. This is to ease the way the logs are periodically sent.
2019-04-24 10:14:33 -04:00
cur_arg += 2;
}
MEDIUM: logs: atomically check and update the log sample index The log server lock is pretty visible in perf top when using log samples because it's taken for each server in turn while trying to validate and update the log server's index. Let's change this for a CAS, since we have the index and the range at hand now. This allow us to remove the logsrv lock. The test on 4 servers now shows a 3.7 times improvement thanks to much lower contention. Without log sampling a test producing 4.4M logs/s delivers 4.4M logs/s at 21 CPUs used, everything spent in the kernel. After enabling 4 samples (1:4, 2:4, 3:4 and 4:4), the throughput would previously drop to 1.13M log/s with 37 CPUs used and 75% spent in process_send_log(). Now with this change, 4.25M logs/s are emitted, using 26 CPUs and 22% in process_send_log(). That's a 3.7x throughput improvement for a 30% global CPU usage reduction, but in practice it mostly shows that the performance drop caused by having samples is much less noticeable (each of the 4 servers has its index updated for each log). Note that in order to even avoid incrementing an index for each log srv that is consulted, it would be more convenient to have a single index per frontend and apply the modulus on each log server in turn to see if the range has to be updated. It would then only perform one write per range switch. However the place where this is done doesn't have access to a frontend, so some changes would need to be performed for this, and it would require to update the current range independently in each logsrv, which is not necessarily easier since we don't know yet if we can commit it.
2023-09-20 14:34:35 -04:00
MINOR: log: move 'log' keyword parsing in dedicated function Now, the function parse_logsrv should be used to parse a "log" line. This function will update the list of loggers passed in argument. It can release all log servers when "no log" line was parsed (by the caller) or it can parse "log global" or "log <address> ... " lines. It takes care of checking the caller context (global or not) to prohibit "log global" usage in the global section.
2018-03-26 09:54:32 -04:00
/* parse the facility */
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
logger->facility = get_log_facility(args[cur_arg]);
if (logger->facility < 0) {
MINOR: log: move 'log' keyword parsing in dedicated function Now, the function parse_logsrv should be used to parse a "log" line. This function will update the list of loggers passed in argument. It can release all log servers when "no log" line was parsed (by the caller) or it can parse "log global" or "log <address> ... " lines. It takes care of checking the caller context (global or not) to prohibit "log global" usage in the global section.
2018-03-26 09:54:32 -04:00
memprintf(err, "unknown log facility '%s'", args[cur_arg]);
goto error;
}
cur_arg++;
/* parse the max syslog level (default: debug) */
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
logger->level = 7;
MINOR: log: move 'log' keyword parsing in dedicated function Now, the function parse_logsrv should be used to parse a "log" line. This function will update the list of loggers passed in argument. It can release all log servers when "no log" line was parsed (by the caller) or it can parse "log global" or "log <address> ... " lines. It takes care of checking the caller context (global or not) to prohibit "log global" usage in the global section.
2018-03-26 09:54:32 -04:00
if (*(args[cur_arg])) {
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
logger->level = get_log_level(args[cur_arg]);
if (logger->level < 0) {
MINOR: log: move 'log' keyword parsing in dedicated function Now, the function parse_logsrv should be used to parse a "log" line. This function will update the list of loggers passed in argument. It can release all log servers when "no log" line was parsed (by the caller) or it can parse "log global" or "log <address> ... " lines. It takes care of checking the caller context (global or not) to prohibit "log global" usage in the global section.
2018-03-26 09:54:32 -04:00
memprintf(err, "unknown optional log level '%s'", args[cur_arg]);
goto error;
}
cur_arg++;
}
/* parse the limit syslog level (default: emerg) */
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
logger->minlvl = 0;
MINOR: log: move 'log' keyword parsing in dedicated function Now, the function parse_logsrv should be used to parse a "log" line. This function will update the list of loggers passed in argument. It can release all log servers when "no log" line was parsed (by the caller) or it can parse "log global" or "log <address> ... " lines. It takes care of checking the caller context (global or not) to prohibit "log global" usage in the global section.
2018-03-26 09:54:32 -04:00
if (*(args[cur_arg])) {
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
logger->minlvl = get_log_level(args[cur_arg]);
if (logger->minlvl < 0) {
MINOR: log: move 'log' keyword parsing in dedicated function Now, the function parse_logsrv should be used to parse a "log" line. This function will update the list of loggers passed in argument. It can release all log servers when "no log" line was parsed (by the caller) or it can parse "log global" or "log <address> ... " lines. It takes care of checking the caller context (global or not) to prohibit "log global" usage in the global section.
2018-03-26 09:54:32 -04:00
memprintf(err, "unknown optional minimum log level '%s'", args[cur_arg]);
goto error;
}
cur_arg++;
}
/* Too many args */
if (*(args[cur_arg])) {
memprintf(err, "cannot handle unexpected argument '%s'", args[cur_arg]);
goto error;
}
MEDIUM: log: introduce log target log targets were immediately embedded in logger struct (previously named logsrv) and could not be used outside of this context. In this patch, we're introducing log_target type with the associated helper functions so that it becomes possible to declare and use log targets outside of loggers scope.
2023-09-11 10:10:37 -04:00
/* now, back to the log target */
if (!parse_log_target(args[1], &logger->target, err))
MINOR: log: move 'log' keyword parsing in dedicated function Now, the function parse_logsrv should be used to parse a "log" line. This function will update the list of loggers passed in argument. It can release all log servers when "no log" line was parsed (by the caller) or it can parse "log global" or "log <address> ... " lines. It takes care of checking the caller context (global or not) to prohibit "log global" usage in the global section.
2018-03-26 09:54:32 -04:00
goto error;
MINOR: log: detect LOG_TARGET_FD from the fd and not from the syntax Now that we have the FD value reported we don't need to cheat and detect "fd@" in the address, we can safely rely on the FD value.
2020-09-15 08:03:26 -04:00
MEDIUM: log: add support for logging to a ring buffer Now by prefixing a log server with "ring@<name>" it's possible to send the logs to a ring buffer. One nice thing is that it allows multiple sessions to consult the logs in real time in parallel over the CLI, and without requiring file system access. At the moment, ring0 is created as a default sink for tracing purposes and is available. No option is provided to create new rings though this is trivial to add to the global section.
2019-08-30 09:24:59 -04:00
done:
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
LIST_APPEND(loggers, &logger->list);
MINOR: log: move 'log' keyword parsing in dedicated function Now, the function parse_logsrv should be used to parse a "log" line. This function will update the list of loggers passed in argument. It can release all log servers when "no log" line was parsed (by the caller) or it can parse "log global" or "log <address> ... " lines. It takes care of checking the caller context (global or not) to prohibit "log global" usage in the global section.
2018-03-26 09:54:32 -04:00
return 1;
error:
BUG/MINOR: log: fix memory leak on logsrv parse error In case of parsing error on logsrv, we can leave parse_logsrv() without releasing logsrv->ring_name or smp_rgs. Let's free them on the error path. This should fix issue #926 detected by Coverity. The impact is only a tiny leak just before reporting a fatal error, so it will essentially annoy valgrind. This can be backported to 2.0 (just drop the ring part).
2020-10-27 04:51:37 -04:00
free(smp_rgs);
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
free_logger(logger);
MINOR: log: move 'log' keyword parsing in dedicated function Now, the function parse_logsrv should be used to parse a "log" line. This function will update the list of loggers passed in argument. It can release all log servers when "no log" line was parsed (by the caller) or it can parse "log global" or "log <address> ... " lines. It takes care of checking the caller context (global or not) to prohibit "log global" usage in the global section.
2018-03-26 09:54:32 -04:00
return 0;
}
MEDIUM: logs: add support for RFC5424 header format per logger The function __send_log() iterates over senders and passes the header as the first vector to sendmsg(), thus it can send a logger-specific header in each message. A new logger arguments "format rfc5424" should be used in order to enable RFC5424 header format. For example: log 10.2.3.4:1234 len 2048 format rfc5424 local2 info
2015-09-22 10:05:32 -04:00
/*
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
* returns log format, LOG_FORMAT_UNSPEC is return if not found.
MEDIUM: logs: add support for RFC5424 header format per logger The function __send_log() iterates over senders and passes the header as the first vector to sendmsg(), thus it can send a logger-specific header in each message. A new logger arguments "format rfc5424" should be used in order to enable RFC5424 header format. For example: log 10.2.3.4:1234 len 2048 format rfc5424 local2 info
2015-09-22 10:05:32 -04:00
*/
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
enum log_fmt get_log_format(const char *fmt)
MEDIUM: logs: add support for RFC5424 header format per logger The function __send_log() iterates over senders and passes the header as the first vector to sendmsg(), thus it can send a logger-specific header in each message. A new logger arguments "format rfc5424" should be used in order to enable RFC5424 header format. For example: log 10.2.3.4:1234 len 2048 format rfc5424 local2 info
2015-09-22 10:05:32 -04:00
{
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
enum log_fmt format;
MEDIUM: logs: add support for RFC5424 header format per logger The function __send_log() iterates over senders and passes the header as the first vector to sendmsg(), thus it can send a logger-specific header in each message. A new logger arguments "format rfc5424" should be used in order to enable RFC5424 header format. For example: log 10.2.3.4:1234 len 2048 format rfc5424 local2 info
2015-09-22 10:05:32 -04:00
format = LOG_FORMATS - 1;
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
while (format > 0 && log_formats[format].name
CLEANUP: Compare the return value of `XXXcmp()` functions with zero According to coding-style.txt it is recommended to use: `strcmp(a, b) == 0` instead of `!strcmp(a, b)` So let's do this. The change was performed by running the following (very long) coccinelle patch on src/: @@ statement S; expression E; expression F; @@ if ( ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) != 0 ) ( S | { ... } ) @@ statement S; expression E; expression F; @@ if ( - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 ) ( S | { ... } ) @@ expression E; expression F; expression G; @@ ( G && ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) != 0 ) @@ expression E; expression F; expression G; @@ ( G || ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) != 0 ) @@ expression E; expression F; expression G; @@ ( ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) != 0 && G ) @@ expression E; expression F; expression G; @@ ( ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) != 0 || G ) @@ expression E; expression F; expression G; @@ ( G && - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 ) @@ expression E; expression F; expression G; @@ ( G || - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 ) @@ expression E; expression F; expression G; @@ ( - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 && G ) @@ expression E; expression F; expression G; @@ ( - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 || G ) @@ expression E; expression F; expression G; @@ ( - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 )
2021-01-02 16:31:53 -05:00
&& strcmp(log_formats[format].name, fmt) != 0)
MEDIUM: logs: add support for RFC5424 header format per logger The function __send_log() iterates over senders and passes the header as the first vector to sendmsg(), thus it can send a logger-specific header in each message. A new logger arguments "format rfc5424" should be used in order to enable RFC5424 header format. For example: log 10.2.3.4:1234 len 2048 format rfc5424 local2 info
2015-09-22 10:05:32 -04:00
format--;
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
/* Note: 0 is LOG_FORMAT_UNSPEC */
MEDIUM: logs: add support for RFC5424 header format per logger The function __send_log() iterates over senders and passes the header as the first vector to sendmsg(), thus it can send a logger-specific header in each message. A new logger arguments "format rfc5424" should be used in order to enable RFC5424 header format. For example: log 10.2.3.4:1234 len 2048 format rfc5424 local2 info
2015-09-22 10:05:32 -04:00
return format;
}
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-25 20:48:02 -04:00
/*
* returns log level for <lev> or -1 if not found.
*/
int get_log_level(const char *lev)
{
int level;
level = NB_LOG_LEVELS - 1;
CLEANUP: Compare the return value of `XXXcmp()` functions with zero According to coding-style.txt it is recommended to use: `strcmp(a, b) == 0` instead of `!strcmp(a, b)` So let's do this. The change was performed by running the following (very long) coccinelle patch on src/: @@ statement S; expression E; expression F; @@ if ( ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) != 0 ) ( S | { ... } ) @@ statement S; expression E; expression F; @@ if ( - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 ) ( S | { ... } ) @@ expression E; expression F; expression G; @@ ( G && ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) != 0 ) @@ expression E; expression F; expression G; @@ ( G || ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) != 0 ) @@ expression E; expression F; expression G; @@ ( ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) != 0 && G ) @@ expression E; expression F; expression G; @@ ( ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) != 0 || G ) @@ expression E; expression F; expression G; @@ ( G && - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 ) @@ expression E; expression F; expression G; @@ ( G || - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 ) @@ expression E; expression F; expression G; @@ ( - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 && G ) @@ expression E; expression F; expression G; @@ ( - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 || G ) @@ expression E; expression F; expression G; @@ ( - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 )
2021-01-02 16:31:53 -05:00
while (level >= 0 && strcmp(log_levels[level], lev) != 0)
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-25 20:48:02 -04:00
level--;
return level;
}
/*
* returns log facility for <fac> or -1 if not found.
*/
int get_log_facility(const char *fac)
{
int facility;
facility = NB_LOG_FACILITIES - 1;
CLEANUP: Compare the return value of `XXXcmp()` functions with zero According to coding-style.txt it is recommended to use: `strcmp(a, b) == 0` instead of `!strcmp(a, b)` So let's do this. The change was performed by running the following (very long) coccinelle patch on src/: @@ statement S; expression E; expression F; @@ if ( ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) != 0 ) ( S | { ... } ) @@ statement S; expression E; expression F; @@ if ( - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 ) ( S | { ... } ) @@ expression E; expression F; expression G; @@ ( G && ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) != 0 ) @@ expression E; expression F; expression G; @@ ( G || ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) != 0 ) @@ expression E; expression F; expression G; @@ ( ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) != 0 && G ) @@ expression E; expression F; expression G; @@ ( ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) != 0 || G ) @@ expression E; expression F; expression G; @@ ( G && - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 ) @@ expression E; expression F; expression G; @@ ( G || - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 ) @@ expression E; expression F; expression G; @@ ( - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 && G ) @@ expression E; expression F; expression G; @@ ( - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 || G ) @@ expression E; expression F; expression G; @@ ( - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 )
2021-01-02 16:31:53 -05:00
while (facility >= 0 && strcmp(log_facilities[facility], fac) != 0)
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-25 20:48:02 -04:00
facility--;
REORG: log: split send_log function send_log function is now splited in 3 functions * hdr_log: generate the syslog header * send_log: send a syslog message with a printf format string * __send_log: send a syslog message
2012-02-06 10:00:33 -05:00
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-25 20:48:02 -04:00
return facility;
}
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
struct lf_buildctx {
OPTIM: log: use lf_buildctx's buffer instead of temporary stack buffers Now that lf_buildctx isn't pushed on the stack anymore, let's take this opportunity to store a small buffer of 256 bytes within it, and then use this buffer as general purpose buffer to build fixed-length strings that are then printed using lf_{raw}text() function. By doing so we stop relying on temporary stack buffers.
2024-05-02 10:48:56 -04:00
char _buf[256];/* fixed size buffer for building small strings */
int options; /* LOG_OPT_* options */
int typecast; /* same as logformat_node->typecast */
int in_text; /* inside variable-length text */
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
union {
struct cbor_encode_ctx cbor; /* cbor-encode specific ctx */
} encode;
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
};
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
static THREAD_LOCAL struct lf_buildctx lf_buildctx;
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
/* helper to encode a single byte in hex form
*
* Returns the position of the last written byte on success and NULL on
* error.
*/
static char *_encode_byte_hex(char *start, char *stop, unsigned char byte)
{
/* hex form requires 2 bytes */
if ((stop - start) < 2)
return NULL;
*start++ = hextab[(byte >> 4) & 15];
*start++ = hextab[byte & 15];
return start;
}
/* lf cbor function ptr used to encode a single byte according to RFC8949
*
* for now only hex form is supported.
*
MINOR: log/cbor: _lf_cbor_encode_byte() explicitly requires non-NULL ctx As shown in GH #2550, Coverity is tempted to think that NULL-dereference can occur in _lf_cbor_encode_byte() due to user-ctx being dereferenced from cbor_ctx, while coverity thinks that cbor_ctx may be NULL. In practise this cannot happen, because _lf_cbor_encode_byte() is only leveraged through a function pointer that is set in conjunction with the function pointer ctx (which ain't NULL). All this logic is done inside lf_buildctx_prepare() when LOG_OPT_ENCODE_CBOR is set. Since coverity doesn't seem to understand the logic properly, then it might as well confuse humans, so let's make it clear in _lf_cbor_encode_byte() that we expect non-NULL ctx by adding a BUG_ON()
2024-04-29 03:46:24 -04:00
* The function may only be called under CBOR context (that is when
* LOG_OPT_ENCODE_CBOR option is set).
*
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
* Returns the position of the last written byte on success and NULL on
* error.
*/
static char *_lf_cbor_encode_byte(struct cbor_encode_ctx *cbor_ctx,
char *start, char *stop, unsigned char byte)
{
MINOR: log/cbor: _lf_cbor_encode_byte() explicitly requires non-NULL ctx As shown in GH #2550, Coverity is tempted to think that NULL-dereference can occur in _lf_cbor_encode_byte() due to user-ctx being dereferenced from cbor_ctx, while coverity thinks that cbor_ctx may be NULL. In practise this cannot happen, because _lf_cbor_encode_byte() is only leveraged through a function pointer that is set in conjunction with the function pointer ctx (which ain't NULL). All this logic is done inside lf_buildctx_prepare() when LOG_OPT_ENCODE_CBOR is set. Since coverity doesn't seem to understand the logic properly, then it might as well confuse humans, so let's make it clear in _lf_cbor_encode_byte() that we expect non-NULL ctx by adding a BUG_ON()
2024-04-29 03:46:24 -04:00
struct lf_buildctx *ctx;
MINOR: log/cbor: _lf_cbor_encode_byte() explicitly requires non-NULL ctx (again) The BUG_ON() statement that was added in 9bdea51 ("MINOR: log/cbor: _lf_cbor_encode_byte() explicitly requires non-NULL ctx") isn't sufficient as Coverity still thinks the lf_buildctx itself may be NULL as shown in GH #2554. In fact the original reports complains about the lf_buildctx itself and I didn't understand it properly, let's add another check in the BUG_ON() to ensure both cbor_ctx and cbor_ctx->ctx are not NULL since it is not expected if used properly.
2024-04-30 02:55:13 -04:00
BUG_ON(!cbor_ctx || !cbor_ctx->e_fct_ctx);
MINOR: log/cbor: _lf_cbor_encode_byte() explicitly requires non-NULL ctx As shown in GH #2550, Coverity is tempted to think that NULL-dereference can occur in _lf_cbor_encode_byte() due to user-ctx being dereferenced from cbor_ctx, while coverity thinks that cbor_ctx may be NULL. In practise this cannot happen, because _lf_cbor_encode_byte() is only leveraged through a function pointer that is set in conjunction with the function pointer ctx (which ain't NULL). All this logic is done inside lf_buildctx_prepare() when LOG_OPT_ENCODE_CBOR is set. Since coverity doesn't seem to understand the logic properly, then it might as well confuse humans, so let's make it clear in _lf_cbor_encode_byte() that we expect non-NULL ctx by adding a BUG_ON()
2024-04-29 03:46:24 -04:00
ctx = cbor_ctx->e_fct_ctx;
MINOR: log: support true cbor binary encoding CBOR in hex format as implemented in previous commit is convenient because the produced output is portable and can easily be embedded in regular syslog payloads. However, one of the goal of CBOR implementation is to be able to produce "Concise Binary" object representation. Here is an excerpt from cbor.io website: "Some applications also benefit from CBOR itself being encoded in binary. This saves bulk and allows faster processing." Currently we don't offer that with '+cbor', quite the opposite actually since a text string encoded with '+cbor' option will be larger than a text string encoded with '+json' or without encoding at all, because for each CBOR binary byte, 2 characters will be emitted. Hopefully, the sink/log API allows for binary data to be passed as parameter, this is because all relevant functions in the chain don't rely on the terminating NULL byte and take a string pointer + string length as parameter. We can actually rely on this property to support the '+bin' option when combined with '+cbor' to produce RAW binary CBOR output. Be careful though, as this is only intended for use with set-var-fmt or to send binary data to capable UDP/ring endpoints. Example: log-format "%{+cbor,+bin}o %(test)[bin(00AABB)]" Will produce: bf64746573745f4300aabbffff (output was piped to `hexdump -ve '1/1 "%.2x"'` to dump raw bytes as HEX characters) With cbor.me pretty printer, it gives us: BF # map(*) 64 # text(4) 74657374 # "test" 5F # bytes(*) 43 # bytes(3) 00AABB # "\u0000\xAA\xBB" FF # primitive(*) FF # primitive(*)
2024-04-23 13:13:40 -04:00
if (ctx->options & LOG_OPT_BIN) {
/* raw output */
if ((stop - start) < 1)
return NULL;
*start++ = byte;
return start;
}
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
return _encode_byte_hex(start, stop, byte);
}
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
/* helper function to prepare lf_buildctx struct based on global options
* and current node settings (may be NULL)
*/
static inline void lf_buildctx_prepare(struct lf_buildctx *ctx,
int g_options,
const struct logformat_node *node)
{
if (node) {
OPTIM: log: resolve logformat options during postparsing In lf_buildctx_prepare(), we perform costly bitwise operations for every nodes to resolve node options and check for incompatibilities with global options. In fact, all this logic may safely be performed during postparsing. This is what we're doing in this commit. Doing so saves us from unnecessary runtime checks and could help speedup sess_build_logline(). Since checks are not as costly as before (due to them being performed during postparsing and not on log building path anymore), an complementary check for OPT_HTTP vs OPT_ENCODE incompatibity was added: encoding is ignored if HTTP option is set, unless HTTP option wasn't set globally and encoding was set globally, which means encoding takes the precedence Thanks to this patch, lf_buildctx_prepare() now only takes care of assigning proper typecast and options settings depending if it's used from global or per-node context, and prepares CBOR-specific structure members when CBOR encode option is set.
2024-04-30 11:42:00 -04:00
/* consider node's options and typecast setting */
ctx->options = node->options;
MINOR: log: expose node typecast in lf_buildctx struct Store node->typecast setting inside lf_buildctx struct so that encoding functions may benefit from it.
2024-04-25 12:52:57 -04:00
ctx->typecast = node->typecast;
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
}
BUG/MEDIUM: log: don't ignore disabled node's options In 3f2e8d0ed ("MEDIUM: log: lf_* build helpers now take a ctx argument") I made a mistake, because starting with this commit it is no longer possible from a node to disable global logformat options. The result is that when an option is set globally, it cannot be disabled anymore. For instance, it is not possible to do this anymore: log-format "%{+X}o %{-X}Ts" The original intent was to prevent encoding options from being disabled once enabled globally, because when encoding is enabled globally we start the object enumeration right away (ie: in CBOR and JSON we announce dynamic map, and for each node we announce the key..), thus it doesn't make sense to mix encoding types there, unless encoding is only used per-node, in which case only the value gets encoded, thus it remains possible to print a value in JSON/CBOR-compatible format while the next one shouldn't be printed as-is. Thus, to restore the original behavior, slightly change the logic in lf_buildctx_prepare() so that only global encoding options take the precedence over node's options (instead of all options). No backport needed.
2024-04-30 11:52:44 -04:00
else {
ctx->options = g_options;
ctx->typecast = SMP_T_SAME; /* default */
}
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
OPTIM: log: resolve logformat options during postparsing In lf_buildctx_prepare(), we perform costly bitwise operations for every nodes to resolve node options and check for incompatibilities with global options. In fact, all this logic may safely be performed during postparsing. This is what we're doing in this commit. Doing so saves us from unnecessary runtime checks and could help speedup sess_build_logline(). Since checks are not as costly as before (due to them being performed during postparsing and not on log building path anymore), an complementary check for OPT_HTTP vs OPT_ENCODE incompatibity was added: encoding is ignored if HTTP option is set, unless HTTP option wasn't set globally and encoding was set globally, which means encoding takes the precedence Thanks to this patch, lf_buildctx_prepare() now only takes care of assigning proper typecast and options settings depending if it's used from global or per-node context, and prepares CBOR-specific structure members when CBOR encode option is set.
2024-04-30 11:42:00 -04:00
if (ctx->options & LOG_OPT_ENCODE_CBOR) {
/* prepare cbor-specific encode ctx */
ctx->encode.cbor.e_fct_byte = _lf_cbor_encode_byte;
ctx->encode.cbor.e_fct_ctx = ctx;
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
}
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
}
MINOR: log: merge lf_encode_string() and lf_encode_chunk() logic lf_encode_string() and lf_encode_chunk() function are pretty similar. The only difference is the stopping behavior, encode_chunk stops at a given position while encode_string stops when encountering '\0'. Moreover, both functions leverage tools.c encode helpers, but because of the LOG_OPT_ESC option, they reimplement those helpers with added logic. Instead of having to deal with code duplication which makes both functions harder to maintain, let's define a _lf_encode_bytes() helper function which satisfies lf_encode_string() and lf_encode_chunk() needs while keeping the function as simple as possible. _lf_encode_bytes() itself is made of multiple static inline helper functions, in the attempt to keep checks outside of core loop for better performance.
2024-04-26 07:53:15 -04:00
/* helper function for _lf_encode_bytes() to escape a single byte
* with <escape>
*/
static inline char *_lf_escape_byte(char *start, char *stop,
char byte, const char escape)
{
if (start + 3 >= stop)
return NULL;
*start++ = escape;
*start++ = hextab[(byte >> 4) & 15];
*start++ = hextab[byte & 15];
return start;
}
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
/* helper function for _lf_encode_bytes() to escape a single byte
* with <escape> and deal with cbor-specific encoding logic
*/
static inline char *_lf_cbor_escape_byte(char *start, char *stop,
char byte, const char escape,
uint8_t cbor_string_prefix,
struct lf_buildctx *ctx)
{
char escaped_byte[3];
escaped_byte[0] = escape;
escaped_byte[1] = hextab[(byte >> 4) & 15];
escaped_byte[2] = hextab[byte & 15];
start = cbor_encode_bytes_prefix(&ctx->encode.cbor, start, stop,
escaped_byte, 3,
cbor_string_prefix);
return start;
}
MINOR: log: merge lf_encode_string() and lf_encode_chunk() logic lf_encode_string() and lf_encode_chunk() function are pretty similar. The only difference is the stopping behavior, encode_chunk stops at a given position while encode_string stops when encountering '\0'. Moreover, both functions leverage tools.c encode helpers, but because of the LOG_OPT_ESC option, they reimplement those helpers with added logic. Instead of having to deal with code duplication which makes both functions harder to maintain, let's define a _lf_encode_bytes() helper function which satisfies lf_encode_string() and lf_encode_chunk() needs while keeping the function as simple as possible. _lf_encode_bytes() itself is made of multiple static inline helper functions, in the attempt to keep checks outside of core loop for better performance.
2024-04-26 07:53:15 -04:00
/* helper function for _lf_encode_bytes() to encode a single byte
* and escape it with <escape> if found in <map>
*
* The function assumes that at least 1 byte is available for writing
*
* Returns the address of the last written byte on success, or NULL
* on error
*/
static inline char *_lf_map_escape_byte(char *start, char *stop,
const char *byte,
const char escape, const long *map,
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
const char **pending, uint8_t cbor_string_prefix,
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
struct lf_buildctx *ctx)
MINOR: log: merge lf_encode_string() and lf_encode_chunk() logic lf_encode_string() and lf_encode_chunk() function are pretty similar. The only difference is the stopping behavior, encode_chunk stops at a given position while encode_string stops when encountering '\0'. Moreover, both functions leverage tools.c encode helpers, but because of the LOG_OPT_ESC option, they reimplement those helpers with added logic. Instead of having to deal with code duplication which makes both functions harder to maintain, let's define a _lf_encode_bytes() helper function which satisfies lf_encode_string() and lf_encode_chunk() needs while keeping the function as simple as possible. _lf_encode_bytes() itself is made of multiple static inline helper functions, in the attempt to keep checks outside of core loop for better performance.
2024-04-26 07:53:15 -04:00
{
if (!ha_bit_test((unsigned char)(*byte), map))
*start++ = *byte;
else
start = _lf_escape_byte(start, stop, *byte, escape);
return start;
}
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
/* helper function for _lf_encode_bytes() to encode a single byte
* and escape it with <escape> if found in <map> and deal with
* cbor-specific encoding logic.
*
* The function assumes that at least 1 byte is available for writing
*
* Returns the address of the last written byte on success, or NULL
* on error
*/
static inline char *_lf_cbor_map_escape_byte(char *start, char *stop,
const char *byte,
const char escape, const long *map,
const char **pending, uint8_t cbor_string_prefix,
struct lf_buildctx *ctx)
{
/* We try our best to minimize the number of chunks produced for the
* indefinite-length byte string as each chunk has an extra overhead
* as per RFC8949.
*
* To achieve that, we try to emit consecutive bytes together
*/
if (!ha_bit_test((unsigned char)(*byte), map)) {
/* do nothing and let the caller continue seeking data,
* pending data will be flushed later
*/
} else {
/* first, flush pending unescaped bytes */
start = cbor_encode_bytes_prefix(&ctx->encode.cbor, start, stop,
*pending, (byte - *pending),
cbor_string_prefix);
if (start == NULL)
return NULL;
*pending = byte + 1;
/* escape current matching byte */
start = _lf_cbor_escape_byte(start, stop, *byte, escape,
cbor_string_prefix,
ctx);
}
return start;
}
MINOR: log: merge lf_encode_string() and lf_encode_chunk() logic lf_encode_string() and lf_encode_chunk() function are pretty similar. The only difference is the stopping behavior, encode_chunk stops at a given position while encode_string stops when encountering '\0'. Moreover, both functions leverage tools.c encode helpers, but because of the LOG_OPT_ESC option, they reimplement those helpers with added logic. Instead of having to deal with code duplication which makes both functions harder to maintain, let's define a _lf_encode_bytes() helper function which satisfies lf_encode_string() and lf_encode_chunk() needs while keeping the function as simple as possible. _lf_encode_bytes() itself is made of multiple static inline helper functions, in the attempt to keep checks outside of core loop for better performance.
2024-04-26 07:53:15 -04:00
/* helper function for _lf_encode_bytes() to encode a single byte
* and escape it with <escape> if found in <map> or escape it with
* '\' if found in rfc5424_escape_map
*
* The function assumes that at least 1 byte is available for writing
*
* Returns the address of the last written byte on success, or NULL
* on error
*/
static inline char *_lf_rfc5424_escape_byte(char *start, char *stop,
const char *byte,
const char escape, const long *map,
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
const char **pending, uint8_t cbor_string_prefix,
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
struct lf_buildctx *ctx)
MINOR: log: merge lf_encode_string() and lf_encode_chunk() logic lf_encode_string() and lf_encode_chunk() function are pretty similar. The only difference is the stopping behavior, encode_chunk stops at a given position while encode_string stops when encountering '\0'. Moreover, both functions leverage tools.c encode helpers, but because of the LOG_OPT_ESC option, they reimplement those helpers with added logic. Instead of having to deal with code duplication which makes both functions harder to maintain, let's define a _lf_encode_bytes() helper function which satisfies lf_encode_string() and lf_encode_chunk() needs while keeping the function as simple as possible. _lf_encode_bytes() itself is made of multiple static inline helper functions, in the attempt to keep checks outside of core loop for better performance.
2024-04-26 07:53:15 -04:00
{
if (!ha_bit_test((unsigned char)(*byte), map)) {
if (!ha_bit_test((unsigned char)(*byte), rfc5424_escape_map))
*start++ = *byte;
else {
if (start + 2 >= stop)
return NULL;
*start++ = '\\';
*start++ = *byte;
}
}
else
start = _lf_escape_byte(start, stop, *byte, escape);
return start;
}
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
/* helper function for _lf_encode_bytes() to encode a single byte
* and escape it with <escape> if found in <map> or escape it with
* '\' if found in json_escape_map
*
* The function assumes that at least 1 byte is available for writing
*
* Returns the address of the last written byte on success, or NULL
* on error
*/
static inline char *_lf_json_escape_byte(char *start, char *stop,
const char *byte,
const char escape, const long *map,
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
const char **pending, uint8_t cbor_string_prefix,
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
struct lf_buildctx *ctx)
{
if (!ha_bit_test((unsigned char)(*byte), map)) {
if (!ha_bit_test((unsigned char)(*byte), json_escape_map))
*start++ = *byte;
else {
if (start + 2 >= stop)
return NULL;
*start++ = '\\';
*start++ = *byte;
}
}
else
start = _lf_escape_byte(start, stop, *byte, escape);
return start;
}
MEDIUM: log: add a new log format flag "E" The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It mostly makes sense to use it in the RFC5424 structured-data log formats. Example: log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 07:23:03 -05:00
/*
MINOR: log: merge lf_encode_string() and lf_encode_chunk() logic lf_encode_string() and lf_encode_chunk() function are pretty similar. The only difference is the stopping behavior, encode_chunk stops at a given position while encode_string stops when encountering '\0'. Moreover, both functions leverage tools.c encode helpers, but because of the LOG_OPT_ESC option, they reimplement those helpers with added logic. Instead of having to deal with code duplication which makes both functions harder to maintain, let's define a _lf_encode_bytes() helper function which satisfies lf_encode_string() and lf_encode_chunk() needs while keeping the function as simple as possible. _lf_encode_bytes() itself is made of multiple static inline helper functions, in the attempt to keep checks outside of core loop for better performance.
2024-04-26 07:53:15 -04:00
* helper for lf_encode_{string,chunk}:
* encode the input bytes, input <bytes> is processed until <bytes_stop>
* is reached. If <bytes_stop> is NULL, <bytes> is expected to be NULL
* terminated.
MEDIUM: log: add a new log format flag "E" The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It mostly makes sense to use it in the RFC5424 structured-data log formats. Example: log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 07:23:03 -05:00
*
* When using the +E log format option, it will try to escape '"\]'
* characters with '\' as prefix. The same prefix should not be used as
* <escape>.
BUG/MINOR: tools/log: invalid encode_{chunk,string} usage encode_{chunk,string}() is often found to be used this way: ret = encode_{chunk,string}(start, stop...) if (ret == NULL || *ret != '\0') { //error } //success Indeed, encode_{chunk,string} will always try to add terminating NULL byte to the output string, unless no space is available for even 1 byte. However, it means that for the caller to be able to spot an error, then it must provide a buffer (here: start) which is already initialized. But this is wrong: not only this is very tricky to use, but since those functions don't return NULL on failure, then if the output buffer was not properly initialized prior to calling the function, the caller will perform invalid reads when checking for failure this way. Moreover, even if the buffer is initialized, we cannot reliably tell if the function actually failed this way because if the buffer was previously initialized with NULL byte, then the caller might think that the call actually succeeded (since the function didn't return NULL and didn't update the buffer). Also, sess_build_logline() relies lf_encode_{chunk,string}() functions which are in fact wrappers for encode_{chunk,string}() functions and thus exhibit the same error handling mechanism. It turns out that sess_build_logline() makes unsafe use of those functions because it uses the error-checking logic mentionned above while buffer (tmplog) is not guaranteed to be initialized when entering the function. This may ultimately cause malfunctions or invalid reads if the output buffer is lacking space. To fix the issue once and for all and prevent similar bugs from being introduced, we make it so encode_{string, chunk} and escape_string() (based on encode_string()) now explicitly return NULL on failure (when the function failed to write at least the ending NULL byte) lf_encode_{string,chunk}() helpers had to be patched as well due to code duplication. This should be backported to all stable versions. [ada: for 2.4 and 2.6 the patch won't apply as-is, it might be helpful to backport ae1e14d65 ("CLEANUP: tools: removing escape_chunk() function") first, considering it's not very relevant to maintain a dead function]
2024-04-09 05:44:54 -04:00
*
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
* When using json encoding, string will be escaped according to
* json escape map
*
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
* When using cbor encoding, escape option is ignored. However bytes found
* in <map> will still be escaped with <escape>.
*
BUG/MINOR: tools/log: invalid encode_{chunk,string} usage encode_{chunk,string}() is often found to be used this way: ret = encode_{chunk,string}(start, stop...) if (ret == NULL || *ret != '\0') { //error } //success Indeed, encode_{chunk,string} will always try to add terminating NULL byte to the output string, unless no space is available for even 1 byte. However, it means that for the caller to be able to spot an error, then it must provide a buffer (here: start) which is already initialized. But this is wrong: not only this is very tricky to use, but since those functions don't return NULL on failure, then if the output buffer was not properly initialized prior to calling the function, the caller will perform invalid reads when checking for failure this way. Moreover, even if the buffer is initialized, we cannot reliably tell if the function actually failed this way because if the buffer was previously initialized with NULL byte, then the caller might think that the call actually succeeded (since the function didn't return NULL and didn't update the buffer). Also, sess_build_logline() relies lf_encode_{chunk,string}() functions which are in fact wrappers for encode_{chunk,string}() functions and thus exhibit the same error handling mechanism. It turns out that sess_build_logline() makes unsafe use of those functions because it uses the error-checking logic mentionned above while buffer (tmplog) is not guaranteed to be initialized when entering the function. This may ultimately cause malfunctions or invalid reads if the output buffer is lacking space. To fix the issue once and for all and prevent similar bugs from being introduced, we make it so encode_{string, chunk} and escape_string() (based on encode_string()) now explicitly return NULL on failure (when the function failed to write at least the ending NULL byte) lf_encode_{string,chunk}() helpers had to be patched as well due to code duplication. This should be backported to all stable versions. [ada: for 2.4 and 2.6 the patch won't apply as-is, it might be helpful to backport ae1e14d65 ("CLEANUP: tools: removing escape_chunk() function") first, considering it's not very relevant to maintain a dead function]
2024-04-09 05:44:54 -04:00
* Return the address of the \0 character, or NULL on error
MEDIUM: log: add a new log format flag "E" The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It mostly makes sense to use it in the RFC5424 structured-data log formats. Example: log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 07:23:03 -05:00
*/
MINOR: log: merge lf_encode_string() and lf_encode_chunk() logic lf_encode_string() and lf_encode_chunk() function are pretty similar. The only difference is the stopping behavior, encode_chunk stops at a given position while encode_string stops when encountering '\0'. Moreover, both functions leverage tools.c encode helpers, but because of the LOG_OPT_ESC option, they reimplement those helpers with added logic. Instead of having to deal with code duplication which makes both functions harder to maintain, let's define a _lf_encode_bytes() helper function which satisfies lf_encode_string() and lf_encode_chunk() needs while keeping the function as simple as possible. _lf_encode_bytes() itself is made of multiple static inline helper functions, in the attempt to keep checks outside of core loop for better performance.
2024-04-26 07:53:15 -04:00
static char *_lf_encode_bytes(char *start, char *stop,
MINOR: logs: use the new bitmap functions instead of fd_sets for encoding maps The fd_sets we've been using in the log encoding functions are not portable and were shown to break at least under Cygwin. This patch gets rid of them in favor of the new bitmap functions. It was verified with the config below that the log output was exactly the same before and after the change : defaults mode http option httplog log stdout local0 timeout client 1s timeout server 1s timeout connect 1s frontend foo bind :8001 capture request header chars len 255 backend bar option httpchk "GET" "/" "HTTP/1.0\r\nchars: \x01\x02\x03\x04\x05\x06\x07\x08\x09\x0b\x0c\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f\x20\x21\x22\x23\x24\x25\x26\x27\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f\x30\x31\x32\x33\x34\x35\x36\x37\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f\x40\x41\x42\x43\x44\x45\x46\x47\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f\x50\x51\x52\x53\x54\x55\x56\x57\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f\x60\x61\x62\x63\x64\x65\x66\x67\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f\x70\x71\x72\x73\x74\x75\x76\x77\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" server foo 127.0.0.1:8001 check
2019-06-07 05:10:07 -04:00
const char escape, const long *map,
MINOR: log: merge lf_encode_string() and lf_encode_chunk() logic lf_encode_string() and lf_encode_chunk() function are pretty similar. The only difference is the stopping behavior, encode_chunk stops at a given position while encode_string stops when encountering '\0'. Moreover, both functions leverage tools.c encode helpers, but because of the LOG_OPT_ESC option, they reimplement those helpers with added logic. Instead of having to deal with code duplication which makes both functions harder to maintain, let's define a _lf_encode_bytes() helper function which satisfies lf_encode_string() and lf_encode_chunk() needs while keeping the function as simple as possible. _lf_encode_bytes() itself is made of multiple static inline helper functions, in the attempt to keep checks outside of core loop for better performance.
2024-04-26 07:53:15 -04:00
const char *bytes, const char *bytes_stop,
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
struct lf_buildctx *ctx)
MEDIUM: log: add a new log format flag "E" The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It mostly makes sense to use it in the RFC5424 structured-data log formats. Example: log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 07:23:03 -05:00
{
MINOR: log: merge lf_encode_string() and lf_encode_chunk() logic lf_encode_string() and lf_encode_chunk() function are pretty similar. The only difference is the stopping behavior, encode_chunk stops at a given position while encode_string stops when encountering '\0'. Moreover, both functions leverage tools.c encode helpers, but because of the LOG_OPT_ESC option, they reimplement those helpers with added logic. Instead of having to deal with code duplication which makes both functions harder to maintain, let's define a _lf_encode_bytes() helper function which satisfies lf_encode_string() and lf_encode_chunk() needs while keeping the function as simple as possible. _lf_encode_bytes() itself is made of multiple static inline helper functions, in the attempt to keep checks outside of core loop for better performance.
2024-04-26 07:53:15 -04:00
char *ret;
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
const char *pending;
uint8_t cbor_string_prefix = 0;
MINOR: log: merge lf_encode_string() and lf_encode_chunk() logic lf_encode_string() and lf_encode_chunk() function are pretty similar. The only difference is the stopping behavior, encode_chunk stops at a given position while encode_string stops when encountering '\0'. Moreover, both functions leverage tools.c encode helpers, but because of the LOG_OPT_ESC option, they reimplement those helpers with added logic. Instead of having to deal with code duplication which makes both functions harder to maintain, let's define a _lf_encode_bytes() helper function which satisfies lf_encode_string() and lf_encode_chunk() needs while keeping the function as simple as possible. _lf_encode_bytes() itself is made of multiple static inline helper functions, in the attempt to keep checks outside of core loop for better performance.
2024-04-26 07:53:15 -04:00
char *(*encode_byte)(char *start, char *stop,
const char *byte,
const char escape, const long *map,
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
const char **pending, uint8_t cbor_string_prefix,
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
struct lf_buildctx *ctx);
MINOR: log: merge lf_encode_string() and lf_encode_chunk() logic lf_encode_string() and lf_encode_chunk() function are pretty similar. The only difference is the stopping behavior, encode_chunk stops at a given position while encode_string stops when encountering '\0'. Moreover, both functions leverage tools.c encode helpers, but because of the LOG_OPT_ESC option, they reimplement those helpers with added logic. Instead of having to deal with code duplication which makes both functions harder to maintain, let's define a _lf_encode_bytes() helper function which satisfies lf_encode_string() and lf_encode_chunk() needs while keeping the function as simple as possible. _lf_encode_bytes() itself is made of multiple static inline helper functions, in the attempt to keep checks outside of core loop for better performance.
2024-04-26 07:53:15 -04:00
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
if (ctx->options & LOG_OPT_ENCODE_JSON)
encode_byte = _lf_json_escape_byte;
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
else if (ctx->options & LOG_OPT_ENCODE_CBOR)
encode_byte = _lf_cbor_map_escape_byte;
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
else if (ctx->options & LOG_OPT_ESC)
MINOR: log: merge lf_encode_string() and lf_encode_chunk() logic lf_encode_string() and lf_encode_chunk() function are pretty similar. The only difference is the stopping behavior, encode_chunk stops at a given position while encode_string stops when encountering '\0'. Moreover, both functions leverage tools.c encode helpers, but because of the LOG_OPT_ESC option, they reimplement those helpers with added logic. Instead of having to deal with code duplication which makes both functions harder to maintain, let's define a _lf_encode_bytes() helper function which satisfies lf_encode_string() and lf_encode_chunk() needs while keeping the function as simple as possible. _lf_encode_bytes() itself is made of multiple static inline helper functions, in the attempt to keep checks outside of core loop for better performance.
2024-04-26 07:53:15 -04:00
encode_byte = _lf_rfc5424_escape_byte;
else
encode_byte = _lf_map_escape_byte;
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
if (ctx->options & LOG_OPT_ENCODE_CBOR) {
if (!bytes_stop) {
/* printable chars: use cbor text */
cbor_string_prefix = 0x60;
}
else {
/* non printable chars: use cbor byte string */
cbor_string_prefix = 0x40;
}
}
MINOR: log: merge lf_encode_string() and lf_encode_chunk() logic lf_encode_string() and lf_encode_chunk() function are pretty similar. The only difference is the stopping behavior, encode_chunk stops at a given position while encode_string stops when encountering '\0'. Moreover, both functions leverage tools.c encode helpers, but because of the LOG_OPT_ESC option, they reimplement those helpers with added logic. Instead of having to deal with code duplication which makes both functions harder to maintain, let's define a _lf_encode_bytes() helper function which satisfies lf_encode_string() and lf_encode_chunk() needs while keeping the function as simple as possible. _lf_encode_bytes() itself is made of multiple static inline helper functions, in the attempt to keep checks outside of core loop for better performance.
2024-04-26 07:53:15 -04:00
if (start < stop) {
stop--; /* reserve one byte for the final '\0' */
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
if ((ctx->options & LOG_OPT_ENCODE_CBOR) && !ctx->in_text) {
/* start indefinite-length cbor byte string or text */
start = _lf_cbor_encode_byte(&ctx->encode.cbor, start, stop,
(cbor_string_prefix | 0x1F));
if (start == NULL)
return NULL;
}
pending = bytes;
MINOR: log: merge lf_encode_string() and lf_encode_chunk() logic lf_encode_string() and lf_encode_chunk() function are pretty similar. The only difference is the stopping behavior, encode_chunk stops at a given position while encode_string stops when encountering '\0'. Moreover, both functions leverage tools.c encode helpers, but because of the LOG_OPT_ESC option, they reimplement those helpers with added logic. Instead of having to deal with code duplication which makes both functions harder to maintain, let's define a _lf_encode_bytes() helper function which satisfies lf_encode_string() and lf_encode_chunk() needs while keeping the function as simple as possible. _lf_encode_bytes() itself is made of multiple static inline helper functions, in the attempt to keep checks outside of core loop for better performance.
2024-04-26 07:53:15 -04:00
/* we have 2 distinct loops to keep checks outside of the loop
* for better performance
*/
if (bytes && !bytes_stop) {
while (start < stop && *bytes != '\0') {
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
ret = encode_byte(start, stop, bytes, escape, map,
&pending, cbor_string_prefix,
ctx);
MINOR: log: merge lf_encode_string() and lf_encode_chunk() logic lf_encode_string() and lf_encode_chunk() function are pretty similar. The only difference is the stopping behavior, encode_chunk stops at a given position while encode_string stops when encountering '\0'. Moreover, both functions leverage tools.c encode helpers, but because of the LOG_OPT_ESC option, they reimplement those helpers with added logic. Instead of having to deal with code duplication which makes both functions harder to maintain, let's define a _lf_encode_bytes() helper function which satisfies lf_encode_string() and lf_encode_chunk() needs while keeping the function as simple as possible. _lf_encode_bytes() itself is made of multiple static inline helper functions, in the attempt to keep checks outside of core loop for better performance.
2024-04-26 07:53:15 -04:00
if (ret == NULL)
break;
start = ret;
bytes++;
}
} else if (bytes) {
while (start < stop && bytes < bytes_stop) {
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
ret = encode_byte(start, stop, bytes, escape, map,
&pending, cbor_string_prefix,
ctx);
MINOR: log: merge lf_encode_string() and lf_encode_chunk() logic lf_encode_string() and lf_encode_chunk() function are pretty similar. The only difference is the stopping behavior, encode_chunk stops at a given position while encode_string stops when encountering '\0'. Moreover, both functions leverage tools.c encode helpers, but because of the LOG_OPT_ESC option, they reimplement those helpers with added logic. Instead of having to deal with code duplication which makes both functions harder to maintain, let's define a _lf_encode_bytes() helper function which satisfies lf_encode_string() and lf_encode_chunk() needs while keeping the function as simple as possible. _lf_encode_bytes() itself is made of multiple static inline helper functions, in the attempt to keep checks outside of core loop for better performance.
2024-04-26 07:53:15 -04:00
if (ret == NULL)
break;
start = ret;
bytes++;
MEDIUM: log: add a new log format flag "E" The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It mostly makes sense to use it in the RFC5424 structured-data log formats. Example: log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 07:23:03 -05:00
}
}
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
if (ctx->options & LOG_OPT_ENCODE_CBOR) {
if (pending != bytes) {
/* flush pending unescaped bytes */
start = cbor_encode_bytes_prefix(&ctx->encode.cbor, start, stop,
pending, (bytes - pending),
cbor_string_prefix);
if (start == NULL)
return NULL;
}
if (!ctx->in_text) {
/* cbor break (to end indefinite-length text or byte string) */
start = _lf_cbor_encode_byte(&ctx->encode.cbor, start, stop, 0xFF);
if (start == NULL)
return NULL;
}
}
MINOR: log: merge lf_encode_string() and lf_encode_chunk() logic lf_encode_string() and lf_encode_chunk() function are pretty similar. The only difference is the stopping behavior, encode_chunk stops at a given position while encode_string stops when encountering '\0'. Moreover, both functions leverage tools.c encode helpers, but because of the LOG_OPT_ESC option, they reimplement those helpers with added logic. Instead of having to deal with code duplication which makes both functions harder to maintain, let's define a _lf_encode_bytes() helper function which satisfies lf_encode_string() and lf_encode_chunk() needs while keeping the function as simple as possible. _lf_encode_bytes() itself is made of multiple static inline helper functions, in the attempt to keep checks outside of core loop for better performance.
2024-04-26 07:53:15 -04:00
*start = '\0';
return start;
MEDIUM: log: add a new log format flag "E" The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It mostly makes sense to use it in the RFC5424 structured-data log formats. Example: log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 07:23:03 -05:00
}
BUG/MINOR: tools/log: invalid encode_{chunk,string} usage encode_{chunk,string}() is often found to be used this way: ret = encode_{chunk,string}(start, stop...) if (ret == NULL || *ret != '\0') { //error } //success Indeed, encode_{chunk,string} will always try to add terminating NULL byte to the output string, unless no space is available for even 1 byte. However, it means that for the caller to be able to spot an error, then it must provide a buffer (here: start) which is already initialized. But this is wrong: not only this is very tricky to use, but since those functions don't return NULL on failure, then if the output buffer was not properly initialized prior to calling the function, the caller will perform invalid reads when checking for failure this way. Moreover, even if the buffer is initialized, we cannot reliably tell if the function actually failed this way because if the buffer was previously initialized with NULL byte, then the caller might think that the call actually succeeded (since the function didn't return NULL and didn't update the buffer). Also, sess_build_logline() relies lf_encode_{chunk,string}() functions which are in fact wrappers for encode_{chunk,string}() functions and thus exhibit the same error handling mechanism. It turns out that sess_build_logline() makes unsafe use of those functions because it uses the error-checking logic mentionned above while buffer (tmplog) is not guaranteed to be initialized when entering the function. This may ultimately cause malfunctions or invalid reads if the output buffer is lacking space. To fix the issue once and for all and prevent similar bugs from being introduced, we make it so encode_{string, chunk} and escape_string() (based on encode_string()) now explicitly return NULL on failure (when the function failed to write at least the ending NULL byte) lf_encode_{string,chunk}() helpers had to be patched as well due to code duplication. This should be backported to all stable versions. [ada: for 2.4 and 2.6 the patch won't apply as-is, it might be helpful to backport ae1e14d65 ("CLEANUP: tools: removing escape_chunk() function") first, considering it's not very relevant to maintain a dead function]
2024-04-09 05:44:54 -04:00
return NULL;
MEDIUM: log: add a new log format flag "E" The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It mostly makes sense to use it in the RFC5424 structured-data log formats. Example: log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 07:23:03 -05:00
}
MINOR: log: merge lf_encode_string() and lf_encode_chunk() logic lf_encode_string() and lf_encode_chunk() function are pretty similar. The only difference is the stopping behavior, encode_chunk stops at a given position while encode_string stops when encountering '\0'. Moreover, both functions leverage tools.c encode helpers, but because of the LOG_OPT_ESC option, they reimplement those helpers with added logic. Instead of having to deal with code duplication which makes both functions harder to maintain, let's define a _lf_encode_bytes() helper function which satisfies lf_encode_string() and lf_encode_chunk() needs while keeping the function as simple as possible. _lf_encode_bytes() itself is made of multiple static inline helper functions, in the attempt to keep checks outside of core loop for better performance.
2024-04-26 07:53:15 -04:00
/*
* Encode the string.
*/
static char *lf_encode_string(char *start, char *stop,
const char escape, const long *map,
const char *string,
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
struct lf_buildctx *ctx)
MINOR: log: merge lf_encode_string() and lf_encode_chunk() logic lf_encode_string() and lf_encode_chunk() function are pretty similar. The only difference is the stopping behavior, encode_chunk stops at a given position while encode_string stops when encountering '\0'. Moreover, both functions leverage tools.c encode helpers, but because of the LOG_OPT_ESC option, they reimplement those helpers with added logic. Instead of having to deal with code duplication which makes both functions harder to maintain, let's define a _lf_encode_bytes() helper function which satisfies lf_encode_string() and lf_encode_chunk() needs while keeping the function as simple as possible. _lf_encode_bytes() itself is made of multiple static inline helper functions, in the attempt to keep checks outside of core loop for better performance.
2024-04-26 07:53:15 -04:00
{
return _lf_encode_bytes(start, stop, escape, map,
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
string, NULL, ctx);
MINOR: log: merge lf_encode_string() and lf_encode_chunk() logic lf_encode_string() and lf_encode_chunk() function are pretty similar. The only difference is the stopping behavior, encode_chunk stops at a given position while encode_string stops when encountering '\0'. Moreover, both functions leverage tools.c encode helpers, but because of the LOG_OPT_ESC option, they reimplement those helpers with added logic. Instead of having to deal with code duplication which makes both functions harder to maintain, let's define a _lf_encode_bytes() helper function which satisfies lf_encode_string() and lf_encode_chunk() needs while keeping the function as simple as possible. _lf_encode_bytes() itself is made of multiple static inline helper functions, in the attempt to keep checks outside of core loop for better performance.
2024-04-26 07:53:15 -04:00
}
MEDIUM: log: add a new log format flag "E" The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It mostly makes sense to use it in the RFC5424 structured-data log formats. Example: log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 07:23:03 -05:00
/*
* Encode the chunk.
*/
static char *lf_encode_chunk(char *start, char *stop,
MINOR: logs: use the new bitmap functions instead of fd_sets for encoding maps The fd_sets we've been using in the log encoding functions are not portable and were shown to break at least under Cygwin. This patch gets rid of them in favor of the new bitmap functions. It was verified with the config below that the log output was exactly the same before and after the change : defaults mode http option httplog log stdout local0 timeout client 1s timeout server 1s timeout connect 1s frontend foo bind :8001 capture request header chars len 255 backend bar option httpchk "GET" "/" "HTTP/1.0\r\nchars: \x01\x02\x03\x04\x05\x06\x07\x08\x09\x0b\x0c\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f\x20\x21\x22\x23\x24\x25\x26\x27\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f\x30\x31\x32\x33\x34\x35\x36\x37\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f\x40\x41\x42\x43\x44\x45\x46\x47\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f\x50\x51\x52\x53\x54\x55\x56\x57\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f\x60\x61\x62\x63\x64\x65\x66\x67\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f\x70\x71\x72\x73\x74\x75\x76\x77\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" server foo 127.0.0.1:8001 check
2019-06-07 05:10:07 -04:00
const char escape, const long *map,
MAJOR: chunks: replace struct chunk with struct buffer Now all the code used to manipulate chunks uses a struct buffer instead. The functions are still called "chunk*", and some of them will progressively move to the generic buffer handling code as they are cleaned up.
2018-07-13 05:56:34 -04:00
const struct buffer *chunk,
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
struct lf_buildctx *ctx)
MEDIUM: log: add a new log format flag "E" The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It mostly makes sense to use it in the RFC5424 structured-data log formats. Example: log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 07:23:03 -05:00
{
MINOR: log: merge lf_encode_string() and lf_encode_chunk() logic lf_encode_string() and lf_encode_chunk() function are pretty similar. The only difference is the stopping behavior, encode_chunk stops at a given position while encode_string stops when encountering '\0'. Moreover, both functions leverage tools.c encode helpers, but because of the LOG_OPT_ESC option, they reimplement those helpers with added logic. Instead of having to deal with code duplication which makes both functions harder to maintain, let's define a _lf_encode_bytes() helper function which satisfies lf_encode_string() and lf_encode_chunk() needs while keeping the function as simple as possible. _lf_encode_bytes() itself is made of multiple static inline helper functions, in the attempt to keep checks outside of core loop for better performance.
2024-04-26 07:53:15 -04:00
return _lf_encode_bytes(start, stop, escape, map,
chunk->area, chunk->area + chunk->data,
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
ctx);
MEDIUM: log: add a new log format flag "E" The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It mostly makes sense to use it in the RFC5424 structured-data log formats. Example: log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 07:23:03 -05:00
}
MEDIUM: log: make http_sess_log use log_format http_sess_log now use the logformat linked list to make the log string, snprintf is not used for speed issue. CLF mode also uses logformat. NOTE: as of now, empty fields in CLF now are "" not "-" anymore.
2012-02-08 10:38:44 -05:00
/*
MINOR: log: add lf_rawtext{_len}() functions similar to lf_text_{len}, except that quoting and mandatory options are ignored. Use this to print the input string without any modification ( except for encoding logic).
2024-04-10 06:07:26 -04:00
* Write a raw string in the log string
* Take care of escape option
MEDIUM: log: make http_sess_log use log_format http_sess_log now use the logformat linked list to make the log string, snprintf is not used for speed issue. CLF mode also uses logformat. NOTE: as of now, empty fields in CLF now are "" not "-" anymore.
2012-02-08 10:38:44 -05:00
*
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
* When using json encoding, string will be escaped according
* to json escape map
*
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
* When using cbor encoding, escape option is ignored.
*
CLEANUP: Fix typos in the log subsystem Fix some misspells in the code comments of the log subsystem.
2018-11-15 15:10:04 -05:00
* Return the address of the \0 character, or NULL on error
MEDIUM: log: make http_sess_log use log_format http_sess_log now use the logformat linked list to make the log string, snprintf is not used for speed issue. CLF mode also uses logformat. NOTE: as of now, empty fields in CLF now are "" not "-" anymore.
2012-02-08 10:38:44 -05:00
*/
MINOR: log: add lf_rawtext{_len}() functions similar to lf_text_{len}, except that quoting and mandatory options are ignored. Use this to print the input string without any modification ( except for encoding logic).
2024-04-10 06:07:26 -04:00
static inline char *_lf_text_len(char *dst, const char *src,
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
size_t len, size_t size, struct lf_buildctx *ctx)
MEDIUM: log: make http_sess_log use log_format http_sess_log now use the logformat linked list to make the log string, snprintf is not used for speed issue. CLF mode also uses logformat. NOTE: as of now, empty fields in CLF now are "" not "-" anymore.
2012-02-08 10:38:44 -05:00
{
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
const long *escape_map = NULL;
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
char *ret;
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
if (ctx->options & LOG_OPT_ENCODE_JSON)
escape_map = json_escape_map;
else if (ctx->options & LOG_OPT_ESC)
escape_map = rfc5424_escape_map;
BUG/MEDIUM: log: emit '-' for empty fields again Commit 2b0108ad accidently got rid of the ability to emit a "-" for empty log fields. This can happen for captured request and response cookies, as well as for fetches. Since we don't want to have this done for headers however, we set the default log method when parsing the format. It is still possible to force the desired mode using +M/-M.
2013-02-05 12:52:25 -05:00
if (src && len) {
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
if (ctx->options & LOG_OPT_ENCODE_CBOR) {
/* it's actually less costly to compute the actual text size to
* write a single fixed length text at once rather than emitting
* indefinite length text in cbor, because indefinite-length text
* has to be made of multiple chunks of known size as per RFC8949...
*/
BUILD: log: get rid of non-portable strnlen() func In c614fd3b9 ("MINOR: log: add +cbor encoding option"), I wrongly used strnlen() without noticing that the function is not portable (requires _POSIX_C_SOURCE >= 2008) and that it was the first occurrence in the entire project. In fact it is not a hard requirement since it's a pretty simple function. Thus to restore build compatibility with minimal/older build systems, let's actually get rid of it and use an equivalent portable code where needed (we cannot simply rely on strlen() because the string might not be NULL terminated, we must take upstream len into account). No backport needed (unless c614fd3b9 gets backported)
2024-05-17 04:24:33 -04:00
{
int _len;
/* strnlen(src, len) portable equivalent: */
for (_len = 0; _len < len && src[_len]; _len++)
;
len = _len;
}
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
ret = cbor_encode_text(&ctx->encode.cbor, dst, dst + size, src, len);
if (ret == NULL)
return NULL;
len = ret - dst;
}
BUG/MINOR: log: improper behavior when escaping log data Patrick Hemmer reported an improper log behavior when using log-format to escape log data (+E option): Some bytes were truncated from the output: - escape_string() function now takes an extra parameter that allow the caller to specify input string stop pointer in case the input string is not guaranteed to be zero-terminated. - Minors checks were added into lf_text_len() to make sure dst string will not overflow. - lf_text_len() now makes proper use of escape_string() function. This should be backported as far as 1.8.
2022-09-20 08:33:18 -04:00
/* escape_string and strlcpy2 will both try to add terminating NULL-byte
BUG/MINOR: log: fix lf_text_len() truncate inconsistency In c5bff8e550 ("BUG/MINOR: log: improper behavior when escaping log data") we fixed lf_text_len() behavior with +E (escape) option. However we introduced an inconsistency if output buffer is too small to hold the whole output and truncation occurs: indeed without +E option up to <size> bytes (including NULL byte) will be used whereas with +E option only <size-1> bytes will be used. Fixing the function and related comment so that the function behaves the same in regards to truncation whether +E option is used or not. This should be backported to all stable versions.
2024-04-09 05:15:23 -04:00
* to dst
BUG/MINOR: log: improper behavior when escaping log data Patrick Hemmer reported an improper log behavior when using log-format to escape log data (+E option): Some bytes were truncated from the output: - escape_string() function now takes an extra parameter that allow the caller to specify input string stop pointer in case the input string is not guaranteed to be zero-terminated. - Minors checks were added into lf_text_len() to make sure dst string will not overflow. - lf_text_len() now makes proper use of escape_string() function. This should be backported as far as 1.8.
2022-09-20 08:33:18 -04:00
*/
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
else if (escape_map) {
MEDIUM: log: add a new log format flag "E" The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It mostly makes sense to use it in the RFC5424 structured-data log formats. Example: log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 07:23:03 -05:00
char *ret;
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
ret = escape_string(dst, dst + size, '\\', escape_map, src, src + len);
CLEANUP: log: remove unused checks for encode_{chunk,string} Thanks to 8226e92eb ("BUG/MINOR: tools/log: invalid encode_{chunk,string} usage"), we only need to check for NULL return value from encode_{chunk,string}() and escape_string() to know if the call failed.
2024-04-09 11:54:41 -04:00
if (ret == NULL)
MEDIUM: log: add a new log format flag "E" The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It mostly makes sense to use it in the RFC5424 structured-data log formats. Example: log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 07:23:03 -05:00
return NULL;
len = ret - dst;
}
else {
BUG/MINOR: log: improper behavior when escaping log data Patrick Hemmer reported an improper log behavior when using log-format to escape log data (+E option): Some bytes were truncated from the output: - escape_string() function now takes an extra parameter that allow the caller to specify input string stop pointer in case the input string is not guaranteed to be zero-terminated. - Minors checks were added into lf_text_len() to make sure dst string will not overflow. - lf_text_len() now makes proper use of escape_string() function. This should be backported as far as 1.8.
2022-09-20 08:33:18 -04:00
if (++len > size)
len = size;
MEDIUM: log: add a new log format flag "E" The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It mostly makes sense to use it in the RFC5424 structured-data log formats. Example: log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 07:23:03 -05:00
len = strlcpy2(dst, src, len);
MINOR: log: add lf_rawtext{_len}() functions similar to lf_text_{len}, except that quoting and mandatory options are ignored. Use this to print the input string without any modification ( except for encoding logic).
2024-04-10 06:07:26 -04:00
if (len == 0)
return NULL;
MEDIUM: log: add a new log format flag "E" The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It mostly makes sense to use it in the RFC5424 structured-data log formats. Example: log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 07:23:03 -05:00
}
MINOR: log: add lf_text_len This function allows to log a text of a specific length.
2012-12-21 13:23:44 -05:00
dst += len;
MINOR: log: add lf_rawtext{_len}() functions similar to lf_text_{len}, except that quoting and mandatory options are ignored. Use this to print the input string without any modification ( except for encoding logic).
2024-04-10 06:07:26 -04:00
size -= len;
MINOR: log: add lf_text_len This function allows to log a text of a specific length.
2012-12-21 13:23:44 -05:00
}
MINOR: log: add lf_rawtext{_len}() functions similar to lf_text_{len}, except that quoting and mandatory options are ignored. Use this to print the input string without any modification ( except for encoding logic).
2024-04-10 06:07:26 -04:00
if (size < 1)
return NULL;
*dst = '\0';
return dst;
}
/*
* Quote a string, then leverage _lf_text_len() to write it
*/
static inline char *_lf_quotetext_len(char *dst, const char *src,
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
size_t len, size_t size, struct lf_buildctx *ctx)
MINOR: log: add lf_rawtext{_len}() functions similar to lf_text_{len}, except that quoting and mandatory options are ignored. Use this to print the input string without any modification ( except for encoding logic).
2024-04-10 06:07:26 -04:00
{
if (size < 2)
return NULL;
*(dst++) = '"';
size--;
if (src && len) {
char *ret;
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
ret = _lf_text_len(dst, src, len, size, ctx);
MINOR: log: add lf_rawtext{_len}() functions similar to lf_text_{len}, except that quoting and mandatory options are ignored. Use this to print the input string without any modification ( except for encoding logic).
2024-04-10 06:07:26 -04:00
if (ret == NULL)
MINOR: log: add lf_text_len This function allows to log a text of a specific length.
2012-12-21 13:23:44 -05:00
return NULL;
MINOR: log: add lf_rawtext{_len}() functions similar to lf_text_{len}, except that quoting and mandatory options are ignored. Use this to print the input string without any modification ( except for encoding logic).
2024-04-10 06:07:26 -04:00
size -= (ret - dst);
dst += (ret - dst);
MINOR: log: add lf_text_len This function allows to log a text of a specific length.
2012-12-21 13:23:44 -05:00
}
MINOR: log: add lf_rawtext{_len}() functions similar to lf_text_{len}, except that quoting and mandatory options are ignored. Use this to print the input string without any modification ( except for encoding logic).
2024-04-10 06:07:26 -04:00
if (size < 2)
return NULL;
*(dst++) = '"';
*dst = '\0';
return dst;
}
/*
* Write a string in the log string
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
* Take care of quote, mandatory and escape and encoding options
MINOR: log: add lf_rawtext{_len}() functions similar to lf_text_{len}, except that quoting and mandatory options are ignored. Use this to print the input string without any modification ( except for encoding logic).
2024-04-10 06:07:26 -04:00
*
* Return the address of the \0 character, or NULL on error
*/
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
static char *lf_text_len(char *dst, const char *src, size_t len, size_t size, struct lf_buildctx *ctx)
MINOR: log: add lf_rawtext{_len}() functions similar to lf_text_{len}, except that quoting and mandatory options are ignored. Use this to print the input string without any modification ( except for encoding logic).
2024-04-10 06:07:26 -04:00
{
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
if ((ctx->options & (LOG_OPT_QUOTE | LOG_OPT_ENCODE_JSON)))
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
return _lf_quotetext_len(dst, src, len, size, ctx);
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
else if ((ctx->options & LOG_OPT_ENCODE_CBOR) ||
(src && len))
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
return _lf_text_len(dst, src, len, size, ctx);
MINOR: log: add lf_rawtext{_len}() functions similar to lf_text_{len}, except that quoting and mandatory options are ignored. Use this to print the input string without any modification ( except for encoding logic).
2024-04-10 06:07:26 -04:00
if (size < 2)
return NULL;
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
if ((ctx->options & LOG_OPT_MANDATORY))
return _lf_text_len(dst, "-", 1, size, ctx);
MINOR: log: add lf_rawtext{_len}() functions similar to lf_text_{len}, except that quoting and mandatory options are ignored. Use this to print the input string without any modification ( except for encoding logic).
2024-04-10 06:07:26 -04:00
MINOR: log: add lf_text_len This function allows to log a text of a specific length.
2012-12-21 13:23:44 -05:00
*dst = '\0';
MINOR: log: add lf_rawtext{_len}() functions similar to lf_text_{len}, except that quoting and mandatory options are ignored. Use this to print the input string without any modification ( except for encoding logic).
2024-04-10 06:07:26 -04:00
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
return dst;
MEDIUM: log: make http_sess_log use log_format http_sess_log now use the logformat linked list to make the log string, snprintf is not used for speed issue. CLF mode also uses logformat. NOTE: as of now, empty fields in CLF now are "" not "-" anymore.
2012-02-08 10:38:44 -05:00
}
MINOR: log: add lf_rawtext{_len}() functions similar to lf_text_{len}, except that quoting and mandatory options are ignored. Use this to print the input string without any modification ( except for encoding logic).
2024-04-10 06:07:26 -04:00
/*
* Same as lf_text_len() except that it ignores mandatory and quoting options.
* Quoting is only performed when strictly required by the encoding method.
*/
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
static char *lf_rawtext_len(char *dst, const char *src, size_t len, size_t size, struct lf_buildctx *ctx)
MINOR: log: add lf_rawtext{_len}() functions similar to lf_text_{len}, except that quoting and mandatory options are ignored. Use this to print the input string without any modification ( except for encoding logic).
2024-04-10 06:07:26 -04:00
{
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
if (!ctx->in_text &&
(ctx->options & LOG_OPT_ENCODE_JSON))
return _lf_quotetext_len(dst, src, len, size, ctx);
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
return _lf_text_len(dst, src, len, size, ctx);
MINOR: log: add lf_rawtext{_len}() functions similar to lf_text_{len}, except that quoting and mandatory options are ignored. Use this to print the input string without any modification ( except for encoding logic).
2024-04-10 06:07:26 -04:00
}
/* lf_text_len() helper when <src> is null-byte terminated */
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
static inline char *lf_text(char *dst, const char *src, size_t size, struct lf_buildctx *ctx)
MINOR: log: add lf_text_len This function allows to log a text of a specific length.
2012-12-21 13:23:44 -05:00
{
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
return lf_text_len(dst, src, size, size, ctx);
MINOR: log: add lf_text_len This function allows to log a text of a specific length.
2012-12-21 13:23:44 -05:00
}
MINOR: log: add lf_rawtext{_len}() functions similar to lf_text_{len}, except that quoting and mandatory options are ignored. Use this to print the input string without any modification ( except for encoding logic).
2024-04-10 06:07:26 -04:00
/* lf_rawtext_len() helper when <src> is null-byte terminated */
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
static inline char *lf_rawtext(char *dst, const char *src, size_t size, struct lf_buildctx *ctx)
MINOR: log: add lf_rawtext{_len}() functions similar to lf_text_{len}, except that quoting and mandatory options are ignored. Use this to print the input string without any modification ( except for encoding logic).
2024-04-10 06:07:26 -04:00
{
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
return lf_rawtext_len(dst, src, size, size, ctx);
MINOR: log: add lf_rawtext{_len}() functions similar to lf_text_{len}, except that quoting and mandatory options are ignored. Use this to print the input string without any modification ( except for encoding logic).
2024-04-10 06:07:26 -04:00
}
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
/*
CLEANUP: Fix typos in the log subsystem Fix some misspells in the code comments of the log subsystem.
2018-11-15 15:10:04 -05:00
* Write a IP address to the log string
CLEANUP: assorted typo fixes in the code and comments This is 8th iteration of typo fixes
2020-04-16 14:51:34 -04:00
* +X option write in hexadecimal notation, most significant byte on the left
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
*/
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
static char *lf_ip(char *dst, const struct sockaddr *sockaddr, size_t size, struct lf_buildctx *ctx)
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
{
char *ret = dst;
int iret;
char pn[INET6_ADDRSTRLEN];
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
if (ctx->options & LOG_OPT_HEXA) {
BUG/MINOR: log: properly format IPv6 address when LOG_OPT_HEXA modifier is used. In lf_ip(), when LOG_OPT_HEXA modifier is used, there is a code to format the IP address as a hexadecimal string. This code does not properly handle cases when the IP address is IPv6. In such case, the code only prints `00000000`. This patch adds support for IPv6. For legacy IPv4, the format remains unchanged. If IPv6 socket is used to accept IPv6 connection, the full IPv6 address is returned. For example, IPv6 localhost, ::1, is printed as 00000000000000000000000000000001. If IPv6 socket accepts IPv4 connection, the IPv4 address is mapped by the kernel into the IPv4-mapped-IPv6 address space (RFC4291, section 2.5.5.2) and is formatted as such. For example, 127.0.0.1 becomes ::ffff:127.0.0.1, which is printed as 00000000000000000000FFFF7F000001. This should be backported to 1.9.
2019-03-22 06:21:54 -04:00
unsigned char *addr = NULL;
switch (sockaddr->sa_family) {
case AF_INET:
MEDIUM: log: use lf_rawtext for lf_ip() and lf_port() hex strings Same as the previous commit, but for ip and port oriented values when +X option is provided. No functional change should be expected. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice, we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-04-03 09:42:35 -04:00
{
BUG/MINOR: log: properly format IPv6 address when LOG_OPT_HEXA modifier is used. In lf_ip(), when LOG_OPT_HEXA modifier is used, there is a code to format the IP address as a hexadecimal string. This code does not properly handle cases when the IP address is IPv6. In such case, the code only prints `00000000`. This patch adds support for IPv6. For legacy IPv4, the format remains unchanged. If IPv6 socket is used to accept IPv6 connection, the full IPv6 address is returned. For example, IPv6 localhost, ::1, is printed as 00000000000000000000000000000001. If IPv6 socket accepts IPv4 connection, the IPv4 address is mapped by the kernel into the IPv4-mapped-IPv6 address space (RFC4291, section 2.5.5.2) and is formatted as such. For example, 127.0.0.1 becomes ::ffff:127.0.0.1, which is printed as 00000000000000000000FFFF7F000001. This should be backported to 1.9.
2019-03-22 06:21:54 -04:00
addr = (unsigned char *)&((struct sockaddr_in *)sockaddr)->sin_addr.s_addr;
OPTIM: log: use lf_buildctx's buffer instead of temporary stack buffers Now that lf_buildctx isn't pushed on the stack anymore, let's take this opportunity to store a small buffer of 256 bytes within it, and then use this buffer as general purpose buffer to build fixed-length strings that are then printed using lf_{raw}text() function. By doing so we stop relying on temporary stack buffers.
2024-05-02 10:48:56 -04:00
iret = snprintf(ctx->_buf, sizeof(ctx->_buf), "%02X%02X%02X%02X",
MEDIUM: log: use lf_rawtext for lf_ip() and lf_port() hex strings Same as the previous commit, but for ip and port oriented values when +X option is provided. No functional change should be expected. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice, we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-04-03 09:42:35 -04:00
addr[0], addr[1], addr[2], addr[3]);
if (iret < 0 || iret >= size)
return NULL;
OPTIM: log: use lf_buildctx's buffer instead of temporary stack buffers Now that lf_buildctx isn't pushed on the stack anymore, let's take this opportunity to store a small buffer of 256 bytes within it, and then use this buffer as general purpose buffer to build fixed-length strings that are then printed using lf_{raw}text() function. By doing so we stop relying on temporary stack buffers.
2024-05-02 10:48:56 -04:00
ret = lf_rawtext(dst, ctx->_buf, size, ctx);
MEDIUM: log: use lf_rawtext for lf_ip() and lf_port() hex strings Same as the previous commit, but for ip and port oriented values when +X option is provided. No functional change should be expected. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice, we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-04-03 09:42:35 -04:00
BUG/MINOR: log: properly format IPv6 address when LOG_OPT_HEXA modifier is used. In lf_ip(), when LOG_OPT_HEXA modifier is used, there is a code to format the IP address as a hexadecimal string. This code does not properly handle cases when the IP address is IPv6. In such case, the code only prints `00000000`. This patch adds support for IPv6. For legacy IPv4, the format remains unchanged. If IPv6 socket is used to accept IPv6 connection, the full IPv6 address is returned. For example, IPv6 localhost, ::1, is printed as 00000000000000000000000000000001. If IPv6 socket accepts IPv4 connection, the IPv4 address is mapped by the kernel into the IPv4-mapped-IPv6 address space (RFC4291, section 2.5.5.2) and is formatted as such. For example, 127.0.0.1 becomes ::ffff:127.0.0.1, which is printed as 00000000000000000000FFFF7F000001. This should be backported to 1.9.
2019-03-22 06:21:54 -04:00
break;
MEDIUM: log: use lf_rawtext for lf_ip() and lf_port() hex strings Same as the previous commit, but for ip and port oriented values when +X option is provided. No functional change should be expected. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice, we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-04-03 09:42:35 -04:00
}
BUG/MINOR: log: properly format IPv6 address when LOG_OPT_HEXA modifier is used. In lf_ip(), when LOG_OPT_HEXA modifier is used, there is a code to format the IP address as a hexadecimal string. This code does not properly handle cases when the IP address is IPv6. In such case, the code only prints `00000000`. This patch adds support for IPv6. For legacy IPv4, the format remains unchanged. If IPv6 socket is used to accept IPv6 connection, the full IPv6 address is returned. For example, IPv6 localhost, ::1, is printed as 00000000000000000000000000000001. If IPv6 socket accepts IPv4 connection, the IPv4 address is mapped by the kernel into the IPv4-mapped-IPv6 address space (RFC4291, section 2.5.5.2) and is formatted as such. For example, 127.0.0.1 becomes ::ffff:127.0.0.1, which is printed as 00000000000000000000FFFF7F000001. This should be backported to 1.9.
2019-03-22 06:21:54 -04:00
case AF_INET6:
MEDIUM: log: use lf_rawtext for lf_ip() and lf_port() hex strings Same as the previous commit, but for ip and port oriented values when +X option is provided. No functional change should be expected. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice, we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-04-03 09:42:35 -04:00
{
BUG/MINOR: log: properly format IPv6 address when LOG_OPT_HEXA modifier is used. In lf_ip(), when LOG_OPT_HEXA modifier is used, there is a code to format the IP address as a hexadecimal string. This code does not properly handle cases when the IP address is IPv6. In such case, the code only prints `00000000`. This patch adds support for IPv6. For legacy IPv4, the format remains unchanged. If IPv6 socket is used to accept IPv6 connection, the full IPv6 address is returned. For example, IPv6 localhost, ::1, is printed as 00000000000000000000000000000001. If IPv6 socket accepts IPv4 connection, the IPv4 address is mapped by the kernel into the IPv4-mapped-IPv6 address space (RFC4291, section 2.5.5.2) and is formatted as such. For example, 127.0.0.1 becomes ::ffff:127.0.0.1, which is printed as 00000000000000000000FFFF7F000001. This should be backported to 1.9.
2019-03-22 06:21:54 -04:00
addr = (unsigned char *)&((struct sockaddr_in6 *)sockaddr)->sin6_addr.s6_addr;
OPTIM: log: use lf_buildctx's buffer instead of temporary stack buffers Now that lf_buildctx isn't pushed on the stack anymore, let's take this opportunity to store a small buffer of 256 bytes within it, and then use this buffer as general purpose buffer to build fixed-length strings that are then printed using lf_{raw}text() function. By doing so we stop relying on temporary stack buffers.
2024-05-02 10:48:56 -04:00
iret = snprintf(ctx->_buf, sizeof(ctx->_buf),
MEDIUM: log: use lf_rawtext for lf_ip() and lf_port() hex strings Same as the previous commit, but for ip and port oriented values when +X option is provided. No functional change should be expected. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice, we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-04-03 09:42:35 -04:00
"%02X%02X%02X%02X%02X%02X%02X%02X%02X%02X%02X%02X%02X%02X%02X%02X",
addr[0], addr[1], addr[2], addr[3],
addr[4], addr[5], addr[6], addr[7],
addr[8], addr[9], addr[10], addr[11],
addr[12], addr[13], addr[14], addr[15]);
if (iret < 0 || iret >= size)
return NULL;
OPTIM: log: use lf_buildctx's buffer instead of temporary stack buffers Now that lf_buildctx isn't pushed on the stack anymore, let's take this opportunity to store a small buffer of 256 bytes within it, and then use this buffer as general purpose buffer to build fixed-length strings that are then printed using lf_{raw}text() function. By doing so we stop relying on temporary stack buffers.
2024-05-02 10:48:56 -04:00
ret = lf_rawtext(dst, ctx->_buf, size, ctx);
MEDIUM: log: use lf_rawtext for lf_ip() and lf_port() hex strings Same as the previous commit, but for ip and port oriented values when +X option is provided. No functional change should be expected. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice, we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-04-03 09:42:35 -04:00
BUG/MINOR: log: properly format IPv6 address when LOG_OPT_HEXA modifier is used. In lf_ip(), when LOG_OPT_HEXA modifier is used, there is a code to format the IP address as a hexadecimal string. This code does not properly handle cases when the IP address is IPv6. In such case, the code only prints `00000000`. This patch adds support for IPv6. For legacy IPv4, the format remains unchanged. If IPv6 socket is used to accept IPv6 connection, the full IPv6 address is returned. For example, IPv6 localhost, ::1, is printed as 00000000000000000000000000000001. If IPv6 socket accepts IPv4 connection, the IPv4 address is mapped by the kernel into the IPv4-mapped-IPv6 address space (RFC4291, section 2.5.5.2) and is formatted as such. For example, 127.0.0.1 becomes ::ffff:127.0.0.1, which is printed as 00000000000000000000FFFF7F000001. This should be backported to 1.9.
2019-03-22 06:21:54 -04:00
break;
MEDIUM: log: use lf_rawtext for lf_ip() and lf_port() hex strings Same as the previous commit, but for ip and port oriented values when +X option is provided. No functional change should be expected. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice, we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-04-03 09:42:35 -04:00
}
BUG/MINOR: log: properly format IPv6 address when LOG_OPT_HEXA modifier is used. In lf_ip(), when LOG_OPT_HEXA modifier is used, there is a code to format the IP address as a hexadecimal string. This code does not properly handle cases when the IP address is IPv6. In such case, the code only prints `00000000`. This patch adds support for IPv6. For legacy IPv4, the format remains unchanged. If IPv6 socket is used to accept IPv6 connection, the full IPv6 address is returned. For example, IPv6 localhost, ::1, is printed as 00000000000000000000000000000001. If IPv6 socket accepts IPv4 connection, the IPv4 address is mapped by the kernel into the IPv4-mapped-IPv6 address space (RFC4291, section 2.5.5.2) and is formatted as such. For example, 127.0.0.1 becomes ::ffff:127.0.0.1, which is printed as 00000000000000000000FFFF7F000001. This should be backported to 1.9.
2019-03-22 06:21:54 -04:00
default:
return NULL;
}
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
} else {
addr_to_str((struct sockaddr_storage *)sockaddr, pn, sizeof(pn));
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
ret = lf_text(dst, pn, size, ctx);
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
}
return ret;
}
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
/* Logformat expr wrapper to write a boolean according to node
* encoding settings
*/
static char *lf_bool_encode(char *dst, size_t size, uint8_t value,
struct lf_buildctx *ctx)
{
/* encode as a regular bool value */
if (ctx->options & LOG_OPT_ENCODE_JSON) {
char *ret = dst;
int iret;
if (value)
iret = snprintf(dst, size, "true");
else
iret = snprintf(dst, size, "false");
if (iret < 0 || iret >= size)
return NULL;
ret += iret;
return ret;
}
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
if (ctx->options & LOG_OPT_ENCODE_CBOR) {
if (value)
return _lf_cbor_encode_byte(&ctx->encode.cbor, dst, dst + size, 0xF5);
return _lf_cbor_encode_byte(&ctx->encode.cbor, dst, dst + size, 0xF4);
}
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
return NULL; /* not supported */
}
/* Logformat expr wrapper to write an integer according to node
* encoding settings and typecast settings.
*/
static char *lf_int_encode(char *dst, size_t size, int64_t value,
struct lf_buildctx *ctx)
{
if (ctx->typecast == SMP_T_BOOL) {
/* either true or false */
return lf_bool_encode(dst, size, !!value, ctx);
}
if (ctx->options & LOG_OPT_ENCODE_JSON) {
char *ret = dst;
int iret = 0;
if (ctx->typecast == SMP_T_STR) {
/* encode as a string number (base10 with "quotes"):
* may be useful to work around the limited resolution
* of JS number types for instance
*/
iret = snprintf(dst, size, "\"%lld\"", (long long int)value);
}
else {
/* encode as a regular int64 number (base10) */
iret = snprintf(dst, size, "%lld", (long long int)value);
}
if (iret < 0 || iret >= size)
return NULL;
ret += iret;
return ret;
}
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
else if (ctx->options & LOG_OPT_ENCODE_CBOR) {
/* Always print as a regular int64 number (STR typecast isn't
* supported)
*/
return cbor_encode_int64(&ctx->encode.cbor, dst, dst + size, value);
}
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
return NULL; /* not supported */
}
MINOR: log: add lf_int() wrapper to print integers Wrap ltoa(), lltoa(), ultoa() and utoa_pad() functions that are used by sess_build_logline() to print numerical values by implementing a dedicated helper named lf_int() that takes <dft_hld> as argument to know how to write the integer by default (when no encoding is specified). LF_INT_UTOA_PAD_4 is used to emulate utoa_pad(x, 4) since it's found only once under sess_build_logline(), thus there is no need to pass an extra parameter to lf_int() function.
2024-03-27 05:17:11 -04:00
enum lf_int_hdl {
LF_INT_LTOA = 0,
LF_INT_LLTOA,
LF_INT_ULTOA,
LF_INT_UTOA_PAD_4,
};
/*
* Logformat expr wrapper to write an integer, uses <dft_hdl> to know
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
* how to encode the value by default (if no encoding is used)
MINOR: log: add lf_int() wrapper to print integers Wrap ltoa(), lltoa(), ultoa() and utoa_pad() functions that are used by sess_build_logline() to print numerical values by implementing a dedicated helper named lf_int() that takes <dft_hld> as argument to know how to write the integer by default (when no encoding is specified). LF_INT_UTOA_PAD_4 is used to emulate utoa_pad(x, 4) since it's found only once under sess_build_logline(), thus there is no need to pass an extra parameter to lf_int() function.
2024-03-27 05:17:11 -04:00
*/
static inline char *lf_int(char *dst, size_t size, int64_t value,
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
struct lf_buildctx *ctx,
MINOR: log: add lf_int() wrapper to print integers Wrap ltoa(), lltoa(), ultoa() and utoa_pad() functions that are used by sess_build_logline() to print numerical values by implementing a dedicated helper named lf_int() that takes <dft_hld> as argument to know how to write the integer by default (when no encoding is specified). LF_INT_UTOA_PAD_4 is used to emulate utoa_pad(x, 4) since it's found only once under sess_build_logline(), thus there is no need to pass an extra parameter to lf_int() function.
2024-03-27 05:17:11 -04:00
enum lf_int_hdl dft_hdl)
{
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
if (ctx->options & LOG_OPT_ENCODE)
return lf_int_encode(dst, size, value, ctx);
MINOR: log: add lf_int() wrapper to print integers Wrap ltoa(), lltoa(), ultoa() and utoa_pad() functions that are used by sess_build_logline() to print numerical values by implementing a dedicated helper named lf_int() that takes <dft_hld> as argument to know how to write the integer by default (when no encoding is specified). LF_INT_UTOA_PAD_4 is used to emulate utoa_pad(x, 4) since it's found only once under sess_build_logline(), thus there is no need to pass an extra parameter to lf_int() function.
2024-03-27 05:17:11 -04:00
switch (dft_hdl) {
case LF_INT_LTOA:
return ltoa_o(value, dst, size);
case LF_INT_LLTOA:
return lltoa(value, dst, size);
case LF_INT_ULTOA:
return ultoa_o(value, dst, size);
case LF_INT_UTOA_PAD_4:
{
if (size < 4)
return NULL;
return utoa_pad(value, dst, 4);
}
}
return NULL;
}
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
/*
* Write a port to the log
CLEANUP: assorted typo fixes in the code and comments This is 8th iteration of typo fixes
2020-04-16 14:51:34 -04:00
* +X option write in hexadecimal notation, most significant byte on the left
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
*/
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
static char *lf_port(char *dst, const struct sockaddr *sockaddr, size_t size, struct lf_buildctx *ctx)
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
{
char *ret = dst;
int iret;
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
if (ctx->options & LOG_OPT_HEXA) {
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
const unsigned char *port = (const unsigned char *)&((struct sockaddr_in *)sockaddr)->sin_port;
MEDIUM: log: use lf_rawtext for lf_ip() and lf_port() hex strings Same as the previous commit, but for ip and port oriented values when +X option is provided. No functional change should be expected. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice, we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-04-03 09:42:35 -04:00
OPTIM: log: use lf_buildctx's buffer instead of temporary stack buffers Now that lf_buildctx isn't pushed on the stack anymore, let's take this opportunity to store a small buffer of 256 bytes within it, and then use this buffer as general purpose buffer to build fixed-length strings that are then printed using lf_{raw}text() function. By doing so we stop relying on temporary stack buffers.
2024-05-02 10:48:56 -04:00
iret = snprintf(ctx->_buf, sizeof(ctx->_buf), "%02X%02X", port[0], port[1]);
BUG/MINOR: log: invalid snprintf() usage in sess_build_logline() According to snprintf() man page: The functions snprintf() and vsnprintf() do not write more than size bytes (including the terminating null byte ('\0')). If the output was truncated due to this limit, then the return value is the number of characters (excluding the terminating null byte) which would have been written to the final string if enough space had been available. Thus, a return value of size or more means that the output was truncated. However, in sess_build_logline(), each time we need to check the return value of snprintf(), here is how we proceed: iret = snprintf(tmplog, max, ...); if (iret < 0 || iret > max) // error // success tmplog += iret; Here is the issue: if snprintf() lacks 1 byte space to write the terminating NULL byte, it will return max. Which means in this case that we fail to know that snprintf() truncated the output in reality, and we still add iret to tmplog pointer. Considering sess_build_logline() should NOT write more than <maxsize> bytes (including the terminating NULL byte) as per the function description, in this case the function would write <maxsize>+1 byte (to write the terminating NULL byte upon return), which may lead to invalid write if <dst> was meant to hold <maxsize> bytes at maximum. Hopefully, this bug wasn't triggered so far because sess_build_logline() is called with logline as <dst> argument and <global.max_syslog_len> as <maxsize> argument, logline being initialized with 1 extra byte upon startup. But we better fix this to comply with the function description and prevent any side-effect since some sess_build_logline() helpers may assume that 'tmplog-dst < maxsize' is always true. Also sess_build_logline() users probably don't expect NULL-byte to be accounted for in the produced logline length. This should be backported to all stable versions. [ada: for backports, the patch needs to be applied manually because of c6a713842 ("MINOR: log: simplify last_isspace in sess_build_logline()")]
2024-04-09 09:59:42 -04:00
if (iret < 0 || iret >= size)
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
return NULL;
OPTIM: log: use lf_buildctx's buffer instead of temporary stack buffers Now that lf_buildctx isn't pushed on the stack anymore, let's take this opportunity to store a small buffer of 256 bytes within it, and then use this buffer as general purpose buffer to build fixed-length strings that are then printed using lf_{raw}text() function. By doing so we stop relying on temporary stack buffers.
2024-05-02 10:48:56 -04:00
ret = lf_rawtext(dst, ctx->_buf, size, ctx);
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
} else {
MINOR: log: add lf_int() wrapper to print integers Wrap ltoa(), lltoa(), ultoa() and utoa_pad() functions that are used by sess_build_logline() to print numerical values by implementing a dedicated helper named lf_int() that takes <dft_hld> as argument to know how to write the integer by default (when no encoding is specified). LF_INT_UTOA_PAD_4 is used to emulate utoa_pad(x, 4) since it's found only once under sess_build_logline(), thus there is no need to pass an extra parameter to lf_int() function.
2024-03-27 05:17:11 -04:00
ret = lf_int(dst, size, get_host_port((struct sockaddr_storage *)sockaddr),
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
ctx, LF_INT_LTOA);
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
}
return ret;
}
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
/*
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
* This function sends a syslog message.
* <target> is the actual log target where log will be sent,
MEDIUM: log/sink: simplify log header handling Introduce log_header struct to easily pass log header data between functions and use that to simplify the logic around log header handling. While at it, some outdated comments were updated as well. No change in behavior should be expected.
2023-08-24 14:46:12 -04:00
*
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
* Message will be prefixed by header according to <hdr> setting.
* Final message will be truncated <maxlen> parameter and will be
* terminated with an LF character.
MEDIUM: log/sink: simplify log header handling Introduce log_header struct to easily pass log header data between functions and use that to simplify the logic around log header handling. While at it, some outdated comments were updated as well. No change in behavior should be expected.
2023-08-24 14:46:12 -04:00
*
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
* Does not return any error
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
*/
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
static inline void __do_send_log(struct log_target *target, struct log_header hdr,
int nblogger, size_t maxlen,
char *message, size_t size)
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
{
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
static THREAD_LOCAL struct iovec iovec[NB_LOG_HDR_MAX_ELEMENTS+1+1] = { }; /* header elements + message + LF */
static THREAD_LOCAL struct msghdr msghdr = {
//.msg_iov = iovec,
.msg_iovlen = NB_LOG_HDR_MAX_ELEMENTS+2
};
static THREAD_LOCAL int logfdunix = -1; /* syslog to AF_UNIX socket */
static THREAD_LOCAL int logfdinet = -1; /* syslog to AF_INET socket */
int *plogfd;
int sent;
size_t nbelem;
struct ist *msg_header = NULL;
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
msghdr.msg_iov = iovec;
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
/* historically some messages used to already contain the trailing LF
* or Zero. Let's remove all trailing LF or Zero
*/
while (size && (message[size-1] == '\n' || (message[size-1] == 0)))
size--;
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
if (target->type == LOG_TARGET_BUFFER) {
plogfd = NULL;
goto send;
}
else if (target->addr->ss_family == AF_CUST_EXISTING_FD) {
/* the socket's address is a file descriptor */
plogfd = (int *)&((struct sockaddr_in *)target->addr)->sin_addr.s_addr;
}
else if (target->addr->ss_family == AF_UNIX)
plogfd = &logfdunix;
else
plogfd = &logfdinet;
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
if (plogfd && unlikely(*plogfd < 0)) {
/* socket not successfully initialized yet */
if ((*plogfd = socket(target->addr->ss_family, SOCK_DGRAM,
(target->addr->ss_family == AF_UNIX) ? 0 : IPPROTO_UDP)) < 0) {
static char once;
if (!once) {
once = 1; /* note: no need for atomic ops here */
ha_alert("socket() failed in logger #%d: %s (errno=%d)\n",
nblogger, strerror(errno), errno);
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
}
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
return;
} else {
/* we don't want to receive anything on this socket */
setsockopt(*plogfd, SOL_SOCKET, SO_RCVBUF, &zero, sizeof(zero));
/* we may want to adjust the output buffer (tune.sndbuf.backend) */
if (global.tune.backend_sndbuf)
setsockopt(*plogfd, SOL_SOCKET, SO_SNDBUF, &global.tune.backend_sndbuf, sizeof(global.tune.backend_sndbuf));
/* does nothing under Linux, maybe needed for others */
shutdown(*plogfd, SHUT_RD);
fd_set_cloexec(*plogfd);
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
}
}
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
msg_header = build_log_header(hdr, &nbelem);
send:
if (target->type == LOG_TARGET_BUFFER) {
struct ist msg;
size_t e_maxlen = maxlen;
msg = ist2(message, size);
/* make room for the final '\n' which may be forcefully inserted
* by tcp forwarder applet (sink_forward_io_handler)
*/
e_maxlen -= 1;
sent = sink_write(target->sink, hdr, e_maxlen, &msg, 1);
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
}
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
else if (target->addr->ss_family == AF_CUST_EXISTING_FD) {
struct ist msg;
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
msg = ist2(message, size);
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
sent = fd_write_frag_line(*plogfd, maxlen, msg_header, nbelem, &msg, 1, 1);
}
else {
int i = 0;
int totlen = maxlen - 1; /* save space for the final '\n' */
for (i = 0 ; i < nbelem ; i++ ) {
iovec[i].iov_base = msg_header[i].ptr;
iovec[i].iov_len = msg_header[i].len;
if (totlen <= iovec[i].iov_len) {
iovec[i].iov_len = totlen;
totlen = 0;
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
break;
}
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
totlen -= iovec[i].iov_len;
}
if (totlen) {
iovec[i].iov_base = message;
iovec[i].iov_len = size;
if (totlen <= iovec[i].iov_len)
iovec[i].iov_len = totlen;
i++;
}
iovec[i].iov_base = "\n"; /* insert a \n at the end of the message */
iovec[i].iov_len = 1;
i++;
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
msghdr.msg_iovlen = i;
msghdr.msg_name = (struct sockaddr *)target->addr;
msghdr.msg_namelen = get_addr_len(target->addr);
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
sent = sendmsg(*plogfd, &msghdr, MSG_DONTWAIT | MSG_NOSIGNAL);
}
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
if (sent < 0) {
static char once;
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
if (errno == EAGAIN || errno == EWOULDBLOCK)
_HA_ATOMIC_INC(&dropped_logs);
else if (!once) {
once = 1; /* note: no need for atomic ops here */
ha_alert("sendmsg()/writev() failed in logger #%d: %s (errno=%d)\n",
nblogger, strerror(errno), errno);
}
}
}
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
/* does the same as __do_send_log() does for a single target, but here the log
* will be sent according to the log backend's lb settings. The function will
* leverage __do_send_log() function to actually send the log messages.
*/
static inline void __do_send_log_backend(struct proxy *be, struct log_header hdr,
int nblogger, size_t maxlen,
char *message, size_t size)
{
struct server *srv = NULL;
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
/* log-balancing logic: */
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
if ((be->lbprm.algo & BE_LB_ALGO) == BE_LB_ALGO_RR) {
srv = fwrr_get_next_server(be, NULL);
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
}
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
else if ((be->lbprm.algo & BE_LB_ALGO) == BE_LB_ALGO_SS) {
/* sticky mode: use first server in the pool, which will always stay
* first during dequeuing and requeuing, unless it becomes unavailable
* and will be replaced by another one
*/
srv = ss_get_server(be);
}
else if ((be->lbprm.algo & BE_LB_ALGO) == BE_LB_ALGO_RND) {
unsigned int hash;
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
hash = statistical_prng(); /* random */
srv = chash_get_server_hash(be, hash, NULL);
}
else if ((be->lbprm.algo & BE_LB_ALGO) == BE_LB_ALGO_LH) {
struct sample result;
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
/* log-balance hash */
memset(&result, 0, sizeof(result));
result.data.type = SMP_T_STR;
result.flags = SMP_F_CONST;
result.data.u.str.area = message;
result.data.u.str.data = size;
result.data.u.str.size = size + 1; /* with terminating NULL byte */
if (sample_process_cnv(be->lbprm.expr, &result)) {
/* gen_hash takes binary input, ensure that we provide such value to it */
if (result.data.type == SMP_T_BIN || sample_casts[result.data.type][SMP_T_BIN]) {
unsigned int hash;
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
sample_casts[result.data.type][SMP_T_BIN](&result);
hash = gen_hash(be, result.data.u.str.area, result.data.u.str.data);
srv = map_get_server_hash(be, hash);
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
}
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
}
}
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
if (!srv) {
/* no srv available, can't log */
goto drop;
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
}
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
__do_send_log(srv->log_target, hdr, nblogger, maxlen, message, size);
return;
drop:
_HA_ATOMIC_INC(&dropped_logs);
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
}
REORG: log: split send_log function send_log function is now splited in 3 functions * hdr_log: generate the syslog header * send_log: send a syslog message with a printf format string * __send_log: send a syslog message
2012-02-06 10:00:33 -05:00
MINOR: log: provide sending log context to process_send_log() when available This is another prerequisite work in preparation for log-profiles: in this patch we make process_send_log() aware of the log origin, primarily aiming for sess and txn logging steps such as error, accept, connect, close, as well as relevant sess and stream pointers.
2024-02-22 05:29:20 -05:00
/* provided to low-level process_send_log() helper, may be NULL */
struct process_send_log_ctx {
struct session *sess;
struct stream *stream;
enum log_orig origin;
};
REORG: log: split send_log function send_log function is now splited in 3 functions * hdr_log: generate the syslog header * send_log: send a syslog message with a printf format string * __send_log: send a syslog message
2012-02-06 10:00:33 -05:00
/*
MINOR: log: support explicit log target as argument in __do_send_log() __do_send_log() now takes an extra target parameter to pass an explicit log target instead of getting it from logger->target. This will allow __do_send_log() to be called multiple times within a logger entry containing multiple log targets.
2023-08-17 07:45:19 -04:00
* This function sends a syslog message.
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
* It doesn't care about errors nor does it report them.
* The argument <metadata> MUST be an array of size
* LOG_META_FIELDS*sizeof(struct ist) containing
* data to build the header.
REORG: log: split send_log function send_log function is now splited in 3 functions * hdr_log: generate the syslog header * send_log: send a syslog message with a printf format string * __send_log: send a syslog message
2012-02-06 10:00:33 -05:00
*/
MINOR: log: provide sending log context to process_send_log() when available This is another prerequisite work in preparation for log-profiles: in this patch we make process_send_log() aware of the log origin, primarily aiming for sess and txn logging steps such as error, accept, connect, close, as well as relevant sess and stream pointers.
2024-02-22 05:29:20 -05:00
static void process_send_log(struct process_send_log_ctx *ctx,
struct list *loggers, int level, int facility,
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
struct ist *metadata, char *message, size_t size)
REORG: log: split send_log function send_log function is now splited in 3 functions * hdr_log: generate the syslog header * send_log: send a syslog message with a printf format string * __send_log: send a syslog message
2012-02-06 10:00:33 -05:00
{
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
struct logger *logger;
int nblogger;
MEDIUM: threads/logs: Make logs thread-safe log buffers and static variables used in log functions are now thread-local. So there is no need to lock anything to log messages. Moreover, per-thread init/deinit functions are now used to initialize these buffers.
2017-06-02 10:20:16 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
/* Send log messages to syslog server. */
nblogger = 0;
list_for_each_entry(logger, loggers, list) {
int in_range = 1;
REORG: log: split send_log function send_log function is now splited in 3 functions * hdr_log: generate the syslog header * send_log: send a syslog message with a printf format string * __send_log: send a syslog message
2012-02-06 10:00:33 -05:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
/* we can filter the level of the messages that are sent to each logger */
if (level > logger->level)
continue;
BUG/MINOR: logs: prevent double line returns in some events. Historically some messages used to already contain the trailing LF but not all, and __do_send_log adds a new one in needed cases. It also does trim a trailing LF in certain cases while computing the max message length, as a result of subtracting 1 to the available room in the destination buffer. But the way it's done is wrong since some messages still contain it. So the code was fixed to always trim the trailing LF from messages if present, and then only subtract 1 from the destination buffer room instead of the size.. Note: new sink API is not designed to receive a trailing LF on event messages This could be backported to relevant stable versions with particular care since the logic of the code changed a bit since 1.6 and there may be other locations that need to be adjusted.
2020-05-12 13:33:15 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
if (logger->lb.smp_rgs) {
struct smp_log_range *smp_rg;
uint next_idx, curr_rg;
ullong curr_rg_idx, next_rg_idx;
MINOR: log: Extract some code to send syslog messages. This patch extracts the code of __send_log() responsible of sending a syslog message to a syslog destination represented as a logsrv struct to define __do_send_log() function. __send_log() calls __do_send_log() for each syslog destination of a proxy after having prepared some of its parameters.
2019-04-10 02:22:17 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
curr_rg_idx = _HA_ATOMIC_LOAD(&logger->lb.curr_rg_idx);
do {
next_idx = (curr_rg_idx & 0xFFFFFFFFU) + 1;
curr_rg = curr_rg_idx >> 32;
smp_rg = &logger->lb.smp_rgs[curr_rg];
MINOR: log: Extract some code to send syslog messages. This patch extracts the code of __send_log() responsible of sending a syslog message to a syslog destination represented as a logsrv struct to define __do_send_log() function. __send_log() calls __do_send_log() for each syslog destination of a proxy after having prepared some of its parameters.
2019-04-10 02:22:17 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
/* check if the index we're going to take is within range */
in_range = smp_rg->low <= next_idx && next_idx <= smp_rg->high;
if (in_range) {
/* Let's consume this range. */
if (next_idx == smp_rg->high) {
/* If consumed, let's select the next range. */
curr_rg = (curr_rg + 1) % logger->lb.smp_rgs_sz;
}
}
next_idx = next_idx % logger->lb.smp_sz;
next_rg_idx = ((ullong)curr_rg << 32) + next_idx;
} while (!_HA_ATOMIC_CAS(&logger->lb.curr_rg_idx, &curr_rg_idx, next_rg_idx) &&
__ha_cpu_relax());
}
if (in_range) {
struct log_header hdr;
hdr.level = MAX(level, logger->minlvl);
hdr.facility = (facility == -1) ? logger->facility : facility;
hdr.format = logger->format;
hdr.metadata = metadata;
nblogger += 1;
if (logger->target.type == LOG_TARGET_BACKEND) {
__do_send_log_backend(logger->target.be, hdr, nblogger, logger->maxlen, message, size);
MINOR: log: Extract some code to send syslog messages. This patch extracts the code of __send_log() responsible of sending a syslog message to a syslog destination represented as a logsrv struct to define __do_send_log() function. __send_log() calls __do_send_log() for each syslog destination of a proxy after having prepared some of its parameters.
2019-04-10 02:22:17 -04:00
}
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
else {
/* normal target */
__do_send_log(&logger->target, hdr, nblogger, logger->maxlen, message, size);
}
}
}
}
/*
* This function sends a syslog message.
* It doesn't care about errors nor does it report them.
* The arguments <sd> and <sd_size> are used for the structured-data part
* in RFC5424 formatted syslog messages.
*/
MINOR: log: provide sending log context to process_send_log() when available This is another prerequisite work in preparation for log-profiles: in this patch we make process_send_log() aware of the log origin, primarily aiming for sess and txn logging steps such as error, accept, connect, close, as well as relevant sess and stream pointers.
2024-02-22 05:29:20 -05:00
static void __send_log(struct process_send_log_ctx *ctx,
struct list *loggers, struct buffer *tagb, int level,
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
char *message, size_t size, char *sd, size_t sd_size)
{
static THREAD_LOCAL pid_t curr_pid;
static THREAD_LOCAL char pidstr[16];
static THREAD_LOCAL struct ist metadata[LOG_META_FIELDS];
if (loggers == NULL) {
if (!LIST_ISEMPTY(&global.loggers)) {
loggers = &global.loggers;
BUG/MEDIUM: logs: segfault writing to log from Lua Michael Ezzell reported a bug causing haproxy to segfault during startup when trying to send syslog message from Lua. The function __send_log() can be called with *p that is NULL and/or when the configuration is not fully parsed, as is the case with Lua. This patch fixes this problem by using individual vectors instead of the pre-generated strings log_htp and log_htp_rfc5424. Also, this patch fixes a problem causing haproxy to write the wrong pid in the logs -- the log_htp(_rfc5424) strings were generated at the haproxy start, but "pid" value would be changed after haproxy is started in daemon/systemd mode.
2015-10-01 07:18:13 -04:00
}
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-25 20:48:02 -04:00
}
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
if (!loggers || LIST_ISEMPTY(loggers))
return;
if (!metadata[LOG_META_HOST].len) {
if (global.log_send_hostname)
metadata[LOG_META_HOST] = ist(global.log_send_hostname);
}
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-25 20:48:02 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
if (!tagb || !tagb->area)
tagb = &global.log_tag;
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
if (tagb)
metadata[LOG_META_TAG] = ist2(tagb->area, tagb->data);
MEDIUM: log: use the new generic fd_write_frag_line() function When logging to a file descriptor, we'd rather use the unified fd_write_frag_line() which uses the FD's lock than perform the writev() ourselves and use a per-server lock, because if several loggers point to the same output (e.g. stdout) they are still not locked and their logs may interleave. The function above instead relies on the fd's lock so this is safer and will even protect against concurrent accesses from other areas (e.g traces). The function also deals with the FD's non-blocking mode so we do not have to keep specific code for this anymore in the logs.
2019-08-30 08:05:35 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
if (unlikely(curr_pid != getpid()))
metadata[LOG_META_PID].len = 0;
BUG/MEDIUM: log: improper use of logsrv->maxlen for buffer targets In e709e1e ("MEDIUM: logs: buffer targets now rely on new sink_write") we started using the sink API instead of using the ring_write function directly. But as indicated in the commit message, the maxlen parameter of the log directive now only applies to the message part and not the complete payload. I don't know what the original intent was (maybe minimizing code changes) but it seems wrong, because the doc doesn't mention this special case, and the result is that the ring->buffer output can differ from all other log output types, making it very confusing. One last issue with this is that log messages can end up being dropped at runtime, only for the buffer target, and even if logsrv->maxlen is correctly set (including default: 1024) because depending on the generated header size the payload can grow bigger than the accepted sink size (sink maxlen is not mandatory) and we have no simple way to detect this at configuration time. First, we partially revert e709e1e: TARGET_BUFFER still leverages the proper sink API, but thanks to "MINOR: sink: pass explicit maxlen parameter to sink_write()" we now explicitly pass the logsrv->maxlen to the sink_write function in order to stop writing as soon as either sink->maxlen or logsrv->maxlen is reached. This restores pre-e709e1e behavior with the added benefit from using the high-level API, which includes automatically announcing dropped message events. Then, we also need to take the ending '\n' into account: it is not explicitly set when generating the logline for TARGET_BUFFER, but it will be forcefully added by the sink_forward_io_handler function from the tcp handler applet when log messages from the buffer are forwarded to tcp endpoints. In current form, because the '\n' is added later in the chain, maxlen is not being considered anymore, so the final log message could exceed maxlen by 1 byte, which could make receiving servers unhappy in logging context. To prevent this, we sacrifice 1 byte from the logsrv->maxlen to ensure that the final message will never exceed log->maxlen, even if the '\n' char is automatically appended later by the forwarding applet. Thanks to this change TCP (over RING/BUFFER) target now behaves like FD and UDP targets. This commit depends on: - "MINOR: sink: pass explicit maxlen parameter to sink_write()" It may be backported as far as 2.2 [For 2.2 and 2.4 the patch does not apply automatically, the sink_write() call must be updated by hand]
2023-06-26 11:15:48 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
if (!metadata[LOG_META_PID].len) {
curr_pid = getpid();
ltoa_o(curr_pid, pidstr, sizeof(pidstr));
metadata[LOG_META_PID] = ist2(pidstr, strlen(pidstr));
MINOR: protocol: add a real family for existing FDs At some places (log fd@XXX, bind fd@XXX) we support using an explicit file descriptor number, that is placed into the sockaddr for later use. The problem is that till now it was done with an AF_UNSPEC family, which is also used for other situations like missing info or rings (for logs). Let's create an "official" family AF_CUST_EXISTING_FD for this case so that we are certain the FD can be found in the address when it is set.
2020-09-04 10:44:20 -04:00
}
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
metadata[LOG_META_STDATA] = ist2(sd, sd_size);
MINOR: protocol: add a real family for existing FDs At some places (log fd@XXX, bind fd@XXX) we support using an explicit file descriptor number, that is placed into the sockaddr for later use. The problem is that till now it was done with an AF_UNSPEC family, which is also used for other situations like missing info or rings (for logs). Let's create an "official" family AF_CUST_EXISTING_FD for this case so that we are certain the FD can be found in the address when it is set.
2020-09-04 10:44:20 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
/* Remove trailing space of structured data */
while (metadata[LOG_META_STDATA].len && metadata[LOG_META_STDATA].ptr[metadata[LOG_META_STDATA].len-1] == ' ')
metadata[LOG_META_STDATA].len--;
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
MINOR: log: provide sending log context to process_send_log() when available This is another prerequisite work in preparation for log-profiles: in this patch we make process_send_log() aware of the log origin, primarily aiming for sess and txn logging steps such as error, accept, connect, close, as well as relevant sess and stream pointers.
2024-02-22 05:29:20 -05:00
return process_send_log(ctx, loggers, level, -1, metadata, message, size);
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
}
MEDIUM: log: use the new generic fd_write_frag_line() function When logging to a file descriptor, we'd rather use the unified fd_write_frag_line() which uses the FD's lock than perform the writev() ourselves and use a per-server lock, because if several loggers point to the same output (e.g. stdout) they are still not locked and their logs may interleave. The function above instead relies on the fd's lock so this is safer and will even protect against concurrent accesses from other areas (e.g traces). The function also deals with the FD's non-blocking mode so we do not have to keep specific code for this anymore in the logs.
2019-08-30 08:05:35 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
/*
* This function sends the syslog message using a printf format string. It
* expects an LF-terminated message.
*/
void send_log(struct proxy *p, int level, const char *format, ...)
{
va_list argp;
int data_len;
BUG/MEDIUM: logs: segfault writing to log from Lua Michael Ezzell reported a bug causing haproxy to segfault during startup when trying to send syslog message from Lua. The function __send_log() can be called with *p that is NULL and/or when the configuration is not fully parsed, as is the case with Lua. This patch fixes this problem by using individual vectors instead of the pre-generated strings log_htp and log_htp_rfc5424. Also, this patch fixes a problem causing haproxy to write the wrong pid in the logs -- the log_htp(_rfc5424) strings were generated at the haproxy start, but "pid" value would be changed after haproxy is started in daemon/systemd mode.
2015-10-01 07:18:13 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
if (level < 0 || format == NULL || logline == NULL)
return;
BUG/MEDIUM: logs: segfault writing to log from Lua Michael Ezzell reported a bug causing haproxy to segfault during startup when trying to send syslog message from Lua. The function __send_log() can be called with *p that is NULL and/or when the configuration is not fully parsed, as is the case with Lua. This patch fixes this problem by using individual vectors instead of the pre-generated strings log_htp and log_htp_rfc5424. Also, this patch fixes a problem causing haproxy to write the wrong pid in the logs -- the log_htp(_rfc5424) strings were generated at the haproxy start, but "pid" value would be changed after haproxy is started in daemon/systemd mode.
2015-10-01 07:18:13 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
va_start(argp, format);
data_len = vsnprintf(logline, global.max_syslog_len, format, argp);
if (data_len < 0 || data_len > global.max_syslog_len)
data_len = global.max_syslog_len;
va_end(argp);
BUG/MEDIUM: logs: segfault writing to log from Lua Michael Ezzell reported a bug causing haproxy to segfault during startup when trying to send syslog message from Lua. The function __send_log() can be called with *p that is NULL and/or when the configuration is not fully parsed, as is the case with Lua. This patch fixes this problem by using individual vectors instead of the pre-generated strings log_htp and log_htp_rfc5424. Also, this patch fixes a problem causing haproxy to write the wrong pid in the logs -- the log_htp(_rfc5424) strings were generated at the haproxy start, but "pid" value would be changed after haproxy is started in daemon/systemd mode.
2015-10-01 07:18:13 -04:00
MINOR: log: provide sending log context to process_send_log() when available This is another prerequisite work in preparation for log-profiles: in this patch we make process_send_log() aware of the log origin, primarily aiming for sess and txn logging steps such as error, accept, connect, close, as well as relevant sess and stream pointers.
2024-02-22 05:29:20 -05:00
__send_log(NULL, (p ? &p->loggers : NULL),
(p ? &p->log_tag : NULL), level,
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
logline, data_len, default_rfc5424_sd_log_format, 2);
MINOR: log: Extract some code to send syslog messages. This patch extracts the code of __send_log() responsible of sending a syslog message to a syslog destination represented as a logsrv struct to define __do_send_log() function. __send_log() calls __do_send_log() for each syslog destination of a proxy after having prepared some of its parameters.
2019-04-10 02:22:17 -04:00
}
MEDIUM: logs: remove the hostname, tag and pid part from the logheader At the moment we have to call snprintf() for every log line just to rebuild a constant. Thanks to sendmsg(), we send the message in 3 parts: time-based header, proxy-specific hostname+log-tag+pid, session-specific message.
2015-09-19 16:35:44 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
/*
* This function builds a log header according to <hdr> settings.
*
* If hdr.format is set to LOG_FORMAT_UNSPEC, it tries to determine
* format based on hdr.metadata. It is useful for log-forwarding to be
* able to forward any format without settings.
*
* This function returns a struct ist array of elements of the header
* nbelem is set to the number of available elements.
* This function returns currently a maximum of NB_LOG_HDR_IST_ELEMENTS
* elements.
MAJOR: log: introduce log backends Using "mode log" in a backend section turns the proxy in a log backend which can be used to log-balance logs between multiple log targets (udp or tcp servers) log backends can be used as regular log targets using the log directive with "backend@be_name" prefix, like so: | log backend@mybackend local0 A log backend will distribute log messages to servers according to the log load-balancing algorithm that can be set using the "log-balance" option from the log backend section. For now, only the roundrobin algorithm is supported and set by default.
2023-09-13 05:52:31 -04:00
*/
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
struct ist *build_log_header(struct log_header hdr, size_t *nbelem)
MAJOR: log: introduce log backends Using "mode log" in a backend section turns the proxy in a log backend which can be used to log-balance logs between multiple log targets (udp or tcp servers) log backends can be used as regular log targets using the log directive with "backend@be_name" prefix, like so: | log backend@mybackend local0 A log backend will distribute log messages to servers according to the log load-balancing algorithm that can be set using the "log-balance" option from the log backend section. For now, only the roundrobin algorithm is supported and set by default.
2023-09-13 05:52:31 -04:00
{
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
static THREAD_LOCAL struct {
struct ist ist_vector[NB_LOG_HDR_MAX_ELEMENTS];
char timestamp_buffer[LOG_LEGACYTIME_LEN+1+1];
time_t cur_legacy_time;
char priority_buffer[6];
} hdr_ctx = { .priority_buffer = "<<<<>" };
MAJOR: log: introduce log backends Using "mode log" in a backend section turns the proxy in a log backend which can be used to log-balance logs between multiple log targets (udp or tcp servers) log backends can be used as regular log targets using the log directive with "backend@be_name" prefix, like so: | log backend@mybackend local0 A log backend will distribute log messages to servers according to the log load-balancing algorithm that can be set using the "log-balance" option from the log backend section. For now, only the roundrobin algorithm is supported and set by default.
2023-09-13 05:52:31 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
struct tm logtime;
int len;
int fac_level = 0;
time_t time = date.tv_sec;
struct ist *metadata = hdr.metadata;
enum log_fmt format = hdr.format;
int facility = hdr.facility;
int level = hdr.level;
MAJOR: log: introduce log backends Using "mode log" in a backend section turns the proxy in a log backend which can be used to log-balance logs between multiple log targets (udp or tcp servers) log backends can be used as regular log targets using the log directive with "backend@be_name" prefix, like so: | log backend@mybackend local0 A log backend will distribute log messages to servers according to the log load-balancing algorithm that can be set using the "log-balance" option from the log backend section. For now, only the roundrobin algorithm is supported and set by default.
2023-09-13 05:52:31 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
*nbelem = 0;
if (format == LOG_FORMAT_UNSPEC) {
format = LOG_FORMAT_RAW;
if (metadata) {
/* If a hostname is set, it appears we want to perform syslog
* because only rfc5427 or rfc3164 support an hostname.
*/
if (metadata[LOG_META_HOST].len) {
/* If a rfc5424 compliant timestamp is used we consider
* that output format is rfc5424, else legacy format
* is used as specified default for local logs
* in documentation.
*/
if ((metadata[LOG_META_TIME].len == 1 && metadata[LOG_META_TIME].ptr[0] == '-')
|| (metadata[LOG_META_TIME].len >= LOG_ISOTIME_MINLEN))
format = LOG_FORMAT_RFC5424;
else
format = LOG_FORMAT_RFC3164;
}
else if (metadata[LOG_META_TAG].len) {
/* Tag is present but no hostname, we should
* consider we try to emit a local log
* in legacy format (analog to RFC3164 but
* with stripped hostname).
*/
format = LOG_FORMAT_LOCAL;
}
else if (metadata[LOG_META_PRIO].len) {
/* the source seems a parsed message
* offering a valid level/prio prefix
* so we consider this format.
*/
format = LOG_FORMAT_PRIO;
}
}
MINOR: log/balance: support for the "sticky" lb algorithm sticky algorithm always tries to send log messages to the first server in the farm. The server will stay in front during queue and dequeue operations (no other server can steal its place), unless it becomes unavailable, in which case it will be replaced by another server from the tree.
2023-09-21 14:24:14 -04:00
}
MEDIUM: log/balance: leverage lbprm api for log load-balancing log load-balancing implementation was not seamlessly integrated within lbprm API. The consequence is that it could become harder to maintain over time since it added some specific cases just for the log backend. Moreover, it resulted in some code duplication since balance algorithms that are common to logs and regular (tcp, http) backends were specifically rewritten for log backends. Thanks to the previous commit, we now have all the prerequisites to make log load-balancing fully leverage lbprm logic. Thus in this patch we make __do_send_log_backend() use existing lbprm algorithms, and we no longer require log-specific lbprm initialization in cfgparse.c and in postcheck_log_backend(). As a bonus, for log backends this allows weighed algorithms to properly support weights (ie: roundrobin, random and log-hash) since we now leverage the same lb algorithms that we use for tcp/http backends (doc was updated).
2024-03-28 10:42:37 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
/* prepare priority, stored into 1 single elem */
switch (format) {
case LOG_FORMAT_LOCAL:
case LOG_FORMAT_RFC3164:
case LOG_FORMAT_RFC5424:
case LOG_FORMAT_PRIO:
fac_level = facility << 3;
/* further format ignore the facility */
__fallthrough;
case LOG_FORMAT_TIMED:
case LOG_FORMAT_SHORT:
fac_level += level;
hdr_ctx.ist_vector[*nbelem].ptr = &hdr_ctx.priority_buffer[3]; /* last digit of the log level */
do {
*hdr_ctx.ist_vector[*nbelem].ptr = '0' + fac_level % 10;
fac_level /= 10;
hdr_ctx.ist_vector[*nbelem].ptr--;
} while (fac_level && hdr_ctx.ist_vector[*nbelem].ptr > &hdr_ctx.priority_buffer[0]);
*hdr_ctx.ist_vector[*nbelem].ptr = '<';
hdr_ctx.ist_vector[(*nbelem)++].len = &hdr_ctx.priority_buffer[5] - hdr_ctx.ist_vector[0].ptr;
break;
case LOG_FORMAT_ISO:
case LOG_FORMAT_RAW:
break;
case LOG_FORMAT_UNSPEC:
case LOG_FORMATS:
ABORT_NOW();
MINOR: log/balance: support for the "random" lb algorithm In this patch we add basic support for the random algorithm: random algorithm picks a random server using the result of the statistical_prng() function as if it was a hash key to then compute the related server ID. There is no support for the <draw> parameter (which is implemented for tcp/http load-balancing), because we don't have the required metrics to evaluate server's load in log backends for the moment. Plus it would add more complexity to the __do_send_log_backend() function so we'll keep it this way for now but this might be needed in the future.
2023-10-04 11:30:02 -04:00
}
MEDIUM: log/balance: support for the "hash" lb algorithm hash lb algorithm can be configured with the "log-balance hash <cnv_list>" directive. With this algorithm, the user specifies a converter list with <cnv_list>. The produced log message will be passed as-is to the provided converter list, and the resulting hash will be used to select the log server that will receive the log message.
2023-09-19 04:51:53 -04:00
MEDIUM: log/balance: leverage lbprm api for log load-balancing log load-balancing implementation was not seamlessly integrated within lbprm API. The consequence is that it could become harder to maintain over time since it added some specific cases just for the log backend. Moreover, it resulted in some code duplication since balance algorithms that are common to logs and regular (tcp, http) backends were specifically rewritten for log backends. Thanks to the previous commit, we now have all the prerequisites to make log load-balancing fully leverage lbprm logic. Thus in this patch we make __do_send_log_backend() use existing lbprm algorithms, and we no longer require log-specific lbprm initialization in cfgparse.c and in postcheck_log_backend(). As a bonus, for log backends this allows weighed algorithms to properly support weights (ie: roundrobin, random and log-hash) since we now leverage the same lb algorithms that we use for tcp/http backends (doc was updated).
2024-03-28 10:42:37 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
/* prepare timestamp, stored into a max of 4 elems */
switch (format) {
case LOG_FORMAT_LOCAL:
case LOG_FORMAT_RFC3164:
/* rfc3164 ex: 'Jan 1 00:00:00 ' */
if (metadata && metadata[LOG_META_TIME].len == LOG_LEGACYTIME_LEN) {
hdr_ctx.ist_vector[(*nbelem)++] = metadata[LOG_META_TIME];
hdr_ctx.ist_vector[(*nbelem)++] = ist2(" ", 1);
/* time is set, break immediately */
break;
}
else if (metadata && metadata[LOG_META_TIME].len >= LOG_ISOTIME_MINLEN) {
int month;
char *timestamp = metadata[LOG_META_TIME].ptr;
/* iso time always begins like this: '1970-01-01T00:00:00' */
/* compute month */
month = 10*(timestamp[5] - '0') + (timestamp[6] - '0');
if (month)
month--;
if (month <= 11) {
/* builds log prefix ex: 'Jan 1 ' */
len = snprintf(hdr_ctx.timestamp_buffer, sizeof(hdr_ctx.timestamp_buffer),
"%s %c%c ", monthname[month],
timestamp[8] != '0' ? timestamp[8] : ' ',
timestamp[9]);
/* we reused the timestamp_buffer, signal that it does not
* contain local time anymore
*/
hdr_ctx.cur_legacy_time = 0;
if (len == 7) {
hdr_ctx.ist_vector[(*nbelem)++] = ist2(&hdr_ctx.timestamp_buffer[0], len);
/* adds 'HH:MM:SS' from iso time */
hdr_ctx.ist_vector[(*nbelem)++] = ist2(&timestamp[11], 8);
hdr_ctx.ist_vector[(*nbelem)++] = ist2(" ", 1);
/* we successfully reuse iso time, we can break */
break;
}
}
/* Failed to reuse isotime time, fallback to local legacy time */
MEDIUM: log/balance: support for the "hash" lb algorithm hash lb algorithm can be configured with the "log-balance hash <cnv_list>" directive. With this algorithm, the user specifies a converter list with <cnv_list>. The produced log message will be passed as-is to the provided converter list, and the resulting hash will be used to select the log server that will receive the log message.
2023-09-19 04:51:53 -04:00
}
MAJOR: log: introduce log backends Using "mode log" in a backend section turns the proxy in a log backend which can be used to log-balance logs between multiple log targets (udp or tcp servers) log backends can be used as regular log targets using the log directive with "backend@be_name" prefix, like so: | log backend@mybackend local0 A log backend will distribute log messages to servers according to the log load-balancing algorithm that can be set using the "log-balance" option from the log backend section. For now, only the roundrobin algorithm is supported and set by default.
2023-09-13 05:52:31 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
if (unlikely(time != hdr_ctx.cur_legacy_time)) {
/* re-builds timestamp from the current local time */
get_localtime(time, &logtime);
MAJOR: log: introduce log backends Using "mode log" in a backend section turns the proxy in a log backend which can be used to log-balance logs between multiple log targets (udp or tcp servers) log backends can be used as regular log targets using the log directive with "backend@be_name" prefix, like so: | log backend@mybackend local0 A log backend will distribute log messages to servers according to the log load-balancing algorithm that can be set using the "log-balance" option from the log backend section. For now, only the roundrobin algorithm is supported and set by default.
2023-09-13 05:52:31 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
len = snprintf(hdr_ctx.timestamp_buffer, sizeof(hdr_ctx.timestamp_buffer),
"%s %2d %02d:%02d:%02d ",
monthname[logtime.tm_mon],
logtime.tm_mday, logtime.tm_hour, logtime.tm_min, logtime.tm_sec);
if (len != LOG_LEGACYTIME_LEN+1)
hdr_ctx.cur_legacy_time = 0;
else
hdr_ctx.cur_legacy_time = time;
}
if (likely(hdr_ctx.cur_legacy_time))
hdr_ctx.ist_vector[(*nbelem)++] = ist2(&hdr_ctx.timestamp_buffer[0], LOG_LEGACYTIME_LEN+1);
else
hdr_ctx.ist_vector[(*nbelem)++] = ist2("Jan 1 00:00:00 ", LOG_LEGACYTIME_LEN+1);
break;
case LOG_FORMAT_RFC5424:
/* adds rfc5425 version prefix */
hdr_ctx.ist_vector[(*nbelem)++] = ist2("1 ", 2);
if (metadata && metadata[LOG_META_TIME].len == 1 && metadata[LOG_META_TIME].ptr[0] == '-') {
/* submitted len is NILVALUE, it is a valid timestamp for rfc5425 */
hdr_ctx.ist_vector[(*nbelem)++] = metadata[LOG_META_TIME];
hdr_ctx.ist_vector[(*nbelem)++] = ist2(" ", 1);
break;
}
/* let continue as 'timed' and 'iso' format for usual timestamp */
__fallthrough;
case LOG_FORMAT_TIMED:
case LOG_FORMAT_ISO:
/* ISO format ex: '1900:01:01T12:00:00.123456Z'
* '1900:01:01T14:00:00+02:00'
* '1900:01:01T10:00:00.123456-02:00'
*/
if (metadata && metadata[LOG_META_TIME].len >= LOG_ISOTIME_MINLEN) {
hdr_ctx.ist_vector[(*nbelem)++] = metadata[LOG_META_TIME];
hdr_ctx.ist_vector[(*nbelem)++] = ist2(" ", 1);
/* time is set, break immediately */
break;
}
else if (metadata && metadata[LOG_META_TIME].len == LOG_LEGACYTIME_LEN) {
int month;
char *timestamp = metadata[LOG_META_TIME].ptr;
MAJOR: log: introduce log backends Using "mode log" in a backend section turns the proxy in a log backend which can be used to log-balance logs between multiple log targets (udp or tcp servers) log backends can be used as regular log targets using the log directive with "backend@be_name" prefix, like so: | log backend@mybackend local0 A log backend will distribute log messages to servers according to the log load-balancing algorithm that can be set using the "log-balance" option from the log backend section. For now, only the roundrobin algorithm is supported and set by default.
2023-09-13 05:52:31 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
for (month = 0; month < 12; month++)
if (!memcmp(monthname[month], timestamp, 3))
break;
MINOR: log: Extract some code to send syslog messages. This patch extracts the code of __send_log() responsible of sending a syslog message to a syslog destination represented as a logsrv struct to define __do_send_log() function. __send_log() calls __do_send_log() for each syslog destination of a proxy after having prepared some of its parameters.
2019-04-10 02:22:17 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
if (month < 12) {
MINOR: log: Enable the log sampling and load-balancing feature. This patch implements the sampling and load-balancing of log servers configured with "sample" new keyword implemented by this commit: 'MINOR: log: Add "sample" new keyword to "log" lines'. As the list of ranges used to sample the log to balance is ordered, we only have to maintain ->curr_idx member of smp_info struct which is the index of the sample and check if it belongs or not to the current range to decide if we must send it to the log server or not.
2019-04-25 01:42:09 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
/* get local time to retrieve year */
get_localtime(time, &logtime);
MINOR: log: Extract some code to send syslog messages. This patch extracts the code of __send_log() responsible of sending a syslog message to a syslog destination represented as a logsrv struct to define __do_send_log() function. __send_log() calls __do_send_log() for each syslog destination of a proxy after having prepared some of its parameters.
2019-04-10 02:22:17 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
/* year seems changed since log */
if (logtime.tm_mon < month)
logtime.tm_year--;
MEDIUM: logs: atomically check and update the log sample index The log server lock is pretty visible in perf top when using log samples because it's taken for each server in turn while trying to validate and update the log server's index. Let's change this for a CAS, since we have the index and the range at hand now. This allow us to remove the logsrv lock. The test on 4 servers now shows a 3.7 times improvement thanks to much lower contention. Without log sampling a test producing 4.4M logs/s delivers 4.4M logs/s at 21 CPUs used, everything spent in the kernel. After enabling 4 samples (1:4, 2:4, 3:4 and 4:4), the throughput would previously drop to 1.13M log/s with 37 CPUs used and 75% spent in process_send_log(). Now with this change, 4.25M logs/s are emitted, using 26 CPUs and 22% in process_send_log(). That's a 3.7x throughput improvement for a 30% global CPU usage reduction, but in practice it mostly shows that the performance drop caused by having samples is much less noticeable (each of the 4 servers has its index updated for each log). Note that in order to even avoid incrementing an index for each log srv that is consulted, it would be more convenient to have a single index per frontend and apply the modulus on each log server in turn to see if the range has to be updated. It would then only perform one write per range switch. However the place where this is done doesn't have access to a frontend, so some changes would need to be performed for this, and it would require to update the current range independently in each logsrv, which is not necessarily easier since we don't know yet if we can commit it.
2023-09-20 14:34:35 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
/* builds rfc5424 prefix ex: '1900-01-01T' */
len = snprintf(hdr_ctx.timestamp_buffer, sizeof(hdr_ctx.timestamp_buffer),
"%4d-%02d-%c%cT",
logtime.tm_year+1900, month+1,
timestamp[4] != ' ' ? timestamp[4] : '0',
timestamp[5]);
MEDIUM: logs: atomically check and update the log sample index The log server lock is pretty visible in perf top when using log samples because it's taken for each server in turn while trying to validate and update the log server's index. Let's change this for a CAS, since we have the index and the range at hand now. This allow us to remove the logsrv lock. The test on 4 servers now shows a 3.7 times improvement thanks to much lower contention. Without log sampling a test producing 4.4M logs/s delivers 4.4M logs/s at 21 CPUs used, everything spent in the kernel. After enabling 4 samples (1:4, 2:4, 3:4 and 4:4), the throughput would previously drop to 1.13M log/s with 37 CPUs used and 75% spent in process_send_log(). Now with this change, 4.25M logs/s are emitted, using 26 CPUs and 22% in process_send_log(). That's a 3.7x throughput improvement for a 30% global CPU usage reduction, but in practice it mostly shows that the performance drop caused by having samples is much less noticeable (each of the 4 servers has its index updated for each log). Note that in order to even avoid incrementing an index for each log srv that is consulted, it would be more convenient to have a single index per frontend and apply the modulus on each log server in turn to see if the range has to be updated. It would then only perform one write per range switch. However the place where this is done doesn't have access to a frontend, so some changes would need to be performed for this, and it would require to update the current range independently in each logsrv, which is not necessarily easier since we don't know yet if we can commit it.
2023-09-20 14:34:35 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
/* we reused the timestamp_buffer, signal that it does not
* contain local time anymore
*/
hdr_ctx.cur_legacy_time = 0;
if (len == 11) {
hdr_ctx.ist_vector[(*nbelem)++] = ist2(&hdr_ctx.timestamp_buffer[0], len);
/* adds HH:MM:SS from legacy timestamp */
hdr_ctx.ist_vector[(*nbelem)++] = ist2(&timestamp[7], 8);
/* skip secfraq because it is optional */
/* according to rfc: -00:00 means we don't know the timezone */
hdr_ctx.ist_vector[(*nbelem)++] = ist2("-00:00 ", 7);
/* we successfully reuse legacy time, we can break */
break;
MEDIUM: logs: atomically check and update the log sample index The log server lock is pretty visible in perf top when using log samples because it's taken for each server in turn while trying to validate and update the log server's index. Let's change this for a CAS, since we have the index and the range at hand now. This allow us to remove the logsrv lock. The test on 4 servers now shows a 3.7 times improvement thanks to much lower contention. Without log sampling a test producing 4.4M logs/s delivers 4.4M logs/s at 21 CPUs used, everything spent in the kernel. After enabling 4 samples (1:4, 2:4, 3:4 and 4:4), the throughput would previously drop to 1.13M log/s with 37 CPUs used and 75% spent in process_send_log(). Now with this change, 4.25M logs/s are emitted, using 26 CPUs and 22% in process_send_log(). That's a 3.7x throughput improvement for a 30% global CPU usage reduction, but in practice it mostly shows that the performance drop caused by having samples is much less noticeable (each of the 4 servers has its index updated for each log). Note that in order to even avoid incrementing an index for each log srv that is consulted, it would be more convenient to have a single index per frontend and apply the modulus on each log server in turn to see if the range has to be updated. It would then only perform one write per range switch. However the place where this is done doesn't have access to a frontend, so some changes would need to be performed for this, and it would require to update the current range independently in each logsrv, which is not necessarily easier since we don't know yet if we can commit it.
2023-09-20 14:34:35 -04:00
}
MINOR: log: Enable the log sampling and load-balancing feature. This patch implements the sampling and load-balancing of log servers configured with "sample" new keyword implemented by this commit: 'MINOR: log: Add "sample" new keyword to "log" lines'. As the list of ranges used to sample the log to balance is ordered, we only have to maintain ->curr_idx member of smp_info struct which is the index of the sample and check if it belongs or not to the current range to decide if we must send it to the log server or not.
2019-04-25 01:42:09 -04:00
}
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
/* Failed to reuse legacy time, fallback to local iso time */
MAJOR: log: introduce log backends Using "mode log" in a backend section turns the proxy in a log backend which can be used to log-balance logs between multiple log targets (udp or tcp servers) log backends can be used as regular log targets using the log directive with "backend@be_name" prefix, like so: | log backend@mybackend local0 A log backend will distribute log messages to servers according to the log load-balancing algorithm that can be set using the "log-balance" option from the log backend section. For now, only the roundrobin algorithm is supported and set by default.
2023-09-13 05:52:31 -04:00
}
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
hdr_ctx.ist_vector[(*nbelem)++] = ist2(timeofday_as_iso_us(1), LOG_ISOTIME_MAXLEN + 1);
break;
case LOG_FORMAT_PRIO:
case LOG_FORMAT_SHORT:
case LOG_FORMAT_RAW:
break;
case LOG_FORMAT_UNSPEC:
case LOG_FORMATS:
ABORT_NOW();
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-25 20:48:02 -04:00
}
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
/* prepare other meta data, stored into a max of 10 elems */
switch (format) {
case LOG_FORMAT_RFC3164:
if (metadata && metadata[LOG_META_HOST].len) {
hdr_ctx.ist_vector[(*nbelem)++] = metadata[LOG_META_HOST];
hdr_ctx.ist_vector[(*nbelem)++] = ist2(" ", 1);
}
else /* the caller MUST fill the hostname, this field is mandatory */
hdr_ctx.ist_vector[(*nbelem)++] = ist2("localhost ", 10);
__fallthrough;
case LOG_FORMAT_LOCAL:
if (!metadata || !metadata[LOG_META_TAG].len)
break;
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
hdr_ctx.ist_vector[(*nbelem)++] = metadata[LOG_META_TAG];
if (metadata[LOG_META_PID].len) {
hdr_ctx.ist_vector[(*nbelem)++] = ist2("[", 1);
hdr_ctx.ist_vector[(*nbelem)++] = metadata[LOG_META_PID];
hdr_ctx.ist_vector[(*nbelem)++] = ist2("]", 1);
}
hdr_ctx.ist_vector[(*nbelem)++] = ist2(": ", 2);
break;
case LOG_FORMAT_RFC5424:
if (metadata && metadata[LOG_META_HOST].len) {
hdr_ctx.ist_vector[(*nbelem)++] = metadata[LOG_META_HOST];
hdr_ctx.ist_vector[(*nbelem)++] = ist2(" ", 1);
}
else
hdr_ctx.ist_vector[(*nbelem)++] = ist2("- ", 2);
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
if (metadata && metadata[LOG_META_TAG].len) {
hdr_ctx.ist_vector[(*nbelem)++] = metadata[LOG_META_TAG];
hdr_ctx.ist_vector[(*nbelem)++] = ist2(" ", 1);
}
else
hdr_ctx.ist_vector[(*nbelem)++] = ist2("- ", 2);
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
if (metadata && metadata[LOG_META_PID].len) {
hdr_ctx.ist_vector[(*nbelem)++] = metadata[LOG_META_PID];
hdr_ctx.ist_vector[(*nbelem)++] = ist2(" ", 1);
}
else
hdr_ctx.ist_vector[(*nbelem)++] = ist2("- ", 2);
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
if (metadata && metadata[LOG_META_MSGID].len) {
hdr_ctx.ist_vector[(*nbelem)++] = metadata[LOG_META_MSGID];
hdr_ctx.ist_vector[(*nbelem)++] = ist2(" ", 1);
}
else
hdr_ctx.ist_vector[(*nbelem)++] = ist2("- ", 2);
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
if (metadata && metadata[LOG_META_STDATA].len) {
hdr_ctx.ist_vector[(*nbelem)++] = metadata[LOG_META_STDATA];
hdr_ctx.ist_vector[(*nbelem)++] = ist2(" ", 1);
}
else
hdr_ctx.ist_vector[(*nbelem)++] = ist2("- ", 2);
break;
case LOG_FORMAT_PRIO:
case LOG_FORMAT_SHORT:
case LOG_FORMAT_TIMED:
case LOG_FORMAT_ISO:
case LOG_FORMAT_RAW:
break;
case LOG_FORMAT_UNSPEC:
case LOG_FORMATS:
ABORT_NOW();
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
}
REORG: log: reorder send log helpers by dependency order This commit looks messy, but all it does is reorganize send_log() helpers by dependency order to remove the need of forward-declaring some of them. Also, since they're all internal helpers, let's explicitly mark them as static to prevent any misuse.
2024-05-22 09:19:32 -04:00
return hdr_ctx.ist_vector;
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
}
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
MEDIUM: log: add a new cookie flag 'U' to report situations where cookie is not used This happens when a "use-server" rule sets the server instead.
2012-04-05 15:18:22 -04:00
const char sess_cookie[8] = "NIDVEOU7"; /* No cookie, Invalid cookie, cookie for a Down server, Valid cookie, Expired cookie, Old cookie, Unused, unknown */
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
const char sess_set_cookie[8] = "NPDIRU67"; /* No set-cookie, Set-cookie found and left unchanged (passive),
Set-cookie Deleted, Set-Cookie Inserted, Set-cookie Rewritten,
Set-cookie Updated, unknown, unknown */
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
/*
* try to write a cbor byte if there is enough space, or goto out
*/
#define LOG_CBOR_BYTE(x) do { \
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = _lf_cbor_encode_byte(&ctx->encode.cbor, \
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
tmplog, \
dst + maxsize, \
(x)); \
if (ret == NULL) \
goto out; \
tmplog = ret; \
} while (0)
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
/*
* try to write a character if there is enough space, or goto out
*/
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
#define LOGCHAR(x) do { \
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
if ((ctx->options & LOG_OPT_ENCODE_CBOR) && \
ctx->in_text) { \
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
char _x[1]; \
/* encode the char as text chunk since we \
* cannot just throw random bytes and expect \
* cbor decoder to know how to handle them \
*/ \
_x[0] = (x); \
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = cbor_encode_text(&ctx->encode.cbor, \
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
tmplog, \
dst + maxsize, \
_x, sizeof(_x)); \
BUG/MINOR: log/encode: fix potential NULL-dereference in LOGCHAR() When CBOR encoding was added in c614fd3b9 ("MINOR: log: add +cbor encoding option"), in LOGCHAR(), we forgot to check that we don't assign the NULL value to tmplog (as we assume that tmplog cannot be NULL at the end of sess_build_logline()) No backport needed.
2024-04-29 14:19:05 -04:00
if (ret == NULL) \
goto out; \
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
tmplog = ret; \
break; \
} \
if (tmplog < dst + maxsize - 1) { \
*(tmplog++) = (x); \
} else { \
goto out; \
} \
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
} while(0)
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-25 20:48:02 -04:00
MINOR: log: use LOG_VARTEXT_{START,END} to enclose text strings Rename LOGQUOTE_{START,END} macros to more generic LOG_VARTEXT_{START,END} in order to prepare for new encoding types that rely on specific treatment for variable-length texts. No functional change should be expected.
2024-04-16 05:17:55 -04:00
/* indicate that a new variable-length text is starting, sets in_text
* variable to indicate that a var text was started and deals with
* encoding and options to know if some special treatment is needed.
MINOR: log: simplify quotes handling in sess_build_logline() quotes building for some log formats is directly performed under each switch case statement so it would become painful to add other conditions to prevent the quotes from being generated when it's not supported by the the data encoding format for instance (ie: JSON). Let's centralize and simplify quotes handling by adding LOGQUOTE_START() and LOGQUOTE_END() helper macros. If a quotation is started and not explicitly ended, it will be automatically ended at the end of the current logformat node: LOGQUOTE_START() sets 'quote' variable to 1, this way LOGQUOTE_END() only prints the ending quote when needed. LOGQUOTE_END() is systematically called after each node switch-case (after each value). LOGQUOTE_START() does nothing if LOG_OPT_QUOTE isn't set, so does LOGQUOTE_END(). Some rare cases such as %hsl (list of captured headers) required special handling: in this case multiple quoted texts are generated for the same field value so explicit LOGQUOTE_START() + LOGQUOTE_END() combination was needed.
2024-01-10 13:14:25 -05:00
*/
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
#define LOG_VARTEXT_START() do { \
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ctx->in_text = 1; \
if (ctx->options & LOG_OPT_ENCODE_CBOR) { \
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
/* start indefinite-length cbor text */ \
LOG_CBOR_BYTE(0x7F); \
break; \
} \
/* put the text within quotes if JSON encoding \
* is used or quoting is enabled \
*/ \
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
if (ctx->options & \
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
(LOG_OPT_QUOTE | LOG_OPT_ENCODE_JSON)) { \
LOGCHAR('"'); \
} \
MINOR: log: simplify quotes handling in sess_build_logline() quotes building for some log formats is directly performed under each switch case statement so it would become painful to add other conditions to prevent the quotes from being generated when it's not supported by the the data encoding format for instance (ie: JSON). Let's centralize and simplify quotes handling by adding LOGQUOTE_START() and LOGQUOTE_END() helper macros. If a quotation is started and not explicitly ended, it will be automatically ended at the end of the current logformat node: LOGQUOTE_START() sets 'quote' variable to 1, this way LOGQUOTE_END() only prints the ending quote when needed. LOGQUOTE_END() is systematically called after each node switch-case (after each value). LOGQUOTE_START() does nothing if LOG_OPT_QUOTE isn't set, so does LOGQUOTE_END(). Some rare cases such as %hsl (list of captured headers) required special handling: in this case multiple quoted texts are generated for the same field value so explicit LOGQUOTE_START() + LOGQUOTE_END() combination was needed.
2024-01-10 13:14:25 -05:00
} while (0)
MINOR: log: use LOG_VARTEXT_{START,END} to enclose text strings Rename LOGQUOTE_{START,END} macros to more generic LOG_VARTEXT_{START,END} in order to prepare for new encoding types that rely on specific treatment for variable-length texts. No functional change should be expected.
2024-04-16 05:17:55 -04:00
/* properly finish a variable text that was started using LOG_VARTEXT_START
* checks the in_text variable to know if a text was started or not, and
* deals with encoding and options to know if some special treatment is
* needed.
*/
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
#define LOG_VARTEXT_END() do { \
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
if (!ctx->in_text) \
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
break; \
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ctx->in_text = 0; \
if (ctx->options & LOG_OPT_ENCODE_CBOR) { \
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
/* end indefinite-length cbor text with break*/\
LOG_CBOR_BYTE(0xFF); \
break; \
} \
/* add the ending quote if JSON encoding is \
* used or quoting is enabled \
*/ \
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
if (ctx->options & \
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
(LOG_OPT_QUOTE | LOG_OPT_ENCODE_JSON)) { \
LOGCHAR('"'); \
} \
MINOR: log: simplify quotes handling in sess_build_logline() quotes building for some log formats is directly performed under each switch case statement so it would become painful to add other conditions to prevent the quotes from being generated when it's not supported by the the data encoding format for instance (ie: JSON). Let's centralize and simplify quotes handling by adding LOGQUOTE_START() and LOGQUOTE_END() helper macros. If a quotation is started and not explicitly ended, it will be automatically ended at the end of the current logformat node: LOGQUOTE_START() sets 'quote' variable to 1, this way LOGQUOTE_END() only prints the ending quote when needed. LOGQUOTE_END() is systematically called after each node switch-case (after each value). LOGQUOTE_START() does nothing if LOG_OPT_QUOTE isn't set, so does LOGQUOTE_END(). Some rare cases such as %hsl (list of captured headers) required special handling: in this case multiple quoted texts are generated for the same field value so explicit LOGQUOTE_START() + LOGQUOTE_END() combination was needed.
2024-01-10 13:14:25 -05:00
} while (0)
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
MINOR: log: print metadata prefixes separately in sess_build_logline() Some log variables may be prefixed with specific chars that represent extra informations that are relevant with it but are are not directly part of the "raw" value. ie: '+' char is prepended before some values when "option logasap" is used to indicate that the value has not yet reached its final value. However, as those "metadata" are printed using the general purpose LOGCHAR() printing helper, it's not easy to tell if they are part of the base value or not. In this patch we add the LOGMETACHAR() helper that is a wrapper for LOGCHAR(). The goal is to prepare for adding some logic to prevent such additional infos from being generated when not relevant or needed.
2024-01-30 12:01:15 -05:00
/* Prints additional logvalue hint represented by <chr>.
* It is useful to express that <chr> is not part of the "raw" value and
* should be considered as optional metadata instead.
*/
#define LOGMETACHAR(chr) do { \
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
/* ignored when encoding is used */ \
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
if (ctx->options & LOG_OPT_ENCODE) \
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
break; \
MINOR: log: print metadata prefixes separately in sess_build_logline() Some log variables may be prefixed with specific chars that represent extra informations that are relevant with it but are are not directly part of the "raw" value. ie: '+' char is prepended before some values when "option logasap" is used to indicate that the value has not yet reached its final value. However, as those "metadata" are printed using the general purpose LOGCHAR() printing helper, it's not easy to tell if they are part of the base value or not. In this patch we add the LOGMETACHAR() helper that is a wrapper for LOGCHAR(). The goal is to prepare for adding some logic to prevent such additional infos from being generated when not relevant or needed.
2024-01-30 12:01:15 -05:00
LOGCHAR(chr); \
} while (0)
MINOR: log: automate string array construction in sess_build_logline() make it so string array construction is performed by dedicated macro helpers instead of manual char insertion between string members. The goal is to easily be able to support multiple forms of array construction depending on the data encoding format (raw, json..). Only %hrl and %hsl logformats are concerned.
2024-01-31 08:37:07 -05:00
/* indicate the start of a string array */
#define LOG_STRARRAY_START() do { \
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
if (ctx->options & LOG_OPT_ENCODE_JSON) \
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
LOGCHAR('['); \
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
if (ctx->options & LOG_OPT_ENCODE_CBOR) { \
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
/* start indefinite-length array */ \
LOG_CBOR_BYTE(0x9F); \
} \
MINOR: log: automate string array construction in sess_build_logline() make it so string array construction is performed by dedicated macro helpers instead of manual char insertion between string members. The goal is to easily be able to support multiple forms of array construction depending on the data encoding format (raw, json..). Only %hrl and %hsl logformats are concerned.
2024-01-31 08:37:07 -05:00
} while (0)
/* indicate that a new element is added to the string array */
#define LOG_STRARRAY_NEXT() do { \
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
if (ctx->options & LOG_OPT_ENCODE_CBOR) \
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
break; \
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
if (ctx->options & LOG_OPT_ENCODE_JSON) { \
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
LOGCHAR(','); \
LOGCHAR(' '); \
} \
else \
LOGCHAR(' '); \
MINOR: log: automate string array construction in sess_build_logline() make it so string array construction is performed by dedicated macro helpers instead of manual char insertion between string members. The goal is to easily be able to support multiple forms of array construction depending on the data encoding format (raw, json..). Only %hrl and %hsl logformats are concerned.
2024-01-31 08:37:07 -05:00
} while (0)
/* indicate the end of a string array */
#define LOG_STRARRAY_END() do { \
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
if (ctx->options & LOG_OPT_ENCODE_JSON) \
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
LOGCHAR(']'); \
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
if (ctx->options & LOG_OPT_ENCODE_CBOR) { \
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
/* cbor break */ \
LOG_CBOR_BYTE(0xFF); \
} \
MINOR: log: automate string array construction in sess_build_logline() make it so string array construction is performed by dedicated macro helpers instead of manual char insertion between string members. The goal is to easily be able to support multiple forms of array construction depending on the data encoding format (raw, json..). Only %hrl and %hsl logformats are concerned.
2024-01-31 08:37:07 -05:00
} while (0)
MEDIUM: initcall: use initcalls for a few initialization functions signal_init(), init_log(), init_stream(), and init_task() all used to only preset some values and lists. This needs to be done very early to provide a reliable interface to all other users. The calls used to be explicit in haproxy.c:init(). Now they're placed in initcalls at the STG_PREPARE stage. The functions are not exported anymore.
2018-11-26 10:31:20 -05:00
/* Initializes some log data at boot */
static void init_log()
MEDIUM: log: add a new log format flag "E" The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It mostly makes sense to use it in the RFC5424 structured-data log formats. Example: log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 07:23:03 -05:00
{
char *tmp;
REORG: http: move the log encoding tables to log.c There are 3 tables in proto_http which are used exclusively by logs : hdr_encode_map[], url_encode_map[] and http_encode_map[]. They indicate what characters are safe to be emitted in logs depending on the part of the message where they are placed. Let's move this to log.c, as well as its initialization. It's worth noting that the rfc5424 map was already initialized there.
2018-09-10 12:16:53 -04:00
int i;
MEDIUM: log: add a new log format flag "E" The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It mostly makes sense to use it in the RFC5424 structured-data log formats. Example: log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 07:23:03 -05:00
MINOR: log: add no_escape_map to bypass escape with _lf_encode_bytes() Providing no_escape_map as <map> argument to _lf_encode_bytes() function will make the function skip escaping since the map is empty. This is for convenience, as it might be useful to call lf_encode_chunk() to encoding binary data without escaping it.
2024-04-26 08:23:43 -04:00
/* Initialize the no escape map, which may be used to bypass escaping */
memset(no_escape_map, 0, sizeof(no_escape_map));
MEDIUM: log: add a new log format flag "E" The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It mostly makes sense to use it in the RFC5424 structured-data log formats. Example: log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 07:23:03 -05:00
/* Initialize the escape map for the RFC5424 structured-data : '"\]'
* inside PARAM-VALUE should be escaped with '\' as prefix.
* See https://tools.ietf.org/html/rfc5424#section-6.3.3 for more
* details.
*/
memset(rfc5424_escape_map, 0, sizeof(rfc5424_escape_map));
tmp = "\"\\]";
while (*tmp) {
MINOR: logs: use the new bitmap functions instead of fd_sets for encoding maps The fd_sets we've been using in the log encoding functions are not portable and were shown to break at least under Cygwin. This patch gets rid of them in favor of the new bitmap functions. It was verified with the config below that the log output was exactly the same before and after the change : defaults mode http option httplog log stdout local0 timeout client 1s timeout server 1s timeout connect 1s frontend foo bind :8001 capture request header chars len 255 backend bar option httpchk "GET" "/" "HTTP/1.0\r\nchars: \x01\x02\x03\x04\x05\x06\x07\x08\x09\x0b\x0c\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f\x20\x21\x22\x23\x24\x25\x26\x27\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f\x30\x31\x32\x33\x34\x35\x36\x37\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f\x40\x41\x42\x43\x44\x45\x46\x47\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f\x50\x51\x52\x53\x54\x55\x56\x57\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f\x60\x61\x62\x63\x64\x65\x66\x67\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f\x70\x71\x72\x73\x74\x75\x76\x77\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" server foo 127.0.0.1:8001 check
2019-06-07 05:10:07 -04:00
ha_bit_set(*tmp, rfc5424_escape_map);
MEDIUM: log: add a new log format flag "E" The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It mostly makes sense to use it in the RFC5424 structured-data log formats. Example: log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 07:23:03 -05:00
tmp++;
}
REORG: http: move the log encoding tables to log.c There are 3 tables in proto_http which are used exclusively by logs : hdr_encode_map[], url_encode_map[] and http_encode_map[]. They indicate what characters are safe to be emitted in logs depending on the part of the message where they are placed. Let's move this to log.c, as well as its initialization. It's worth noting that the rfc5424 map was already initialized there.
2018-09-10 12:16:53 -04:00
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
/* Initialize the escape map for JSON strings : '"\' */
memset(json_escape_map, 0, sizeof(json_escape_map));
tmp = "\"\\";
while (*tmp) {
ha_bit_set(*tmp, json_escape_map);
tmp++;
}
REORG: http: move the log encoding tables to log.c There are 3 tables in proto_http which are used exclusively by logs : hdr_encode_map[], url_encode_map[] and http_encode_map[]. They indicate what characters are safe to be emitted in logs depending on the part of the message where they are placed. Let's move this to log.c, as well as its initialization. It's worth noting that the rfc5424 map was already initialized there.
2018-09-10 12:16:53 -04:00
/* initialize the log header encoding map : '{|}"#' should be encoded with
* '#' as prefix, as well as non-printable characters ( <32 or >= 127 ).
* URL encoding only requires '"', '#' to be encoded as well as non-
* printable characters above.
*/
memset(hdr_encode_map, 0, sizeof(hdr_encode_map));
memset(url_encode_map, 0, sizeof(url_encode_map));
for (i = 0; i < 32; i++) {
MINOR: logs: use the new bitmap functions instead of fd_sets for encoding maps The fd_sets we've been using in the log encoding functions are not portable and were shown to break at least under Cygwin. This patch gets rid of them in favor of the new bitmap functions. It was verified with the config below that the log output was exactly the same before and after the change : defaults mode http option httplog log stdout local0 timeout client 1s timeout server 1s timeout connect 1s frontend foo bind :8001 capture request header chars len 255 backend bar option httpchk "GET" "/" "HTTP/1.0\r\nchars: \x01\x02\x03\x04\x05\x06\x07\x08\x09\x0b\x0c\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f\x20\x21\x22\x23\x24\x25\x26\x27\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f\x30\x31\x32\x33\x34\x35\x36\x37\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f\x40\x41\x42\x43\x44\x45\x46\x47\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f\x50\x51\x52\x53\x54\x55\x56\x57\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f\x60\x61\x62\x63\x64\x65\x66\x67\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f\x70\x71\x72\x73\x74\x75\x76\x77\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" server foo 127.0.0.1:8001 check
2019-06-07 05:10:07 -04:00
ha_bit_set(i, hdr_encode_map);
ha_bit_set(i, url_encode_map);
REORG: http: move the log encoding tables to log.c There are 3 tables in proto_http which are used exclusively by logs : hdr_encode_map[], url_encode_map[] and http_encode_map[]. They indicate what characters are safe to be emitted in logs depending on the part of the message where they are placed. Let's move this to log.c, as well as its initialization. It's worth noting that the rfc5424 map was already initialized there.
2018-09-10 12:16:53 -04:00
}
for (i = 127; i < 256; i++) {
MINOR: logs: use the new bitmap functions instead of fd_sets for encoding maps The fd_sets we've been using in the log encoding functions are not portable and were shown to break at least under Cygwin. This patch gets rid of them in favor of the new bitmap functions. It was verified with the config below that the log output was exactly the same before and after the change : defaults mode http option httplog log stdout local0 timeout client 1s timeout server 1s timeout connect 1s frontend foo bind :8001 capture request header chars len 255 backend bar option httpchk "GET" "/" "HTTP/1.0\r\nchars: \x01\x02\x03\x04\x05\x06\x07\x08\x09\x0b\x0c\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f\x20\x21\x22\x23\x24\x25\x26\x27\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f\x30\x31\x32\x33\x34\x35\x36\x37\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f\x40\x41\x42\x43\x44\x45\x46\x47\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f\x50\x51\x52\x53\x54\x55\x56\x57\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f\x60\x61\x62\x63\x64\x65\x66\x67\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f\x70\x71\x72\x73\x74\x75\x76\x77\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" server foo 127.0.0.1:8001 check
2019-06-07 05:10:07 -04:00
ha_bit_set(i, hdr_encode_map);
ha_bit_set(i, url_encode_map);
REORG: http: move the log encoding tables to log.c There are 3 tables in proto_http which are used exclusively by logs : hdr_encode_map[], url_encode_map[] and http_encode_map[]. They indicate what characters are safe to be emitted in logs depending on the part of the message where they are placed. Let's move this to log.c, as well as its initialization. It's worth noting that the rfc5424 map was already initialized there.
2018-09-10 12:16:53 -04:00
}
tmp = "\"#{|}";
while (*tmp) {
MINOR: logs: use the new bitmap functions instead of fd_sets for encoding maps The fd_sets we've been using in the log encoding functions are not portable and were shown to break at least under Cygwin. This patch gets rid of them in favor of the new bitmap functions. It was verified with the config below that the log output was exactly the same before and after the change : defaults mode http option httplog log stdout local0 timeout client 1s timeout server 1s timeout connect 1s frontend foo bind :8001 capture request header chars len 255 backend bar option httpchk "GET" "/" "HTTP/1.0\r\nchars: \x01\x02\x03\x04\x05\x06\x07\x08\x09\x0b\x0c\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f\x20\x21\x22\x23\x24\x25\x26\x27\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f\x30\x31\x32\x33\x34\x35\x36\x37\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f\x40\x41\x42\x43\x44\x45\x46\x47\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f\x50\x51\x52\x53\x54\x55\x56\x57\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f\x60\x61\x62\x63\x64\x65\x66\x67\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f\x70\x71\x72\x73\x74\x75\x76\x77\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" server foo 127.0.0.1:8001 check
2019-06-07 05:10:07 -04:00
ha_bit_set(*tmp, hdr_encode_map);
REORG: http: move the log encoding tables to log.c There are 3 tables in proto_http which are used exclusively by logs : hdr_encode_map[], url_encode_map[] and http_encode_map[]. They indicate what characters are safe to be emitted in logs depending on the part of the message where they are placed. Let's move this to log.c, as well as its initialization. It's worth noting that the rfc5424 map was already initialized there.
2018-09-10 12:16:53 -04:00
tmp++;
}
tmp = "\"#";
while (*tmp) {
MINOR: logs: use the new bitmap functions instead of fd_sets for encoding maps The fd_sets we've been using in the log encoding functions are not portable and were shown to break at least under Cygwin. This patch gets rid of them in favor of the new bitmap functions. It was verified with the config below that the log output was exactly the same before and after the change : defaults mode http option httplog log stdout local0 timeout client 1s timeout server 1s timeout connect 1s frontend foo bind :8001 capture request header chars len 255 backend bar option httpchk "GET" "/" "HTTP/1.0\r\nchars: \x01\x02\x03\x04\x05\x06\x07\x08\x09\x0b\x0c\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f\x20\x21\x22\x23\x24\x25\x26\x27\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f\x30\x31\x32\x33\x34\x35\x36\x37\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f\x40\x41\x42\x43\x44\x45\x46\x47\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f\x50\x51\x52\x53\x54\x55\x56\x57\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f\x60\x61\x62\x63\x64\x65\x66\x67\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f\x70\x71\x72\x73\x74\x75\x76\x77\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" server foo 127.0.0.1:8001 check
2019-06-07 05:10:07 -04:00
ha_bit_set(*tmp, url_encode_map);
REORG: http: move the log encoding tables to log.c There are 3 tables in proto_http which are used exclusively by logs : hdr_encode_map[], url_encode_map[] and http_encode_map[]. They indicate what characters are safe to be emitted in logs depending on the part of the message where they are placed. Let's move this to log.c, as well as its initialization. It's worth noting that the rfc5424 map was already initialized there.
2018-09-10 12:16:53 -04:00
tmp++;
}
/* initialize the http header encoding map. The draft httpbis define the
* header content as:
*
* HTTP-message = start-line
* *( header-field CRLF )
* CRLF
* [ message-body ]
* header-field = field-name ":" OWS field-value OWS
* field-value = *( field-content / obs-fold )
* field-content = field-vchar [ 1*( SP / HTAB ) field-vchar ]
* obs-fold = CRLF 1*( SP / HTAB )
* field-vchar = VCHAR / obs-text
* VCHAR = %x21-7E
* obs-text = %x80-FF
*
* All the chars are encoded except "VCHAR", "obs-text", SP and HTAB.
* The encoded chars are form 0x00 to 0x08, 0x0a to 0x1f and 0x7f. The
CLEANUP: Fix typos in the log subsystem Fix some misspells in the code comments of the log subsystem.
2018-11-15 15:10:04 -05:00
* "obs-fold" is voluntarily forgotten because haproxy remove this.
REORG: http: move the log encoding tables to log.c There are 3 tables in proto_http which are used exclusively by logs : hdr_encode_map[], url_encode_map[] and http_encode_map[]. They indicate what characters are safe to be emitted in logs depending on the part of the message where they are placed. Let's move this to log.c, as well as its initialization. It's worth noting that the rfc5424 map was already initialized there.
2018-09-10 12:16:53 -04:00
*/
memset(http_encode_map, 0, sizeof(http_encode_map));
for (i = 0x00; i <= 0x08; i++)
MINOR: logs: use the new bitmap functions instead of fd_sets for encoding maps The fd_sets we've been using in the log encoding functions are not portable and were shown to break at least under Cygwin. This patch gets rid of them in favor of the new bitmap functions. It was verified with the config below that the log output was exactly the same before and after the change : defaults mode http option httplog log stdout local0 timeout client 1s timeout server 1s timeout connect 1s frontend foo bind :8001 capture request header chars len 255 backend bar option httpchk "GET" "/" "HTTP/1.0\r\nchars: \x01\x02\x03\x04\x05\x06\x07\x08\x09\x0b\x0c\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f\x20\x21\x22\x23\x24\x25\x26\x27\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f\x30\x31\x32\x33\x34\x35\x36\x37\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f\x40\x41\x42\x43\x44\x45\x46\x47\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f\x50\x51\x52\x53\x54\x55\x56\x57\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f\x60\x61\x62\x63\x64\x65\x66\x67\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f\x70\x71\x72\x73\x74\x75\x76\x77\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" server foo 127.0.0.1:8001 check
2019-06-07 05:10:07 -04:00
ha_bit_set(i, http_encode_map);
REORG: http: move the log encoding tables to log.c There are 3 tables in proto_http which are used exclusively by logs : hdr_encode_map[], url_encode_map[] and http_encode_map[]. They indicate what characters are safe to be emitted in logs depending on the part of the message where they are placed. Let's move this to log.c, as well as its initialization. It's worth noting that the rfc5424 map was already initialized there.
2018-09-10 12:16:53 -04:00
for (i = 0x0a; i <= 0x1f; i++)
MINOR: logs: use the new bitmap functions instead of fd_sets for encoding maps The fd_sets we've been using in the log encoding functions are not portable and were shown to break at least under Cygwin. This patch gets rid of them in favor of the new bitmap functions. It was verified with the config below that the log output was exactly the same before and after the change : defaults mode http option httplog log stdout local0 timeout client 1s timeout server 1s timeout connect 1s frontend foo bind :8001 capture request header chars len 255 backend bar option httpchk "GET" "/" "HTTP/1.0\r\nchars: \x01\x02\x03\x04\x05\x06\x07\x08\x09\x0b\x0c\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f\x20\x21\x22\x23\x24\x25\x26\x27\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f\x30\x31\x32\x33\x34\x35\x36\x37\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f\x40\x41\x42\x43\x44\x45\x46\x47\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f\x50\x51\x52\x53\x54\x55\x56\x57\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f\x60\x61\x62\x63\x64\x65\x66\x67\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f\x70\x71\x72\x73\x74\x75\x76\x77\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" server foo 127.0.0.1:8001 check
2019-06-07 05:10:07 -04:00
ha_bit_set(i, http_encode_map);
ha_bit_set(0x7f, http_encode_map);
MEDIUM: log: add a new log format flag "E" The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It mostly makes sense to use it in the RFC5424 structured-data log formats. Example: log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 07:23:03 -05:00
}
MEDIUM: initcall: use initcalls for a few initialization functions signal_init(), init_log(), init_stream(), and init_task() all used to only preset some values and lists. This needs to be done very early to provide a reliable interface to all other users. The calls used to be explicit in haproxy.c:init(). Now they're placed in initcalls at the STG_PREPARE stage. The functions are not exported anymore.
2018-11-26 10:31:20 -05:00
INITCALL0(STG_PREPARE, init_log);
MINOR: logs: Use dedicated function to init/deinit log buffers Now, we use init_log_buffers and deinit_log_buffers to, respectively, initialize and deinitialize log buffers used for syslog messages. These functions have been introduced to be used by threads, to deal with thread-local log buffers.
2017-07-26 09:33:35 -04:00
/* Initialize log buffers used for syslog messages */
int init_log_buffers()
{
logline = my_realloc2(logline, global.max_syslog_len + 1);
logline_rfc5424 = my_realloc2(logline_rfc5424, global.max_syslog_len + 1);
MEDIUM: log/sink: re-work and merge of build message API. This patch merges build message code between sink and log and introduce a new API based on struct ist array to prepare message header with zero copy, targeting the log forwarding feature. Log format 'iso' and 'timed' are now avalaible on logs line. A new log format 'priority' is also added.
2020-07-06 09:54:06 -04:00
if (!logline || !logline_rfc5424)
MINOR: logs: Use dedicated function to init/deinit log buffers Now, we use init_log_buffers and deinit_log_buffers to, respectively, initialize and deinitialize log buffers used for syslog messages. These functions have been introduced to be used by threads, to deal with thread-local log buffers.
2017-07-26 09:33:35 -04:00
return 0;
return 1;
}
/* Deinitialize log buffers used for syslog messages */
void deinit_log_buffers()
{
free(logline);
free(logline_rfc5424);
logline = NULL;
logline_rfc5424 = NULL;
}
MINOR: log: move log-forwarders cleanup in log.c Move the log-forwarded proxies cleanup from global deinit() function into log dedicated deinit function. No backport needed.
2023-07-03 12:07:30 -04:00
/* Deinitialize log forwarder proxies used for syslog messages */
void deinit_log_forward()
{
struct proxy *p, *p0;
p = cfg_log_forward;
/* we need to manually clean cfg_log_forward proxy list */
while (p) {
p0 = p;
p = p->next;
free_proxy(p0);
}
}
MINOR: log: add free_logformat_node() helper function Function may be used to free a single logformat node.
2024-02-22 09:03:29 -05:00
/* Releases memory for a single log-format node */
void free_logformat_node(struct logformat_node *node)
{
if (!node)
return;
release_sample_expr(node->expr);
node->expr = NULL;
ha_free(&node->name);
ha_free(&node->arg);
ha_free(&node);
}
CLEANUP: log: deinitialization of the log buffer in one function In several places in the source, there was the same block of code that was used to deinitialize the log buffer. There were even two functions that did this, but they were called only from the code that is in the same source file (free_tcpcheck_fmt() in src/tcpcheck.c and free_logformat_list() in src/proxy.c - they were both static functions). The function free_logformat_list() was moved from the file src/proxy.c to src/log.c, and a check of the list before freeing the memory was added to that function.
2024-01-29 21:14:09 -05:00
/* Releases memory allocated for a log-format string */
void free_logformat_list(struct list *fmt)
{
struct logformat_node *lf, *lfb;
if ((fmt == NULL) || LIST_ISEMPTY(fmt))
return;
list_for_each_entry_safe(lf, lfb, fmt, list) {
LIST_DELETE(&lf->list);
MINOR: log: add free_logformat_node() helper function Function may be used to free a single logformat node.
2024-02-22 09:03:29 -05:00
free_logformat_node(lf);
CLEANUP: log: deinitialization of the log buffer in one function In several places in the source, there was the same block of code that was used to deinitialize the log buffer. There were even two functions that did this, but they were called only from the code that is in the same source file (free_tcpcheck_fmt() in src/tcpcheck.c and free_logformat_list() in src/proxy.c - they were both static functions). The function free_logformat_list() was moved from the file src/proxy.c to src/log.c, and a check of the list before freeing the memory was added to that function.
2024-01-29 21:14:09 -05:00
}
}
MEDIUM: tree-wide: add logformat expressions wrapper log format expressions are broadly used within the code: once they are parsed from input string, they are converted to a linked list of logformat nodes. We're starting to face some limitations because we're simply storing the converted expression as a generic logformat_node list. The first issue we're facing is that storing logformat expressions that way doesn't allow us to add metadata alongside the list, which is part of the prerequites for implementing log-profiles. Another issue with storing logformat expressions as generic lists of logformat_node elements is that it's starting to become really hard to tell when we rely on logformat expressions or not in the code given that there isn't always a comment near the list declaration or manipulation to indicate that it's relying on logformat expressions under the hood, so this adds some complexity for code maintenance. This patch looks quite impressive due to changes in a lot of header and source files (since logformat expressions are broadly used), but it does a simple thing: it defines the lf_expr structure which itself holds a generic list of logformat nodes, and then declares some helpers to manipulate lf_expr elements and fixes the code so that we now exclusively manipulate logformat_node lists as lf_expr elements outside of log.c. For now, lf_expr struct only contains the list of logformat nodes (no additional metadata), but now that we have dedicated type and helpers, doing so in the future won't be problematic at all and won't require extensive code changes.
2024-02-23 09:57:21 -05:00
/* Prepares log-format expression struct */
void lf_expr_init(struct lf_expr *expr)
{
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
LIST_INIT(&expr->list);
MEDIUM: log: add compiling logic to logformat expressions split parse_logformat_string() into two functions: parse_logformat_string() sticks to the same behavior, but now becomes an helper for lf_expr_compile() which uses explicit arguments so that it becomes possible to use lf_expr_compile() without a proxy, but also compile an expression which was previously prepared for compiling (set string and config hints within the logformat expression to avoid manually storing string and config context if the compiling step happens later). lf_expr_dup() may be used to duplicate an expression before it is compiled, lf_expr_xfer() now makes sure that the input logformat is already compiled. This is some prerequisite works for log-profiles implementation, no functional change should be expected.
2024-02-29 08:54:43 -05:00
expr->flags = LF_FL_NONE;
expr->str = NULL;
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
expr->conf.file = NULL;
expr->conf.line = 0;
MEDIUM: tree-wide: add logformat expressions wrapper log format expressions are broadly used within the code: once they are parsed from input string, they are converted to a linked list of logformat nodes. We're starting to face some limitations because we're simply storing the converted expression as a generic logformat_node list. The first issue we're facing is that storing logformat expressions that way doesn't allow us to add metadata alongside the list, which is part of the prerequites for implementing log-profiles. Another issue with storing logformat expressions as generic lists of logformat_node elements is that it's starting to become really hard to tell when we rely on logformat expressions or not in the code given that there isn't always a comment near the list declaration or manipulation to indicate that it's relying on logformat expressions under the hood, so this adds some complexity for code maintenance. This patch looks quite impressive due to changes in a lot of header and source files (since logformat expressions are broadly used), but it does a simple thing: it defines the lf_expr structure which itself holds a generic list of logformat nodes, and then declares some helpers to manipulate lf_expr elements and fixes the code so that we now exclusively manipulate logformat_node lists as lf_expr elements outside of log.c. For now, lf_expr struct only contains the list of logformat nodes (no additional metadata), but now that we have dedicated type and helpers, doing so in the future won't be problematic at all and won't require extensive code changes.
2024-02-23 09:57:21 -05:00
}
/* Releases and resets a log-format expression */
void lf_expr_deinit(struct lf_expr *expr)
{
MEDIUM: log: add compiling logic to logformat expressions split parse_logformat_string() into two functions: parse_logformat_string() sticks to the same behavior, but now becomes an helper for lf_expr_compile() which uses explicit arguments so that it becomes possible to use lf_expr_compile() without a proxy, but also compile an expression which was previously prepared for compiling (set string and config hints within the logformat expression to avoid manually storing string and config context if the compiling step happens later). lf_expr_dup() may be used to duplicate an expression before it is compiled, lf_expr_xfer() now makes sure that the input logformat is already compiled. This is some prerequisite works for log-profiles implementation, no functional change should be expected.
2024-02-29 08:54:43 -05:00
if ((expr->flags & LF_FL_COMPILED))
MINOR: log: store lf_expr nodes inside substruct Add another struct level inside lf_expr struct to allow new information to be stored alongside lf_expr nodes.
2024-03-25 06:29:58 -04:00
free_logformat_list(&expr->nodes.list);
MEDIUM: log: add compiling logic to logformat expressions split parse_logformat_string() into two functions: parse_logformat_string() sticks to the same behavior, but now becomes an helper for lf_expr_compile() which uses explicit arguments so that it becomes possible to use lf_expr_compile() without a proxy, but also compile an expression which was previously prepared for compiling (set string and config hints within the logformat expression to avoid manually storing string and config context if the compiling step happens later). lf_expr_dup() may be used to duplicate an expression before it is compiled, lf_expr_xfer() now makes sure that the input logformat is already compiled. This is some prerequisite works for log-profiles implementation, no functional change should be expected.
2024-02-29 08:54:43 -05:00
else
MEDIUM: proxy/log: leverage lf_expr API for logformat preparsing Currently, the way proxy-oriented logformat directives are handled is way too complicated. Indeed, "log-format", "log-format-error", "log-format-sd" and "unique-id-format" all rely on preparsing hints stored inside proxy->conf member struct. Those preparsing hints include the original string that should be compiled once the proxy parameters are known plus the config file and line number where the string was found to generate precise error messages in case of failure during the compiling process that happens within check_config_validity(). Now that lf_expr API permits to compile a lf_expr struct that was previously prepared (with original string and config hints), let's leverage lf_expr_compile() from check_config_validity() and instead of relying on individual proxy->conf hints for each logformat expression, store string and config hints in the lf_expr struct directly and use lf_expr helpers funcs to handle them when relevant (ie: original logformat string freeing is now done at a central place inside lf_expr_deinit(), which allows for some simplifications) Doing so allows us to greatly simplify the preparsing logic for those 4 proxy directives, and to finally save some space in the proxy struct. Also, since httpclient proxy has its "logformat" automatically compiled in check_config_validity(), we now use the file hint from the logformat expression struct to set an explicit name that will be reported in case of error ("parsing [httpclient:0] : ...") and remove the extraneous check in httpclient_precheck() (logformat was parsed twice previously..)
2024-03-05 09:44:43 -05:00
logformat_str_free(&expr->str);
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
free(expr->conf.file);
/* remove from parent list (if any) */
LIST_DEL_INIT(&expr->list);
MEDIUM: tree-wide: add logformat expressions wrapper log format expressions are broadly used within the code: once they are parsed from input string, they are converted to a linked list of logformat nodes. We're starting to face some limitations because we're simply storing the converted expression as a generic logformat_node list. The first issue we're facing is that storing logformat expressions that way doesn't allow us to add metadata alongside the list, which is part of the prerequites for implementing log-profiles. Another issue with storing logformat expressions as generic lists of logformat_node elements is that it's starting to become really hard to tell when we rely on logformat expressions or not in the code given that there isn't always a comment near the list declaration or manipulation to indicate that it's relying on logformat expressions under the hood, so this adds some complexity for code maintenance. This patch looks quite impressive due to changes in a lot of header and source files (since logformat expressions are broadly used), but it does a simple thing: it defines the lf_expr structure which itself holds a generic list of logformat nodes, and then declares some helpers to manipulate lf_expr elements and fixes the code so that we now exclusively manipulate logformat_node lists as lf_expr elements outside of log.c. For now, lf_expr struct only contains the list of logformat nodes (no additional metadata), but now that we have dedicated type and helpers, doing so in the future won't be problematic at all and won't require extensive code changes.
2024-02-23 09:57:21 -05:00
lf_expr_init(expr);
}
MEDIUM: log: add compiling logic to logformat expressions split parse_logformat_string() into two functions: parse_logformat_string() sticks to the same behavior, but now becomes an helper for lf_expr_compile() which uses explicit arguments so that it becomes possible to use lf_expr_compile() without a proxy, but also compile an expression which was previously prepared for compiling (set string and config hints within the logformat expression to avoid manually storing string and config context if the compiling step happens later). lf_expr_dup() may be used to duplicate an expression before it is compiled, lf_expr_xfer() now makes sure that the input logformat is already compiled. This is some prerequisite works for log-profiles implementation, no functional change should be expected.
2024-02-29 08:54:43 -05:00
/* Transfer a compiled log-format expression from <src> to <dst>
MEDIUM: tree-wide: add logformat expressions wrapper log format expressions are broadly used within the code: once they are parsed from input string, they are converted to a linked list of logformat nodes. We're starting to face some limitations because we're simply storing the converted expression as a generic logformat_node list. The first issue we're facing is that storing logformat expressions that way doesn't allow us to add metadata alongside the list, which is part of the prerequites for implementing log-profiles. Another issue with storing logformat expressions as generic lists of logformat_node elements is that it's starting to become really hard to tell when we rely on logformat expressions or not in the code given that there isn't always a comment near the list declaration or manipulation to indicate that it's relying on logformat expressions under the hood, so this adds some complexity for code maintenance. This patch looks quite impressive due to changes in a lot of header and source files (since logformat expressions are broadly used), but it does a simple thing: it defines the lf_expr structure which itself holds a generic list of logformat nodes, and then declares some helpers to manipulate lf_expr elements and fixes the code so that we now exclusively manipulate logformat_node lists as lf_expr elements outside of log.c. For now, lf_expr struct only contains the list of logformat nodes (no additional metadata), but now that we have dedicated type and helpers, doing so in the future won't be problematic at all and won't require extensive code changes.
2024-02-23 09:57:21 -05:00
* at the end of the operation, <src> is reset
*/
void lf_expr_xfer(struct lf_expr *src, struct lf_expr *dst)
{
struct logformat_node *lf, *lfb;
/* first, reset any existing expr */
lf_expr_deinit(dst);
MEDIUM: log: add compiling logic to logformat expressions split parse_logformat_string() into two functions: parse_logformat_string() sticks to the same behavior, but now becomes an helper for lf_expr_compile() which uses explicit arguments so that it becomes possible to use lf_expr_compile() without a proxy, but also compile an expression which was previously prepared for compiling (set string and config hints within the logformat expression to avoid manually storing string and config context if the compiling step happens later). lf_expr_dup() may be used to duplicate an expression before it is compiled, lf_expr_xfer() now makes sure that the input logformat is already compiled. This is some prerequisite works for log-profiles implementation, no functional change should be expected.
2024-02-29 08:54:43 -05:00
BUG_ON(!(src->flags & LF_FL_COMPILED));
MEDIUM: tree-wide: add logformat expressions wrapper log format expressions are broadly used within the code: once they are parsed from input string, they are converted to a linked list of logformat nodes. We're starting to face some limitations because we're simply storing the converted expression as a generic logformat_node list. The first issue we're facing is that storing logformat expressions that way doesn't allow us to add metadata alongside the list, which is part of the prerequites for implementing log-profiles. Another issue with storing logformat expressions as generic lists of logformat_node elements is that it's starting to become really hard to tell when we rely on logformat expressions or not in the code given that there isn't always a comment near the list declaration or manipulation to indicate that it's relying on logformat expressions under the hood, so this adds some complexity for code maintenance. This patch looks quite impressive due to changes in a lot of header and source files (since logformat expressions are broadly used), but it does a simple thing: it defines the lf_expr structure which itself holds a generic list of logformat nodes, and then declares some helpers to manipulate lf_expr elements and fixes the code so that we now exclusively manipulate logformat_node lists as lf_expr elements outside of log.c. For now, lf_expr struct only contains the list of logformat nodes (no additional metadata), but now that we have dedicated type and helpers, doing so in the future won't be problematic at all and won't require extensive code changes.
2024-02-23 09:57:21 -05:00
/* then proceed with transfer between <src> and <dst> */
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
dst->conf.file = src->conf.file;
dst->conf.line = src->conf.line;
MEDIUM: log: add compiling logic to logformat expressions split parse_logformat_string() into two functions: parse_logformat_string() sticks to the same behavior, but now becomes an helper for lf_expr_compile() which uses explicit arguments so that it becomes possible to use lf_expr_compile() without a proxy, but also compile an expression which was previously prepared for compiling (set string and config hints within the logformat expression to avoid manually storing string and config context if the compiling step happens later). lf_expr_dup() may be used to duplicate an expression before it is compiled, lf_expr_xfer() now makes sure that the input logformat is already compiled. This is some prerequisite works for log-profiles implementation, no functional change should be expected.
2024-02-29 08:54:43 -05:00
dst->flags |= LF_FL_COMPILED;
MINOR: log: store lf_expr nodes inside substruct Add another struct level inside lf_expr struct to allow new information to be stored alongside lf_expr nodes.
2024-03-25 06:29:58 -04:00
LIST_INIT(&dst->nodes.list);
MEDIUM: log: add compiling logic to logformat expressions split parse_logformat_string() into two functions: parse_logformat_string() sticks to the same behavior, but now becomes an helper for lf_expr_compile() which uses explicit arguments so that it becomes possible to use lf_expr_compile() without a proxy, but also compile an expression which was previously prepared for compiling (set string and config hints within the logformat expression to avoid manually storing string and config context if the compiling step happens later). lf_expr_dup() may be used to duplicate an expression before it is compiled, lf_expr_xfer() now makes sure that the input logformat is already compiled. This is some prerequisite works for log-profiles implementation, no functional change should be expected.
2024-02-29 08:54:43 -05:00
MINOR: log: store lf_expr nodes inside substruct Add another struct level inside lf_expr struct to allow new information to be stored alongside lf_expr nodes.
2024-03-25 06:29:58 -04:00
list_for_each_entry_safe(lf, lfb, &src->nodes.list, list) {
MEDIUM: tree-wide: add logformat expressions wrapper log format expressions are broadly used within the code: once they are parsed from input string, they are converted to a linked list of logformat nodes. We're starting to face some limitations because we're simply storing the converted expression as a generic logformat_node list. The first issue we're facing is that storing logformat expressions that way doesn't allow us to add metadata alongside the list, which is part of the prerequites for implementing log-profiles. Another issue with storing logformat expressions as generic lists of logformat_node elements is that it's starting to become really hard to tell when we rely on logformat expressions or not in the code given that there isn't always a comment near the list declaration or manipulation to indicate that it's relying on logformat expressions under the hood, so this adds some complexity for code maintenance. This patch looks quite impressive due to changes in a lot of header and source files (since logformat expressions are broadly used), but it does a simple thing: it defines the lf_expr structure which itself holds a generic list of logformat nodes, and then declares some helpers to manipulate lf_expr elements and fixes the code so that we now exclusively manipulate logformat_node lists as lf_expr elements outside of log.c. For now, lf_expr struct only contains the list of logformat nodes (no additional metadata), but now that we have dedicated type and helpers, doing so in the future won't be problematic at all and won't require extensive code changes.
2024-02-23 09:57:21 -05:00
LIST_DELETE(&lf->list);
MINOR: log: store lf_expr nodes inside substruct Add another struct level inside lf_expr struct to allow new information to be stored alongside lf_expr nodes.
2024-03-25 06:29:58 -04:00
LIST_APPEND(&dst->nodes.list, &lf->list);
MEDIUM: tree-wide: add logformat expressions wrapper log format expressions are broadly used within the code: once they are parsed from input string, they are converted to a linked list of logformat nodes. We're starting to face some limitations because we're simply storing the converted expression as a generic logformat_node list. The first issue we're facing is that storing logformat expressions that way doesn't allow us to add metadata alongside the list, which is part of the prerequites for implementing log-profiles. Another issue with storing logformat expressions as generic lists of logformat_node elements is that it's starting to become really hard to tell when we rely on logformat expressions or not in the code given that there isn't always a comment near the list declaration or manipulation to indicate that it's relying on logformat expressions under the hood, so this adds some complexity for code maintenance. This patch looks quite impressive due to changes in a lot of header and source files (since logformat expressions are broadly used), but it does a simple thing: it defines the lf_expr structure which itself holds a generic list of logformat nodes, and then declares some helpers to manipulate lf_expr elements and fixes the code so that we now exclusively manipulate logformat_node lists as lf_expr elements outside of log.c. For now, lf_expr struct only contains the list of logformat nodes (no additional metadata), but now that we have dedicated type and helpers, doing so in the future won't be problematic at all and won't require extensive code changes.
2024-02-23 09:57:21 -05:00
}
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
/* replace <src> with <dst> in <src>'s list by first adding
* <dst> after <src>, then removing <src>...
*/
LIST_INSERT(&src->list, &dst->list);
LIST_DEL_INIT(&src->list);
MEDIUM: tree-wide: add logformat expressions wrapper log format expressions are broadly used within the code: once they are parsed from input string, they are converted to a linked list of logformat nodes. We're starting to face some limitations because we're simply storing the converted expression as a generic logformat_node list. The first issue we're facing is that storing logformat expressions that way doesn't allow us to add metadata alongside the list, which is part of the prerequites for implementing log-profiles. Another issue with storing logformat expressions as generic lists of logformat_node elements is that it's starting to become really hard to tell when we rely on logformat expressions or not in the code given that there isn't always a comment near the list declaration or manipulation to indicate that it's relying on logformat expressions under the hood, so this adds some complexity for code maintenance. This patch looks quite impressive due to changes in a lot of header and source files (since logformat expressions are broadly used), but it does a simple thing: it defines the lf_expr structure which itself holds a generic list of logformat nodes, and then declares some helpers to manipulate lf_expr elements and fixes the code so that we now exclusively manipulate logformat_node lists as lf_expr elements outside of log.c. For now, lf_expr struct only contains the list of logformat nodes (no additional metadata), but now that we have dedicated type and helpers, doing so in the future won't be problematic at all and won't require extensive code changes.
2024-02-23 09:57:21 -05:00
/* src is now empty, perform an explicit reset */
lf_expr_init(src);
}
MEDIUM: log: add compiling logic to logformat expressions split parse_logformat_string() into two functions: parse_logformat_string() sticks to the same behavior, but now becomes an helper for lf_expr_compile() which uses explicit arguments so that it becomes possible to use lf_expr_compile() without a proxy, but also compile an expression which was previously prepared for compiling (set string and config hints within the logformat expression to avoid manually storing string and config context if the compiling step happens later). lf_expr_dup() may be used to duplicate an expression before it is compiled, lf_expr_xfer() now makes sure that the input logformat is already compiled. This is some prerequisite works for log-profiles implementation, no functional change should be expected.
2024-02-29 08:54:43 -05:00
/* tries to duplicate an uncompiled logformat expression from <orig> to <dest>
*
* Returns 1 on success and 0 on failure.
*/
int lf_expr_dup(const struct lf_expr *orig, struct lf_expr *dest)
{
BUG_ON((orig->flags & LF_FL_COMPILED));
lf_expr_deinit(dest);
if (orig->str) {
MEDIUM: proxy/log: leverage lf_expr API for logformat preparsing Currently, the way proxy-oriented logformat directives are handled is way too complicated. Indeed, "log-format", "log-format-error", "log-format-sd" and "unique-id-format" all rely on preparsing hints stored inside proxy->conf member struct. Those preparsing hints include the original string that should be compiled once the proxy parameters are known plus the config file and line number where the string was found to generate precise error messages in case of failure during the compiling process that happens within check_config_validity(). Now that lf_expr API permits to compile a lf_expr struct that was previously prepared (with original string and config hints), let's leverage lf_expr_compile() from check_config_validity() and instead of relying on individual proxy->conf hints for each logformat expression, store string and config hints in the lf_expr struct directly and use lf_expr helpers funcs to handle them when relevant (ie: original logformat string freeing is now done at a central place inside lf_expr_deinit(), which allows for some simplifications) Doing so allows us to greatly simplify the preparsing logic for those 4 proxy directives, and to finally save some space in the proxy struct. Also, since httpclient proxy has its "logformat" automatically compiled in check_config_validity(), we now use the file hint from the logformat expression struct to set an explicit name that will be reported in case of error ("parsing [httpclient:0] : ...") and remove the extraneous check in httpclient_precheck() (logformat was parsed twice previously..)
2024-03-05 09:44:43 -05:00
dest->str = logformat_str_dup(orig->str);
MEDIUM: log: add compiling logic to logformat expressions split parse_logformat_string() into two functions: parse_logformat_string() sticks to the same behavior, but now becomes an helper for lf_expr_compile() which uses explicit arguments so that it becomes possible to use lf_expr_compile() without a proxy, but also compile an expression which was previously prepared for compiling (set string and config hints within the logformat expression to avoid manually storing string and config context if the compiling step happens later). lf_expr_dup() may be used to duplicate an expression before it is compiled, lf_expr_xfer() now makes sure that the input logformat is already compiled. This is some prerequisite works for log-profiles implementation, no functional change should be expected.
2024-02-29 08:54:43 -05:00
if (!dest->str)
goto error;
}
if (orig->conf.file) {
dest->conf.file = strdup(orig->conf.file);
if (!dest->conf.file)
goto error;
}
dest->conf.line = orig->conf.line;
return 1;
error:
lf_expr_deinit(dest);
return 0;
}
MEDIUM: tree-wide: add logformat expressions wrapper log format expressions are broadly used within the code: once they are parsed from input string, they are converted to a linked list of logformat nodes. We're starting to face some limitations because we're simply storing the converted expression as a generic logformat_node list. The first issue we're facing is that storing logformat expressions that way doesn't allow us to add metadata alongside the list, which is part of the prerequites for implementing log-profiles. Another issue with storing logformat expressions as generic lists of logformat_node elements is that it's starting to become really hard to tell when we rely on logformat expressions or not in the code given that there isn't always a comment near the list declaration or manipulation to indicate that it's relying on logformat expressions under the hood, so this adds some complexity for code maintenance. This patch looks quite impressive due to changes in a lot of header and source files (since logformat expressions are broadly used), but it does a simple thing: it defines the lf_expr structure which itself holds a generic list of logformat nodes, and then declares some helpers to manipulate lf_expr elements and fixes the code so that we now exclusively manipulate logformat_node lists as lf_expr elements outside of log.c. For now, lf_expr struct only contains the list of logformat nodes (no additional metadata), but now that we have dedicated type and helpers, doing so in the future won't be problematic at all and won't require extensive code changes.
2024-02-23 09:57:21 -05:00
/* Builds a log line in <dst> based on <lf_expr>, and stops before reaching
BUG/MINOR: http: http-request add-header emits a corrupted header David BERARD reported that http-request add-header passes a \0 along with the header field, which of course is not appropriate. This is caused by build_logline() which sometimes returns the size with the trailing zero and sometimes can return an empty string. Let's fix this function instead of fixing the places where it's used.
2012-12-27 20:44:01 -05:00
* <maxsize> characters. Returns the size of the output string in characters,
* not counting the trailing zero which is always added if the resulting size
MEDIUM: log: make sess_build_logline() support being called with no stream Till now it was impossible to emit logs from the lower layers only because a stream was mandatory. From now on it will at least be possible to emit a log to report a bad request or some timings for example. When the stream is null, sess_build_logline() will use default values and will extract the timing information from the session just like stream_new() does, so the resulting log line is perfectly valid. The termination state will indicate a proxy error during the request phase since it is the only realistic use for such a call with no stream.
2018-09-05 10:55:15 -04:00
* is not zero. It requires a valid session and optionally a stream. If the
* stream is NULL, default values will be assumed for the stream part.
BUG/MINOR: http: http-request add-header emits a corrupted header David BERARD reported that http-request add-header passes a \0 along with the header field, which of course is not appropriate. This is caused by build_logline() which sometimes returns the size with the trailing zero and sometimes can return an empty string. Let's fix this function instead of fixing the places where it's used.
2012-12-27 20:44:01 -05:00
*/
MINOR: log: provide log origin in logformat expressions using '%OG' '%OG' logformat alias may be used to report the log origin (when/where) that triggered log generation using sess_build_logline(). Possible values are: - "sess_error": log was generated during session error handling - "sess_killed": log was generated during session abortion (killed embryonic session) - "txn_accept": log was generated right after frontend conn was accepted - "txn_request": log was generated after client request was received - "txn_connect": log was generated after backend connection establishment - "txn_response": log was generated during server response handling - "txn_close": log was generated at the final txn step, before closing - "unspec": unknown or not specified Documentation was updated.
2024-05-06 08:13:11 -04:00
int sess_build_logline_orig(struct session *sess, struct stream *s,
char *dst, size_t maxsize, struct lf_expr *lf_expr,
enum log_orig log_orig)
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-25 20:48:02 -04:00
{
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
struct lf_buildctx *ctx = &lf_buildctx;
MEDIUM: stream: move the frontend's pointer to the session Just like for the listener, the frontend is session-wide so let's move it to the session. There are a lot of places which were changed but the changes are minimal in fact.
2015-04-03 09:40:56 -04:00
struct proxy *fe = sess->fe;
MEDIUM: log: make sess_build_logline() support being called with no stream Till now it was impossible to emit logs from the lower layers only because a stream was mandatory. From now on it will at least be possible to emit a log to report a bad request or some timings for example. When the stream is null, sess_build_logline() will use default values and will extract the timing information from the session just like stream_new() does, so the resulting log line is perfectly valid. The termination state will indicate a proxy error during the request phase since it is the only realistic use for such a call with no stream.
2018-09-05 10:55:15 -04:00
struct proxy *be;
struct http_txn *txn;
const struct strm_logs *logs;
MINOR: logs: Get the multiplexer exist status when no stream is provided When a log message is emitted from the session level, by a multiplexer, there is no stream. Thus for HTTP session, there no status code and the termination flags are not correctly set. Thanks to previous patch, the HTTP status code is deduced from the mux exist status, using the MUX_EXIT_STATE ctl param. This is only done for HTTP frontends. If it is defined ( != 0), it is used to deduce the termination flags.
2020-10-06 09:11:43 -04:00
struct connection *fe_conn, *be_conn;
MINOR: log: store lf_expr nodes inside substruct Add another struct level inside lf_expr struct to allow new information to be stored alongside lf_expr nodes.
2024-03-25 06:29:58 -04:00
struct list *list_format = &lf_expr->nodes.list;
MEDIUM: log: make sess_build_logline() support being called with no stream Till now it was impossible to emit logs from the lower layers only because a stream was mandatory. From now on it will at least be possible to emit a log to report a bad request or some timings for example. When the stream is null, sess_build_logline() will use default values and will extract the timing information from the session just like stream_new() does, so the resulting log line is perfectly valid. The termination state will indicate a proxy error during the request phase since it is the only realistic use for such a call with no stream.
2018-09-05 10:55:15 -04:00
unsigned int s_flags;
unsigned int uniq_id;
MAJOR: chunks: replace struct chunk with struct buffer Now all the code used to manipulate chunks uses a struct buffer instead. The functions are still called "chunk*", and some of them will progressively move to the generic buffer handling code as they are cleaned up.
2018-07-13 05:56:34 -04:00
struct buffer chunk;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
char *uri;
MEDIUM: logs: Add HTTP request-line log format directives This commit adds 4 new log format variables that parse the HTTP Request-Line for more specific logging than "%r" provides. For example, we can parse the following HTTP Request-Line with these new variables: "GET /foo?bar=baz HTTP/1.1" - %HM: HTTP Method ("GET") - %HV: HTTP Version ("HTTP/1.1") - %HU: HTTP Request-URI ("/foo?bar=baz") - %HP: HTTP Request-URI without query string ("/foo")
2015-04-27 17:37:03 -04:00
char *spc;
MINOR: log: Add log-format variable %HQ, to log HTTP query strings Since sample fetches are not always available in the response phase, this patch implements %HQ such that: GET /foo?bar=baz HTTP/1.0 ...would be logged as: ?bar=baz
2015-07-31 12:14:16 -04:00
char *qmark;
MEDIUM: logs: Add HTTP request-line log format directives This commit adds 4 new log format variables that parse the HTTP Request-Line for more specific logging than "%r" provides. For example, we can parse the following HTTP Request-Line with these new variables: "GET /foo?bar=baz HTTP/1.1" - %HM: HTTP Method ("GET") - %HV: HTTP Version ("HTTP/1.1") - %HU: HTTP Request-URI ("/foo?bar=baz") - %HP: HTTP Request-URI without query string ("/foo")
2015-04-27 17:37:03 -04:00
char *end;
[BUG] fix calls to localtime() localtime() was called with pointers to tv_sec, which is time_t on some platforms and long on others. A problem was encountered on Sparc64 under OpenBSD where tv_sec is long (64 bits) and time_t is 32 bits. Since this architecture is big-endian, it exhibited the bug because localtime() always worked with the high part of the value which is always zero. This problem was identified and debugged by Thierry Fournier. The correct solution is to pass the date by value and not by pointer, through an intermediate function. The use of localtime_r() instead of localtime() also made it possible to get rid of the first call to localtime() since it does not need to allocate memory anymore.
2007-10-25 04:34:16 -04:00
struct tm tm;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
int t_request;
int hdr;
int last_isspace = 1;
MEDIUM: logs: Add HTTP request-line log format directives This commit adds 4 new log format variables that parse the HTTP Request-Line for more specific logging than "%r" provides. For example, we can parse the following HTTP Request-Line with these new variables: "GET /foo?bar=baz HTTP/1.1" - %HM: HTTP Method ("GET") - %HV: HTTP Version ("HTTP/1.1") - %HU: HTTP Request-URI ("/foo?bar=baz") - %HP: HTTP Request-URI without query string ("/foo")
2015-04-27 17:37:03 -04:00
int nspaces = 0;
BUG/CRITICAL: log: fix risk of crash in development snapshot Commit a1cc38 introduced a regression which was easy to trigger till ad4cd58 (snapshots 20120222 to 20120311 included). The bug was still present after that but harder to trigger. The bug is caused by the use of two distinct log buffers due to intermediary changes. The issue happens when an HTTP request is logged just after a TCP request during the same second and the HTTP request is too large for the buffer. In this case, it happens that the HTTP request is logged into the TCP buffer instead and that length controls can't detect anything. Starting with bddd4f, the issue is still possible when logging too large an HTTP request just after a send_log() call (typically a server status change). We owe a big thanks to Sander Klein for testing several snapshots and more specifically for taking significant risks in production by letting the buggy version crash several times in order to provide an exploitable core ! The bug could not have been found without this precious help. Thank you Sander ! This fix does not need to be backported, it did not affect any released version.
2012-03-19 11:51:53 -04:00
char *tmplog;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
char *ret;
int iret;
MINOR: logs: Get the multiplexer exist status when no stream is provided When a log message is emitted from the session level, by a multiplexer, there is no stream. Thus for HTTP session, there no status code and the termination flags are not correctly set. Thanks to previous patch, the HTTP status code is deduced from the mux exist status, using the MUX_EXIT_STATE ctl param. This is only done for HTTP frontends. If it is defined ( != 0), it is used to deduce the termination flags.
2020-10-06 09:11:43 -04:00
int status;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
struct logformat_node *tmp;
MEDIUM: log: Decompose %Tq in %Th %Ti %TR Tq is the time between the instant the connection is accepted and a complete valid request is received. This time includes the handshake (SSL / Proxy-Protocol), the idle when the browser does preconnect and the request reception. This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR which returns respectively the handshake time, the idle time and the duration of valid request reception. It also adds %Ta which reports the request's active time, which is the total time without %Th nor %Ti. It replaces %Tt as the total time, reporting accurate measurements for HTTP persistent connections. %Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only avalaible for HTTP connections. In addition to this, we have new timestamps %tr, %trg and %trl, which log the date of start of receipt of the request, respectively in the default format, in GMT time and in local time (by analogy with %t, %T and %Tl). All of them are obviously only available for HTTP. These values are more relevant as they more accurately represent the request date without being skewed by a browser's preconnect nor a keep-alive idle time. The HTTP log format and the CLF log format have been modified to use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This way the default log formats now produce the expected output for users who don't want to manually fiddle with the log-format directive. Example with the following log-format : log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r" The request was sent by hand using "openssl s_client -connect" : Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1" => 6 ms of SSL handshake, 2375 waiting before sending the first char (in fact the time to type the first line), 261 ms before the end of the request, no time spent in queue, 1 ms spend connecting to the server, immediate response, total active time for this request = 262ms. Total time from accept to close : 2643 ms. The timing now decomposes like this : first request 2nd request |<-------------------------------->|<-------------- ... t tr t tr ... ---|----|----|----|----|----|----|----|----|-- : Th Ti TR Tw Tc Tr Td : Ti ... :<---- Tq ---->: : :<-------------- Tt -------------->: :<--------- Ta --------->:
2016-07-28 11:19:45 -04:00
struct timeval tv;
MEDIUM: log: make sess_build_logline() support being called with no stream Till now it was impossible to emit logs from the lower layers only because a stream was mandatory. From now on it will at least be possible to emit a log to report a bad request or some timings for example. When the stream is null, sess_build_logline() will use default values and will extract the timing information from the session just like stream_new() does, so the resulting log line is perfectly valid. The termination state will indicate a proxy error during the request phase since it is the only realistic use for such a call with no stream.
2018-09-05 10:55:15 -04:00
struct strm_logs tmp_strm_log;
MINOR: log: Logging HTTP path only with %HPO This patch adds a new logging variable '%HPO' for logging HTTP path only (without query string) from relative or absolute URI. For example: log-format "hpo=%HPO hp=%HP hu=%HU hq=%HQ" GET /r/1 HTTP/1.1 => hpo=/r/1 hp=/r/1 hu=/r/1 hq= GET /r/2?q=2 HTTP/1.1 => hpo=/r/2 hp=/r/2 hu=/r/2?q=2 hq=?q=2 GET http://host/r/3 HTTP/1.1 => hpo=/r/3 hp=http://host/r/3 hu=http://host/r/3 hq= GET http://host/r/4?q=4 HTTP/1.1 => hpo=/r/4 hp=http://host/r/4 hu=http://host/r/4?q=4 hq=?q=4
2020-11-30 13:27:47 -05:00
struct ist path;
MINOR: http: use http uri parser for path Replace http_get_path by the http_uri_parser API. The new functions is renamed http_parse_path. Replace duplicated code for scheme and authority parsing by invocations to http_parse_scheme/authority. If no scheme is found for an URI detected as an absolute-uri/authority, consider it to be an authority format : no path will be found. For an absolute-uri or absolute-path, use the remaining of the string as the path. A new http_uri_parser state is declared to mark the path parsing as done.
2021-07-06 05:40:12 -04:00
struct http_uri_parser parser;
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
int g_options = lf_expr->nodes.options; /* global */
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
int first_node = 1;
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-25 20:48:02 -04:00
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
/* FIXME: let's limit ourselves to frontend logging for now. */
[MEDIUM] splitted logs into two versions : TCP and HTTP logs are handled better with dedicated functions. The HTTP implementation moved to proto_http.c. It has been cleaned up a bit. Now a frontend with option httplog and no log will not call the function anymore.
2007-03-31 19:30:43 -04:00
MEDIUM: log: make sess_build_logline() support being called with no stream Till now it was impossible to emit logs from the lower layers only because a stream was mandatory. From now on it will at least be possible to emit a log to report a bad request or some timings for example. When the stream is null, sess_build_logline() will use default values and will extract the timing information from the session just like stream_new() does, so the resulting log line is perfectly valid. The termination state will indicate a proxy error during the request phase since it is the only realistic use for such a call with no stream.
2018-09-05 10:55:15 -04:00
if (likely(s)) {
be = s->be;
txn = s->txn;
CLEANUP: stconn: rename cs_conn() to sc_conn() It's mostly used from upper layers. Both the checked and unchecked functions were updated, or ~150 entries.
2022-05-18 10:23:22 -04:00
be_conn = sc_conn(s->scb);
MINOR: logs: Get the multiplexer exist status when no stream is provided When a log message is emitted from the session level, by a multiplexer, there is no stream. Thus for HTTP session, there no status code and the termination flags are not correctly set. Thanks to previous patch, the HTTP status code is deduced from the mux exist status, using the MUX_EXIT_STATE ctl param. This is only done for HTTP frontends. If it is defined ( != 0), it is used to deduce the termination flags.
2020-10-06 09:11:43 -04:00
status = (txn ? txn->status : 0);
MEDIUM: log: make sess_build_logline() support being called with no stream Till now it was impossible to emit logs from the lower layers only because a stream was mandatory. From now on it will at least be possible to emit a log to report a bad request or some timings for example. When the stream is null, sess_build_logline() will use default values and will extract the timing information from the session just like stream_new() does, so the resulting log line is perfectly valid. The termination state will indicate a proxy error during the request phase since it is the only realistic use for such a call with no stream.
2018-09-05 10:55:15 -04:00
s_flags = s->flags;
uniq_id = s->uniq_id;
logs = &s->logs;
} else {
/* we have no stream so we first need to initialize a few
* things that are needed later. We do increment the request
* ID so that it's uniquely assigned to this request just as
* if the request had reached the point of being processed.
* A request error is reported as it's the only element we have
* here and which justifies emitting such a log.
*/
MINOR: logs: Add support of checks as session origin to format lf strings When a log-format string is built from an health-check, the session origin is the health-check itself and not a connection. In addition, there is no stream. It means for now some formats are not supported: %s, %sc, %b, %bi, %bp, %si and %sp. Thanks to this patch, the session origin is converted to a check. So it is possible to retrieve the backend and the backend connection. Note this session have no listener, thus %ft format must be guarded. This patch is light and standalone, thus it may be backported as far as 2.2 if required. However, because the error is human, it is probably better to wait a bit to be sure everything is properly protected.
2021-04-14 08:01:41 -04:00
be = ((obj_type(sess->origin) == OBJ_TYPE_CHECK) ? __objt_check(sess->origin)->proxy : fe);
MEDIUM: log: make sess_build_logline() support being called with no stream Till now it was impossible to emit logs from the lower layers only because a stream was mandatory. From now on it will at least be possible to emit a log to report a bad request or some timings for example. When the stream is null, sess_build_logline() will use default values and will extract the timing information from the session just like stream_new() does, so the resulting log line is perfectly valid. The termination state will indicate a proxy error during the request phase since it is the only realistic use for such a call with no stream.
2018-09-05 10:55:15 -04:00
txn = NULL;
MINOR: logs: Get the multiplexer exist status when no stream is provided When a log message is emitted from the session level, by a multiplexer, there is no stream. Thus for HTTP session, there no status code and the termination flags are not correctly set. Thanks to previous patch, the HTTP status code is deduced from the mux exist status, using the MUX_EXIT_STATE ctl param. This is only done for HTTP frontends. If it is defined ( != 0), it is used to deduce the termination flags.
2020-10-06 09:11:43 -04:00
fe_conn = objt_conn(sess->origin);
CLEANUP: check: rename all occurrences of stconn "cs" to "sc" The check struct had a "cs" field renamed to "sc", which also required a tiny update to a few functions using it to distinguish a check from a stream (log.c, payload.c, ssl_sample.c, tcp_sample.c, tcpcheck.c, connection.c). Function arguments and local variables called "cs" were renamed to "sc". The presence of one "cs=" in the debugging traces was also turned to "sc=" for consistency.
2022-05-27 04:04:04 -04:00
be_conn = ((obj_type(sess->origin) == OBJ_TYPE_CHECK) ? sc_conn(__objt_check(sess->origin)->sc) : NULL);
MINOR: logs: Get the multiplexer exist status when no stream is provided When a log message is emitted from the session level, by a multiplexer, there is no stream. Thus for HTTP session, there no status code and the termination flags are not correctly set. Thanks to previous patch, the HTTP status code is deduced from the mux exist status, using the MUX_EXIT_STATE ctl param. This is only done for HTTP frontends. If it is defined ( != 0), it is used to deduce the termination flags.
2020-10-06 09:11:43 -04:00
status = 0;
MEDIUM: log: make sess_build_logline() support being called with no stream Till now it was impossible to emit logs from the lower layers only because a stream was mandatory. From now on it will at least be possible to emit a log to report a bad request or some timings for example. When the stream is null, sess_build_logline() will use default values and will extract the timing information from the session just like stream_new() does, so the resulting log line is perfectly valid. The termination state will indicate a proxy error during the request phase since it is the only realistic use for such a call with no stream.
2018-09-05 10:55:15 -04:00
s_flags = SF_ERR_PRXCOND | SF_FINST_R;
CLEANUP: atomic: add a fetch-and-xxx variant for common operations The fetch_and_xxx variant is often missing for add/sub/and/or. In fact it was only provided for ADD under the name XADD which corresponds to the x86 instruction name. But for destructive operations like AND and OR it's missing even more as it's not possible to know the value before modifying it. This patch explicitly adds HA_ATOMIC_FETCH_{OR,AND,ADD,SUB} which cover these standard operations, and renames XADD to FETCH_ADD (there were only 6 call places). In the future, backport of fixes involving such operations could simply remap FETCH_ADD(x) to XADD(x), FETCH_SUB(x) to XADD(-x), and for the OR/AND if needed, these could possibly be done using BTS/BTR. It's worth noting that xchg could have been renamed to fetch_and_store() but xchg already has well understood semantics and it wasn't needed to go further.
2021-04-06 05:57:41 -04:00
uniq_id = _HA_ATOMIC_FETCH_ADD(&global.req_count, 1);
MEDIUM: log: make sess_build_logline() support being called with no stream Till now it was impossible to emit logs from the lower layers only because a stream was mandatory. From now on it will at least be possible to emit a log to report a bad request or some timings for example. When the stream is null, sess_build_logline() will use default values and will extract the timing information from the session just like stream_new() does, so the resulting log line is perfectly valid. The termination state will indicate a proxy error during the request phase since it is the only realistic use for such a call with no stream.
2018-09-05 10:55:15 -04:00
/* prepare a valid log structure */
MEDIUM: tree-wide: replace timeval with nanoseconds in tv_accept and tv_request Let's get rid of timeval in storage of internal timestamps so that they are no longer mistaken for wall clock time. These were exclusively used subtracted from each other or to/from "now" after being converted to ns, so this patch removes the tv_to_ns() conversion to use them natively. Two occurrences of tv_isge() were turned to a regular wrapping subtract.
2023-04-27 03:46:02 -04:00
tmp_strm_log.accept_ts = sess->accept_ts;
MEDIUM: log: make sess_build_logline() support being called with no stream Till now it was impossible to emit logs from the lower layers only because a stream was mandatory. From now on it will at least be possible to emit a log to report a bad request or some timings for example. When the stream is null, sess_build_logline() will use default values and will extract the timing information from the session just like stream_new() does, so the resulting log line is perfectly valid. The termination state will indicate a proxy error during the request phase since it is the only realistic use for such a call with no stream.
2018-09-05 10:55:15 -04:00
tmp_strm_log.accept_date = sess->accept_date;
tmp_strm_log.t_handshake = sess->t_handshake;
MINOR: logs: Use session idle duration when no stream is provided When a log message is emitted from the session, using sess_log() function, there is no stream available. In this case, instead of deducing the idle duration from the accept date, we use the one provided by the session. 0 is used if it is undefined (i.e set to -1).
2020-09-30 09:10:07 -04:00
tmp_strm_log.t_idle = (sess->t_idle >= 0 ? sess->t_idle : 0);
MEDIUM: tree-wide: replace timeval with nanoseconds in tv_accept and tv_request Let's get rid of timeval in storage of internal timestamps so that they are no longer mistaken for wall clock time. These were exclusively used subtracted from each other or to/from "now" after being converted to ns, so this patch removes the tv_to_ns() conversion to use them natively. Two occurrences of tv_isge() were turned to a regular wrapping subtract.
2023-04-27 03:46:02 -04:00
tmp_strm_log.request_ts = 0;
MEDIUM: log: make sess_build_logline() support being called with no stream Till now it was impossible to emit logs from the lower layers only because a stream was mandatory. From now on it will at least be possible to emit a log to report a bad request or some timings for example. When the stream is null, sess_build_logline() will use default values and will extract the timing information from the session just like stream_new() does, so the resulting log line is perfectly valid. The termination state will indicate a proxy error during the request phase since it is the only realistic use for such a call with no stream.
2018-09-05 10:55:15 -04:00
tmp_strm_log.t_queue = -1;
tmp_strm_log.t_connect = -1;
tmp_strm_log.t_data = -1;
MEDIUM: clock: replace timeval "now" with integer "now_ns" This puts an end to the occasional confusion between the "now" date that is internal, monotonic and not synchronized with the system's date, and "date" which is the system's date and not necessarily monotonic. Variable "now" was removed and replaced with a 64-bit integer "now_ns" which is a counter of nanoseconds. It wraps every 585 years, so if all goes well (i.e. if humanity does not need haproxy anymore in 500 years), it will just never wrap. This implies that now_ns is never nul and that the zero value can reliably be used as "not set yet" for a timestamp if needed. This will also simplify date checks where it becomes possible again to do "date1<date2". All occurrences of "tv_to_ns(&now)" were simply replaced by "now_ns". Due to the intricacies between now, global_now and now_offset, all 3 had to be turned to nanoseconds at once. It's not a problem since all of them were solely used in 3 functions in clock.c, but they make the patch look bigger than it really is. The clock_update_local_date() and clock_update_global_date() functions are now much simpler as there's no need anymore to perform conversions nor to round the timeval up or down. The wrapping continues to happen by presetting the internal offset in the short future so that the 32-bit now_ms continues to wrap 20 seconds after boot. The start_time used to calculate uptime can still be turned to nanoseconds now. One interrogation concerns global_now_ms which is used only for the freq counters. It's unclear whether there's more value in using two variables that need to be synchronized sequentially like today or to just use global_now_ns divided by 1 million. Both approaches will work equally well on modern systems, the difference might come from smaller ones. Better not change anyhting for now. One benefit of the new approach is that we now have an internal date with a resolution of the nanosecond and the precision of the microsecond, which can be useful to extend some measurements given that timestamps also have this resolution.
2023-04-28 03:16:15 -04:00
tmp_strm_log.t_close = ns_to_ms(now_ns - sess->accept_ts);
MEDIUM: log: make sess_build_logline() support being called with no stream Till now it was impossible to emit logs from the lower layers only because a stream was mandatory. From now on it will at least be possible to emit a log to report a bad request or some timings for example. When the stream is null, sess_build_logline() will use default values and will extract the timing information from the session just like stream_new() does, so the resulting log line is perfectly valid. The termination state will indicate a proxy error during the request phase since it is the only realistic use for such a call with no stream.
2018-09-05 10:55:15 -04:00
tmp_strm_log.bytes_in = 0;
tmp_strm_log.bytes_out = 0;
tmp_strm_log.prx_queue_pos = 0;
tmp_strm_log.srv_queue_pos = 0;
logs = &tmp_strm_log;
MINOR: logs: Get the multiplexer exist status when no stream is provided When a log message is emitted from the session level, by a multiplexer, there is no stream. Thus for HTTP session, there no status code and the termination flags are not correctly set. Thanks to previous patch, the HTTP status code is deduced from the mux exist status, using the MUX_EXIT_STATE ctl param. This is only done for HTTP frontends. If it is defined ( != 0), it is used to deduce the termination flags.
2020-10-06 09:11:43 -04:00
if ((fe->mode == PR_MODE_HTTP) && fe_conn && fe_conn->mux && fe_conn->mux->ctl) {
MINOR: muxes: Rename mux_ctl_type values to use MUX_CTL_ prefix Instead of the generic MUX_, we now use MUX_CTL_ prefix for all mux_ctl_type value. This will avoid any ambiguities with other enums, especially with a new one that will be added to get information on mux streams.
2023-11-28 08:27:51 -05:00
enum mux_exit_status es = fe_conn->mux->ctl(fe_conn, MUX_CTL_EXIT_STATUS, &status);
MINOR: logs: Get the multiplexer exist status when no stream is provided When a log message is emitted from the session level, by a multiplexer, there is no stream. Thus for HTTP session, there no status code and the termination flags are not correctly set. Thanks to previous patch, the HTTP status code is deduced from the mux exist status, using the MUX_EXIT_STATE ctl param. This is only done for HTTP frontends. If it is defined ( != 0), it is used to deduce the termination flags.
2020-10-06 09:11:43 -04:00
switch (es) {
case MUX_ES_SUCCESS:
break;
case MUX_ES_INVALID_ERR:
MINOR: log: Try to get the status code when MUX_EXIT_STATUS is retrieved The mux .ctl callback can provide some information about the mux to the caller if the third parameter is provided. Thus, when MUX_EXIT_STATUS is retrieved, a pointer on the status is now passed. The mux may fill it. It will be pretty handy to provide custom error code from h1 mux instead of default ones (400/408/500/501).
2021-09-28 05:36:28 -04:00
status = (status ? status : 400);
MINOR: logs: Get the multiplexer exist status when no stream is provided When a log message is emitted from the session level, by a multiplexer, there is no stream. Thus for HTTP session, there no status code and the termination flags are not correctly set. Thanks to previous patch, the HTTP status code is deduced from the mux exist status, using the MUX_EXIT_STATE ctl param. This is only done for HTTP frontends. If it is defined ( != 0), it is used to deduce the termination flags.
2020-10-06 09:11:43 -04:00
if ((fe_conn->flags & CO_FL_ERROR) || conn_xprt_read0_pending(fe_conn))
s_flags = SF_ERR_CLICL | SF_FINST_R;
else
s_flags = SF_ERR_PRXCOND | SF_FINST_R;
break;
case MUX_ES_TOUT_ERR:
MINOR: log: Try to get the status code when MUX_EXIT_STATUS is retrieved The mux .ctl callback can provide some information about the mux to the caller if the third parameter is provided. Thus, when MUX_EXIT_STATUS is retrieved, a pointer on the status is now passed. The mux may fill it. It will be pretty handy to provide custom error code from h1 mux instead of default ones (400/408/500/501).
2021-09-28 05:36:28 -04:00
status = (status ? status : 408);
MINOR: logs: Get the multiplexer exist status when no stream is provided When a log message is emitted from the session level, by a multiplexer, there is no stream. Thus for HTTP session, there no status code and the termination flags are not correctly set. Thanks to previous patch, the HTTP status code is deduced from the mux exist status, using the MUX_EXIT_STATE ctl param. This is only done for HTTP frontends. If it is defined ( != 0), it is used to deduce the termination flags.
2020-10-06 09:11:43 -04:00
s_flags = SF_ERR_CLITO | SF_FINST_R;
break;
MINOR: muxes: Add exit status for errors about not implemented features The MUX_ES_NOTIMPL_ERR exit status is added to allow the multiplexers to report errors about not implemented features. This will be used by the H1 mux to return 501-not-implemented errors.
2020-12-07 05:24:37 -05:00
case MUX_ES_NOTIMPL_ERR:
MINOR: log: Try to get the status code when MUX_EXIT_STATUS is retrieved The mux .ctl callback can provide some information about the mux to the caller if the third parameter is provided. Thus, when MUX_EXIT_STATUS is retrieved, a pointer on the status is now passed. The mux may fill it. It will be pretty handy to provide custom error code from h1 mux instead of default ones (400/408/500/501).
2021-09-28 05:36:28 -04:00
status = (status ? status : 501);
MINOR: muxes: Add exit status for errors about not implemented features The MUX_ES_NOTIMPL_ERR exit status is added to allow the multiplexers to report errors about not implemented features. This will be used by the H1 mux to return 501-not-implemented errors.
2020-12-07 05:24:37 -05:00
s_flags = SF_ERR_PRXCOND | SF_FINST_R;
break;
MINOR: logs: Get the multiplexer exist status when no stream is provided When a log message is emitted from the session level, by a multiplexer, there is no stream. Thus for HTTP session, there no status code and the termination flags are not correctly set. Thanks to previous patch, the HTTP status code is deduced from the mux exist status, using the MUX_EXIT_STATE ctl param. This is only done for HTTP frontends. If it is defined ( != 0), it is used to deduce the termination flags.
2020-10-06 09:11:43 -04:00
case MUX_ES_INTERNAL_ERR:
MINOR: log: Try to get the status code when MUX_EXIT_STATUS is retrieved The mux .ctl callback can provide some information about the mux to the caller if the third parameter is provided. Thus, when MUX_EXIT_STATUS is retrieved, a pointer on the status is now passed. The mux may fill it. It will be pretty handy to provide custom error code from h1 mux instead of default ones (400/408/500/501).
2021-09-28 05:36:28 -04:00
status = (status ? status : 500);
MINOR: logs: Get the multiplexer exist status when no stream is provided When a log message is emitted from the session level, by a multiplexer, there is no stream. Thus for HTTP session, there no status code and the termination flags are not correctly set. Thanks to previous patch, the HTTP status code is deduced from the mux exist status, using the MUX_EXIT_STATE ctl param. This is only done for HTTP frontends. If it is defined ( != 0), it is used to deduce the termination flags.
2020-10-06 09:11:43 -04:00
s_flags = SF_ERR_INTERNAL | SF_FINST_R;
break;
default:
break;
}
}
MEDIUM: log: make sess_build_logline() support being called with no stream Till now it was impossible to emit logs from the lower layers only because a stream was mandatory. From now on it will at least be possible to emit a log to report a bad request or some timings for example. When the stream is null, sess_build_logline() will use default values and will extract the timing information from the session just like stream_new() does, so the resulting log line is perfectly valid. The termination state will indicate a proxy error during the request phase since it is the only realistic use for such a call with no stream.
2018-09-05 10:55:15 -04:00
}
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
t_request = -1;
MEDIUM: tree-wide: replace timeval with nanoseconds in tv_accept and tv_request Let's get rid of timeval in storage of internal timestamps so that they are no longer mistaken for wall clock time. These were exclusively used subtracted from each other or to/from "now" after being converted to ns, so this patch removes the tv_to_ns() conversion to use them natively. Two occurrences of tv_isge() were turned to a regular wrapping subtract.
2023-04-27 03:46:02 -04:00
if ((llong)(logs->request_ts - logs->accept_ts) >= 0)
t_request = ns_to_ms(logs->request_ts - logs->accept_ts);
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
tmplog = dst;
[MINOR] add options dontlog-normal and log-separate-errors Some big traffic sites have trouble dealing with logs and tend to disable them. Here are two new options to help cope with massive logs. - dontlog-normal only disables logging for 100% successful connections, other ones will still be logged - log-separate-errors will cause non-100% successful connections to be logged at level "err" instead of level "info" so that a properly configured syslog daemon can send them to a different file for longer conservation.
2009-05-10 05:57:02 -04:00
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
/* reset static ctx struct */
ctx->in_text = 0;
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
/* start with global ctx by default */
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
lf_buildctx_prepare(ctx, g_options, NULL);
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
/* fill logbuffer */
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
if (!(ctx->options & LOG_OPT_ENCODE) && lf_expr_isempty(lf_expr))
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
return 0;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
if (ctx->options & LOG_OPT_ENCODE_JSON)
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
LOGCHAR('{');
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
else if (ctx->options & LOG_OPT_ENCODE_CBOR) {
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
/* start indefinite-length map */
LOG_CBOR_BYTE(0xBF);
}
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
list_for_each_entry(tmp, list_format, list) {
BUILD: log: Fix compilation without SSL support When compiled without SSL support, a variable is reported as not used by GCC. src/log.c: In function ‘sess_build_logline’: src/log.c:2056:36: error: unused variable ‘conn’ [-Werror=unused-variable] 2056 | struct connection *conn; | ^~~~ This does not need to be backported.
2021-10-27 05:58:05 -04:00
#ifdef USE_OPENSSL
MAJOR: stream-int: stop using si->conn and use si->end instead The connection will only remain there as a pre-allocated entity whose goal is to be placed in ->end when establishing an outgoing connection. All connection initialization can be made on this connection, but all information retrieved should be applied to the end point only. This change is huge because there were many users of si->conn. Now the only users are those who initialize the new connection. The difficulty appears in a few places such as backend.c, proto_http.c, peers.c where si->conn is used to hold the connection's target address before assigning the connection to the stream interface. This is why we have to keep si->conn for now. A future improvement might consist in dynamically allocating the connection when it is needed.
2013-10-01 04:45:07 -04:00
struct connection *conn;
BUILD: log: Fix compilation without SSL support When compiled without SSL support, a variable is reported as not used by GCC. src/log.c: In function ‘sess_build_logline’: src/log.c:2056:36: error: unused variable ‘conn’ [-Werror=unused-variable] 2056 | struct connection *conn; | ^~~~ This does not need to be backported.
2021-10-27 05:58:05 -04:00
#endif
MINOR: log: Rely on client addresses at the appropriate level to log messages When a log message is emitted, if the stream exits, we use the frontend stream-interface to retrieve the client source and destination addresses. Otherwise, the session is used. For now, stream-interface or session addresses are never set. So, thanks to the fallback mechanism, no changes are expected with this patch. But its purpose is to rely on addresses at the appropriate level when set instead of those at the connection level.
2021-10-22 11:43:22 -04:00
const struct sockaddr_storage *addr;
MINOR: log: make lf_text use a const char * lf_text() should use a const char * otherwise it makes it more complex to use data coming from const strings.
2012-10-12 13:48:16 -04:00
const char *src = NULL;
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
const char *value_beg = NULL;
MEDIUM: log: add the ability to include samples in logs Using %[expression] it becomes possible to make the log engine fetch some samples from the request or the response and provide them in the logs. Note that this feature is still limited, it does not yet allow to apply converters, to limit the output length, nor to specify the direction which should be fetched when a fetch function works in both directions. However it's quite convenient to log SSL information or to include some information that are used in stick tables. It is worth noting that this has been done in the generic log format handler, which means that the same information may be used to build the unique-id header and to pass the information to a backend server.
2012-12-20 18:09:23 -05:00
struct sample *key;
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
MEDIUM: log: optimizing tmp->type handling in sess_build_logline() Instead of chaining 2 switchcases and performing encoding checks for all nodes let's actually split the logic in 2: first handle simple node types (text/separator), and then handle dynamic node types (tag, expr). Encoding options are only evaluated for dynamic node types. Also, last_isspace is always set to 0 after next_fmt label, since next_fmt label is only used for dynamic nodes, thus != LOG_FMT_SEPARATOR. Since LF_NODE_WITH_OPT() macro (which was introduced recently) is now unused, let's get rid of it. No functional change should be expected. (Use diff -w to check patch changes since reindentation makes the patch look heavy, but in fact it remains fairly small)
2024-04-30 09:52:57 -04:00
/* first start with basic types (use continue statement to skip
* the current node)
*/
if (tmp->type == LOG_FMT_SEPARATOR) {
if (g_options & LOG_OPT_ENCODE) {
/* ignored when global encoding is set */
continue;
}
if (!last_isspace) {
LOGCHAR(' ');
last_isspace = 1;
}
continue;
}
else if (tmp->type == LOG_FMT_TEXT) {
/* text */
if (g_options & LOG_OPT_ENCODE) {
/* ignored when global encoding is set */
continue;
}
src = tmp->arg;
iret = strlcpy2(tmplog, src, dst + maxsize - tmplog);
if (iret == 0)
goto out;
tmplog += iret;
last_isspace = 0; /* data was written */
continue;
}
/* dynamic types handling (use "goto next_fmt" statement to skip
* the current node)
*/
BUG/MINOR: log/encode: consider global options for key encoding In sess_build_logline(), contrary to what's stated in the comment "only consider global ctx for key encoding", we check for LOG_OPT_ENCODE flag on the current ctx options instead of global ones. Because of this, we could end up doing the wrong thing if the previous node had encoding enabled but it isn't set globally for instance. To fix the issue, let's simply check the presence of the flag on g_options before entering the "key encoding" block. This bug was introduced with 3f7c8387 ("MINOR: log: add +json encoding option"), no backport needed.
2024-04-29 09:31:17 -04:00
if (g_options & LOG_OPT_ENCODE) {
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
/* only consider global ctx for key encoding */
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
lf_buildctx_prepare(ctx, g_options, NULL);
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
if (!tmp->name)
goto next_fmt; /* cannot represent anonymous field, ignore */
if (!first_node) {
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
if (ctx->options & LOG_OPT_ENCODE_JSON) {
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
LOGCHAR(',');
LOGCHAR(' ');
}
}
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
if (ctx->options & LOG_OPT_ENCODE_JSON) {
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
LOGCHAR('"');
iret = strlcpy2(tmplog, tmp->name, dst + maxsize - tmplog);
if (iret == 0)
goto out;
tmplog += iret;
LOGCHAR('"');
LOGCHAR(':');
LOGCHAR(' ');
}
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
else if (ctx->options & LOG_OPT_ENCODE_CBOR) {
ret = cbor_encode_text(&ctx->encode.cbor, tmplog,
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
dst + maxsize, tmp->name,
strlen(tmp->name));
if (ret == NULL)
goto out;
tmplog = ret;
}
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
first_node = 0;
}
value_beg = tmplog;
MEDIUM: log: lf_* build helpers now take a ctx argument Add internal lf_buildctx struct that is only used inside sess_build_logline() scope and is passed to lf_* log building helpers to expose current building context. For now, node options and the in_text counter are stored in the ctx struct. Thanks to this change, lf_* building functions don't depend on a logformat_node struct pointer, and may be used in a standalone manner as long as a build context is provided. Also, global options are now handled explictly in sess_build_logline() to make sure that global options are always considered even if they were not duplicated on every nodes. No functional change should be expected.
2024-04-22 04:12:42 -04:00
/* get the chance to consider per-node options (if not already
* set globally) for printing the value
*/
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
lf_buildctx_prepare(ctx, g_options, tmp);
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
MEDIUM: log: optimizing tmp->type handling in sess_build_logline() Instead of chaining 2 switchcases and performing encoding checks for all nodes let's actually split the logic in 2: first handle simple node types (text/separator), and then handle dynamic node types (tag, expr). Encoding options are only evaluated for dynamic node types. Also, last_isspace is always set to 0 after next_fmt label, since next_fmt label is only used for dynamic nodes, thus != LOG_FMT_SEPARATOR. Since LF_NODE_WITH_OPT() macro (which was introduced recently) is now unused, let's get rid of it. No functional change should be expected. (Use diff -w to check patch changes since reindentation makes the patch look heavy, but in fact it remains fairly small)
2024-04-30 09:52:57 -04:00
if (tmp->type == LOG_FMT_EXPR) {
/* sample expression, may be request or response */
int type;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
MEDIUM: log: optimizing tmp->type handling in sess_build_logline() Instead of chaining 2 switchcases and performing encoding checks for all nodes let's actually split the logic in 2: first handle simple node types (text/separator), and then handle dynamic node types (tag, expr). Encoding options are only evaluated for dynamic node types. Also, last_isspace is always set to 0 after next_fmt label, since next_fmt label is only used for dynamic nodes, thus != LOG_FMT_SEPARATOR. Since LF_NODE_WITH_OPT() macro (which was introduced recently) is now unused, let's get rid of it. No functional change should be expected. (Use diff -w to check patch changes since reindentation makes the patch look heavy, but in fact it remains fairly small)
2024-04-30 09:52:57 -04:00
key = NULL;
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
if (ctx->options & LOG_OPT_REQ_CAP)
MEDIUM: log: optimizing tmp->type handling in sess_build_logline() Instead of chaining 2 switchcases and performing encoding checks for all nodes let's actually split the logic in 2: first handle simple node types (text/separator), and then handle dynamic node types (tag, expr). Encoding options are only evaluated for dynamic node types. Also, last_isspace is always set to 0 after next_fmt label, since next_fmt label is only used for dynamic nodes, thus != LOG_FMT_SEPARATOR. Since LF_NODE_WITH_OPT() macro (which was introduced recently) is now unused, let's get rid of it. No functional change should be expected. (Use diff -w to check patch changes since reindentation makes the patch look heavy, but in fact it remains fairly small)
2024-04-30 09:52:57 -04:00
key = sample_process(be, sess, s, SMP_OPT_DIR_REQ|SMP_OPT_FINAL, tmp->expr, NULL);
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
if (!key && (ctx->options & LOG_OPT_RES_CAP))
MEDIUM: log: optimizing tmp->type handling in sess_build_logline() Instead of chaining 2 switchcases and performing encoding checks for all nodes let's actually split the logic in 2: first handle simple node types (text/separator), and then handle dynamic node types (tag, expr). Encoding options are only evaluated for dynamic node types. Also, last_isspace is always set to 0 after next_fmt label, since next_fmt label is only used for dynamic nodes, thus != LOG_FMT_SEPARATOR. Since LF_NODE_WITH_OPT() macro (which was introduced recently) is now unused, let's get rid of it. No functional change should be expected. (Use diff -w to check patch changes since reindentation makes the patch look heavy, but in fact it remains fairly small)
2024-04-30 09:52:57 -04:00
key = sample_process(be, sess, s, SMP_OPT_DIR_RES|SMP_OPT_FINAL, tmp->expr, NULL);
MINOR: log: postpone conversion for sample expressions in sess_build_logline() In sess_build_logline(), for sample expression nodes, instead of directly calling sample_fetch_as_type(... SMP_T_STR), let's first process the sample using sample_process(), and then proceed with the conversion to str if required. Doing so will allow us to implement type casting and preserving logic.
2024-04-25 10:20:11 -04:00
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
if (!key && !(ctx->options & (LOG_OPT_REQ_CAP|LOG_OPT_RES_CAP))) // cfg, cli
MEDIUM: log: optimizing tmp->type handling in sess_build_logline() Instead of chaining 2 switchcases and performing encoding checks for all nodes let's actually split the logic in 2: first handle simple node types (text/separator), and then handle dynamic node types (tag, expr). Encoding options are only evaluated for dynamic node types. Also, last_isspace is always set to 0 after next_fmt label, since next_fmt label is only used for dynamic nodes, thus != LOG_FMT_SEPARATOR. Since LF_NODE_WITH_OPT() macro (which was introduced recently) is now unused, let's get rid of it. No functional change should be expected. (Use diff -w to check patch changes since reindentation makes the patch look heavy, but in fact it remains fairly small)
2024-04-30 09:52:57 -04:00
key = sample_process(be, sess, s, SMP_OPT_FINAL, tmp->expr, NULL);
MINOR: log: make log-format expressions completely usable outside of req/resp The log-format strings are usable at plenty of places, but the expressions using %[] were restricted to request or response context and nothing else. This prevents from using them from the config context or the CLI, let's relax this.
2021-09-03 02:53:29 -04:00
MEDIUM: log: optimizing tmp->type handling in sess_build_logline() Instead of chaining 2 switchcases and performing encoding checks for all nodes let's actually split the logic in 2: first handle simple node types (text/separator), and then handle dynamic node types (tag, expr). Encoding options are only evaluated for dynamic node types. Also, last_isspace is always set to 0 after next_fmt label, since next_fmt label is only used for dynamic nodes, thus != LOG_FMT_SEPARATOR. Since LF_NODE_WITH_OPT() macro (which was introduced recently) is now unused, let's get rid of it. No functional change should be expected. (Use diff -w to check patch changes since reindentation makes the patch look heavy, but in fact it remains fairly small)
2024-04-30 09:52:57 -04:00
type = SMP_T_STR; // default
MINOR: log: make log-format expressions completely usable outside of req/resp The log-format strings are usable at plenty of places, but the expressions using %[] were restricted to request or response context and nothing else. This prevents from using them from the config context or the CLI, let's relax this.
2021-09-03 02:53:29 -04:00
MEDIUM: log: optimizing tmp->type handling in sess_build_logline() Instead of chaining 2 switchcases and performing encoding checks for all nodes let's actually split the logic in 2: first handle simple node types (text/separator), and then handle dynamic node types (tag, expr). Encoding options are only evaluated for dynamic node types. Also, last_isspace is always set to 0 after next_fmt label, since next_fmt label is only used for dynamic nodes, thus != LOG_FMT_SEPARATOR. Since LF_NODE_WITH_OPT() macro (which was introduced recently) is now unused, let's get rid of it. No functional change should be expected. (Use diff -w to check patch changes since reindentation makes the patch look heavy, but in fact it remains fairly small)
2024-04-30 09:52:57 -04:00
if (key && key->data.type == SMP_T_BIN &&
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
(ctx->options & LOG_OPT_BIN)) {
MEDIUM: log: optimizing tmp->type handling in sess_build_logline() Instead of chaining 2 switchcases and performing encoding checks for all nodes let's actually split the logic in 2: first handle simple node types (text/separator), and then handle dynamic node types (tag, expr). Encoding options are only evaluated for dynamic node types. Also, last_isspace is always set to 0 after next_fmt label, since next_fmt label is only used for dynamic nodes, thus != LOG_FMT_SEPARATOR. Since LF_NODE_WITH_OPT() macro (which was introduced recently) is now unused, let's get rid of it. No functional change should be expected. (Use diff -w to check patch changes since reindentation makes the patch look heavy, but in fact it remains fairly small)
2024-04-30 09:52:57 -04:00
/* output type is binary, and binary option is set:
* preserve output type unless typecast is set to
* force output type to string
*/
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
if (ctx->typecast != SMP_T_STR)
MEDIUM: log: optimizing tmp->type handling in sess_build_logline() Instead of chaining 2 switchcases and performing encoding checks for all nodes let's actually split the logic in 2: first handle simple node types (text/separator), and then handle dynamic node types (tag, expr). Encoding options are only evaluated for dynamic node types. Also, last_isspace is always set to 0 after next_fmt label, since next_fmt label is only used for dynamic nodes, thus != LOG_FMT_SEPARATOR. Since LF_NODE_WITH_OPT() macro (which was introduced recently) is now unused, let's get rid of it. No functional change should be expected. (Use diff -w to check patch changes since reindentation makes the patch look heavy, but in fact it remains fairly small)
2024-04-30 09:52:57 -04:00
type = SMP_T_BIN;
}
MINOR: log: postpone conversion for sample expressions in sess_build_logline() In sess_build_logline(), for sample expression nodes, instead of directly calling sample_fetch_as_type(... SMP_T_STR), let's first process the sample using sample_process(), and then proceed with the conversion to str if required. Doing so will allow us to implement type casting and preserving logic.
2024-04-25 10:20:11 -04:00
MEDIUM: log: optimizing tmp->type handling in sess_build_logline() Instead of chaining 2 switchcases and performing encoding checks for all nodes let's actually split the logic in 2: first handle simple node types (text/separator), and then handle dynamic node types (tag, expr). Encoding options are only evaluated for dynamic node types. Also, last_isspace is always set to 0 after next_fmt label, since next_fmt label is only used for dynamic nodes, thus != LOG_FMT_SEPARATOR. Since LF_NODE_WITH_OPT() macro (which was introduced recently) is now unused, let's get rid of it. No functional change should be expected. (Use diff -w to check patch changes since reindentation makes the patch look heavy, but in fact it remains fairly small)
2024-04-30 09:52:57 -04:00
/* if encoding is set, try to preserve output type
* with respect to typecast settings
* (ie: str, sint, bool)
*
* Special case for cbor encoding: we also try to
* preserve bin output type since cbor encoders
* know how to deal with binary data.
*/
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
if (ctx->options & LOG_OPT_ENCODE) {
if (ctx->typecast == SMP_T_STR ||
ctx->typecast == SMP_T_SINT ||
ctx->typecast == SMP_T_BOOL) {
MEDIUM: log: optimizing tmp->type handling in sess_build_logline() Instead of chaining 2 switchcases and performing encoding checks for all nodes let's actually split the logic in 2: first handle simple node types (text/separator), and then handle dynamic node types (tag, expr). Encoding options are only evaluated for dynamic node types. Also, last_isspace is always set to 0 after next_fmt label, since next_fmt label is only used for dynamic nodes, thus != LOG_FMT_SEPARATOR. Since LF_NODE_WITH_OPT() macro (which was introduced recently) is now unused, let's get rid of it. No functional change should be expected. (Use diff -w to check patch changes since reindentation makes the patch look heavy, but in fact it remains fairly small)
2024-04-30 09:52:57 -04:00
/* enforce type */
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
type = ctx->typecast;
MINOR: log: add +bin logformat node option Support '+bin' option argument on logformat nodes to try to preserve binary output type with binary sample expressions. For this, we rely on the log/sink API which is capable of conveying binary data since all related functions don't search for a terminating NULL byte in provided log payload as they take a string pointer and a string length as argument. Example: log-format "%{+bin}o %[bin(00AABB)]" Will produce: 00aabb (output was piped to `hexdump -ve '1/1 "%.2x"'` to dump raw bytes as HEX characters) This should be used carefully, because many syslog endpoints don't expect binary data (especially NULL bytes). This is mainly intended for use with set-var-fmt actions or with ring/udp log endpoints that know how to deal with such binary payloads. Also, this option is only supported globally (for use with '%o'), it will not have any effect when set on an individual node. (it makes no sense to have binary data in the middle of log payload that was started without binary data option)
2024-04-25 10:29:01 -04:00
}
MEDIUM: log: optimizing tmp->type handling in sess_build_logline() Instead of chaining 2 switchcases and performing encoding checks for all nodes let's actually split the logic in 2: first handle simple node types (text/separator), and then handle dynamic node types (tag, expr). Encoding options are only evaluated for dynamic node types. Also, last_isspace is always set to 0 after next_fmt label, since next_fmt label is only used for dynamic nodes, thus != LOG_FMT_SEPARATOR. Since LF_NODE_WITH_OPT() macro (which was introduced recently) is now unused, let's get rid of it. No functional change should be expected. (Use diff -w to check patch changes since reindentation makes the patch look heavy, but in fact it remains fairly small)
2024-04-30 09:52:57 -04:00
else if (key &&
(key->data.type == SMP_T_SINT ||
key->data.type == SMP_T_BOOL ||
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
((ctx->options & LOG_OPT_ENCODE_CBOR) &&
MEDIUM: log: optimizing tmp->type handling in sess_build_logline() Instead of chaining 2 switchcases and performing encoding checks for all nodes let's actually split the logic in 2: first handle simple node types (text/separator), and then handle dynamic node types (tag, expr). Encoding options are only evaluated for dynamic node types. Also, last_isspace is always set to 0 after next_fmt label, since next_fmt label is only used for dynamic nodes, thus != LOG_FMT_SEPARATOR. Since LF_NODE_WITH_OPT() macro (which was introduced recently) is now unused, let's get rid of it. No functional change should be expected. (Use diff -w to check patch changes since reindentation makes the patch look heavy, but in fact it remains fairly small)
2024-04-30 09:52:57 -04:00
key->data.type == SMP_T_BIN))) {
/* preserve type */
type = key->data.type;
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
}
MEDIUM: log: optimizing tmp->type handling in sess_build_logline() Instead of chaining 2 switchcases and performing encoding checks for all nodes let's actually split the logic in 2: first handle simple node types (text/separator), and then handle dynamic node types (tag, expr). Encoding options are only evaluated for dynamic node types. Also, last_isspace is always set to 0 after next_fmt label, since next_fmt label is only used for dynamic nodes, thus != LOG_FMT_SEPARATOR. Since LF_NODE_WITH_OPT() macro (which was introduced recently) is now unused, let's get rid of it. No functional change should be expected. (Use diff -w to check patch changes since reindentation makes the patch look heavy, but in fact it remains fairly small)
2024-04-30 09:52:57 -04:00
}
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
MEDIUM: log: optimizing tmp->type handling in sess_build_logline() Instead of chaining 2 switchcases and performing encoding checks for all nodes let's actually split the logic in 2: first handle simple node types (text/separator), and then handle dynamic node types (tag, expr). Encoding options are only evaluated for dynamic node types. Also, last_isspace is always set to 0 after next_fmt label, since next_fmt label is only used for dynamic nodes, thus != LOG_FMT_SEPARATOR. Since LF_NODE_WITH_OPT() macro (which was introduced recently) is now unused, let's get rid of it. No functional change should be expected. (Use diff -w to check patch changes since reindentation makes the patch look heavy, but in fact it remains fairly small)
2024-04-30 09:52:57 -04:00
if (key && !sample_convert(key, type))
key = NULL;
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
if (ctx->options & LOG_OPT_HTTP)
MEDIUM: log: optimizing tmp->type handling in sess_build_logline() Instead of chaining 2 switchcases and performing encoding checks for all nodes let's actually split the logic in 2: first handle simple node types (text/separator), and then handle dynamic node types (tag, expr). Encoding options are only evaluated for dynamic node types. Also, last_isspace is always set to 0 after next_fmt label, since next_fmt label is only used for dynamic nodes, thus != LOG_FMT_SEPARATOR. Since LF_NODE_WITH_OPT() macro (which was introduced recently) is now unused, let's get rid of it. No functional change should be expected. (Use diff -w to check patch changes since reindentation makes the patch look heavy, but in fact it remains fairly small)
2024-04-30 09:52:57 -04:00
ret = lf_encode_chunk(tmplog, dst + maxsize,
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
'%', http_encode_map, key ? &key->data.u.str : &empty, ctx);
MEDIUM: log: optimizing tmp->type handling in sess_build_logline() Instead of chaining 2 switchcases and performing encoding checks for all nodes let's actually split the logic in 2: first handle simple node types (text/separator), and then handle dynamic node types (tag, expr). Encoding options are only evaluated for dynamic node types. Also, last_isspace is always set to 0 after next_fmt label, since next_fmt label is only used for dynamic nodes, thus != LOG_FMT_SEPARATOR. Since LF_NODE_WITH_OPT() macro (which was introduced recently) is now unused, let's get rid of it. No functional change should be expected. (Use diff -w to check patch changes since reindentation makes the patch look heavy, but in fact it remains fairly small)
2024-04-30 09:52:57 -04:00
else {
if (key && type == SMP_T_BIN)
MEDIUM: log: add a new log format flag "E" The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It mostly makes sense to use it in the RFC5424 structured-data log formats. Example: log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 07:23:03 -05:00
ret = lf_encode_chunk(tmplog, dst + maxsize,
MEDIUM: log: optimizing tmp->type handling in sess_build_logline() Instead of chaining 2 switchcases and performing encoding checks for all nodes let's actually split the logic in 2: first handle simple node types (text/separator), and then handle dynamic node types (tag, expr). Encoding options are only evaluated for dynamic node types. Also, last_isspace is always set to 0 after next_fmt label, since next_fmt label is only used for dynamic nodes, thus != LOG_FMT_SEPARATOR. Since LF_NODE_WITH_OPT() macro (which was introduced recently) is now unused, let's get rid of it. No functional change should be expected. (Use diff -w to check patch changes since reindentation makes the patch look heavy, but in fact it remains fairly small)
2024-04-30 09:52:57 -04:00
0, no_escape_map,
&key->data.u.str,
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ctx);
MEDIUM: log: optimizing tmp->type handling in sess_build_logline() Instead of chaining 2 switchcases and performing encoding checks for all nodes let's actually split the logic in 2: first handle simple node types (text/separator), and then handle dynamic node types (tag, expr). Encoding options are only evaluated for dynamic node types. Also, last_isspace is always set to 0 after next_fmt label, since next_fmt label is only used for dynamic nodes, thus != LOG_FMT_SEPARATOR. Since LF_NODE_WITH_OPT() macro (which was introduced recently) is now unused, let's get rid of it. No functional change should be expected. (Use diff -w to check patch changes since reindentation makes the patch look heavy, but in fact it remains fairly small)
2024-04-30 09:52:57 -04:00
else if (key && type == SMP_T_SINT)
ret = lf_int_encode(tmplog, dst + maxsize - tmplog,
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
key->data.u.sint, ctx);
MEDIUM: log: optimizing tmp->type handling in sess_build_logline() Instead of chaining 2 switchcases and performing encoding checks for all nodes let's actually split the logic in 2: first handle simple node types (text/separator), and then handle dynamic node types (tag, expr). Encoding options are only evaluated for dynamic node types. Also, last_isspace is always set to 0 after next_fmt label, since next_fmt label is only used for dynamic nodes, thus != LOG_FMT_SEPARATOR. Since LF_NODE_WITH_OPT() macro (which was introduced recently) is now unused, let's get rid of it. No functional change should be expected. (Use diff -w to check patch changes since reindentation makes the patch look heavy, but in fact it remains fairly small)
2024-04-30 09:52:57 -04:00
else if (key && type == SMP_T_BOOL)
ret = lf_bool_encode(tmplog, dst + maxsize - tmplog,
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
key->data.u.sint, ctx);
MEDIUM: log: optimizing tmp->type handling in sess_build_logline() Instead of chaining 2 switchcases and performing encoding checks for all nodes let's actually split the logic in 2: first handle simple node types (text/separator), and then handle dynamic node types (tag, expr). Encoding options are only evaluated for dynamic node types. Also, last_isspace is always set to 0 after next_fmt label, since next_fmt label is only used for dynamic nodes, thus != LOG_FMT_SEPARATOR. Since LF_NODE_WITH_OPT() macro (which was introduced recently) is now unused, let's get rid of it. No functional change should be expected. (Use diff -w to check patch changes since reindentation makes the patch look heavy, but in fact it remains fairly small)
2024-04-30 09:52:57 -04:00
else
ret = lf_text_len(tmplog,
key ? key->data.u.str.area : NULL,
key ? key->data.u.str.data : 0,
dst + maxsize - tmplog,
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ctx);
MINOR: log: postpone conversion for sample expressions in sess_build_logline() In sess_build_logline(), for sample expression nodes, instead of directly calling sample_fetch_as_type(... SMP_T_STR), let's first process the sample using sample_process(), and then proceed with the conversion to str if required. Doing so will allow us to implement type casting and preserving logic.
2024-04-25 10:20:11 -04:00
}
MEDIUM: log: optimizing tmp->type handling in sess_build_logline() Instead of chaining 2 switchcases and performing encoding checks for all nodes let's actually split the logic in 2: first handle simple node types (text/separator), and then handle dynamic node types (tag, expr). Encoding options are only evaluated for dynamic node types. Also, last_isspace is always set to 0 after next_fmt label, since next_fmt label is only used for dynamic nodes, thus != LOG_FMT_SEPARATOR. Since LF_NODE_WITH_OPT() macro (which was introduced recently) is now unused, let's get rid of it. No functional change should be expected. (Use diff -w to check patch changes since reindentation makes the patch look heavy, but in fact it remains fairly small)
2024-04-30 09:52:57 -04:00
if (ret == NULL)
goto out;
tmplog = ret;
BUG/MINOR: log: prevent double spaces emission in sess_build_logline() Christian reported in GH #2556 that since 3.0-dev double spaces may be found in log messages on some cases where it was not the case before. As we were able to easily reproduce, a quick bisect led us to c6a7138 ("MINOR: log: simplify last_isspace in sess_build_logline()"). While it is true that all switch cases set the last_isspace variable to 0, there was a subtelty for some fields such as '%hr', '%hrl', '%hs' or '%hsl' and I overlooked it. Indeed, for '%hr', last_isspace was only set to 0 if data was emitted, else the assignment didn't occur. But with c6a7138, last_isspace is always set to 0 as long as the current node type is not a separator. Because of that, if no data is emitted for the current node value, and a space was already emitted prior to the current node, then an extra space could be emitted after the node, resulting in two spaces being emitted. Note that while c6a7138 introduces a slight behavior regression regarding last_isspace logic with the specific fields mentionned above, this behavior could already be triggered with a failing or empty logformat node sample expression. Consider this logformat expression: log-format "%{-M}o | %[str()] |" str() will not print anything, and since we disabled mandatory option with '-M', nothing gets printed for the node sample expression. As a result, we have the following output: "| |" Instead of (when mandatory option is enabled): "| - |" Thus in order to stick to the historical behavior, systematically set last_isspace to 0 for EXPR nodes, and only set last_isspace to 0 when data was written for TAG nodes. This way, '%hr', '%hrl', '%hs' or '%hsl' should behave as before. No backport needed.
2024-05-02 03:30:28 -04:00
last_isspace = 0; /* consider that data was written */
MEDIUM: log: optimizing tmp->type handling in sess_build_logline() Instead of chaining 2 switchcases and performing encoding checks for all nodes let's actually split the logic in 2: first handle simple node types (text/separator), and then handle dynamic node types (tag, expr). Encoding options are only evaluated for dynamic node types. Also, last_isspace is always set to 0 after next_fmt label, since next_fmt label is only used for dynamic nodes, thus != LOG_FMT_SEPARATOR. Since LF_NODE_WITH_OPT() macro (which was introduced recently) is now unused, let's get rid of it. No functional change should be expected. (Use diff -w to check patch changes since reindentation makes the patch look heavy, but in fact it remains fairly small)
2024-04-30 09:52:57 -04:00
goto next_fmt;
MEDIUM: log: carry tag context in logformat node This is a pretty simple patch despite requiring to make some visible changes in the code: When parsing a logformat string, log tags (ie: '%tag', AKA log tags) are turned into logformat nodes with their type set to the type of the corresponding logformat_tag element which was matched by name. Thus, when "compiling" a logformat tag, we only keep a reference to the tag type from the original logformat_tag. For example, for "%B" log tag, we have the following logformat_tag element: { .name = "B", .type = LOG_FMT_BYTES, .mode = PR_MODE_TCP, .lw = LW_BYTES, .config_callback = NULL } When parsing "%B" string, we search for a matching logformat tag inside logformat_tags[] array using the provided name, once we find a matching element, we craft a logformat node whose type will be LOG_FMT_BYTES, but from the node itself, we no longer have access to other informations that are set in the logformat_tag struct element. Thus from a logformat_node resulting from a log tag, with current implementation, we cannot easily get back to matching logformat_tag struct element as it would require us to scan the whole logformat_tags array at runtime using node->type to find the matching element. Let's take a simpler path and consider all tag-specific LOG_FMT_* subtypes as being part of the same logformat node type: LOG_FMT_TAG. Thanks to that, we're now able to distinguish logformat nodes made from logformat tag from other logformat nodes, and link them to their corresponding logformat_tag element from logformat_tags[] array. All it costs is a simple indirection and an extra pointer in logformat_node struct. While at it, all LOG_FMT_* types related to logformat tags were moved inside log.c as they have no use outside of it since they are simply lookup indexes for sess_build_logline() and could even be replaced by function pointers some day...
2024-02-22 14:20:41 -05:00
}
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
BUG_ON(tmp->type != LOG_FMT_ALIAS);
MEDIUM: log: add the ability to include samples in logs Using %[expression] it becomes possible to make the log engine fetch some samples from the request or the response and provide them in the logs. Note that this feature is still limited, it does not yet allow to apply converters, to limit the output length, nor to specify the direction which should be fetched when a fetch function works in both directions. However it's quite convenient to log SSL information or to include some information that are used in stick tables. It is worth noting that this has been done in the generic log format handler, which means that the same information may be used to build the unique-id header and to pass the information to a backend server.
2012-12-20 18:09:23 -05:00
MINOR: log: rename 'log-format tag' to 'log-format alias' In 2.9 we started to introduce an ambiguity in the documentation by referring to historical log-format variables ('%var') as log-format tags in 739c4e5b1e ("MINOR: sample: accept_date / request_date return %Ts / %tr timestamp values") and 454c372b60 ("DOC: configuration: add sample fetches for timing events"). In fact, we've had this confusion between log-format tag and log-format var for more than 10 years now, but in 2.9 it was the first time the confusion was exposed in the documentation. Indeed, both 'log-format variable' and 'log-format tag' actually refer to the same feature (that is: '%B' and friends that can be used for direct access to some log-oriented predefined fetches instead of using %[expr] with generic sample expressions). This feature was first implemented in 723b73ad75 ("MINOR: config: Parse the string of the log-format config keyword") and later documented in 4894040fa ("DOC: log-format documentation"). At that time, it was clear that we used to name it 'log-format variable'. But later the same year, 'log-format tag' naming started to appear in some commit messages (while still referring to the same feature), for instance with ffc3fcd6d ("MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv"). Unfortunately in 2.9 when we added (and documented) new log-format variables we officially started drifting to the misleading 'log-format tag' naming (perhaps because it was the most recent naming found for this feature in git log history, or because the confusion has always been there) Even worse, in 3.0 this confusion led us to rename all 'var' occurrences to 'tag' in log-format related code to unify the code with the doc. Hopefully William quickly noticed that we made a mistake there, but instead of reverting to historical naming (log-format variable), it was decided that we must use a different name that is less confusing than 'tags' or 'variables' (tags and variables are keywords that are already used to designate other features in the code and that are not very explicit under log-format context today). Now we refer to '%B' and friends as a logformat alias, which is essentially a handy way to print some log oriented information in the log string instead of leveraging '%[expr]' with generic sample expressions made of fetches and converters. Of course, there are some subtelties, such as a few log-format aliases that still don't have sample fetch equivalent for historical reasons, and some aliases that may be a little faster than their generic sample expression equivalents because most aliases are pretty much hardcoded in the log building function. But in general logformat aliases should be simply considered as an alternative to using expressions (with '%[expr']') Also, under log-format context, when we want to refer to either an alias ('%alias') or an expression ('%[expr]'), we should use the generic term 'logformat item', which in fact designates a single item within the logformat string provided by the user. Indeed, a logformat item (whether is is an alias or an expression) always starts with '%' and may accept optional flags / arguments Both the code and the documentation were updated in that sense, hopefully this will clarify things and prevent future confusions.
2024-05-27 04:18:10 -04:00
/* logformat alias */
switch (tmp->alias->type) {
MEDIUM: log: change a few log tokens to make them easier to remember Some log tokens have evolved in a way that is not completely logical. For example, frontend tokens sometimes begin with an 'f' and sometimes with an 'F'. Same for backend and server. So let's change a few cases without disrupting compatibility with existing setups : Bi => bi Bp => bp Ci => ci Cp => cp Fi => fi Fp => fp Si => si Sp => sp cc => CC cs => CS st => ST The old ones are still supported but deprecated and will be unsupported by the 1.5 release. However, a warning message is emitted when they're encounterd and it indicates what token should be used to replace them.
2012-12-20 11:22:52 -05:00
case LOG_FMT_CLIENTIP: // %ci
CLEANUP: stconn: rename cs{,_get}_{src,dst} to sc_* The following functions were renamed: cs_src() -> sc_src() cs_dst() -> sc_dst() cs_get_src() -> sc_get_src() cs_get_dst() -> sc_get_dst()
2022-05-27 02:57:21 -04:00
addr = (s ? sc_src(s->scf) : sess_src(sess));
MINOR: log: Rely on client addresses at the appropriate level to log messages When a log message is emitted, if the stream exits, we use the frontend stream-interface to retrieve the client source and destination addresses. Otherwise, the session is used. For now, stream-interface or session addresses are never set. So, thanks to the fallback mechanism, no changes are expected with this patch. But its purpose is to rely on addresses at the appropriate level when set instead of those at the connection level.
2021-10-22 11:43:22 -04:00
if (addr)
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_ip(tmplog, (struct sockaddr *)addr, dst + maxsize - tmplog, ctx);
MAJOR: stream-int: stop using si->conn and use si->end instead The connection will only remain there as a pre-allocated entity whose goal is to be placed in ->end when establishing an outgoing connection. All connection initialization can be made on this connection, but all information retrieved should be applied to the end point only. This change is huge because there were many users of si->conn. Now the only users are those who initialize the new connection. The difficulty appears in a few places such as backend.c, proto_http.c, peers.c where si->conn is used to hold the connection's target address before assigning the connection to the stream interface. This is why we have to keep si->conn for now. A future improvement might consist in dynamically allocating the connection when it is needed.
2013-10-01 04:45:07 -04:00
else
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_text_len(tmplog, NULL, 0, dst + maxsize - tmplog, ctx);
MINOR: log: Rely on client addresses at the appropriate level to log messages When a log message is emitted, if the stream exits, we use the frontend stream-interface to retrieve the client source and destination addresses. Otherwise, the session is used. For now, stream-interface or session addresses are never set. So, thanks to the fallback mechanism, no changes are expected with this patch. But its purpose is to rely on addresses at the appropriate level when set instead of those at the connection level.
2021-10-22 11:43:22 -04:00
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
if (ret == NULL)
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
goto out;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
tmplog = ret;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
break;
MEDIUM: log: change a few log tokens to make them easier to remember Some log tokens have evolved in a way that is not completely logical. For example, frontend tokens sometimes begin with an 'f' and sometimes with an 'F'. Same for backend and server. So let's change a few cases without disrupting compatibility with existing setups : Bi => bi Bp => bp Ci => ci Cp => cp Fi => fi Fp => fp Si => si Sp => sp cc => CC cs => CS st => ST The old ones are still supported but deprecated and will be unsupported by the 1.5 release. However, a warning message is emitted when they're encounterd and it indicates what token should be used to replace them.
2012-12-20 11:22:52 -05:00
case LOG_FMT_CLIENTPORT: // %cp
CLEANUP: stconn: rename cs{,_get}_{src,dst} to sc_* The following functions were renamed: cs_src() -> sc_src() cs_dst() -> sc_dst() cs_get_src() -> sc_get_src() cs_get_dst() -> sc_get_dst()
2022-05-27 02:57:21 -04:00
addr = (s ? sc_src(s->scf) : sess_src(sess));
MINOR: log: Rely on client addresses at the appropriate level to log messages When a log message is emitted, if the stream exits, we use the frontend stream-interface to retrieve the client source and destination addresses. Otherwise, the session is used. For now, stream-interface or session addresses are never set. So, thanks to the fallback mechanism, no changes are expected with this patch. But its purpose is to rely on addresses at the appropriate level when set instead of those at the connection level.
2021-10-22 11:43:22 -04:00
if (addr) {
DOC: log: Add comments to specify when session's listener is defined or not When a log message is emitted, The session's listener is always defined when the session's owner is an inbound connection while it is undefined for a health-check. It is not obvious. So, comments have been added to make it clear. This patch is related to the issue #1434.
2021-11-15 05:31:08 -05:00
/* sess->listener is always defined when the session's owner is an inbound connections */
MINOR: log: Rely on client addresses at the appropriate level to log messages When a log message is emitted, if the stream exits, we use the frontend stream-interface to retrieve the client source and destination addresses. Otherwise, the session is used. For now, stream-interface or session addresses are never set. So, thanks to the fallback mechanism, no changes are expected with this patch. But its purpose is to rely on addresses at the appropriate level when set instead of those at the connection level.
2021-10-22 11:43:22 -04:00
if (addr->ss_family == AF_UNIX)
MINOR: log: add lf_int() wrapper to print integers Wrap ltoa(), lltoa(), ultoa() and utoa_pad() functions that are used by sess_build_logline() to print numerical values by implementing a dedicated helper named lf_int() that takes <dft_hld> as argument to know how to write the integer by default (when no encoding is specified). LF_INT_UTOA_PAD_4 is used to emulate utoa_pad(x, 4) since it's found only once under sess_build_logline(), thus there is no need to pass an extra parameter to lf_int() function.
2024-03-27 05:17:11 -04:00
ret = lf_int(tmplog, dst + maxsize - tmplog,
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
sess->listener->luid, ctx, LF_INT_LTOA);
MINOR: log: Rely on client addresses at the appropriate level to log messages When a log message is emitted, if the stream exits, we use the frontend stream-interface to retrieve the client source and destination addresses. Otherwise, the session is used. For now, stream-interface or session addresses are never set. So, thanks to the fallback mechanism, no changes are expected with this patch. But its purpose is to rely on addresses at the appropriate level when set instead of those at the connection level.
2021-10-22 11:43:22 -04:00
else
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_port(tmplog, (struct sockaddr *)addr, dst + maxsize - tmplog, ctx);
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
}
MAJOR: stream-int: stop using si->conn and use si->end instead The connection will only remain there as a pre-allocated entity whose goal is to be placed in ->end when establishing an outgoing connection. All connection initialization can be made on this connection, but all information retrieved should be applied to the end point only. This change is huge because there were many users of si->conn. Now the only users are those who initialize the new connection. The difficulty appears in a few places such as backend.c, proto_http.c, peers.c where si->conn is used to hold the connection's target address before assigning the connection to the stream interface. This is why we have to keep si->conn for now. A future improvement might consist in dynamically allocating the connection when it is needed.
2013-10-01 04:45:07 -04:00
else
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_text_len(tmplog, NULL, 0, dst + maxsize - tmplog, ctx);
MAJOR: stream-int: stop using si->conn and use si->end instead The connection will only remain there as a pre-allocated entity whose goal is to be placed in ->end when establishing an outgoing connection. All connection initialization can be made on this connection, but all information retrieved should be applied to the end point only. This change is huge because there were many users of si->conn. Now the only users are those who initialize the new connection. The difficulty appears in a few places such as backend.c, proto_http.c, peers.c where si->conn is used to hold the connection's target address before assigning the connection to the stream interface. This is why we have to keep si->conn for now. A future improvement might consist in dynamically allocating the connection when it is needed.
2013-10-01 04:45:07 -04:00
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
if (ret == NULL)
goto out;
tmplog = ret;
break;
MEDIUM: log: change a few log tokens to make them easier to remember Some log tokens have evolved in a way that is not completely logical. For example, frontend tokens sometimes begin with an 'f' and sometimes with an 'F'. Same for backend and server. So let's change a few cases without disrupting compatibility with existing setups : Bi => bi Bp => bp Ci => ci Cp => cp Fi => fi Fp => fp Si => si Sp => sp cc => CC cs => CS st => ST The old ones are still supported but deprecated and will be unsupported by the 1.5 release. However, a warning message is emitted when they're encounterd and it indicates what token should be used to replace them.
2012-12-20 11:22:52 -05:00
case LOG_FMT_FRONTENDIP: // %fi
CLEANUP: stconn: rename cs{,_get}_{src,dst} to sc_* The following functions were renamed: cs_src() -> sc_src() cs_dst() -> sc_dst() cs_get_src() -> sc_get_src() cs_get_dst() -> sc_get_dst()
2022-05-27 02:57:21 -04:00
addr = (s ? sc_dst(s->scf) : sess_dst(sess));
MINOR: log: Rely on client addresses at the appropriate level to log messages When a log message is emitted, if the stream exits, we use the frontend stream-interface to retrieve the client source and destination addresses. Otherwise, the session is used. For now, stream-interface or session addresses are never set. So, thanks to the fallback mechanism, no changes are expected with this patch. But its purpose is to rely on addresses at the appropriate level when set instead of those at the connection level.
2021-10-22 11:43:22 -04:00
if (addr)
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_ip(tmplog, (struct sockaddr *)addr, dst + maxsize - tmplog, ctx);
MAJOR: stream-int: stop using si->conn and use si->end instead The connection will only remain there as a pre-allocated entity whose goal is to be placed in ->end when establishing an outgoing connection. All connection initialization can be made on this connection, but all information retrieved should be applied to the end point only. This change is huge because there were many users of si->conn. Now the only users are those who initialize the new connection. The difficulty appears in a few places such as backend.c, proto_http.c, peers.c where si->conn is used to hold the connection's target address before assigning the connection to the stream interface. This is why we have to keep si->conn for now. A future improvement might consist in dynamically allocating the connection when it is needed.
2013-10-01 04:45:07 -04:00
else
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_text_len(tmplog, NULL, 0, dst + maxsize - tmplog, ctx);
MAJOR: stream-int: stop using si->conn and use si->end instead The connection will only remain there as a pre-allocated entity whose goal is to be placed in ->end when establishing an outgoing connection. All connection initialization can be made on this connection, but all information retrieved should be applied to the end point only. This change is huge because there were many users of si->conn. Now the only users are those who initialize the new connection. The difficulty appears in a few places such as backend.c, proto_http.c, peers.c where si->conn is used to hold the connection's target address before assigning the connection to the stream interface. This is why we have to keep si->conn for now. A future improvement might consist in dynamically allocating the connection when it is needed.
2013-10-01 04:45:07 -04:00
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
if (ret == NULL)
goto out;
tmplog = ret;
break;
MEDIUM: log: change a few log tokens to make them easier to remember Some log tokens have evolved in a way that is not completely logical. For example, frontend tokens sometimes begin with an 'f' and sometimes with an 'F'. Same for backend and server. So let's change a few cases without disrupting compatibility with existing setups : Bi => bi Bp => bp Ci => ci Cp => cp Fi => fi Fp => fp Si => si Sp => sp cc => CC cs => CS st => ST The old ones are still supported but deprecated and will be unsupported by the 1.5 release. However, a warning message is emitted when they're encounterd and it indicates what token should be used to replace them.
2012-12-20 11:22:52 -05:00
case LOG_FMT_FRONTENDPORT: // %fp
CLEANUP: stconn: rename cs{,_get}_{src,dst} to sc_* The following functions were renamed: cs_src() -> sc_src() cs_dst() -> sc_dst() cs_get_src() -> sc_get_src() cs_get_dst() -> sc_get_dst()
2022-05-27 02:57:21 -04:00
addr = (s ? sc_dst(s->scf) : sess_dst(sess));
MINOR: log: Rely on client addresses at the appropriate level to log messages When a log message is emitted, if the stream exits, we use the frontend stream-interface to retrieve the client source and destination addresses. Otherwise, the session is used. For now, stream-interface or session addresses are never set. So, thanks to the fallback mechanism, no changes are expected with this patch. But its purpose is to rely on addresses at the appropriate level when set instead of those at the connection level.
2021-10-22 11:43:22 -04:00
if (addr) {
DOC: log: Add comments to specify when session's listener is defined or not When a log message is emitted, The session's listener is always defined when the session's owner is an inbound connection while it is undefined for a health-check. It is not obvious. So, comments have been added to make it clear. This patch is related to the issue #1434.
2021-11-15 05:31:08 -05:00
/* sess->listener is always defined when the session's owner is an inbound connections */
MINOR: log: Rely on client addresses at the appropriate level to log messages When a log message is emitted, if the stream exits, we use the frontend stream-interface to retrieve the client source and destination addresses. Otherwise, the session is used. For now, stream-interface or session addresses are never set. So, thanks to the fallback mechanism, no changes are expected with this patch. But its purpose is to rely on addresses at the appropriate level when set instead of those at the connection level.
2021-10-22 11:43:22 -04:00
if (addr->ss_family == AF_UNIX)
MINOR: log: add lf_int() wrapper to print integers Wrap ltoa(), lltoa(), ultoa() and utoa_pad() functions that are used by sess_build_logline() to print numerical values by implementing a dedicated helper named lf_int() that takes <dft_hld> as argument to know how to write the integer by default (when no encoding is specified). LF_INT_UTOA_PAD_4 is used to emulate utoa_pad(x, 4) since it's found only once under sess_build_logline(), thus there is no need to pass an extra parameter to lf_int() function.
2024-03-27 05:17:11 -04:00
ret = lf_int(tmplog, dst + maxsize - tmplog,
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
sess->listener->luid, ctx, LF_INT_LTOA);
MAJOR: stream-int: stop using si->conn and use si->end instead The connection will only remain there as a pre-allocated entity whose goal is to be placed in ->end when establishing an outgoing connection. All connection initialization can be made on this connection, but all information retrieved should be applied to the end point only. This change is huge because there were many users of si->conn. Now the only users are those who initialize the new connection. The difficulty appears in a few places such as backend.c, proto_http.c, peers.c where si->conn is used to hold the connection's target address before assigning the connection to the stream interface. This is why we have to keep si->conn for now. A future improvement might consist in dynamically allocating the connection when it is needed.
2013-10-01 04:45:07 -04:00
else
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_port(tmplog, (struct sockaddr *)addr, dst + maxsize - tmplog, ctx);
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
}
MAJOR: stream-int: stop using si->conn and use si->end instead The connection will only remain there as a pre-allocated entity whose goal is to be placed in ->end when establishing an outgoing connection. All connection initialization can be made on this connection, but all information retrieved should be applied to the end point only. This change is huge because there were many users of si->conn. Now the only users are those who initialize the new connection. The difficulty appears in a few places such as backend.c, proto_http.c, peers.c where si->conn is used to hold the connection's target address before assigning the connection to the stream interface. This is why we have to keep si->conn for now. A future improvement might consist in dynamically allocating the connection when it is needed.
2013-10-01 04:45:07 -04:00
else
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_text_len(tmplog, NULL, 0, dst + maxsize - tmplog, ctx);
MAJOR: stream-int: stop using si->conn and use si->end instead The connection will only remain there as a pre-allocated entity whose goal is to be placed in ->end when establishing an outgoing connection. All connection initialization can be made on this connection, but all information retrieved should be applied to the end point only. This change is huge because there were many users of si->conn. Now the only users are those who initialize the new connection. The difficulty appears in a few places such as backend.c, proto_http.c, peers.c where si->conn is used to hold the connection's target address before assigning the connection to the stream interface. This is why we have to keep si->conn for now. A future improvement might consist in dynamically allocating the connection when it is needed.
2013-10-01 04:45:07 -04:00
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
if (ret == NULL)
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
goto out;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
tmplog = ret;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
break;
MEDIUM: log: change a few log tokens to make them easier to remember Some log tokens have evolved in a way that is not completely logical. For example, frontend tokens sometimes begin with an 'f' and sometimes with an 'F'. Same for backend and server. So let's change a few cases without disrupting compatibility with existing setups : Bi => bi Bp => bp Ci => ci Cp => cp Fi => fi Fp => fp Si => si Sp => sp cc => CC cs => CS st => ST The old ones are still supported but deprecated and will be unsupported by the 1.5 release. However, a warning message is emitted when they're encounterd and it indicates what token should be used to replace them.
2012-12-20 11:22:52 -05:00
case LOG_FMT_BACKENDIP: // %bi
MINOR: log: use conn_get_{dst,src}() to retrieve the cli/frt/bck/srv/ addresses This also allows us to check that the operation succeeded without logging whatever remained in the memory area in case of failure.
2019-07-17 05:47:11 -04:00
if (be_conn && conn_get_src(be_conn))
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_ip(tmplog, (const struct sockaddr *)be_conn->src, dst + maxsize - tmplog, ctx);
MAJOR: stream-int: stop using si->conn and use si->end instead The connection will only remain there as a pre-allocated entity whose goal is to be placed in ->end when establishing an outgoing connection. All connection initialization can be made on this connection, but all information retrieved should be applied to the end point only. This change is huge because there were many users of si->conn. Now the only users are those who initialize the new connection. The difficulty appears in a few places such as backend.c, proto_http.c, peers.c where si->conn is used to hold the connection's target address before assigning the connection to the stream interface. This is why we have to keep si->conn for now. A future improvement might consist in dynamically allocating the connection when it is needed.
2013-10-01 04:45:07 -04:00
else
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_text_len(tmplog, NULL, 0, dst + maxsize - tmplog, ctx);
MAJOR: stream-int: stop using si->conn and use si->end instead The connection will only remain there as a pre-allocated entity whose goal is to be placed in ->end when establishing an outgoing connection. All connection initialization can be made on this connection, but all information retrieved should be applied to the end point only. This change is huge because there were many users of si->conn. Now the only users are those who initialize the new connection. The difficulty appears in a few places such as backend.c, proto_http.c, peers.c where si->conn is used to hold the connection's target address before assigning the connection to the stream interface. This is why we have to keep si->conn for now. A future improvement might consist in dynamically allocating the connection when it is needed.
2013-10-01 04:45:07 -04:00
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
if (ret == NULL)
MEDIUM: log-format: backend source address %Bi %Bp %Bi return the backend source IP %Bp return the backend source port Add a function pointer in logformat_type to do additional configuration during the log-format variable parsing.
2012-03-02 08:35:21 -05:00
goto out;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
tmplog = ret;
MEDIUM: log-format: backend source address %Bi %Bp %Bi return the backend source IP %Bp return the backend source port Add a function pointer in logformat_type to do additional configuration during the log-format variable parsing.
2012-03-02 08:35:21 -05:00
break;
MEDIUM: log: change a few log tokens to make them easier to remember Some log tokens have evolved in a way that is not completely logical. For example, frontend tokens sometimes begin with an 'f' and sometimes with an 'F'. Same for backend and server. So let's change a few cases without disrupting compatibility with existing setups : Bi => bi Bp => bp Ci => ci Cp => cp Fi => fi Fp => fp Si => si Sp => sp cc => CC cs => CS st => ST The old ones are still supported but deprecated and will be unsupported by the 1.5 release. However, a warning message is emitted when they're encounterd and it indicates what token should be used to replace them.
2012-12-20 11:22:52 -05:00
case LOG_FMT_BACKENDPORT: // %bp
MINOR: log: use conn_get_{dst,src}() to retrieve the cli/frt/bck/srv/ addresses This also allows us to check that the operation succeeded without logging whatever remained in the memory area in case of failure.
2019-07-17 05:47:11 -04:00
if (be_conn && conn_get_src(be_conn))
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_port(tmplog, (struct sockaddr *)be_conn->src, dst + maxsize - tmplog, ctx);
MAJOR: stream-int: stop using si->conn and use si->end instead The connection will only remain there as a pre-allocated entity whose goal is to be placed in ->end when establishing an outgoing connection. All connection initialization can be made on this connection, but all information retrieved should be applied to the end point only. This change is huge because there were many users of si->conn. Now the only users are those who initialize the new connection. The difficulty appears in a few places such as backend.c, proto_http.c, peers.c where si->conn is used to hold the connection's target address before assigning the connection to the stream interface. This is why we have to keep si->conn for now. A future improvement might consist in dynamically allocating the connection when it is needed.
2013-10-01 04:45:07 -04:00
else
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_text_len(tmplog, NULL, 0, dst + maxsize - tmplog, ctx);
MAJOR: stream-int: stop using si->conn and use si->end instead The connection will only remain there as a pre-allocated entity whose goal is to be placed in ->end when establishing an outgoing connection. All connection initialization can be made on this connection, but all information retrieved should be applied to the end point only. This change is huge because there were many users of si->conn. Now the only users are those who initialize the new connection. The difficulty appears in a few places such as backend.c, proto_http.c, peers.c where si->conn is used to hold the connection's target address before assigning the connection to the stream interface. This is why we have to keep si->conn for now. A future improvement might consist in dynamically allocating the connection when it is needed.
2013-10-01 04:45:07 -04:00
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
if (ret == NULL)
goto out;
tmplog = ret;
break;
MEDIUM: log: change a few log tokens to make them easier to remember Some log tokens have evolved in a way that is not completely logical. For example, frontend tokens sometimes begin with an 'f' and sometimes with an 'F'. Same for backend and server. So let's change a few cases without disrupting compatibility with existing setups : Bi => bi Bp => bp Ci => ci Cp => cp Fi => fi Fp => fp Si => si Sp => sp cc => CC cs => CS st => ST The old ones are still supported but deprecated and will be unsupported by the 1.5 release. However, a warning message is emitted when they're encounterd and it indicates what token should be used to replace them.
2012-12-20 11:22:52 -05:00
case LOG_FMT_SERVERIP: // %si
MINOR: log: use conn_get_{dst,src}() to retrieve the cli/frt/bck/srv/ addresses This also allows us to check that the operation succeeded without logging whatever remained in the memory area in case of failure.
2019-07-17 05:47:11 -04:00
if (be_conn && conn_get_dst(be_conn))
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_ip(tmplog, (struct sockaddr *)be_conn->dst, dst + maxsize - tmplog, ctx);
MAJOR: stream-int: stop using si->conn and use si->end instead The connection will only remain there as a pre-allocated entity whose goal is to be placed in ->end when establishing an outgoing connection. All connection initialization can be made on this connection, but all information retrieved should be applied to the end point only. This change is huge because there were many users of si->conn. Now the only users are those who initialize the new connection. The difficulty appears in a few places such as backend.c, proto_http.c, peers.c where si->conn is used to hold the connection's target address before assigning the connection to the stream interface. This is why we have to keep si->conn for now. A future improvement might consist in dynamically allocating the connection when it is needed.
2013-10-01 04:45:07 -04:00
else
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_text_len(tmplog, NULL, 0, dst + maxsize - tmplog, ctx);
MAJOR: stream-int: stop using si->conn and use si->end instead The connection will only remain there as a pre-allocated entity whose goal is to be placed in ->end when establishing an outgoing connection. All connection initialization can be made on this connection, but all information retrieved should be applied to the end point only. This change is huge because there were many users of si->conn. Now the only users are those who initialize the new connection. The difficulty appears in a few places such as backend.c, proto_http.c, peers.c where si->conn is used to hold the connection's target address before assigning the connection to the stream interface. This is why we have to keep si->conn for now. A future improvement might consist in dynamically allocating the connection when it is needed.
2013-10-01 04:45:07 -04:00
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
if (ret == NULL)
goto out;
tmplog = ret;
break;
MEDIUM: log: change a few log tokens to make them easier to remember Some log tokens have evolved in a way that is not completely logical. For example, frontend tokens sometimes begin with an 'f' and sometimes with an 'F'. Same for backend and server. So let's change a few cases without disrupting compatibility with existing setups : Bi => bi Bp => bp Ci => ci Cp => cp Fi => fi Fp => fp Si => si Sp => sp cc => CC cs => CS st => ST The old ones are still supported but deprecated and will be unsupported by the 1.5 release. However, a warning message is emitted when they're encounterd and it indicates what token should be used to replace them.
2012-12-20 11:22:52 -05:00
case LOG_FMT_SERVERPORT: // %sp
MINOR: log: use conn_get_{dst,src}() to retrieve the cli/frt/bck/srv/ addresses This also allows us to check that the operation succeeded without logging whatever remained in the memory area in case of failure.
2019-07-17 05:47:11 -04:00
if (be_conn && conn_get_dst(be_conn))
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_port(tmplog, (struct sockaddr *)be_conn->dst, dst + maxsize - tmplog, ctx);
MAJOR: stream-int: stop using si->conn and use si->end instead The connection will only remain there as a pre-allocated entity whose goal is to be placed in ->end when establishing an outgoing connection. All connection initialization can be made on this connection, but all information retrieved should be applied to the end point only. This change is huge because there were many users of si->conn. Now the only users are those who initialize the new connection. The difficulty appears in a few places such as backend.c, proto_http.c, peers.c where si->conn is used to hold the connection's target address before assigning the connection to the stream interface. This is why we have to keep si->conn for now. A future improvement might consist in dynamically allocating the connection when it is needed.
2013-10-01 04:45:07 -04:00
else
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_text_len(tmplog, NULL, 0, dst + maxsize - tmplog, ctx);
MAJOR: stream-int: stop using si->conn and use si->end instead The connection will only remain there as a pre-allocated entity whose goal is to be placed in ->end when establishing an outgoing connection. All connection initialization can be made on this connection, but all information retrieved should be applied to the end point only. This change is huge because there were many users of si->conn. Now the only users are those who initialize the new connection. The difficulty appears in a few places such as backend.c, proto_http.c, peers.c where si->conn is used to hold the connection's target address before assigning the connection to the stream interface. This is why we have to keep si->conn for now. A future improvement might consist in dynamically allocating the connection when it is needed.
2013-10-01 04:45:07 -04:00
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
if (ret == NULL)
MEDIUM: log-format: backend source address %Bi %Bp %Bi return the backend source IP %Bp return the backend source port Add a function pointer in logformat_type to do additional configuration during the log-format variable parsing.
2012-03-02 08:35:21 -05:00
goto out;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
tmplog = ret;
MEDIUM: log-format: backend source address %Bi %Bp %Bi return the backend source IP %Bp return the backend source port Add a function pointer in logformat_type to do additional configuration during the log-format variable parsing.
2012-03-02 08:35:21 -05:00
break;
MEDIUM: log: Decompose %Tq in %Th %Ti %TR Tq is the time between the instant the connection is accepted and a complete valid request is received. This time includes the handshake (SSL / Proxy-Protocol), the idle when the browser does preconnect and the request reception. This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR which returns respectively the handshake time, the idle time and the duration of valid request reception. It also adds %Ta which reports the request's active time, which is the total time without %Th nor %Ti. It replaces %Tt as the total time, reporting accurate measurements for HTTP persistent connections. %Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only avalaible for HTTP connections. In addition to this, we have new timestamps %tr, %trg and %trl, which log the date of start of receipt of the request, respectively in the default format, in GMT time and in local time (by analogy with %t, %T and %Tl). All of them are obviously only available for HTTP. These values are more relevant as they more accurately represent the request date without being skewed by a browser's preconnect nor a keep-alive idle time. The HTTP log format and the CLF log format have been modified to use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This way the default log formats now produce the expected output for users who don't want to manually fiddle with the log-format directive. Example with the following log-format : log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r" The request was sent by hand using "openssl s_client -connect" : Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1" => 6 ms of SSL handshake, 2375 waiting before sending the first char (in fact the time to type the first line), 261 ms before the end of the request, no time spent in queue, 1 ms spend connecting to the server, immediate response, total active time for this request = 262ms. Total time from accept to close : 2643 ms. The timing now decomposes like this : first request 2nd request |<-------------------------------->|<-------------- ... t tr t tr ... ---|----|----|----|----|----|----|----|----|-- : Th Ti TR Tw Tc Tr Td : Ti ... :<---- Tq ---->: : :<-------------- Tt -------------->: :<--------- Ta --------->:
2016-07-28 11:19:45 -04:00
case LOG_FMT_DATE: // %t = accept date
MEDIUM: log: pass date strings to lf_rawtext() Don't directly call functions that take date as argument and output the string representation to the log output buffer under sess_build_logline(), and instead build the strings in temporary buffers of fixed size (hopefully such functions, such as date2str_log() and gmt2str_log() procuce strings of known size), and then print the result using lf_rawtext() helper function. This way, we will be able to encode them automatically as regular string/text when new encoding methods are added. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice (< 30), we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-04-03 05:08:15 -04:00
{
OPTIM: log: use lf_buildctx's buffer instead of temporary stack buffers Now that lf_buildctx isn't pushed on the stack anymore, let's take this opportunity to store a small buffer of 256 bytes within it, and then use this buffer as general purpose buffer to build fixed-length strings that are then printed using lf_{raw}text() function. By doing so we stop relying on temporary stack buffers.
2024-05-02 10:48:56 -04:00
// "26/Apr/2024:09:39:58.774"
MEDIUM: log: pass date strings to lf_rawtext() Don't directly call functions that take date as argument and output the string representation to the log output buffer under sess_build_logline(), and instead build the strings in temporary buffers of fixed size (hopefully such functions, such as date2str_log() and gmt2str_log() procuce strings of known size), and then print the result using lf_rawtext() helper function. This way, we will be able to encode them automatically as regular string/text when new encoding methods are added. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice (< 30), we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-04-03 05:08:15 -04:00
MINOR: log: don't unconditionally pick log info from s->logs We'll soon support s==NULL so let's use an intermediary variable for the logs structure. For now it only points to s->logs but will support a local variable as an alternative later.
2018-09-05 09:16:23 -04:00
get_localtime(logs->accept_date.tv_sec, &tm);
OPTIM: log: speedup date printing in sess_build_logline() when no encoding is used In sess_build_logline(), we have multiple fieds such as '%t' that build a fixed-length string out of a date struct and then print it using lf_rawtext(). In fact, printing it using lf_rawtext() is only mandatory to deal with encoding options, but when no encoding is used we can output the result to tmplog directly. Since most dates generate between 25 and 30 chars, doing so spares us from writing them twice and could help make sess_build_logline() a bit faster when no encoding is used. (to match with pre-encoding patch series performance).
2024-05-02 11:04:28 -04:00
if (ctx->options & LOG_OPT_ENCODE) {
if (!date2str_log(ctx->_buf, &tm, &logs->accept_date, sizeof(ctx->_buf)))
goto out;
ret = lf_rawtext(tmplog, ctx->_buf, dst + maxsize - tmplog, ctx);
}
else // speedup
ret = date2str_log(tmplog, &tm, &logs->accept_date, dst + maxsize - tmplog);
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
if (ret == NULL)
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
goto out;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
tmplog = ret;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
break;
MEDIUM: log: pass date strings to lf_rawtext() Don't directly call functions that take date as argument and output the string representation to the log output buffer under sess_build_logline(), and instead build the strings in temporary buffers of fixed size (hopefully such functions, such as date2str_log() and gmt2str_log() procuce strings of known size), and then print the result using lf_rawtext() helper function. This way, we will be able to encode them automatically as regular string/text when new encoding methods are added. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice (< 30), we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-04-03 05:08:15 -04:00
}
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
MEDIUM: log: Decompose %Tq in %Th %Ti %TR Tq is the time between the instant the connection is accepted and a complete valid request is received. This time includes the handshake (SSL / Proxy-Protocol), the idle when the browser does preconnect and the request reception. This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR which returns respectively the handshake time, the idle time and the duration of valid request reception. It also adds %Ta which reports the request's active time, which is the total time without %Th nor %Ti. It replaces %Tt as the total time, reporting accurate measurements for HTTP persistent connections. %Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only avalaible for HTTP connections. In addition to this, we have new timestamps %tr, %trg and %trl, which log the date of start of receipt of the request, respectively in the default format, in GMT time and in local time (by analogy with %t, %T and %Tl). All of them are obviously only available for HTTP. These values are more relevant as they more accurately represent the request date without being skewed by a browser's preconnect nor a keep-alive idle time. The HTTP log format and the CLF log format have been modified to use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This way the default log formats now produce the expected output for users who don't want to manually fiddle with the log-format directive. Example with the following log-format : log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r" The request was sent by hand using "openssl s_client -connect" : Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1" => 6 ms of SSL handshake, 2375 waiting before sending the first char (in fact the time to type the first line), 261 ms before the end of the request, no time spent in queue, 1 ms spend connecting to the server, immediate response, total active time for this request = 262ms. Total time from accept to close : 2643 ms. The timing now decomposes like this : first request 2nd request |<-------------------------------->|<-------------- ... t tr t tr ... ---|----|----|----|----|----|----|----|----|-- : Th Ti TR Tw Tc Tr Td : Ti ... :<---- Tq ---->: : :<-------------- Tt -------------->: :<--------- Ta --------->:
2016-07-28 11:19:45 -04:00
case LOG_FMT_tr: // %tr = start of request date
MEDIUM: log: pass date strings to lf_rawtext() Don't directly call functions that take date as argument and output the string representation to the log output buffer under sess_build_logline(), and instead build the strings in temporary buffers of fixed size (hopefully such functions, such as date2str_log() and gmt2str_log() procuce strings of known size), and then print the result using lf_rawtext() helper function. This way, we will be able to encode them automatically as regular string/text when new encoding methods are added. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice (< 30), we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-04-03 05:08:15 -04:00
{
OPTIM: log: use lf_buildctx's buffer instead of temporary stack buffers Now that lf_buildctx isn't pushed on the stack anymore, let's take this opportunity to store a small buffer of 256 bytes within it, and then use this buffer as general purpose buffer to build fixed-length strings that are then printed using lf_{raw}text() function. By doing so we stop relying on temporary stack buffers.
2024-05-02 10:48:56 -04:00
// "26/Apr/2024:09:39:58.774"
MEDIUM: log: pass date strings to lf_rawtext() Don't directly call functions that take date as argument and output the string representation to the log output buffer under sess_build_logline(), and instead build the strings in temporary buffers of fixed size (hopefully such functions, such as date2str_log() and gmt2str_log() procuce strings of known size), and then print the result using lf_rawtext() helper function. This way, we will be able to encode them automatically as regular string/text when new encoding methods are added. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice (< 30), we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-04-03 05:08:15 -04:00
MEDIUM: log: Decompose %Tq in %Th %Ti %TR Tq is the time between the instant the connection is accepted and a complete valid request is received. This time includes the handshake (SSL / Proxy-Protocol), the idle when the browser does preconnect and the request reception. This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR which returns respectively the handshake time, the idle time and the duration of valid request reception. It also adds %Ta which reports the request's active time, which is the total time without %Th nor %Ti. It replaces %Tt as the total time, reporting accurate measurements for HTTP persistent connections. %Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only avalaible for HTTP connections. In addition to this, we have new timestamps %tr, %trg and %trl, which log the date of start of receipt of the request, respectively in the default format, in GMT time and in local time (by analogy with %t, %T and %Tl). All of them are obviously only available for HTTP. These values are more relevant as they more accurately represent the request date without being skewed by a browser's preconnect nor a keep-alive idle time. The HTTP log format and the CLF log format have been modified to use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This way the default log formats now produce the expected output for users who don't want to manually fiddle with the log-format directive. Example with the following log-format : log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r" The request was sent by hand using "openssl s_client -connect" : Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1" => 6 ms of SSL handshake, 2375 waiting before sending the first char (in fact the time to type the first line), 261 ms before the end of the request, no time spent in queue, 1 ms spend connecting to the server, immediate response, total active time for this request = 262ms. Total time from accept to close : 2643 ms. The timing now decomposes like this : first request 2nd request |<-------------------------------->|<-------------- ... t tr t tr ... ---|----|----|----|----|----|----|----|----|-- : Th Ti TR Tw Tc Tr Td : Ti ... :<---- Tq ---->: : :<-------------- Tt -------------->: :<--------- Ta --------->:
2016-07-28 11:19:45 -04:00
/* Note that the timers are valid if we get here */
MINOR: log: don't unconditionally pick log info from s->logs We'll soon support s==NULL so let's use an intermediary variable for the logs structure. For now it only points to s->logs but will support a local variable as an alternative later.
2018-09-05 09:16:23 -04:00
tv_ms_add(&tv, &logs->accept_date, logs->t_idle >= 0 ? logs->t_idle + logs->t_handshake : 0);
MEDIUM: log: Decompose %Tq in %Th %Ti %TR Tq is the time between the instant the connection is accepted and a complete valid request is received. This time includes the handshake (SSL / Proxy-Protocol), the idle when the browser does preconnect and the request reception. This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR which returns respectively the handshake time, the idle time and the duration of valid request reception. It also adds %Ta which reports the request's active time, which is the total time without %Th nor %Ti. It replaces %Tt as the total time, reporting accurate measurements for HTTP persistent connections. %Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only avalaible for HTTP connections. In addition to this, we have new timestamps %tr, %trg and %trl, which log the date of start of receipt of the request, respectively in the default format, in GMT time and in local time (by analogy with %t, %T and %Tl). All of them are obviously only available for HTTP. These values are more relevant as they more accurately represent the request date without being skewed by a browser's preconnect nor a keep-alive idle time. The HTTP log format and the CLF log format have been modified to use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This way the default log formats now produce the expected output for users who don't want to manually fiddle with the log-format directive. Example with the following log-format : log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r" The request was sent by hand using "openssl s_client -connect" : Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1" => 6 ms of SSL handshake, 2375 waiting before sending the first char (in fact the time to type the first line), 261 ms before the end of the request, no time spent in queue, 1 ms spend connecting to the server, immediate response, total active time for this request = 262ms. Total time from accept to close : 2643 ms. The timing now decomposes like this : first request 2nd request |<-------------------------------->|<-------------- ... t tr t tr ... ---|----|----|----|----|----|----|----|----|-- : Th Ti TR Tw Tc Tr Td : Ti ... :<---- Tq ---->: : :<-------------- Tt -------------->: :<--------- Ta --------->:
2016-07-28 11:19:45 -04:00
get_localtime(tv.tv_sec, &tm);
OPTIM: log: speedup date printing in sess_build_logline() when no encoding is used In sess_build_logline(), we have multiple fieds such as '%t' that build a fixed-length string out of a date struct and then print it using lf_rawtext(). In fact, printing it using lf_rawtext() is only mandatory to deal with encoding options, but when no encoding is used we can output the result to tmplog directly. Since most dates generate between 25 and 30 chars, doing so spares us from writing them twice and could help make sess_build_logline() a bit faster when no encoding is used. (to match with pre-encoding patch series performance).
2024-05-02 11:04:28 -04:00
if (ctx->options & LOG_OPT_ENCODE) {
if (!date2str_log(ctx->_buf, &tm, &tv, sizeof(ctx->_buf)))
goto out;
ret = lf_rawtext(tmplog, ctx->_buf, dst + maxsize - tmplog, ctx);
}
else // speedup
ret = date2str_log(tmplog, &tm, &tv, dst + maxsize - tmplog);
MEDIUM: log: Decompose %Tq in %Th %Ti %TR Tq is the time between the instant the connection is accepted and a complete valid request is received. This time includes the handshake (SSL / Proxy-Protocol), the idle when the browser does preconnect and the request reception. This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR which returns respectively the handshake time, the idle time and the duration of valid request reception. It also adds %Ta which reports the request's active time, which is the total time without %Th nor %Ti. It replaces %Tt as the total time, reporting accurate measurements for HTTP persistent connections. %Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only avalaible for HTTP connections. In addition to this, we have new timestamps %tr, %trg and %trl, which log the date of start of receipt of the request, respectively in the default format, in GMT time and in local time (by analogy with %t, %T and %Tl). All of them are obviously only available for HTTP. These values are more relevant as they more accurately represent the request date without being skewed by a browser's preconnect nor a keep-alive idle time. The HTTP log format and the CLF log format have been modified to use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This way the default log formats now produce the expected output for users who don't want to manually fiddle with the log-format directive. Example with the following log-format : log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r" The request was sent by hand using "openssl s_client -connect" : Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1" => 6 ms of SSL handshake, 2375 waiting before sending the first char (in fact the time to type the first line), 261 ms before the end of the request, no time spent in queue, 1 ms spend connecting to the server, immediate response, total active time for this request = 262ms. Total time from accept to close : 2643 ms. The timing now decomposes like this : first request 2nd request |<-------------------------------->|<-------------- ... t tr t tr ... ---|----|----|----|----|----|----|----|----|-- : Th Ti TR Tw Tc Tr Td : Ti ... :<---- Tq ---->: : :<-------------- Tt -------------->: :<--------- Ta --------->:
2016-07-28 11:19:45 -04:00
if (ret == NULL)
goto out;
tmplog = ret;
break;
MEDIUM: log: pass date strings to lf_rawtext() Don't directly call functions that take date as argument and output the string representation to the log output buffer under sess_build_logline(), and instead build the strings in temporary buffers of fixed size (hopefully such functions, such as date2str_log() and gmt2str_log() procuce strings of known size), and then print the result using lf_rawtext() helper function. This way, we will be able to encode them automatically as regular string/text when new encoding methods are added. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice (< 30), we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-04-03 05:08:15 -04:00
}
MEDIUM: log: Decompose %Tq in %Th %Ti %TR Tq is the time between the instant the connection is accepted and a complete valid request is received. This time includes the handshake (SSL / Proxy-Protocol), the idle when the browser does preconnect and the request reception. This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR which returns respectively the handshake time, the idle time and the duration of valid request reception. It also adds %Ta which reports the request's active time, which is the total time without %Th nor %Ti. It replaces %Tt as the total time, reporting accurate measurements for HTTP persistent connections. %Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only avalaible for HTTP connections. In addition to this, we have new timestamps %tr, %trg and %trl, which log the date of start of receipt of the request, respectively in the default format, in GMT time and in local time (by analogy with %t, %T and %Tl). All of them are obviously only available for HTTP. These values are more relevant as they more accurately represent the request date without being skewed by a browser's preconnect nor a keep-alive idle time. The HTTP log format and the CLF log format have been modified to use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This way the default log formats now produce the expected output for users who don't want to manually fiddle with the log-format directive. Example with the following log-format : log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r" The request was sent by hand using "openssl s_client -connect" : Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1" => 6 ms of SSL handshake, 2375 waiting before sending the first char (in fact the time to type the first line), 261 ms before the end of the request, no time spent in queue, 1 ms spend connecting to the server, immediate response, total active time for this request = 262ms. Total time from accept to close : 2643 ms. The timing now decomposes like this : first request 2nd request |<-------------------------------->|<-------------- ... t tr t tr ... ---|----|----|----|----|----|----|----|----|-- : Th Ti TR Tw Tc Tr Td : Ti ... :<---- Tq ---->: : :<-------------- Tt -------------->: :<--------- Ta --------->:
2016-07-28 11:19:45 -04:00
case LOG_FMT_DATEGMT: // %T = accept date, GMT
MEDIUM: log: pass date strings to lf_rawtext() Don't directly call functions that take date as argument and output the string representation to the log output buffer under sess_build_logline(), and instead build the strings in temporary buffers of fixed size (hopefully such functions, such as date2str_log() and gmt2str_log() procuce strings of known size), and then print the result using lf_rawtext() helper function. This way, we will be able to encode them automatically as regular string/text when new encoding methods are added. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice (< 30), we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-04-03 05:08:15 -04:00
{
OPTIM: log: use lf_buildctx's buffer instead of temporary stack buffers Now that lf_buildctx isn't pushed on the stack anymore, let's take this opportunity to store a small buffer of 256 bytes within it, and then use this buffer as general purpose buffer to build fixed-length strings that are then printed using lf_{raw}text() function. By doing so we stop relying on temporary stack buffers.
2024-05-02 10:48:56 -04:00
// "26/Apr/2024:07:41:11 +0000"
MEDIUM: log: pass date strings to lf_rawtext() Don't directly call functions that take date as argument and output the string representation to the log output buffer under sess_build_logline(), and instead build the strings in temporary buffers of fixed size (hopefully such functions, such as date2str_log() and gmt2str_log() procuce strings of known size), and then print the result using lf_rawtext() helper function. This way, we will be able to encode them automatically as regular string/text when new encoding methods are added. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice (< 30), we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-04-03 05:08:15 -04:00
MINOR: log: don't unconditionally pick log info from s->logs We'll soon support s==NULL so let's use an intermediary variable for the logs structure. For now it only points to s->logs but will support a local variable as an alternative later.
2018-09-05 09:16:23 -04:00
get_gmtime(logs->accept_date.tv_sec, &tm);
OPTIM: log: speedup date printing in sess_build_logline() when no encoding is used In sess_build_logline(), we have multiple fieds such as '%t' that build a fixed-length string out of a date struct and then print it using lf_rawtext(). In fact, printing it using lf_rawtext() is only mandatory to deal with encoding options, but when no encoding is used we can output the result to tmplog directly. Since most dates generate between 25 and 30 chars, doing so spares us from writing them twice and could help make sess_build_logline() a bit faster when no encoding is used. (to match with pre-encoding patch series performance).
2024-05-02 11:04:28 -04:00
if (ctx->options & LOG_OPT_ENCODE) {
if (!gmt2str_log(ctx->_buf, &tm, sizeof(ctx->_buf)))
goto out;
ret = lf_rawtext(tmplog, ctx->_buf, dst + maxsize - tmplog, ctx);
}
else // speedup
ret = gmt2str_log(tmplog, &tm, dst + maxsize - tmplog);
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
if (ret == NULL)
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
goto out;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
tmplog = ret;
MINOR: log: add '%Tl' to log-format The '%Tl' is similar to '%T', but using local timezone.
2012-10-18 22:36:09 -04:00
break;
MEDIUM: log: pass date strings to lf_rawtext() Don't directly call functions that take date as argument and output the string representation to the log output buffer under sess_build_logline(), and instead build the strings in temporary buffers of fixed size (hopefully such functions, such as date2str_log() and gmt2str_log() procuce strings of known size), and then print the result using lf_rawtext() helper function. This way, we will be able to encode them automatically as regular string/text when new encoding methods are added. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice (< 30), we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-04-03 05:08:15 -04:00
}
MINOR: log: add '%Tl' to log-format The '%Tl' is similar to '%T', but using local timezone.
2012-10-18 22:36:09 -04:00
MEDIUM: log: Decompose %Tq in %Th %Ti %TR Tq is the time between the instant the connection is accepted and a complete valid request is received. This time includes the handshake (SSL / Proxy-Protocol), the idle when the browser does preconnect and the request reception. This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR which returns respectively the handshake time, the idle time and the duration of valid request reception. It also adds %Ta which reports the request's active time, which is the total time without %Th nor %Ti. It replaces %Tt as the total time, reporting accurate measurements for HTTP persistent connections. %Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only avalaible for HTTP connections. In addition to this, we have new timestamps %tr, %trg and %trl, which log the date of start of receipt of the request, respectively in the default format, in GMT time and in local time (by analogy with %t, %T and %Tl). All of them are obviously only available for HTTP. These values are more relevant as they more accurately represent the request date without being skewed by a browser's preconnect nor a keep-alive idle time. The HTTP log format and the CLF log format have been modified to use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This way the default log formats now produce the expected output for users who don't want to manually fiddle with the log-format directive. Example with the following log-format : log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r" The request was sent by hand using "openssl s_client -connect" : Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1" => 6 ms of SSL handshake, 2375 waiting before sending the first char (in fact the time to type the first line), 261 ms before the end of the request, no time spent in queue, 1 ms spend connecting to the server, immediate response, total active time for this request = 262ms. Total time from accept to close : 2643 ms. The timing now decomposes like this : first request 2nd request |<-------------------------------->|<-------------- ... t tr t tr ... ---|----|----|----|----|----|----|----|----|-- : Th Ti TR Tw Tc Tr Td : Ti ... :<---- Tq ---->: : :<-------------- Tt -------------->: :<--------- Ta --------->:
2016-07-28 11:19:45 -04:00
case LOG_FMT_trg: // %trg = start of request date, GMT
MEDIUM: log: pass date strings to lf_rawtext() Don't directly call functions that take date as argument and output the string representation to the log output buffer under sess_build_logline(), and instead build the strings in temporary buffers of fixed size (hopefully such functions, such as date2str_log() and gmt2str_log() procuce strings of known size), and then print the result using lf_rawtext() helper function. This way, we will be able to encode them automatically as regular string/text when new encoding methods are added. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice (< 30), we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-04-03 05:08:15 -04:00
{
OPTIM: log: use lf_buildctx's buffer instead of temporary stack buffers Now that lf_buildctx isn't pushed on the stack anymore, let's take this opportunity to store a small buffer of 256 bytes within it, and then use this buffer as general purpose buffer to build fixed-length strings that are then printed using lf_{raw}text() function. By doing so we stop relying on temporary stack buffers.
2024-05-02 10:48:56 -04:00
// "26/Apr/2024:07:41:11 +0000"
MEDIUM: log: pass date strings to lf_rawtext() Don't directly call functions that take date as argument and output the string representation to the log output buffer under sess_build_logline(), and instead build the strings in temporary buffers of fixed size (hopefully such functions, such as date2str_log() and gmt2str_log() procuce strings of known size), and then print the result using lf_rawtext() helper function. This way, we will be able to encode them automatically as regular string/text when new encoding methods are added. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice (< 30), we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-04-03 05:08:15 -04:00
MINOR: log: don't unconditionally pick log info from s->logs We'll soon support s==NULL so let's use an intermediary variable for the logs structure. For now it only points to s->logs but will support a local variable as an alternative later.
2018-09-05 09:16:23 -04:00
tv_ms_add(&tv, &logs->accept_date, logs->t_idle >= 0 ? logs->t_idle + logs->t_handshake : 0);
MEDIUM: log: Decompose %Tq in %Th %Ti %TR Tq is the time between the instant the connection is accepted and a complete valid request is received. This time includes the handshake (SSL / Proxy-Protocol), the idle when the browser does preconnect and the request reception. This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR which returns respectively the handshake time, the idle time and the duration of valid request reception. It also adds %Ta which reports the request's active time, which is the total time without %Th nor %Ti. It replaces %Tt as the total time, reporting accurate measurements for HTTP persistent connections. %Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only avalaible for HTTP connections. In addition to this, we have new timestamps %tr, %trg and %trl, which log the date of start of receipt of the request, respectively in the default format, in GMT time and in local time (by analogy with %t, %T and %Tl). All of them are obviously only available for HTTP. These values are more relevant as they more accurately represent the request date without being skewed by a browser's preconnect nor a keep-alive idle time. The HTTP log format and the CLF log format have been modified to use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This way the default log formats now produce the expected output for users who don't want to manually fiddle with the log-format directive. Example with the following log-format : log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r" The request was sent by hand using "openssl s_client -connect" : Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1" => 6 ms of SSL handshake, 2375 waiting before sending the first char (in fact the time to type the first line), 261 ms before the end of the request, no time spent in queue, 1 ms spend connecting to the server, immediate response, total active time for this request = 262ms. Total time from accept to close : 2643 ms. The timing now decomposes like this : first request 2nd request |<-------------------------------->|<-------------- ... t tr t tr ... ---|----|----|----|----|----|----|----|----|-- : Th Ti TR Tw Tc Tr Td : Ti ... :<---- Tq ---->: : :<-------------- Tt -------------->: :<--------- Ta --------->:
2016-07-28 11:19:45 -04:00
get_gmtime(tv.tv_sec, &tm);
OPTIM: log: speedup date printing in sess_build_logline() when no encoding is used In sess_build_logline(), we have multiple fieds such as '%t' that build a fixed-length string out of a date struct and then print it using lf_rawtext(). In fact, printing it using lf_rawtext() is only mandatory to deal with encoding options, but when no encoding is used we can output the result to tmplog directly. Since most dates generate between 25 and 30 chars, doing so spares us from writing them twice and could help make sess_build_logline() a bit faster when no encoding is used. (to match with pre-encoding patch series performance).
2024-05-02 11:04:28 -04:00
if (ctx->options & LOG_OPT_ENCODE) {
if (!gmt2str_log(ctx->_buf, &tm, sizeof(ctx->_buf)))
goto out;
ret = lf_rawtext(tmplog, ctx->_buf, dst + maxsize - tmplog, ctx);
}
else // speedup
ret = gmt2str_log(tmplog, &tm, dst + maxsize - tmplog);
MEDIUM: log: Decompose %Tq in %Th %Ti %TR Tq is the time between the instant the connection is accepted and a complete valid request is received. This time includes the handshake (SSL / Proxy-Protocol), the idle when the browser does preconnect and the request reception. This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR which returns respectively the handshake time, the idle time and the duration of valid request reception. It also adds %Ta which reports the request's active time, which is the total time without %Th nor %Ti. It replaces %Tt as the total time, reporting accurate measurements for HTTP persistent connections. %Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only avalaible for HTTP connections. In addition to this, we have new timestamps %tr, %trg and %trl, which log the date of start of receipt of the request, respectively in the default format, in GMT time and in local time (by analogy with %t, %T and %Tl). All of them are obviously only available for HTTP. These values are more relevant as they more accurately represent the request date without being skewed by a browser's preconnect nor a keep-alive idle time. The HTTP log format and the CLF log format have been modified to use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This way the default log formats now produce the expected output for users who don't want to manually fiddle with the log-format directive. Example with the following log-format : log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r" The request was sent by hand using "openssl s_client -connect" : Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1" => 6 ms of SSL handshake, 2375 waiting before sending the first char (in fact the time to type the first line), 261 ms before the end of the request, no time spent in queue, 1 ms spend connecting to the server, immediate response, total active time for this request = 262ms. Total time from accept to close : 2643 ms. The timing now decomposes like this : first request 2nd request |<-------------------------------->|<-------------- ... t tr t tr ... ---|----|----|----|----|----|----|----|----|-- : Th Ti TR Tw Tc Tr Td : Ti ... :<---- Tq ---->: : :<-------------- Tt -------------->: :<--------- Ta --------->:
2016-07-28 11:19:45 -04:00
if (ret == NULL)
goto out;
tmplog = ret;
break;
MEDIUM: log: pass date strings to lf_rawtext() Don't directly call functions that take date as argument and output the string representation to the log output buffer under sess_build_logline(), and instead build the strings in temporary buffers of fixed size (hopefully such functions, such as date2str_log() and gmt2str_log() procuce strings of known size), and then print the result using lf_rawtext() helper function. This way, we will be able to encode them automatically as regular string/text when new encoding methods are added. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice (< 30), we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-04-03 05:08:15 -04:00
}
MEDIUM: log: Decompose %Tq in %Th %Ti %TR Tq is the time between the instant the connection is accepted and a complete valid request is received. This time includes the handshake (SSL / Proxy-Protocol), the idle when the browser does preconnect and the request reception. This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR which returns respectively the handshake time, the idle time and the duration of valid request reception. It also adds %Ta which reports the request's active time, which is the total time without %Th nor %Ti. It replaces %Tt as the total time, reporting accurate measurements for HTTP persistent connections. %Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only avalaible for HTTP connections. In addition to this, we have new timestamps %tr, %trg and %trl, which log the date of start of receipt of the request, respectively in the default format, in GMT time and in local time (by analogy with %t, %T and %Tl). All of them are obviously only available for HTTP. These values are more relevant as they more accurately represent the request date without being skewed by a browser's preconnect nor a keep-alive idle time. The HTTP log format and the CLF log format have been modified to use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This way the default log formats now produce the expected output for users who don't want to manually fiddle with the log-format directive. Example with the following log-format : log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r" The request was sent by hand using "openssl s_client -connect" : Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1" => 6 ms of SSL handshake, 2375 waiting before sending the first char (in fact the time to type the first line), 261 ms before the end of the request, no time spent in queue, 1 ms spend connecting to the server, immediate response, total active time for this request = 262ms. Total time from accept to close : 2643 ms. The timing now decomposes like this : first request 2nd request |<-------------------------------->|<-------------- ... t tr t tr ... ---|----|----|----|----|----|----|----|----|-- : Th Ti TR Tw Tc Tr Td : Ti ... :<---- Tq ---->: : :<-------------- Tt -------------->: :<--------- Ta --------->:
2016-07-28 11:19:45 -04:00
case LOG_FMT_DATELOCAL: // %Tl = accept date, local
MEDIUM: log: pass date strings to lf_rawtext() Don't directly call functions that take date as argument and output the string representation to the log output buffer under sess_build_logline(), and instead build the strings in temporary buffers of fixed size (hopefully such functions, such as date2str_log() and gmt2str_log() procuce strings of known size), and then print the result using lf_rawtext() helper function. This way, we will be able to encode them automatically as regular string/text when new encoding methods are added. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice (< 30), we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-04-03 05:08:15 -04:00
{
OPTIM: log: use lf_buildctx's buffer instead of temporary stack buffers Now that lf_buildctx isn't pushed on the stack anymore, let's take this opportunity to store a small buffer of 256 bytes within it, and then use this buffer as general purpose buffer to build fixed-length strings that are then printed using lf_{raw}text() function. By doing so we stop relying on temporary stack buffers.
2024-05-02 10:48:56 -04:00
// "26/Apr/2024:09:42:32 +0200"
MEDIUM: log: pass date strings to lf_rawtext() Don't directly call functions that take date as argument and output the string representation to the log output buffer under sess_build_logline(), and instead build the strings in temporary buffers of fixed size (hopefully such functions, such as date2str_log() and gmt2str_log() procuce strings of known size), and then print the result using lf_rawtext() helper function. This way, we will be able to encode them automatically as regular string/text when new encoding methods are added. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice (< 30), we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-04-03 05:08:15 -04:00
MINOR: log: don't unconditionally pick log info from s->logs We'll soon support s==NULL so let's use an intermediary variable for the logs structure. For now it only points to s->logs but will support a local variable as an alternative later.
2018-09-05 09:16:23 -04:00
get_localtime(logs->accept_date.tv_sec, &tm);
OPTIM: log: speedup date printing in sess_build_logline() when no encoding is used In sess_build_logline(), we have multiple fieds such as '%t' that build a fixed-length string out of a date struct and then print it using lf_rawtext(). In fact, printing it using lf_rawtext() is only mandatory to deal with encoding options, but when no encoding is used we can output the result to tmplog directly. Since most dates generate between 25 and 30 chars, doing so spares us from writing them twice and could help make sess_build_logline() a bit faster when no encoding is used. (to match with pre-encoding patch series performance).
2024-05-02 11:04:28 -04:00
if (ctx->options & LOG_OPT_ENCODE) {
if (!localdate2str_log(ctx->_buf, logs->accept_date.tv_sec,
&tm, sizeof(ctx->_buf)))
goto out;
ret = lf_rawtext(tmplog, ctx->_buf, dst + maxsize - tmplog, ctx);
}
else // speedup
ret = localdate2str_log(tmplog, logs->accept_date.tv_sec,
&tm, dst + maxsize - tmplog);
MINOR: log: add '%Tl' to log-format The '%Tl' is similar to '%T', but using local timezone.
2012-10-18 22:36:09 -04:00
if (ret == NULL)
goto out;
tmplog = ret;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
break;
MEDIUM: log: pass date strings to lf_rawtext() Don't directly call functions that take date as argument and output the string representation to the log output buffer under sess_build_logline(), and instead build the strings in temporary buffers of fixed size (hopefully such functions, such as date2str_log() and gmt2str_log() procuce strings of known size), and then print the result using lf_rawtext() helper function. This way, we will be able to encode them automatically as regular string/text when new encoding methods are added. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice (< 30), we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-04-03 05:08:15 -04:00
}
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
MEDIUM: log: Decompose %Tq in %Th %Ti %TR Tq is the time between the instant the connection is accepted and a complete valid request is received. This time includes the handshake (SSL / Proxy-Protocol), the idle when the browser does preconnect and the request reception. This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR which returns respectively the handshake time, the idle time and the duration of valid request reception. It also adds %Ta which reports the request's active time, which is the total time without %Th nor %Ti. It replaces %Tt as the total time, reporting accurate measurements for HTTP persistent connections. %Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only avalaible for HTTP connections. In addition to this, we have new timestamps %tr, %trg and %trl, which log the date of start of receipt of the request, respectively in the default format, in GMT time and in local time (by analogy with %t, %T and %Tl). All of them are obviously only available for HTTP. These values are more relevant as they more accurately represent the request date without being skewed by a browser's preconnect nor a keep-alive idle time. The HTTP log format and the CLF log format have been modified to use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This way the default log formats now produce the expected output for users who don't want to manually fiddle with the log-format directive. Example with the following log-format : log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r" The request was sent by hand using "openssl s_client -connect" : Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1" => 6 ms of SSL handshake, 2375 waiting before sending the first char (in fact the time to type the first line), 261 ms before the end of the request, no time spent in queue, 1 ms spend connecting to the server, immediate response, total active time for this request = 262ms. Total time from accept to close : 2643 ms. The timing now decomposes like this : first request 2nd request |<-------------------------------->|<-------------- ... t tr t tr ... ---|----|----|----|----|----|----|----|----|-- : Th Ti TR Tw Tc Tr Td : Ti ... :<---- Tq ---->: : :<-------------- Tt -------------->: :<--------- Ta --------->:
2016-07-28 11:19:45 -04:00
case LOG_FMT_trl: // %trl = start of request date, local
MEDIUM: log: pass date strings to lf_rawtext() Don't directly call functions that take date as argument and output the string representation to the log output buffer under sess_build_logline(), and instead build the strings in temporary buffers of fixed size (hopefully such functions, such as date2str_log() and gmt2str_log() procuce strings of known size), and then print the result using lf_rawtext() helper function. This way, we will be able to encode them automatically as regular string/text when new encoding methods are added. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice (< 30), we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-04-03 05:08:15 -04:00
{
OPTIM: log: use lf_buildctx's buffer instead of temporary stack buffers Now that lf_buildctx isn't pushed on the stack anymore, let's take this opportunity to store a small buffer of 256 bytes within it, and then use this buffer as general purpose buffer to build fixed-length strings that are then printed using lf_{raw}text() function. By doing so we stop relying on temporary stack buffers.
2024-05-02 10:48:56 -04:00
// "26/Apr/2024:09:42:32 +0200"
MEDIUM: log: pass date strings to lf_rawtext() Don't directly call functions that take date as argument and output the string representation to the log output buffer under sess_build_logline(), and instead build the strings in temporary buffers of fixed size (hopefully such functions, such as date2str_log() and gmt2str_log() procuce strings of known size), and then print the result using lf_rawtext() helper function. This way, we will be able to encode them automatically as regular string/text when new encoding methods are added. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice (< 30), we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-04-03 05:08:15 -04:00
MINOR: log: don't unconditionally pick log info from s->logs We'll soon support s==NULL so let's use an intermediary variable for the logs structure. For now it only points to s->logs but will support a local variable as an alternative later.
2018-09-05 09:16:23 -04:00
tv_ms_add(&tv, &logs->accept_date, logs->t_idle >= 0 ? logs->t_idle + logs->t_handshake : 0);
MEDIUM: log: Decompose %Tq in %Th %Ti %TR Tq is the time between the instant the connection is accepted and a complete valid request is received. This time includes the handshake (SSL / Proxy-Protocol), the idle when the browser does preconnect and the request reception. This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR which returns respectively the handshake time, the idle time and the duration of valid request reception. It also adds %Ta which reports the request's active time, which is the total time without %Th nor %Ti. It replaces %Tt as the total time, reporting accurate measurements for HTTP persistent connections. %Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only avalaible for HTTP connections. In addition to this, we have new timestamps %tr, %trg and %trl, which log the date of start of receipt of the request, respectively in the default format, in GMT time and in local time (by analogy with %t, %T and %Tl). All of them are obviously only available for HTTP. These values are more relevant as they more accurately represent the request date without being skewed by a browser's preconnect nor a keep-alive idle time. The HTTP log format and the CLF log format have been modified to use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This way the default log formats now produce the expected output for users who don't want to manually fiddle with the log-format directive. Example with the following log-format : log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r" The request was sent by hand using "openssl s_client -connect" : Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1" => 6 ms of SSL handshake, 2375 waiting before sending the first char (in fact the time to type the first line), 261 ms before the end of the request, no time spent in queue, 1 ms spend connecting to the server, immediate response, total active time for this request = 262ms. Total time from accept to close : 2643 ms. The timing now decomposes like this : first request 2nd request |<-------------------------------->|<-------------- ... t tr t tr ... ---|----|----|----|----|----|----|----|----|-- : Th Ti TR Tw Tc Tr Td : Ti ... :<---- Tq ---->: : :<-------------- Tt -------------->: :<--------- Ta --------->:
2016-07-28 11:19:45 -04:00
get_localtime(tv.tv_sec, &tm);
OPTIM: log: speedup date printing in sess_build_logline() when no encoding is used In sess_build_logline(), we have multiple fieds such as '%t' that build a fixed-length string out of a date struct and then print it using lf_rawtext(). In fact, printing it using lf_rawtext() is only mandatory to deal with encoding options, but when no encoding is used we can output the result to tmplog directly. Since most dates generate between 25 and 30 chars, doing so spares us from writing them twice and could help make sess_build_logline() a bit faster when no encoding is used. (to match with pre-encoding patch series performance).
2024-05-02 11:04:28 -04:00
if (ctx->options & LOG_OPT_ENCODE) {
if (!localdate2str_log(ctx->_buf, tv.tv_sec, &tm, sizeof(ctx->_buf)))
goto out;
ret = lf_rawtext(tmplog, ctx->_buf, dst + maxsize - tmplog, ctx);
}
else // speedup
ret = localdate2str_log(tmplog, tv.tv_sec, &tm, dst + maxsize - tmplog);
MEDIUM: log: Decompose %Tq in %Th %Ti %TR Tq is the time between the instant the connection is accepted and a complete valid request is received. This time includes the handshake (SSL / Proxy-Protocol), the idle when the browser does preconnect and the request reception. This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR which returns respectively the handshake time, the idle time and the duration of valid request reception. It also adds %Ta which reports the request's active time, which is the total time without %Th nor %Ti. It replaces %Tt as the total time, reporting accurate measurements for HTTP persistent connections. %Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only avalaible for HTTP connections. In addition to this, we have new timestamps %tr, %trg and %trl, which log the date of start of receipt of the request, respectively in the default format, in GMT time and in local time (by analogy with %t, %T and %Tl). All of them are obviously only available for HTTP. These values are more relevant as they more accurately represent the request date without being skewed by a browser's preconnect nor a keep-alive idle time. The HTTP log format and the CLF log format have been modified to use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This way the default log formats now produce the expected output for users who don't want to manually fiddle with the log-format directive. Example with the following log-format : log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r" The request was sent by hand using "openssl s_client -connect" : Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1" => 6 ms of SSL handshake, 2375 waiting before sending the first char (in fact the time to type the first line), 261 ms before the end of the request, no time spent in queue, 1 ms spend connecting to the server, immediate response, total active time for this request = 262ms. Total time from accept to close : 2643 ms. The timing now decomposes like this : first request 2nd request |<-------------------------------->|<-------------- ... t tr t tr ... ---|----|----|----|----|----|----|----|----|-- : Th Ti TR Tw Tc Tr Td : Ti ... :<---- Tq ---->: : :<-------------- Tt -------------->: :<--------- Ta --------->:
2016-07-28 11:19:45 -04:00
if (ret == NULL)
goto out;
tmplog = ret;
break;
MEDIUM: log: pass date strings to lf_rawtext() Don't directly call functions that take date as argument and output the string representation to the log output buffer under sess_build_logline(), and instead build the strings in temporary buffers of fixed size (hopefully such functions, such as date2str_log() and gmt2str_log() procuce strings of known size), and then print the result using lf_rawtext() helper function. This way, we will be able to encode them automatically as regular string/text when new encoding methods are added. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice (< 30), we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-04-03 05:08:15 -04:00
}
MEDIUM: log: Decompose %Tq in %Th %Ti %TR Tq is the time between the instant the connection is accepted and a complete valid request is received. This time includes the handshake (SSL / Proxy-Protocol), the idle when the browser does preconnect and the request reception. This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR which returns respectively the handshake time, the idle time and the duration of valid request reception. It also adds %Ta which reports the request's active time, which is the total time without %Th nor %Ti. It replaces %Tt as the total time, reporting accurate measurements for HTTP persistent connections. %Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only avalaible for HTTP connections. In addition to this, we have new timestamps %tr, %trg and %trl, which log the date of start of receipt of the request, respectively in the default format, in GMT time and in local time (by analogy with %t, %T and %Tl). All of them are obviously only available for HTTP. These values are more relevant as they more accurately represent the request date without being skewed by a browser's preconnect nor a keep-alive idle time. The HTTP log format and the CLF log format have been modified to use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This way the default log formats now produce the expected output for users who don't want to manually fiddle with the log-format directive. Example with the following log-format : log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r" The request was sent by hand using "openssl s_client -connect" : Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1" => 6 ms of SSL handshake, 2375 waiting before sending the first char (in fact the time to type the first line), 261 ms before the end of the request, no time spent in queue, 1 ms spend connecting to the server, immediate response, total active time for this request = 262ms. Total time from accept to close : 2643 ms. The timing now decomposes like this : first request 2nd request |<-------------------------------->|<-------------- ... t tr t tr ... ---|----|----|----|----|----|----|----|----|-- : Th Ti TR Tw Tc Tr Td : Ti ... :<---- Tq ---->: : :<-------------- Tt -------------->: :<--------- Ta --------->:
2016-07-28 11:19:45 -04:00
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
case LOG_FMT_TS: // %Ts
CLEANUP: log: simplify complex values usages in sess_build_logline() make sess_build_logline() switch case more readable by performing some simplifications: complex values are first extracted in a temporary variable so that it's easier to refer to them and at a single place.
2024-03-26 06:50:50 -04:00
{
unsigned long value = logs->accept_date.tv_sec;
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
if (ctx->options & LOG_OPT_HEXA) {
OPTIM: log: use lf_buildctx's buffer instead of temporary stack buffers Now that lf_buildctx isn't pushed on the stack anymore, let's take this opportunity to store a small buffer of 256 bytes within it, and then use this buffer as general purpose buffer to build fixed-length strings that are then printed using lf_{raw}text() function. By doing so we stop relying on temporary stack buffers.
2024-05-02 10:48:56 -04:00
iret = snprintf(ctx->_buf, sizeof(ctx->_buf), "%04X", (unsigned int)value);
BUG/MINOR: log: invalid snprintf() usage in sess_build_logline() According to snprintf() man page: The functions snprintf() and vsnprintf() do not write more than size bytes (including the terminating null byte ('\0')). If the output was truncated due to this limit, then the return value is the number of characters (excluding the terminating null byte) which would have been written to the final string if enough space had been available. Thus, a return value of size or more means that the output was truncated. However, in sess_build_logline(), each time we need to check the return value of snprintf(), here is how we proceed: iret = snprintf(tmplog, max, ...); if (iret < 0 || iret > max) // error // success tmplog += iret; Here is the issue: if snprintf() lacks 1 byte space to write the terminating NULL byte, it will return max. Which means in this case that we fail to know that snprintf() truncated the output in reality, and we still add iret to tmplog pointer. Considering sess_build_logline() should NOT write more than <maxsize> bytes (including the terminating NULL byte) as per the function description, in this case the function would write <maxsize>+1 byte (to write the terminating NULL byte upon return), which may lead to invalid write if <dst> was meant to hold <maxsize> bytes at maximum. Hopefully, this bug wasn't triggered so far because sess_build_logline() is called with logline as <dst> argument and <global.max_syslog_len> as <maxsize> argument, logline being initialized with 1 extra byte upon startup. But we better fix this to comply with the function description and prevent any side-effect since some sess_build_logline() helpers may assume that 'tmplog-dst < maxsize' is always true. Also sess_build_logline() users probably don't expect NULL-byte to be accounted for in the produced logline length. This should be backported to all stable versions. [ada: for backports, the patch needs to be applied manually because of c6a713842 ("MINOR: log: simplify last_isspace in sess_build_logline()")]
2024-04-09 09:59:42 -04:00
if (iret < 0 || iret >= dst + maxsize - tmplog)
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
goto out;
OPTIM: log: use lf_buildctx's buffer instead of temporary stack buffers Now that lf_buildctx isn't pushed on the stack anymore, let's take this opportunity to store a small buffer of 256 bytes within it, and then use this buffer as general purpose buffer to build fixed-length strings that are then printed using lf_{raw}text() function. By doing so we stop relying on temporary stack buffers.
2024-05-02 10:48:56 -04:00
ret = lf_rawtext(tmplog, ctx->_buf, dst + maxsize - tmplog, ctx);
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
} else {
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_int(tmplog, dst + maxsize - tmplog, value, ctx, LF_INT_LTOA);
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
}
MEDIUM: log: write raw strings using lf_rawtext() Make use of the previous commit to print strings that should not be modified. For instance, when +X option is provided, we have to print numerical values in ASCII HEX form. For that, we used snprintf() to output the result to the log output buffer directly, but now we build the string in a temporary buffer of fixed-size and then print it using lf_rawtext() which will take care of encoding options. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice, we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-03-27 05:28:59 -04:00
if (ret == NULL)
goto out;
tmplog = ret;
CLEANUP: log: simplify complex values usages in sess_build_logline() make sess_build_logline() switch case more readable by performing some simplifications: complex values are first extracted in a temporary variable so that it's easier to refer to them and at a single place.
2024-03-26 06:50:50 -04:00
break;
}
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
case LOG_FMT_MS: // %ms
CLEANUP: log: simplify complex values usages in sess_build_logline() make sess_build_logline() switch case more readable by performing some simplifications: complex values are first extracted in a temporary variable so that it's easier to refer to them and at a single place.
2024-03-26 06:50:50 -04:00
{
unsigned int value = (unsigned int)logs->accept_date.tv_usec/1000;
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
if (ctx->options & LOG_OPT_HEXA) {
OPTIM: log: use lf_buildctx's buffer instead of temporary stack buffers Now that lf_buildctx isn't pushed on the stack anymore, let's take this opportunity to store a small buffer of 256 bytes within it, and then use this buffer as general purpose buffer to build fixed-length strings that are then printed using lf_{raw}text() function. By doing so we stop relying on temporary stack buffers.
2024-05-02 10:48:56 -04:00
iret = snprintf(ctx->_buf, sizeof(ctx->_buf), "%02X", value);
BUG/MINOR: log: invalid snprintf() usage in sess_build_logline() According to snprintf() man page: The functions snprintf() and vsnprintf() do not write more than size bytes (including the terminating null byte ('\0')). If the output was truncated due to this limit, then the return value is the number of characters (excluding the terminating null byte) which would have been written to the final string if enough space had been available. Thus, a return value of size or more means that the output was truncated. However, in sess_build_logline(), each time we need to check the return value of snprintf(), here is how we proceed: iret = snprintf(tmplog, max, ...); if (iret < 0 || iret > max) // error // success tmplog += iret; Here is the issue: if snprintf() lacks 1 byte space to write the terminating NULL byte, it will return max. Which means in this case that we fail to know that snprintf() truncated the output in reality, and we still add iret to tmplog pointer. Considering sess_build_logline() should NOT write more than <maxsize> bytes (including the terminating NULL byte) as per the function description, in this case the function would write <maxsize>+1 byte (to write the terminating NULL byte upon return), which may lead to invalid write if <dst> was meant to hold <maxsize> bytes at maximum. Hopefully, this bug wasn't triggered so far because sess_build_logline() is called with logline as <dst> argument and <global.max_syslog_len> as <maxsize> argument, logline being initialized with 1 extra byte upon startup. But we better fix this to comply with the function description and prevent any side-effect since some sess_build_logline() helpers may assume that 'tmplog-dst < maxsize' is always true. Also sess_build_logline() users probably don't expect NULL-byte to be accounted for in the produced logline length. This should be backported to all stable versions. [ada: for backports, the patch needs to be applied manually because of c6a713842 ("MINOR: log: simplify last_isspace in sess_build_logline()")]
2024-04-09 09:59:42 -04:00
if (iret < 0 || iret >= dst + maxsize - tmplog)
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
goto out;
OPTIM: log: use lf_buildctx's buffer instead of temporary stack buffers Now that lf_buildctx isn't pushed on the stack anymore, let's take this opportunity to store a small buffer of 256 bytes within it, and then use this buffer as general purpose buffer to build fixed-length strings that are then printed using lf_{raw}text() function. By doing so we stop relying on temporary stack buffers.
2024-05-02 10:48:56 -04:00
ret = lf_rawtext(tmplog, ctx->_buf, dst + maxsize - tmplog, ctx);
CLEANUP: log: simplify complex values usages in sess_build_logline() make sess_build_logline() switch case more readable by performing some simplifications: complex values are first extracted in a temporary variable so that it's easier to refer to them and at a single place.
2024-03-26 06:50:50 -04:00
} else {
MINOR: log: add lf_int() wrapper to print integers Wrap ltoa(), lltoa(), ultoa() and utoa_pad() functions that are used by sess_build_logline() to print numerical values by implementing a dedicated helper named lf_int() that takes <dft_hld> as argument to know how to write the integer by default (when no encoding is specified). LF_INT_UTOA_PAD_4 is used to emulate utoa_pad(x, 4) since it's found only once under sess_build_logline(), thus there is no need to pass an extra parameter to lf_int() function.
2024-03-27 05:17:11 -04:00
ret = lf_int(tmplog, dst + maxsize - tmplog, value,
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ctx, LF_INT_UTOA_PAD_4);
CLEANUP: log: simplify complex values usages in sess_build_logline() make sess_build_logline() switch case more readable by performing some simplifications: complex values are first extracted in a temporary variable so that it's easier to refer to them and at a single place.
2024-03-26 06:50:50 -04:00
}
MEDIUM: log: write raw strings using lf_rawtext() Make use of the previous commit to print strings that should not be modified. For instance, when +X option is provided, we have to print numerical values in ASCII HEX form. For that, we used snprintf() to output the result to the log output buffer directly, but now we build the string in a temporary buffer of fixed-size and then print it using lf_rawtext() which will take care of encoding options. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice, we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-03-27 05:28:59 -04:00
if (ret == NULL)
goto out;
tmplog = ret;
CLEANUP: log: simplify complex values usages in sess_build_logline() make sess_build_logline() switch case more readable by performing some simplifications: complex values are first extracted in a temporary variable so that it's easier to refer to them and at a single place.
2024-03-26 06:50:50 -04:00
break;
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
}
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
case LOG_FMT_FRONTEND: // %f
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
src = fe->id;
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_text(tmplog, src, dst + maxsize - tmplog, ctx);
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
if (ret == NULL)
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
goto out;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
tmplog = ret;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
break;
MEDIUM: log: suffix the frontend's name with '~' when using SSL Until now it was not possible to know from the logs whether the incoming connection was made over SSL or not. In order to address this in the existing log formats, a new log format %ft was introduced, to log the frontend's name suffixed with its transport layer. The only transport layer in use right now is '~' for SSL, so that existing log formats for non-SSL traffic are not affected at all, and SSL log formats have the frontend's name suffixed with '~'. The TCP, HTTP and CLF log format now use %ft instead of %f. This does not affect existing log formats which still make use of %f however.
2012-10-12 08:56:11 -04:00
case LOG_FMT_FRONTEND_XPRT: // %ft
src = fe->id;
MINOR: log: use LOG_VARTEXT_{START,END} to enclose text strings Rename LOGQUOTE_{START,END} macros to more generic LOG_VARTEXT_{START,END} in order to prepare for new encoding types that rely on specific treatment for variable-length texts. No functional change should be expected.
2024-04-16 05:17:55 -04:00
LOG_VARTEXT_START();
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_rawtext(tmplog, src, dst + maxsize - tmplog, ctx);
MEDIUM: log: write raw strings using lf_rawtext() Make use of the previous commit to print strings that should not be modified. For instance, when +X option is provided, we have to print numerical values in ASCII HEX form. For that, we used snprintf() to output the result to the log output buffer directly, but now we build the string in a temporary buffer of fixed-size and then print it using lf_rawtext() which will take care of encoding options. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice, we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-03-27 05:28:59 -04:00
if (ret == NULL)
MEDIUM: log: suffix the frontend's name with '~' when using SSL Until now it was not possible to know from the logs whether the incoming connection was made over SSL or not. In order to address this in the existing log formats, a new log format %ft was introduced, to log the frontend's name suffixed with its transport layer. The only transport layer in use right now is '~' for SSL, so that existing log formats for non-SSL traffic are not affected at all, and SSL log formats have the frontend's name suffixed with '~'. The TCP, HTTP and CLF log format now use %ft instead of %f. This does not affect existing log formats which still make use of %f however.
2012-10-12 08:56:11 -04:00
goto out;
MEDIUM: log: write raw strings using lf_rawtext() Make use of the previous commit to print strings that should not be modified. For instance, when +X option is provided, we have to print numerical values in ASCII HEX form. For that, we used snprintf() to output the result to the log output buffer directly, but now we build the string in a temporary buffer of fixed-size and then print it using lf_rawtext() which will take care of encoding options. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice, we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-03-27 05:28:59 -04:00
tmplog = ret;
DOC: log: Add comments to specify when session's listener is defined or not When a log message is emitted, The session's listener is always defined when the session's owner is an inbound connection while it is undefined for a health-check. It is not obvious. So, comments have been added to make it clear. This patch is related to the issue #1434.
2021-11-15 05:31:08 -05:00
/* sess->listener may be undefined if the session's owner is a health-check */
MINOR: log: add '~' to frontend when the transport layer provides SSL We used to check if the transport layer was ssl_sock to decide to log "~" after a frontend's name. Now that QUIC is present, this doesn't work anymore. Better rely on the transport layer's get_ssl_sock_ctx() method.
2022-04-12 02:08:33 -04:00
if (sess->listener && sess->listener->bind_conf->xprt->get_ssl_sock_ctx)
MEDIUM: log: suffix the frontend's name with '~' when using SSL Until now it was not possible to know from the logs whether the incoming connection was made over SSL or not. In order to address this in the existing log formats, a new log format %ft was introduced, to log the frontend's name suffixed with its transport layer. The only transport layer in use right now is '~' for SSL, so that existing log formats for non-SSL traffic are not affected at all, and SSL log formats have the frontend's name suffixed with '~'. The TCP, HTTP and CLF log format now use %ft instead of %f. This does not affect existing log formats which still make use of %f however.
2012-10-12 08:56:11 -04:00
LOGCHAR('~');
break;
MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv These two new log-format tags report the SSL protocol version (%sslv) and the SSL ciphers (%sslc) used for the connection with the client. For instance, to append these information just after the client's IP/port address information on an HTTP log line, use the following configuration : log-format %Ci:%Cp\ %sslv:%sslc\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %st\ %B\ %cc\ \ %cs\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %{+Q}r It will report a line such as the following one : Oct 12 20:47:30 haproxy[9643]: 127.0.0.1:43602 TLSv1:AES-SHA [12/Oct/2012:20:47:30.303] stick2~ stick2/s1 7/0/12/0/19 200 145 - - ---- 0/0/0/0/0 0/0 "GET /?t=0 HTTP/1.0"
2012-10-12 14:17:54 -04:00
#ifdef USE_OPENSSL
case LOG_FMT_SSL_CIPHER: // %sslc
src = NULL;
MEDIUM: session: use the pointer to the origin instead of s->si[0].end When s->si[0].end was dereferenced as a connection or anything in order to retrieve information about the originating session, we'll now use sess->origin instead so that when we have to chain multiple streams in HTTP/2, we'll keep accessing the same origin.
2015-04-03 13:19:59 -04:00
conn = objt_conn(sess->origin);
MAJOR: stream-int: stop using si->conn and use si->end instead The connection will only remain there as a pre-allocated entity whose goal is to be placed in ->end when establishing an outgoing connection. All connection initialization can be made on this connection, but all information retrieved should be applied to the end point only. This change is huge because there were many users of si->conn. Now the only users are those who initialize the new connection. The difficulty appears in a few places such as backend.c, proto_http.c, peers.c where si->conn is used to hold the connection's target address before assigning the connection to the stream interface. This is why we have to keep si->conn for now. A future improvement might consist in dynamically allocating the connection when it is needed.
2013-10-01 04:45:07 -04:00
if (conn) {
MINOR: merge ssl_sock_get calls for log and ppv2 Merge ssl_sock_get_version and ssl_sock_get_proto_version. Change ssl_sock_get_cipher to be used in ppv2.
2017-10-13 10:59:49 -04:00
src = ssl_sock_get_cipher_name(conn);
MAJOR: stream-int: stop using si->conn and use si->end instead The connection will only remain there as a pre-allocated entity whose goal is to be placed in ->end when establishing an outgoing connection. All connection initialization can be made on this connection, but all information retrieved should be applied to the end point only. This change is huge because there were many users of si->conn. Now the only users are those who initialize the new connection. The difficulty appears in a few places such as backend.c, proto_http.c, peers.c where si->conn is used to hold the connection's target address before assigning the connection to the stream interface. This is why we have to keep si->conn for now. A future improvement might consist in dynamically allocating the connection when it is needed.
2013-10-01 04:45:07 -04:00
}
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_text(tmplog, src, dst + maxsize - tmplog, ctx);
MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv These two new log-format tags report the SSL protocol version (%sslv) and the SSL ciphers (%sslc) used for the connection with the client. For instance, to append these information just after the client's IP/port address information on an HTTP log line, use the following configuration : log-format %Ci:%Cp\ %sslv:%sslc\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %st\ %B\ %cc\ \ %cs\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %{+Q}r It will report a line such as the following one : Oct 12 20:47:30 haproxy[9643]: 127.0.0.1:43602 TLSv1:AES-SHA [12/Oct/2012:20:47:30.303] stick2~ stick2/s1 7/0/12/0/19 200 145 - - ---- 0/0/0/0/0 0/0 "GET /?t=0 HTTP/1.0"
2012-10-12 14:17:54 -04:00
if (ret == NULL)
goto out;
tmplog = ret;
break;
MEDIUM: log: suffix the frontend's name with '~' when using SSL Until now it was not possible to know from the logs whether the incoming connection was made over SSL or not. In order to address this in the existing log formats, a new log format %ft was introduced, to log the frontend's name suffixed with its transport layer. The only transport layer in use right now is '~' for SSL, so that existing log formats for non-SSL traffic are not affected at all, and SSL log formats have the frontend's name suffixed with '~'. The TCP, HTTP and CLF log format now use %ft instead of %f. This does not affect existing log formats which still make use of %f however.
2012-10-12 08:56:11 -04:00
MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv These two new log-format tags report the SSL protocol version (%sslv) and the SSL ciphers (%sslc) used for the connection with the client. For instance, to append these information just after the client's IP/port address information on an HTTP log line, use the following configuration : log-format %Ci:%Cp\ %sslv:%sslc\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %st\ %B\ %cc\ \ %cs\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %{+Q}r It will report a line such as the following one : Oct 12 20:47:30 haproxy[9643]: 127.0.0.1:43602 TLSv1:AES-SHA [12/Oct/2012:20:47:30.303] stick2~ stick2/s1 7/0/12/0/19 200 145 - - ---- 0/0/0/0/0 0/0 "GET /?t=0 HTTP/1.0"
2012-10-12 14:17:54 -04:00
case LOG_FMT_SSL_VERSION: // %sslv
src = NULL;
MEDIUM: session: use the pointer to the origin instead of s->si[0].end When s->si[0].end was dereferenced as a connection or anything in order to retrieve information about the originating session, we'll now use sess->origin instead so that when we have to chain multiple streams in HTTP/2, we'll keep accessing the same origin.
2015-04-03 13:19:59 -04:00
conn = objt_conn(sess->origin);
MAJOR: stream-int: stop using si->conn and use si->end instead The connection will only remain there as a pre-allocated entity whose goal is to be placed in ->end when establishing an outgoing connection. All connection initialization can be made on this connection, but all information retrieved should be applied to the end point only. This change is huge because there were many users of si->conn. Now the only users are those who initialize the new connection. The difficulty appears in a few places such as backend.c, proto_http.c, peers.c where si->conn is used to hold the connection's target address before assigning the connection to the stream interface. This is why we have to keep si->conn for now. A future improvement might consist in dynamically allocating the connection when it is needed.
2013-10-01 04:45:07 -04:00
if (conn) {
MINOR: merge ssl_sock_get calls for log and ppv2 Merge ssl_sock_get_version and ssl_sock_get_proto_version. Change ssl_sock_get_cipher to be used in ppv2.
2017-10-13 10:59:49 -04:00
src = ssl_sock_get_proto_version(conn);
MAJOR: stream-int: stop using si->conn and use si->end instead The connection will only remain there as a pre-allocated entity whose goal is to be placed in ->end when establishing an outgoing connection. All connection initialization can be made on this connection, but all information retrieved should be applied to the end point only. This change is huge because there were many users of si->conn. Now the only users are those who initialize the new connection. The difficulty appears in a few places such as backend.c, proto_http.c, peers.c where si->conn is used to hold the connection's target address before assigning the connection to the stream interface. This is why we have to keep si->conn for now. A future improvement might consist in dynamically allocating the connection when it is needed.
2013-10-01 04:45:07 -04:00
}
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_text(tmplog, src, dst + maxsize - tmplog, ctx);
MEDIUM: log: report SSL ciphers and version in logs using logformat %sslc/%sslv These two new log-format tags report the SSL protocol version (%sslv) and the SSL ciphers (%sslc) used for the connection with the client. For instance, to append these information just after the client's IP/port address information on an HTTP log line, use the following configuration : log-format %Ci:%Cp\ %sslv:%sslc\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %st\ %B\ %cc\ \ %cs\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %{+Q}r It will report a line such as the following one : Oct 12 20:47:30 haproxy[9643]: 127.0.0.1:43602 TLSv1:AES-SHA [12/Oct/2012:20:47:30.303] stick2~ stick2/s1 7/0/12/0/19 200 145 - - ---- 0/0/0/0/0 0/0 "GET /?t=0 HTTP/1.0"
2012-10-12 14:17:54 -04:00
if (ret == NULL)
goto out;
tmplog = ret;
break;
#endif
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
case LOG_FMT_BACKEND: // %b
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
src = be->id;
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_text(tmplog, src, dst + maxsize - tmplog, ctx);
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
if (ret == NULL)
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
goto out;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
tmplog = ret;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
break;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
case LOG_FMT_SERVER: // %s
MINOR: logs: Add support of checks as session origin to format lf strings When a log-format string is built from an health-check, the session origin is the health-check itself and not a connection. In addition, there is no stream. It means for now some formats are not supported: %s, %sc, %b, %bi, %bp, %si and %sp. Thanks to this patch, the session origin is converted to a check. So it is possible to retrieve the backend and the backend connection. Note this session have no listener, thus %ft format must be guarded. This patch is light and standalone, thus it may be backported as far as 2.2 if required. However, because the error is human, it is probably better to wait a bit to be sure everything is properly protected.
2021-04-14 08:01:41 -04:00
switch (obj_type(s ? s->target : sess->origin)) {
BUG/MINOR: log: make log-format, unique-id-format and add-header more independant It happens that all of them call parse_logformat_line() which sets proxy->to_log with a number of flags affecting the line format for all three users. For example, having a unique-id specified disables the default log-format since fe->to_log is tested when the session is established. Similarly, having "option logasap" will cause "+" to be inserted in unique-id or headers referencing some of the fields depending on LW_BYTES. This patch first removes most of the dependency on fe->to_log whenever possible. The first possible cleanup is to stop checking fe->to_log for being null, considering that it always contains at least LW_INIT when any such usage is made of the log-format! Also, some checks are wrong. s->logs.logwait cannot be nulled by "logwait &= ~LW_*" since LW_INIT is always there. This results in getting the wrong log at the end of a request or session when a unique-id or add-header is set, because logwait is still not null but the log-format is not checked. Further cleanups are required. Most LW_* flags should be removed or at least replaced with what they really mean (eg: depend on client-side connection, depend on server-side connection, etc...) and this should only affect logging, not other mechanisms. This patch fixes the default log-format and tries to limit interferences between the log formats, but does not pretend to do more for the moment, since it's the most visible breakage.
2012-12-28 03:40:16 -05:00
case OBJ_TYPE_SERVER:
BUILD: log: silent build warnings due to unchecked __objt_{server,applet} These ones are safe to use there since the same check is performed in the switch/case they're used it. Let's use the unguarded versions instead.
2018-09-20 05:13:58 -04:00
src = __objt_server(s->target)->id;
BUG/MINOR: log: make log-format, unique-id-format and add-header more independant It happens that all of them call parse_logformat_line() which sets proxy->to_log with a number of flags affecting the line format for all three users. For example, having a unique-id specified disables the default log-format since fe->to_log is tested when the session is established. Similarly, having "option logasap" will cause "+" to be inserted in unique-id or headers referencing some of the fields depending on LW_BYTES. This patch first removes most of the dependency on fe->to_log whenever possible. The first possible cleanup is to stop checking fe->to_log for being null, considering that it always contains at least LW_INIT when any such usage is made of the log-format! Also, some checks are wrong. s->logs.logwait cannot be nulled by "logwait &= ~LW_*" since LW_INIT is always there. This results in getting the wrong log at the end of a request or session when a unique-id or add-header is set, because logwait is still not null but the log-format is not checked. Further cleanups are required. Most LW_* flags should be removed or at least replaced with what they really mean (eg: depend on client-side connection, depend on server-side connection, etc...) and this should only affect logging, not other mechanisms. This patch fixes the default log-format and tries to limit interferences between the log formats, but does not pretend to do more for the moment, since it's the most visible breakage.
2012-12-28 03:40:16 -05:00
break;
case OBJ_TYPE_APPLET:
BUILD: log: silent build warnings due to unchecked __objt_{server,applet} These ones are safe to use there since the same check is performed in the switch/case they're used it. Let's use the unguarded versions instead.
2018-09-20 05:13:58 -04:00
src = __objt_applet(s->target)->name;
BUG/MINOR: log: make log-format, unique-id-format and add-header more independant It happens that all of them call parse_logformat_line() which sets proxy->to_log with a number of flags affecting the line format for all three users. For example, having a unique-id specified disables the default log-format since fe->to_log is tested when the session is established. Similarly, having "option logasap" will cause "+" to be inserted in unique-id or headers referencing some of the fields depending on LW_BYTES. This patch first removes most of the dependency on fe->to_log whenever possible. The first possible cleanup is to stop checking fe->to_log for being null, considering that it always contains at least LW_INIT when any such usage is made of the log-format! Also, some checks are wrong. s->logs.logwait cannot be nulled by "logwait &= ~LW_*" since LW_INIT is always there. This results in getting the wrong log at the end of a request or session when a unique-id or add-header is set, because logwait is still not null but the log-format is not checked. Further cleanups are required. Most LW_* flags should be removed or at least replaced with what they really mean (eg: depend on client-side connection, depend on server-side connection, etc...) and this should only affect logging, not other mechanisms. This patch fixes the default log-format and tries to limit interferences between the log formats, but does not pretend to do more for the moment, since it's the most visible breakage.
2012-12-28 03:40:16 -05:00
break;
MINOR: logs: Add support of checks as session origin to format lf strings When a log-format string is built from an health-check, the session origin is the health-check itself and not a connection. In addition, there is no stream. It means for now some formats are not supported: %s, %sc, %b, %bi, %bp, %si and %sp. Thanks to this patch, the session origin is converted to a check. So it is possible to retrieve the backend and the backend connection. Note this session have no listener, thus %ft format must be guarded. This patch is light and standalone, thus it may be backported as far as 2.2 if required. However, because the error is human, it is probably better to wait a bit to be sure everything is properly protected.
2021-04-14 08:01:41 -04:00
case OBJ_TYPE_CHECK:
src = (__objt_check(sess->origin)->server
? __objt_check(sess->origin)->server->id
: "<NOSRV>");
break;
BUG/MINOR: log: make log-format, unique-id-format and add-header more independant It happens that all of them call parse_logformat_line() which sets proxy->to_log with a number of flags affecting the line format for all three users. For example, having a unique-id specified disables the default log-format since fe->to_log is tested when the session is established. Similarly, having "option logasap" will cause "+" to be inserted in unique-id or headers referencing some of the fields depending on LW_BYTES. This patch first removes most of the dependency on fe->to_log whenever possible. The first possible cleanup is to stop checking fe->to_log for being null, considering that it always contains at least LW_INIT when any such usage is made of the log-format! Also, some checks are wrong. s->logs.logwait cannot be nulled by "logwait &= ~LW_*" since LW_INIT is always there. This results in getting the wrong log at the end of a request or session when a unique-id or add-header is set, because logwait is still not null but the log-format is not checked. Further cleanups are required. Most LW_* flags should be removed or at least replaced with what they really mean (eg: depend on client-side connection, depend on server-side connection, etc...) and this should only affect logging, not other mechanisms. This patch fixes the default log-format and tries to limit interferences between the log formats, but does not pretend to do more for the moment, since it's the most visible breakage.
2012-12-28 03:40:16 -05:00
default:
src = "<NOSRV>";
break;
}
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_text(tmplog, src, dst + maxsize - tmplog, ctx);
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
if (ret == NULL)
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
goto out;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
tmplog = ret;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
break;
MEDIUM: log: Decompose %Tq in %Th %Ti %TR Tq is the time between the instant the connection is accepted and a complete valid request is received. This time includes the handshake (SSL / Proxy-Protocol), the idle when the browser does preconnect and the request reception. This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR which returns respectively the handshake time, the idle time and the duration of valid request reception. It also adds %Ta which reports the request's active time, which is the total time without %Th nor %Ti. It replaces %Tt as the total time, reporting accurate measurements for HTTP persistent connections. %Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only avalaible for HTTP connections. In addition to this, we have new timestamps %tr, %trg and %trl, which log the date of start of receipt of the request, respectively in the default format, in GMT time and in local time (by analogy with %t, %T and %Tl). All of them are obviously only available for HTTP. These values are more relevant as they more accurately represent the request date without being skewed by a browser's preconnect nor a keep-alive idle time. The HTTP log format and the CLF log format have been modified to use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This way the default log formats now produce the expected output for users who don't want to manually fiddle with the log-format directive. Example with the following log-format : log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r" The request was sent by hand using "openssl s_client -connect" : Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1" => 6 ms of SSL handshake, 2375 waiting before sending the first char (in fact the time to type the first line), 261 ms before the end of the request, no time spent in queue, 1 ms spend connecting to the server, immediate response, total active time for this request = 262ms. Total time from accept to close : 2643 ms. The timing now decomposes like this : first request 2nd request |<-------------------------------->|<-------------- ... t tr t tr ... ---|----|----|----|----|----|----|----|----|-- : Th Ti TR Tw Tc Tr Td : Ti ... :<---- Tq ---->: : :<-------------- Tt -------------->: :<--------- Ta --------->:
2016-07-28 11:19:45 -04:00
case LOG_FMT_Th: // %Th = handshake time
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_int(tmplog, dst + maxsize - tmplog, logs->t_handshake, ctx, LF_INT_LTOA);
MEDIUM: log: Decompose %Tq in %Th %Ti %TR Tq is the time between the instant the connection is accepted and a complete valid request is received. This time includes the handshake (SSL / Proxy-Protocol), the idle when the browser does preconnect and the request reception. This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR which returns respectively the handshake time, the idle time and the duration of valid request reception. It also adds %Ta which reports the request's active time, which is the total time without %Th nor %Ti. It replaces %Tt as the total time, reporting accurate measurements for HTTP persistent connections. %Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only avalaible for HTTP connections. In addition to this, we have new timestamps %tr, %trg and %trl, which log the date of start of receipt of the request, respectively in the default format, in GMT time and in local time (by analogy with %t, %T and %Tl). All of them are obviously only available for HTTP. These values are more relevant as they more accurately represent the request date without being skewed by a browser's preconnect nor a keep-alive idle time. The HTTP log format and the CLF log format have been modified to use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This way the default log formats now produce the expected output for users who don't want to manually fiddle with the log-format directive. Example with the following log-format : log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r" The request was sent by hand using "openssl s_client -connect" : Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1" => 6 ms of SSL handshake, 2375 waiting before sending the first char (in fact the time to type the first line), 261 ms before the end of the request, no time spent in queue, 1 ms spend connecting to the server, immediate response, total active time for this request = 262ms. Total time from accept to close : 2643 ms. The timing now decomposes like this : first request 2nd request |<-------------------------------->|<-------------- ... t tr t tr ... ---|----|----|----|----|----|----|----|----|-- : Th Ti TR Tw Tc Tr Td : Ti ... :<---- Tq ---->: : :<-------------- Tt -------------->: :<--------- Ta --------->:
2016-07-28 11:19:45 -04:00
if (ret == NULL)
goto out;
tmplog = ret;
break;
case LOG_FMT_Ti: // %Ti = HTTP idle time
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_int(tmplog, dst + maxsize - tmplog, logs->t_idle, ctx, LF_INT_LTOA);
MEDIUM: log: Decompose %Tq in %Th %Ti %TR Tq is the time between the instant the connection is accepted and a complete valid request is received. This time includes the handshake (SSL / Proxy-Protocol), the idle when the browser does preconnect and the request reception. This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR which returns respectively the handshake time, the idle time and the duration of valid request reception. It also adds %Ta which reports the request's active time, which is the total time without %Th nor %Ti. It replaces %Tt as the total time, reporting accurate measurements for HTTP persistent connections. %Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only avalaible for HTTP connections. In addition to this, we have new timestamps %tr, %trg and %trl, which log the date of start of receipt of the request, respectively in the default format, in GMT time and in local time (by analogy with %t, %T and %Tl). All of them are obviously only available for HTTP. These values are more relevant as they more accurately represent the request date without being skewed by a browser's preconnect nor a keep-alive idle time. The HTTP log format and the CLF log format have been modified to use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This way the default log formats now produce the expected output for users who don't want to manually fiddle with the log-format directive. Example with the following log-format : log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r" The request was sent by hand using "openssl s_client -connect" : Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1" => 6 ms of SSL handshake, 2375 waiting before sending the first char (in fact the time to type the first line), 261 ms before the end of the request, no time spent in queue, 1 ms spend connecting to the server, immediate response, total active time for this request = 262ms. Total time from accept to close : 2643 ms. The timing now decomposes like this : first request 2nd request |<-------------------------------->|<-------------- ... t tr t tr ... ---|----|----|----|----|----|----|----|----|-- : Th Ti TR Tw Tc Tr Td : Ti ... :<---- Tq ---->: : :<-------------- Tt -------------->: :<--------- Ta --------->:
2016-07-28 11:19:45 -04:00
if (ret == NULL)
goto out;
tmplog = ret;
break;
case LOG_FMT_TR: // %TR = HTTP request time
CLEANUP: log: simplify complex values usages in sess_build_logline() make sess_build_logline() switch case more readable by performing some simplifications: complex values are first extracted in a temporary variable so that it's easier to refer to them and at a single place.
2024-03-26 06:50:50 -04:00
{
long value = (t_request >= 0) ? t_request - logs->t_idle - logs->t_handshake : -1;
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_int(tmplog, dst + maxsize - tmplog, value, ctx, LF_INT_LTOA);
MEDIUM: log: Decompose %Tq in %Th %Ti %TR Tq is the time between the instant the connection is accepted and a complete valid request is received. This time includes the handshake (SSL / Proxy-Protocol), the idle when the browser does preconnect and the request reception. This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR which returns respectively the handshake time, the idle time and the duration of valid request reception. It also adds %Ta which reports the request's active time, which is the total time without %Th nor %Ti. It replaces %Tt as the total time, reporting accurate measurements for HTTP persistent connections. %Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only avalaible for HTTP connections. In addition to this, we have new timestamps %tr, %trg and %trl, which log the date of start of receipt of the request, respectively in the default format, in GMT time and in local time (by analogy with %t, %T and %Tl). All of them are obviously only available for HTTP. These values are more relevant as they more accurately represent the request date without being skewed by a browser's preconnect nor a keep-alive idle time. The HTTP log format and the CLF log format have been modified to use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This way the default log formats now produce the expected output for users who don't want to manually fiddle with the log-format directive. Example with the following log-format : log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r" The request was sent by hand using "openssl s_client -connect" : Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1" => 6 ms of SSL handshake, 2375 waiting before sending the first char (in fact the time to type the first line), 261 ms before the end of the request, no time spent in queue, 1 ms spend connecting to the server, immediate response, total active time for this request = 262ms. Total time from accept to close : 2643 ms. The timing now decomposes like this : first request 2nd request |<-------------------------------->|<-------------- ... t tr t tr ... ---|----|----|----|----|----|----|----|----|-- : Th Ti TR Tw Tc Tr Td : Ti ... :<---- Tq ---->: : :<-------------- Tt -------------->: :<--------- Ta --------->:
2016-07-28 11:19:45 -04:00
if (ret == NULL)
goto out;
tmplog = ret;
break;
CLEANUP: log: simplify complex values usages in sess_build_logline() make sess_build_logline() switch case more readable by performing some simplifications: complex values are first extracted in a temporary variable so that it's easier to refer to them and at a single place.
2024-03-26 06:50:50 -04:00
}
MEDIUM: log: Decompose %Tq in %Th %Ti %TR Tq is the time between the instant the connection is accepted and a complete valid request is received. This time includes the handshake (SSL / Proxy-Protocol), the idle when the browser does preconnect and the request reception. This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR which returns respectively the handshake time, the idle time and the duration of valid request reception. It also adds %Ta which reports the request's active time, which is the total time without %Th nor %Ti. It replaces %Tt as the total time, reporting accurate measurements for HTTP persistent connections. %Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only avalaible for HTTP connections. In addition to this, we have new timestamps %tr, %trg and %trl, which log the date of start of receipt of the request, respectively in the default format, in GMT time and in local time (by analogy with %t, %T and %Tl). All of them are obviously only available for HTTP. These values are more relevant as they more accurately represent the request date without being skewed by a browser's preconnect nor a keep-alive idle time. The HTTP log format and the CLF log format have been modified to use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This way the default log formats now produce the expected output for users who don't want to manually fiddle with the log-format directive. Example with the following log-format : log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r" The request was sent by hand using "openssl s_client -connect" : Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1" => 6 ms of SSL handshake, 2375 waiting before sending the first char (in fact the time to type the first line), 261 ms before the end of the request, no time spent in queue, 1 ms spend connecting to the server, immediate response, total active time for this request = 262ms. Total time from accept to close : 2643 ms. The timing now decomposes like this : first request 2nd request |<-------------------------------->|<-------------- ... t tr t tr ... ---|----|----|----|----|----|----|----|----|-- : Th Ti TR Tw Tc Tr Td : Ti ... :<---- Tq ---->: : :<-------------- Tt -------------->: :<--------- Ta --------->:
2016-07-28 11:19:45 -04:00
case LOG_FMT_TQ: // %Tq = Th + Ti + TR
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_int(tmplog, dst + maxsize - tmplog, t_request, ctx, LF_INT_LTOA);
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
if (ret == NULL)
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
goto out;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
tmplog = ret;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
break;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
case LOG_FMT_TW: // %Tw
CLEANUP: log: simplify complex values usages in sess_build_logline() make sess_build_logline() switch case more readable by performing some simplifications: complex values are first extracted in a temporary variable so that it's easier to refer to them and at a single place.
2024-03-26 06:50:50 -04:00
{
long value = (logs->t_queue >= 0) ? logs->t_queue - t_request : -1;
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_int(tmplog, dst + maxsize - tmplog, value, ctx, LF_INT_LTOA);
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
if (ret == NULL)
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
goto out;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
tmplog = ret;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
break;
CLEANUP: log: simplify complex values usages in sess_build_logline() make sess_build_logline() switch case more readable by performing some simplifications: complex values are first extracted in a temporary variable so that it's easier to refer to them and at a single place.
2024-03-26 06:50:50 -04:00
}
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
case LOG_FMT_TC: // %Tc
CLEANUP: log: simplify complex values usages in sess_build_logline() make sess_build_logline() switch case more readable by performing some simplifications: complex values are first extracted in a temporary variable so that it's easier to refer to them and at a single place.
2024-03-26 06:50:50 -04:00
{
long value = (logs->t_connect >= 0) ? logs->t_connect - logs->t_queue : -1;
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_int(tmplog, dst + maxsize - tmplog, value, ctx, LF_INT_LTOA);
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
if (ret == NULL)
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
goto out;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
tmplog = ret;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
break;
CLEANUP: log: simplify complex values usages in sess_build_logline() make sess_build_logline() switch case more readable by performing some simplifications: complex values are first extracted in a temporary variable so that it's easier to refer to them and at a single place.
2024-03-26 06:50:50 -04:00
}
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
MEDIUM: log: Decompose %Tq in %Th %Ti %TR Tq is the time between the instant the connection is accepted and a complete valid request is received. This time includes the handshake (SSL / Proxy-Protocol), the idle when the browser does preconnect and the request reception. This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR which returns respectively the handshake time, the idle time and the duration of valid request reception. It also adds %Ta which reports the request's active time, which is the total time without %Th nor %Ti. It replaces %Tt as the total time, reporting accurate measurements for HTTP persistent connections. %Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only avalaible for HTTP connections. In addition to this, we have new timestamps %tr, %trg and %trl, which log the date of start of receipt of the request, respectively in the default format, in GMT time and in local time (by analogy with %t, %T and %Tl). All of them are obviously only available for HTTP. These values are more relevant as they more accurately represent the request date without being skewed by a browser's preconnect nor a keep-alive idle time. The HTTP log format and the CLF log format have been modified to use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This way the default log formats now produce the expected output for users who don't want to manually fiddle with the log-format directive. Example with the following log-format : log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r" The request was sent by hand using "openssl s_client -connect" : Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1" => 6 ms of SSL handshake, 2375 waiting before sending the first char (in fact the time to type the first line), 261 ms before the end of the request, no time spent in queue, 1 ms spend connecting to the server, immediate response, total active time for this request = 262ms. Total time from accept to close : 2643 ms. The timing now decomposes like this : first request 2nd request |<-------------------------------->|<-------------- ... t tr t tr ... ---|----|----|----|----|----|----|----|----|-- : Th Ti TR Tw Tc Tr Td : Ti ... :<---- Tq ---->: : :<-------------- Tt -------------->: :<--------- Ta --------->:
2016-07-28 11:19:45 -04:00
case LOG_FMT_Tr: // %Tr
CLEANUP: log: simplify complex values usages in sess_build_logline() make sess_build_logline() switch case more readable by performing some simplifications: complex values are first extracted in a temporary variable so that it's easier to refer to them and at a single place.
2024-03-26 06:50:50 -04:00
{
long value = (logs->t_data >= 0) ? logs->t_data - logs->t_connect : -1;
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_int(tmplog, dst + maxsize - tmplog, value, ctx, LF_INT_LTOA);
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
if (ret == NULL)
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
goto out;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
tmplog = ret;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
break;
CLEANUP: log: simplify complex values usages in sess_build_logline() make sess_build_logline() switch case more readable by performing some simplifications: complex values are first extracted in a temporary variable so that it's easier to refer to them and at a single place.
2024-03-26 06:50:50 -04:00
}
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
MINOR: log: add the %Td log-format specifier As suggested by Pavlos, it's too bad that we didn't have a %Td log format tag given that there are a few mentions of Td corresponding to the data transmission time already in the doc, so this is now done. Just like the other specifiers, we report -1 if the connection failed before reaching the data transmission state.
2016-05-17 11:55:27 -04:00
case LOG_FMT_TD: // %Td
CLEANUP: log: simplify complex values usages in sess_build_logline() make sess_build_logline() switch case more readable by performing some simplifications: complex values are first extracted in a temporary variable so that it's easier to refer to them and at a single place.
2024-03-26 06:50:50 -04:00
{
long value;
MINOR: log: make the backend fall back to the frontend when there's no stream This is already what happens before the backend is assigned, except that now we don't need to dereference a NULL stream to figure this.
2018-09-05 09:07:15 -04:00
if (be->mode == PR_MODE_HTTP)
CLEANUP: log: simplify complex values usages in sess_build_logline() make sess_build_logline() switch case more readable by performing some simplifications: complex values are first extracted in a temporary variable so that it's easier to refer to them and at a single place.
2024-03-26 06:50:50 -04:00
value = (logs->t_data >= 0) ? logs->t_close - logs->t_data : -1;
MINOR: log: add the %Td log-format specifier As suggested by Pavlos, it's too bad that we didn't have a %Td log format tag given that there are a few mentions of Td corresponding to the data transmission time already in the doc, so this is now done. Just like the other specifiers, we report -1 if the connection failed before reaching the data transmission state.
2016-05-17 11:55:27 -04:00
else
CLEANUP: log: simplify complex values usages in sess_build_logline() make sess_build_logline() switch case more readable by performing some simplifications: complex values are first extracted in a temporary variable so that it's easier to refer to them and at a single place.
2024-03-26 06:50:50 -04:00
value = (logs->t_connect >= 0) ? logs->t_close - logs->t_connect : -1;
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_int(tmplog, dst + maxsize - tmplog, value, ctx, LF_INT_LTOA);
CLEANUP: log: simplify complex values usages in sess_build_logline() make sess_build_logline() switch case more readable by performing some simplifications: complex values are first extracted in a temporary variable so that it's easier to refer to them and at a single place.
2024-03-26 06:50:50 -04:00
MINOR: log: add the %Td log-format specifier As suggested by Pavlos, it's too bad that we didn't have a %Td log format tag given that there are a few mentions of Td corresponding to the data transmission time already in the doc, so this is now done. Just like the other specifiers, we report -1 if the connection failed before reaching the data transmission state.
2016-05-17 11:55:27 -04:00
if (ret == NULL)
goto out;
tmplog = ret;
break;
CLEANUP: log: simplify complex values usages in sess_build_logline() make sess_build_logline() switch case more readable by performing some simplifications: complex values are first extracted in a temporary variable so that it's easier to refer to them and at a single place.
2024-03-26 06:50:50 -04:00
}
MINOR: log: add the %Td log-format specifier As suggested by Pavlos, it's too bad that we didn't have a %Td log format tag given that there are a few mentions of Td corresponding to the data transmission time already in the doc, so this is now done. Just like the other specifiers, we report -1 if the connection failed before reaching the data transmission state.
2016-05-17 11:55:27 -04:00
MEDIUM: log: Decompose %Tq in %Th %Ti %TR Tq is the time between the instant the connection is accepted and a complete valid request is received. This time includes the handshake (SSL / Proxy-Protocol), the idle when the browser does preconnect and the request reception. This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR which returns respectively the handshake time, the idle time and the duration of valid request reception. It also adds %Ta which reports the request's active time, which is the total time without %Th nor %Ti. It replaces %Tt as the total time, reporting accurate measurements for HTTP persistent connections. %Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only avalaible for HTTP connections. In addition to this, we have new timestamps %tr, %trg and %trl, which log the date of start of receipt of the request, respectively in the default format, in GMT time and in local time (by analogy with %t, %T and %Tl). All of them are obviously only available for HTTP. These values are more relevant as they more accurately represent the request date without being skewed by a browser's preconnect nor a keep-alive idle time. The HTTP log format and the CLF log format have been modified to use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This way the default log formats now produce the expected output for users who don't want to manually fiddle with the log-format directive. Example with the following log-format : log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r" The request was sent by hand using "openssl s_client -connect" : Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1" => 6 ms of SSL handshake, 2375 waiting before sending the first char (in fact the time to type the first line), 261 ms before the end of the request, no time spent in queue, 1 ms spend connecting to the server, immediate response, total active time for this request = 262ms. Total time from accept to close : 2643 ms. The timing now decomposes like this : first request 2nd request |<-------------------------------->|<-------------- ... t tr t tr ... ---|----|----|----|----|----|----|----|----|-- : Th Ti TR Tw Tc Tr Td : Ti ... :<---- Tq ---->: : :<-------------- Tt -------------->: :<--------- Ta --------->:
2016-07-28 11:19:45 -04:00
case LOG_FMT_Ta: // %Ta = active time = Tt - Th - Ti
CLEANUP: log: simplify complex values usages in sess_build_logline() make sess_build_logline() switch case more readable by performing some simplifications: complex values are first extracted in a temporary variable so that it's easier to refer to them and at a single place.
2024-03-26 06:50:50 -04:00
{
long value = logs->t_close - (logs->t_idle >= 0 ? logs->t_idle + logs->t_handshake : 0);
MEDIUM: log: Decompose %Tq in %Th %Ti %TR Tq is the time between the instant the connection is accepted and a complete valid request is received. This time includes the handshake (SSL / Proxy-Protocol), the idle when the browser does preconnect and the request reception. This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR which returns respectively the handshake time, the idle time and the duration of valid request reception. It also adds %Ta which reports the request's active time, which is the total time without %Th nor %Ti. It replaces %Tt as the total time, reporting accurate measurements for HTTP persistent connections. %Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only avalaible for HTTP connections. In addition to this, we have new timestamps %tr, %trg and %trl, which log the date of start of receipt of the request, respectively in the default format, in GMT time and in local time (by analogy with %t, %T and %Tl). All of them are obviously only available for HTTP. These values are more relevant as they more accurately represent the request date without being skewed by a browser's preconnect nor a keep-alive idle time. The HTTP log format and the CLF log format have been modified to use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This way the default log formats now produce the expected output for users who don't want to manually fiddle with the log-format directive. Example with the following log-format : log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r" The request was sent by hand using "openssl s_client -connect" : Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1" => 6 ms of SSL handshake, 2375 waiting before sending the first char (in fact the time to type the first line), 261 ms before the end of the request, no time spent in queue, 1 ms spend connecting to the server, immediate response, total active time for this request = 262ms. Total time from accept to close : 2643 ms. The timing now decomposes like this : first request 2nd request |<-------------------------------->|<-------------- ... t tr t tr ... ---|----|----|----|----|----|----|----|----|-- : Th Ti TR Tw Tc Tr Td : Ti ... :<---- Tq ---->: : :<-------------- Tt -------------->: :<--------- Ta --------->:
2016-07-28 11:19:45 -04:00
if (!(fe->to_log & LW_BYTES))
MINOR: log: print metadata prefixes separately in sess_build_logline() Some log variables may be prefixed with specific chars that represent extra informations that are relevant with it but are are not directly part of the "raw" value. ie: '+' char is prepended before some values when "option logasap" is used to indicate that the value has not yet reached its final value. However, as those "metadata" are printed using the general purpose LOGCHAR() printing helper, it's not easy to tell if they are part of the base value or not. In this patch we add the LOGMETACHAR() helper that is a wrapper for LOGCHAR(). The goal is to prepare for adding some logic to prevent such additional infos from being generated when not relevant or needed.
2024-01-30 12:01:15 -05:00
LOGMETACHAR('+');
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_int(tmplog, dst + maxsize - tmplog, value, ctx, LF_INT_LTOA);
MEDIUM: log: Decompose %Tq in %Th %Ti %TR Tq is the time between the instant the connection is accepted and a complete valid request is received. This time includes the handshake (SSL / Proxy-Protocol), the idle when the browser does preconnect and the request reception. This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR which returns respectively the handshake time, the idle time and the duration of valid request reception. It also adds %Ta which reports the request's active time, which is the total time without %Th nor %Ti. It replaces %Tt as the total time, reporting accurate measurements for HTTP persistent connections. %Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only avalaible for HTTP connections. In addition to this, we have new timestamps %tr, %trg and %trl, which log the date of start of receipt of the request, respectively in the default format, in GMT time and in local time (by analogy with %t, %T and %Tl). All of them are obviously only available for HTTP. These values are more relevant as they more accurately represent the request date without being skewed by a browser's preconnect nor a keep-alive idle time. The HTTP log format and the CLF log format have been modified to use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This way the default log formats now produce the expected output for users who don't want to manually fiddle with the log-format directive. Example with the following log-format : log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r" The request was sent by hand using "openssl s_client -connect" : Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1" => 6 ms of SSL handshake, 2375 waiting before sending the first char (in fact the time to type the first line), 261 ms before the end of the request, no time spent in queue, 1 ms spend connecting to the server, immediate response, total active time for this request = 262ms. Total time from accept to close : 2643 ms. The timing now decomposes like this : first request 2nd request |<-------------------------------->|<-------------- ... t tr t tr ... ---|----|----|----|----|----|----|----|----|-- : Th Ti TR Tw Tc Tr Td : Ti ... :<---- Tq ---->: : :<-------------- Tt -------------->: :<--------- Ta --------->:
2016-07-28 11:19:45 -04:00
if (ret == NULL)
goto out;
tmplog = ret;
break;
CLEANUP: log: simplify complex values usages in sess_build_logline() make sess_build_logline() switch case more readable by performing some simplifications: complex values are first extracted in a temporary variable so that it's easier to refer to them and at a single place.
2024-03-26 06:50:50 -04:00
}
MEDIUM: log: Decompose %Tq in %Th %Ti %TR Tq is the time between the instant the connection is accepted and a complete valid request is received. This time includes the handshake (SSL / Proxy-Protocol), the idle when the browser does preconnect and the request reception. This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR which returns respectively the handshake time, the idle time and the duration of valid request reception. It also adds %Ta which reports the request's active time, which is the total time without %Th nor %Ti. It replaces %Tt as the total time, reporting accurate measurements for HTTP persistent connections. %Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only avalaible for HTTP connections. In addition to this, we have new timestamps %tr, %trg and %trl, which log the date of start of receipt of the request, respectively in the default format, in GMT time and in local time (by analogy with %t, %T and %Tl). All of them are obviously only available for HTTP. These values are more relevant as they more accurately represent the request date without being skewed by a browser's preconnect nor a keep-alive idle time. The HTTP log format and the CLF log format have been modified to use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This way the default log formats now produce the expected output for users who don't want to manually fiddle with the log-format directive. Example with the following log-format : log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r" The request was sent by hand using "openssl s_client -connect" : Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1" => 6 ms of SSL handshake, 2375 waiting before sending the first char (in fact the time to type the first line), 261 ms before the end of the request, no time spent in queue, 1 ms spend connecting to the server, immediate response, total active time for this request = 262ms. Total time from accept to close : 2643 ms. The timing now decomposes like this : first request 2nd request |<-------------------------------->|<-------------- ... t tr t tr ... ---|----|----|----|----|----|----|----|----|-- : Th Ti TR Tw Tc Tr Td : Ti ... :<---- Tq ---->: : :<-------------- Tt -------------->: :<--------- Ta --------->:
2016-07-28 11:19:45 -04:00
case LOG_FMT_TT: // %Tt = total time
BUG/MINOR: log: make log-format, unique-id-format and add-header more independant It happens that all of them call parse_logformat_line() which sets proxy->to_log with a number of flags affecting the line format for all three users. For example, having a unique-id specified disables the default log-format since fe->to_log is tested when the session is established. Similarly, having "option logasap" will cause "+" to be inserted in unique-id or headers referencing some of the fields depending on LW_BYTES. This patch first removes most of the dependency on fe->to_log whenever possible. The first possible cleanup is to stop checking fe->to_log for being null, considering that it always contains at least LW_INIT when any such usage is made of the log-format! Also, some checks are wrong. s->logs.logwait cannot be nulled by "logwait &= ~LW_*" since LW_INIT is always there. This results in getting the wrong log at the end of a request or session when a unique-id or add-header is set, because logwait is still not null but the log-format is not checked. Further cleanups are required. Most LW_* flags should be removed or at least replaced with what they really mean (eg: depend on client-side connection, depend on server-side connection, etc...) and this should only affect logging, not other mechanisms. This patch fixes the default log-format and tries to limit interferences between the log formats, but does not pretend to do more for the moment, since it's the most visible breakage.
2012-12-28 03:40:16 -05:00
if (!(fe->to_log & LW_BYTES))
MINOR: log: print metadata prefixes separately in sess_build_logline() Some log variables may be prefixed with specific chars that represent extra informations that are relevant with it but are are not directly part of the "raw" value. ie: '+' char is prepended before some values when "option logasap" is used to indicate that the value has not yet reached its final value. However, as those "metadata" are printed using the general purpose LOGCHAR() printing helper, it's not easy to tell if they are part of the base value or not. In this patch we add the LOGMETACHAR() helper that is a wrapper for LOGCHAR(). The goal is to prepare for adding some logic to prevent such additional infos from being generated when not relevant or needed.
2024-01-30 12:01:15 -05:00
LOGMETACHAR('+');
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_int(tmplog, dst + maxsize - tmplog, logs->t_close, ctx, LF_INT_LTOA);
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
if (ret == NULL)
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
goto out;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
tmplog = ret;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
break;
MINOR: log: Add "Tu" timer It can be sometimes useful to measure total time of a request as seen from an end user, including TCP/TLS negotiation, server response time and transfer time. "Tt" currently provides something close to that, but it also takes client idle time into account, which is problematic for keep-alive requests as idle time can be very long. "Ta" is also not sufficient as it hides TCP/TLS negotiationtime. To improve that, introduce a "Tu" timer, without idle time and everything else. It roughly estimates time spent time spent from user point of view (without DNS resolution time), assuming network latency is the same in both directions.
2020-04-28 08:09:19 -04:00
case LOG_FMT_TU: // %Tu = total time seen by user = Tt - Ti
CLEANUP: log: simplify complex values usages in sess_build_logline() make sess_build_logline() switch case more readable by performing some simplifications: complex values are first extracted in a temporary variable so that it's easier to refer to them and at a single place.
2024-03-26 06:50:50 -04:00
{
long value = logs->t_close - (logs->t_idle >= 0 ? logs->t_idle : 0);
MINOR: log: Add "Tu" timer It can be sometimes useful to measure total time of a request as seen from an end user, including TCP/TLS negotiation, server response time and transfer time. "Tt" currently provides something close to that, but it also takes client idle time into account, which is problematic for keep-alive requests as idle time can be very long. "Ta" is also not sufficient as it hides TCP/TLS negotiationtime. To improve that, introduce a "Tu" timer, without idle time and everything else. It roughly estimates time spent time spent from user point of view (without DNS resolution time), assuming network latency is the same in both directions.
2020-04-28 08:09:19 -04:00
if (!(fe->to_log & LW_BYTES))
MINOR: log: print metadata prefixes separately in sess_build_logline() Some log variables may be prefixed with specific chars that represent extra informations that are relevant with it but are are not directly part of the "raw" value. ie: '+' char is prepended before some values when "option logasap" is used to indicate that the value has not yet reached its final value. However, as those "metadata" are printed using the general purpose LOGCHAR() printing helper, it's not easy to tell if they are part of the base value or not. In this patch we add the LOGMETACHAR() helper that is a wrapper for LOGCHAR(). The goal is to prepare for adding some logic to prevent such additional infos from being generated when not relevant or needed.
2024-01-30 12:01:15 -05:00
LOGMETACHAR('+');
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_int(tmplog, dst + maxsize - tmplog, value, ctx, LF_INT_LTOA);
MINOR: log: Add "Tu" timer It can be sometimes useful to measure total time of a request as seen from an end user, including TCP/TLS negotiation, server response time and transfer time. "Tt" currently provides something close to that, but it also takes client idle time into account, which is problematic for keep-alive requests as idle time can be very long. "Ta" is also not sufficient as it hides TCP/TLS negotiationtime. To improve that, introduce a "Tu" timer, without idle time and everything else. It roughly estimates time spent time spent from user point of view (without DNS resolution time), assuming network latency is the same in both directions.
2020-04-28 08:09:19 -04:00
if (ret == NULL)
goto out;
tmplog = ret;
break;
CLEANUP: log: simplify complex values usages in sess_build_logline() make sess_build_logline() switch case more readable by performing some simplifications: complex values are first extracted in a temporary variable so that it's easier to refer to them and at a single place.
2024-03-26 06:50:50 -04:00
}
MINOR: log: Add "Tu" timer It can be sometimes useful to measure total time of a request as seen from an end user, including TCP/TLS negotiation, server response time and transfer time. "Tt" currently provides something close to that, but it also takes client idle time into account, which is problematic for keep-alive requests as idle time can be very long. "Ta" is also not sufficient as it hides TCP/TLS negotiationtime. To improve that, introduce a "Tu" timer, without idle time and everything else. It roughly estimates time spent time spent from user point of view (without DNS resolution time), assuming network latency is the same in both directions.
2020-04-28 08:09:19 -04:00
MEDIUM: log: change a few log tokens to make them easier to remember Some log tokens have evolved in a way that is not completely logical. For example, frontend tokens sometimes begin with an 'f' and sometimes with an 'F'. Same for backend and server. So let's change a few cases without disrupting compatibility with existing setups : Bi => bi Bp => bp Ci => ci Cp => cp Fi => fi Fp => fp Si => si Sp => sp cc => CC cs => CS st => ST The old ones are still supported but deprecated and will be unsupported by the 1.5 release. However, a warning message is emitted when they're encounterd and it indicates what token should be used to replace them.
2012-12-20 11:22:52 -05:00
case LOG_FMT_STATUS: // %ST
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_int(tmplog, dst + maxsize - tmplog, status, ctx, LF_INT_LTOA);
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
if (ret == NULL)
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
goto out;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
tmplog = ret;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
break;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
case LOG_FMT_BYTES: // %B
BUG/MINOR: log: make log-format, unique-id-format and add-header more independant It happens that all of them call parse_logformat_line() which sets proxy->to_log with a number of flags affecting the line format for all three users. For example, having a unique-id specified disables the default log-format since fe->to_log is tested when the session is established. Similarly, having "option logasap" will cause "+" to be inserted in unique-id or headers referencing some of the fields depending on LW_BYTES. This patch first removes most of the dependency on fe->to_log whenever possible. The first possible cleanup is to stop checking fe->to_log for being null, considering that it always contains at least LW_INIT when any such usage is made of the log-format! Also, some checks are wrong. s->logs.logwait cannot be nulled by "logwait &= ~LW_*" since LW_INIT is always there. This results in getting the wrong log at the end of a request or session when a unique-id or add-header is set, because logwait is still not null but the log-format is not checked. Further cleanups are required. Most LW_* flags should be removed or at least replaced with what they really mean (eg: depend on client-side connection, depend on server-side connection, etc...) and this should only affect logging, not other mechanisms. This patch fixes the default log-format and tries to limit interferences between the log formats, but does not pretend to do more for the moment, since it's the most visible breakage.
2012-12-28 03:40:16 -05:00
if (!(fe->to_log & LW_BYTES))
MINOR: log: print metadata prefixes separately in sess_build_logline() Some log variables may be prefixed with specific chars that represent extra informations that are relevant with it but are are not directly part of the "raw" value. ie: '+' char is prepended before some values when "option logasap" is used to indicate that the value has not yet reached its final value. However, as those "metadata" are printed using the general purpose LOGCHAR() printing helper, it's not easy to tell if they are part of the base value or not. In this patch we add the LOGMETACHAR() helper that is a wrapper for LOGCHAR(). The goal is to prepare for adding some logic to prevent such additional infos from being generated when not relevant or needed.
2024-01-30 12:01:15 -05:00
LOGMETACHAR('+');
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_int(tmplog, dst + maxsize - tmplog, logs->bytes_out, ctx, LF_INT_LLTOA);
MINOR: log: add a tag for amount of bytes uploaded from client to server For POST, PUT, CONNECT or tunnelled connections, it's annoying not to have the amount of uploaded bytes in the logs. %U now reports this value.
2012-12-20 09:38:04 -05:00
if (ret == NULL)
goto out;
tmplog = ret;
break;
case LOG_FMT_BYTES_UP: // %U
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_int(tmplog, dst + maxsize - tmplog, logs->bytes_in, ctx, LF_INT_LLTOA);
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
if (ret == NULL)
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
goto out;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
tmplog = ret;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
break;
MEDIUM: log: change a few log tokens to make them easier to remember Some log tokens have evolved in a way that is not completely logical. For example, frontend tokens sometimes begin with an 'f' and sometimes with an 'F'. Same for backend and server. So let's change a few cases without disrupting compatibility with existing setups : Bi => bi Bp => bp Ci => ci Cp => cp Fi => fi Fp => fp Si => si Sp => sp cc => CC cs => CS st => ST The old ones are still supported but deprecated and will be unsupported by the 1.5 release. However, a warning message is emitted when they're encounterd and it indicates what token should be used to replace them.
2012-12-20 11:22:52 -05:00
case LOG_FMT_CCLIENT: // %CC
BUG/MEDIUM: log: fix risk of segfault when logging HTTP fields in TCP mode David Torgerson faced an issue when using HTTP fields in log-format in TCP sections. The txn is dereferenced while it's null, resulting in a crash of the process. Such configurations are invalid and a warning is emitted, but nevertheless the process must not crash. As found by Lukas Tribus, this is a side effect of the split between the stream and the HTTP transaction that happened in 1.6, making it possible to have txn==NULL there. The fix consists in checking that txn is valid before using it. Fortunately it's easy since almost all places already used to check for the existence of a field (eg: txn->uri). This patch should be backported to 1.6.
2016-04-25 11:09:40 -04:00
src = txn ? txn->cli_cookie : NULL;
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_text(tmplog, src, dst + maxsize - tmplog, ctx);
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
if (ret == NULL)
BUG/MAJOR: log: possible segfault with logformat Possible zero-pointer deference in sess_log(). Checks of return values in sess_log() fix the issue. Fix bad computation in logformat_write_string(). This issue is 1.5-specific and was introduced just before 1.5-dev8. No backport is needed.
2012-03-26 11:52:55 -04:00
goto out;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
tmplog = ret;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
break;
MEDIUM: log: change a few log tokens to make them easier to remember Some log tokens have evolved in a way that is not completely logical. For example, frontend tokens sometimes begin with an 'f' and sometimes with an 'F'. Same for backend and server. So let's change a few cases without disrupting compatibility with existing setups : Bi => bi Bp => bp Ci => ci Cp => cp Fi => fi Fp => fp Si => si Sp => sp cc => CC cs => CS st => ST The old ones are still supported but deprecated and will be unsupported by the 1.5 release. However, a warning message is emitted when they're encounterd and it indicates what token should be used to replace them.
2012-12-20 11:22:52 -05:00
case LOG_FMT_CSERVER: // %CS
BUG/MEDIUM: log: fix risk of segfault when logging HTTP fields in TCP mode David Torgerson faced an issue when using HTTP fields in log-format in TCP sections. The txn is dereferenced while it's null, resulting in a crash of the process. Such configurations are invalid and a warning is emitted, but nevertheless the process must not crash. As found by Lukas Tribus, this is a side effect of the split between the stream and the HTTP transaction that happened in 1.6, making it possible to have txn==NULL there. The fix consists in checking that txn is valid before using it. Fortunately it's easy since almost all places already used to check for the existence of a field (eg: txn->uri). This patch should be backported to 1.6.
2016-04-25 11:09:40 -04:00
src = txn ? txn->srv_cookie : NULL;
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_text(tmplog, src, dst + maxsize - tmplog, ctx);
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
if (ret == NULL)
BUG/MAJOR: log: possible segfault with logformat Possible zero-pointer deference in sess_log(). Checks of return values in sess_log() fix the issue. Fix bad computation in logformat_write_string(). This issue is 1.5-specific and was introduced just before 1.5-dev8. No backport is needed.
2012-03-26 11:52:55 -04:00
goto out;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
tmplog = ret;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
break;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
case LOG_FMT_TERMSTATE: // %ts
MINOR: log: explicitly handle %ts and %tsc as text strings Build fixed-length strings for %ts and %tsc to be able to print them using lf_rawtext_len(), this way it will be easier to encode them when new encoding options will be added. No functional change should be expected.
2024-04-23 12:25:32 -04:00
{
OPTIM: log: use lf_buildctx's buffer instead of temporary stack buffers Now that lf_buildctx isn't pushed on the stack anymore, let's take this opportunity to store a small buffer of 256 bytes within it, and then use this buffer as general purpose buffer to build fixed-length strings that are then printed using lf_{raw}text() function. By doing so we stop relying on temporary stack buffers.
2024-05-02 10:48:56 -04:00
ctx->_buf[0] = sess_term_cond[(s_flags & SF_ERR_MASK) >> SF_ERR_SHIFT];
ctx->_buf[1] = sess_fin_state[(s_flags & SF_FINST_MASK) >> SF_FINST_SHIFT];
ret = lf_rawtext_len(tmplog, ctx->_buf, 2, maxsize - (tmplog - dst), ctx);
MINOR: log: explicitly handle %ts and %tsc as text strings Build fixed-length strings for %ts and %tsc to be able to print them using lf_rawtext_len(), this way it will be easier to encode them when new encoding options will be added. No functional change should be expected.
2024-04-23 12:25:32 -04:00
if (ret == NULL)
goto out;
tmplog = ret;
MINOR: log: use "%ts" to log term status only and "%tsc" to log with cookie The difference could be seen when logging a request in HTTP mode with option tcplog, as it would keep emitting 4 chars. Better use two distinct flags to clear the confusion.
2012-03-12 10:09:42 -04:00
break;
MINOR: log: explicitly handle %ts and %tsc as text strings Build fixed-length strings for %ts and %tsc to be able to print them using lf_rawtext_len(), this way it will be easier to encode them when new encoding options will be added. No functional change should be expected.
2024-04-23 12:25:32 -04:00
}
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
case LOG_FMT_TERMSTATE_CK: // %tsc, same as TS with cookie state (for mode HTTP)
MINOR: log: explicitly handle %ts and %tsc as text strings Build fixed-length strings for %ts and %tsc to be able to print them using lf_rawtext_len(), this way it will be easier to encode them when new encoding options will be added. No functional change should be expected.
2024-04-23 12:25:32 -04:00
{
OPTIM: log: use lf_buildctx's buffer instead of temporary stack buffers Now that lf_buildctx isn't pushed on the stack anymore, let's take this opportunity to store a small buffer of 256 bytes within it, and then use this buffer as general purpose buffer to build fixed-length strings that are then printed using lf_{raw}text() function. By doing so we stop relying on temporary stack buffers.
2024-05-02 10:48:56 -04:00
ctx->_buf[0] = sess_term_cond[(s_flags & SF_ERR_MASK) >> SF_ERR_SHIFT];
ctx->_buf[1] = sess_fin_state[(s_flags & SF_FINST_MASK) >> SF_FINST_SHIFT];
ctx->_buf[2] = (txn && (be->ck_opts & PR_CK_ANY)) ? sess_cookie[(txn->flags & TX_CK_MASK) >> TX_CK_SHIFT] : '-';
ctx->_buf[3] = (txn && (be->ck_opts & PR_CK_ANY)) ? sess_set_cookie[(txn->flags & TX_SCK_MASK) >> TX_SCK_SHIFT] : '-';
ret = lf_rawtext_len(tmplog, ctx->_buf, 4, maxsize - (tmplog - dst), ctx);
MINOR: log: explicitly handle %ts and %tsc as text strings Build fixed-length strings for %ts and %tsc to be able to print them using lf_rawtext_len(), this way it will be easier to encode them when new encoding options will be added. No functional change should be expected.
2024-04-23 12:25:32 -04:00
if (ret == NULL)
goto out;
tmplog = ret;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
break;
MINOR: log: explicitly handle %ts and %tsc as text strings Build fixed-length strings for %ts and %tsc to be able to print them using lf_rawtext_len(), this way it will be easier to encode them when new encoding options will be added. No functional change should be expected.
2024-04-23 12:25:32 -04:00
}
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
case LOG_FMT_ACTCONN: // %ac
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_int(tmplog, dst + maxsize - tmplog, actconn, ctx, LF_INT_LTOA);
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
if (ret == NULL)
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
goto out;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
tmplog = ret;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
break;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
case LOG_FMT_FECONN: // %fc
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_int(tmplog, dst + maxsize - tmplog, fe->feconn, ctx, LF_INT_LTOA);
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
if (ret == NULL)
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
goto out;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
tmplog = ret;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
break;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
case LOG_FMT_BECONN: // %bc
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_int(tmplog, dst + maxsize - tmplog, be->beconn, ctx, LF_INT_LTOA);
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
if (ret == NULL)
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
goto out;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
tmplog = ret;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
break;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
case LOG_FMT_SRVCONN: // %sc
CLEANUP: log: simplify complex values usages in sess_build_logline() make sess_build_logline() switch case more readable by performing some simplifications: complex values are first extracted in a temporary variable so that it's easier to refer to them and at a single place.
2024-03-26 06:50:50 -04:00
{
unsigned long value;
MINOR: logs: Add support of checks as session origin to format lf strings When a log-format string is built from an health-check, the session origin is the health-check itself and not a connection. In addition, there is no stream. It means for now some formats are not supported: %s, %sc, %b, %bi, %bp, %si and %sp. Thanks to this patch, the session origin is converted to a check. So it is possible to retrieve the backend and the backend connection. Note this session have no listener, thus %ft format must be guarded. This patch is light and standalone, thus it may be backported as far as 2.2 if required. However, because the error is human, it is probably better to wait a bit to be sure everything is properly protected.
2021-04-14 08:01:41 -04:00
switch (obj_type(s ? s->target : sess->origin)) {
case OBJ_TYPE_SERVER:
CLEANUP: log: simplify complex values usages in sess_build_logline() make sess_build_logline() switch case more readable by performing some simplifications: complex values are first extracted in a temporary variable so that it's easier to refer to them and at a single place.
2024-03-26 06:50:50 -04:00
value = __objt_server(s->target)->cur_sess;
MINOR: logs: Add support of checks as session origin to format lf strings When a log-format string is built from an health-check, the session origin is the health-check itself and not a connection. In addition, there is no stream. It means for now some formats are not supported: %s, %sc, %b, %bi, %bp, %si and %sp. Thanks to this patch, the session origin is converted to a check. So it is possible to retrieve the backend and the backend connection. Note this session have no listener, thus %ft format must be guarded. This patch is light and standalone, thus it may be backported as far as 2.2 if required. However, because the error is human, it is probably better to wait a bit to be sure everything is properly protected.
2021-04-14 08:01:41 -04:00
break;
case OBJ_TYPE_CHECK:
CLEANUP: log: simplify complex values usages in sess_build_logline() make sess_build_logline() switch case more readable by performing some simplifications: complex values are first extracted in a temporary variable so that it's easier to refer to them and at a single place.
2024-03-26 06:50:50 -04:00
value = (__objt_check(sess->origin)->server
? __objt_check(sess->origin)->server->cur_sess
: 0);
MINOR: logs: Add support of checks as session origin to format lf strings When a log-format string is built from an health-check, the session origin is the health-check itself and not a connection. In addition, there is no stream. It means for now some formats are not supported: %s, %sc, %b, %bi, %bp, %si and %sp. Thanks to this patch, the session origin is converted to a check. So it is possible to retrieve the backend and the backend connection. Note this session have no listener, thus %ft format must be guarded. This patch is light and standalone, thus it may be backported as far as 2.2 if required. However, because the error is human, it is probably better to wait a bit to be sure everything is properly protected.
2021-04-14 08:01:41 -04:00
break;
default:
CLEANUP: log: simplify complex values usages in sess_build_logline() make sess_build_logline() switch case more readable by performing some simplifications: complex values are first extracted in a temporary variable so that it's easier to refer to them and at a single place.
2024-03-26 06:50:50 -04:00
value = 0;
MINOR: logs: Add support of checks as session origin to format lf strings When a log-format string is built from an health-check, the session origin is the health-check itself and not a connection. In addition, there is no stream. It means for now some formats are not supported: %s, %sc, %b, %bi, %bp, %si and %sp. Thanks to this patch, the session origin is converted to a check. So it is possible to retrieve the backend and the backend connection. Note this session have no listener, thus %ft format must be guarded. This patch is light and standalone, thus it may be backported as far as 2.2 if required. However, because the error is human, it is probably better to wait a bit to be sure everything is properly protected.
2021-04-14 08:01:41 -04:00
break;
}
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_int(tmplog, dst + maxsize - tmplog, value, ctx, LF_INT_ULTOA);
CLEANUP: log: simplify complex values usages in sess_build_logline() make sess_build_logline() switch case more readable by performing some simplifications: complex values are first extracted in a temporary variable so that it's easier to refer to them and at a single place.
2024-03-26 06:50:50 -04:00
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
if (ret == NULL)
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
goto out;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
tmplog = ret;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
break;
CLEANUP: log: simplify complex values usages in sess_build_logline() make sess_build_logline() switch case more readable by performing some simplifications: complex values are first extracted in a temporary variable so that it's easier to refer to them and at a single place.
2024-03-26 06:50:50 -04:00
}
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
CLEANUP: log: Fix %rc comment in sess_build_logline() %rq was used instead of %rc.
2023-11-28 05:21:40 -05:00
case LOG_FMT_RETRIES: // %rc
CLEANUP: log: simplify complex values usages in sess_build_logline() make sess_build_logline() switch case more readable by performing some simplifications: complex values are first extracted in a temporary variable so that it's easier to refer to them and at a single place.
2024-03-26 06:50:50 -04:00
{
long int value = (s ? s->conn_retries : 0);
MINOR: log: keep a copy of s->flags early to avoid a dereference By placing s->flags into a local variable we'll be able to force it new values when s is NULL.
2018-09-05 09:51:28 -04:00
if (s_flags & SF_REDISP)
MINOR: log: print metadata prefixes separately in sess_build_logline() Some log variables may be prefixed with specific chars that represent extra informations that are relevant with it but are are not directly part of the "raw" value. ie: '+' char is prepended before some values when "option logasap" is used to indicate that the value has not yet reached its final value. However, as those "metadata" are printed using the general purpose LOGCHAR() printing helper, it's not easy to tell if they are part of the base value or not. In this patch we add the LOGMETACHAR() helper that is a wrapper for LOGCHAR(). The goal is to prepare for adding some logic to prevent such additional infos from being generated when not relevant or needed.
2024-01-30 12:01:15 -05:00
LOGMETACHAR('+');
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_int(tmplog, dst + maxsize - tmplog, value, ctx, LF_INT_LTOA);
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
if (ret == NULL)
BUG/MAJOR: log: possible segfault with logformat Possible zero-pointer deference in sess_log(). Checks of return values in sess_log() fix the issue. Fix bad computation in logformat_write_string(). This issue is 1.5-specific and was introduced just before 1.5-dev8. No backport is needed.
2012-03-26 11:52:55 -04:00
goto out;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
tmplog = ret;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
break;
CLEANUP: log: simplify complex values usages in sess_build_logline() make sess_build_logline() switch case more readable by performing some simplifications: complex values are first extracted in a temporary variable so that it's easier to refer to them and at a single place.
2024-03-26 06:50:50 -04:00
}
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
case LOG_FMT_SRVQUEUE: // %sq
MINOR: log: add lf_int() wrapper to print integers Wrap ltoa(), lltoa(), ultoa() and utoa_pad() functions that are used by sess_build_logline() to print numerical values by implementing a dedicated helper named lf_int() that takes <dft_hld> as argument to know how to write the integer by default (when no encoding is specified). LF_INT_UTOA_PAD_4 is used to emulate utoa_pad(x, 4) since it's found only once under sess_build_logline(), thus there is no need to pass an extra parameter to lf_int() function.
2024-03-27 05:17:11 -04:00
ret = lf_int(tmplog, dst + maxsize - tmplog, logs->srv_queue_pos,
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ctx, LF_INT_LTOA);
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
if (ret == NULL)
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
goto out;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
tmplog = ret;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
break;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
case LOG_FMT_BCKQUEUE: // %bq
MINOR: log: add lf_int() wrapper to print integers Wrap ltoa(), lltoa(), ultoa() and utoa_pad() functions that are used by sess_build_logline() to print numerical values by implementing a dedicated helper named lf_int() that takes <dft_hld> as argument to know how to write the integer by default (when no encoding is specified). LF_INT_UTOA_PAD_4 is used to emulate utoa_pad(x, 4) since it's found only once under sess_build_logline(), thus there is no need to pass an extra parameter to lf_int() function.
2024-03-27 05:17:11 -04:00
ret = lf_int(tmplog, dst + maxsize - tmplog, logs->prx_queue_pos,
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ctx, LF_INT_LTOA);
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
if (ret == NULL)
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
goto out;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
tmplog = ret;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
break;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
case LOG_FMT_HDRREQUEST: // %hr
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
/* request header */
MINOR: log: do not dereference a null stream to access captures If the stream is null, let's simply not check captures. That's already done if there is no capture.
2018-09-05 09:28:07 -04:00
if (fe->nb_req_cap && s && s->req_cap) {
MINOR: log: use LOG_VARTEXT_{START,END} to enclose text strings Rename LOGQUOTE_{START,END} macros to more generic LOG_VARTEXT_{START,END} in order to prepare for new encoding types that rely on specific treatment for variable-length texts. No functional change should be expected.
2024-04-16 05:17:55 -04:00
LOG_VARTEXT_START();
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
LOGCHAR('{');
for (hdr = 0; hdr < fe->nb_req_cap; hdr++) {
if (hdr)
LOGCHAR('|');
MEDIUM: http: move header captures from http_txn to struct stream The header captures are now general purpose captures since tcp rules can use them to capture various contents. That removes a dependency on http_txn that appeared in some sample fetch functions and in the order by which captures and http_txn were allocated. Interestingly the reset of the header captures were done at too many places as http_init_txn() used to do it while it was done previously in every call place.
2015-04-03 16:16:32 -04:00
if (s->req_cap[hdr] != NULL) {
MEDIUM: log: add a new log format flag "E" The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It mostly makes sense to use it in the RFC5424 structured-data log formats. Example: log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 07:23:03 -05:00
ret = lf_encode_string(tmplog, dst + maxsize,
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
'#', hdr_encode_map, s->req_cap[hdr], ctx);
CLEANUP: log: remove unused checks for encode_{chunk,string} Thanks to 8226e92eb ("BUG/MINOR: tools/log: invalid encode_{chunk,string} usage"), we only need to check for NULL return value from encode_{chunk,string}() and escape_string() to know if the call failed.
2024-04-09 11:54:41 -04:00
if (ret == NULL)
BUG/MAJOR: log: possible segfault with logformat Possible zero-pointer deference in sess_log(). Checks of return values in sess_log() fix the issue. Fix bad computation in logformat_write_string(). This issue is 1.5-specific and was introduced just before 1.5-dev8. No backport is needed.
2012-03-26 11:52:55 -04:00
goto out;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
tmplog = ret;
BUG/MAJOR: log: possible segfault with logformat Possible zero-pointer deference in sess_log(). Checks of return values in sess_log() fix the issue. Fix bad computation in logformat_write_string(). This issue is 1.5-specific and was introduced just before 1.5-dev8. No backport is needed.
2012-03-26 11:52:55 -04:00
}
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
}
LOGCHAR('}');
}
break;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
case LOG_FMT_HDRREQUESTLIST: // %hrl
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
/* request header list */
MINOR: log: do not dereference a null stream to access captures If the stream is null, let's simply not check captures. That's already done if there is no capture.
2018-09-05 09:28:07 -04:00
if (fe->nb_req_cap && s && s->req_cap) {
MINOR: log: automate string array construction in sess_build_logline() make it so string array construction is performed by dedicated macro helpers instead of manual char insertion between string members. The goal is to easily be able to support multiple forms of array construction depending on the data encoding format (raw, json..). Only %hrl and %hsl logformats are concerned.
2024-01-31 08:37:07 -05:00
LOG_STRARRAY_START();
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
for (hdr = 0; hdr < fe->nb_req_cap; hdr++) {
if (hdr > 0)
MINOR: log: automate string array construction in sess_build_logline() make it so string array construction is performed by dedicated macro helpers instead of manual char insertion between string members. The goal is to easily be able to support multiple forms of array construction depending on the data encoding format (raw, json..). Only %hrl and %hsl logformats are concerned.
2024-01-31 08:37:07 -05:00
LOG_STRARRAY_NEXT();
MINOR: log: use LOG_VARTEXT_{START,END} to enclose text strings Rename LOGQUOTE_{START,END} macros to more generic LOG_VARTEXT_{START,END} in order to prepare for new encoding types that rely on specific treatment for variable-length texts. No functional change should be expected.
2024-04-16 05:17:55 -04:00
LOG_VARTEXT_START();
MEDIUM: http: move header captures from http_txn to struct stream The header captures are now general purpose captures since tcp rules can use them to capture various contents. That removes a dependency on http_txn that appeared in some sample fetch functions and in the order by which captures and http_txn were allocated. Interestingly the reset of the header captures were done at too many places as http_init_txn() used to do it while it was done previously in every call place.
2015-04-03 16:16:32 -04:00
if (s->req_cap[hdr] != NULL) {
MEDIUM: log: add a new log format flag "E" The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It mostly makes sense to use it in the RFC5424 structured-data log formats. Example: log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 07:23:03 -05:00
ret = lf_encode_string(tmplog, dst + maxsize,
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
'#', hdr_encode_map, s->req_cap[hdr], ctx);
CLEANUP: log: remove unused checks for encode_{chunk,string} Thanks to 8226e92eb ("BUG/MINOR: tools/log: invalid encode_{chunk,string} usage"), we only need to check for NULL return value from encode_{chunk,string}() and escape_string() to know if the call failed.
2024-04-09 11:54:41 -04:00
if (ret == NULL)
BUG/MAJOR: log: possible segfault with logformat Possible zero-pointer deference in sess_log(). Checks of return values in sess_log() fix the issue. Fix bad computation in logformat_write_string(). This issue is 1.5-specific and was introduced just before 1.5-dev8. No backport is needed.
2012-03-26 11:52:55 -04:00
goto out;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
tmplog = ret;
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
} else if (!(ctx->options & LOG_OPT_QUOTE))
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
LOGCHAR('-');
MINOR: log: use LOG_VARTEXT_{START,END} to enclose text strings Rename LOGQUOTE_{START,END} macros to more generic LOG_VARTEXT_{START,END} in order to prepare for new encoding types that rely on specific treatment for variable-length texts. No functional change should be expected.
2024-04-16 05:17:55 -04:00
/* Manually end variable text as we're emitting multiple
* texts at once
MINOR: log: simplify quotes handling in sess_build_logline() quotes building for some log formats is directly performed under each switch case statement so it would become painful to add other conditions to prevent the quotes from being generated when it's not supported by the the data encoding format for instance (ie: JSON). Let's centralize and simplify quotes handling by adding LOGQUOTE_START() and LOGQUOTE_END() helper macros. If a quotation is started and not explicitly ended, it will be automatically ended at the end of the current logformat node: LOGQUOTE_START() sets 'quote' variable to 1, this way LOGQUOTE_END() only prints the ending quote when needed. LOGQUOTE_END() is systematically called after each node switch-case (after each value). LOGQUOTE_START() does nothing if LOG_OPT_QUOTE isn't set, so does LOGQUOTE_END(). Some rare cases such as %hsl (list of captured headers) required special handling: in this case multiple quoted texts are generated for the same field value so explicit LOGQUOTE_START() + LOGQUOTE_END() combination was needed.
2024-01-10 13:14:25 -05:00
*/
MINOR: log: use LOG_VARTEXT_{START,END} to enclose text strings Rename LOGQUOTE_{START,END} macros to more generic LOG_VARTEXT_{START,END} in order to prepare for new encoding types that rely on specific treatment for variable-length texts. No functional change should be expected.
2024-04-16 05:17:55 -04:00
LOG_VARTEXT_END();
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
}
MINOR: log: automate string array construction in sess_build_logline() make it so string array construction is performed by dedicated macro helpers instead of manual char insertion between string members. The goal is to easily be able to support multiple forms of array construction depending on the data encoding format (raw, json..). Only %hrl and %hsl logformats are concerned.
2024-01-31 08:37:07 -05:00
LOG_STRARRAY_END();
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
}
break;
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-25 20:48:02 -04:00
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
case LOG_FMT_HDRRESPONS: // %hs
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
/* response header */
MINOR: log: do not dereference a null stream to access captures If the stream is null, let's simply not check captures. That's already done if there is no capture.
2018-09-05 09:28:07 -04:00
if (fe->nb_rsp_cap && s && s->res_cap) {
MINOR: log: use LOG_VARTEXT_{START,END} to enclose text strings Rename LOGQUOTE_{START,END} macros to more generic LOG_VARTEXT_{START,END} in order to prepare for new encoding types that rely on specific treatment for variable-length texts. No functional change should be expected.
2024-04-16 05:17:55 -04:00
LOG_VARTEXT_START();
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
LOGCHAR('{');
for (hdr = 0; hdr < fe->nb_rsp_cap; hdr++) {
if (hdr)
LOGCHAR('|');
MEDIUM: http: move header captures from http_txn to struct stream The header captures are now general purpose captures since tcp rules can use them to capture various contents. That removes a dependency on http_txn that appeared in some sample fetch functions and in the order by which captures and http_txn were allocated. Interestingly the reset of the header captures were done at too many places as http_init_txn() used to do it while it was done previously in every call place.
2015-04-03 16:16:32 -04:00
if (s->res_cap[hdr] != NULL) {
MEDIUM: log: add a new log format flag "E" The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It mostly makes sense to use it in the RFC5424 structured-data log formats. Example: log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 07:23:03 -05:00
ret = lf_encode_string(tmplog, dst + maxsize,
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
'#', hdr_encode_map, s->res_cap[hdr], ctx);
CLEANUP: log: remove unused checks for encode_{chunk,string} Thanks to 8226e92eb ("BUG/MINOR: tools/log: invalid encode_{chunk,string} usage"), we only need to check for NULL return value from encode_{chunk,string}() and escape_string() to know if the call failed.
2024-04-09 11:54:41 -04:00
if (ret == NULL)
BUG/MAJOR: log: possible segfault with logformat Possible zero-pointer deference in sess_log(). Checks of return values in sess_log() fix the issue. Fix bad computation in logformat_write_string(). This issue is 1.5-specific and was introduced just before 1.5-dev8. No backport is needed.
2012-03-26 11:52:55 -04:00
goto out;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
tmplog = ret;
BUG/MAJOR: log: possible segfault with logformat Possible zero-pointer deference in sess_log(). Checks of return values in sess_log() fix the issue. Fix bad computation in logformat_write_string(). This issue is 1.5-specific and was introduced just before 1.5-dev8. No backport is needed.
2012-03-26 11:52:55 -04:00
}
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
}
LOGCHAR('}');
}
break;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
case LOG_FMT_HDRRESPONSLIST: // %hsl
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
/* response header list */
MINOR: log: do not dereference a null stream to access captures If the stream is null, let's simply not check captures. That's already done if there is no capture.
2018-09-05 09:28:07 -04:00
if (fe->nb_rsp_cap && s && s->res_cap) {
MINOR: log: automate string array construction in sess_build_logline() make it so string array construction is performed by dedicated macro helpers instead of manual char insertion between string members. The goal is to easily be able to support multiple forms of array construction depending on the data encoding format (raw, json..). Only %hrl and %hsl logformats are concerned.
2024-01-31 08:37:07 -05:00
LOG_STRARRAY_START();
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
for (hdr = 0; hdr < fe->nb_rsp_cap; hdr++) {
if (hdr > 0)
MINOR: log: automate string array construction in sess_build_logline() make it so string array construction is performed by dedicated macro helpers instead of manual char insertion between string members. The goal is to easily be able to support multiple forms of array construction depending on the data encoding format (raw, json..). Only %hrl and %hsl logformats are concerned.
2024-01-31 08:37:07 -05:00
LOG_STRARRAY_NEXT();
MINOR: log: use LOG_VARTEXT_{START,END} to enclose text strings Rename LOGQUOTE_{START,END} macros to more generic LOG_VARTEXT_{START,END} in order to prepare for new encoding types that rely on specific treatment for variable-length texts. No functional change should be expected.
2024-04-16 05:17:55 -04:00
LOG_VARTEXT_START();
MEDIUM: http: move header captures from http_txn to struct stream The header captures are now general purpose captures since tcp rules can use them to capture various contents. That removes a dependency on http_txn that appeared in some sample fetch functions and in the order by which captures and http_txn were allocated. Interestingly the reset of the header captures were done at too many places as http_init_txn() used to do it while it was done previously in every call place.
2015-04-03 16:16:32 -04:00
if (s->res_cap[hdr] != NULL) {
MEDIUM: log: add a new log format flag "E" The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It mostly makes sense to use it in the RFC5424 structured-data log formats. Example: log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 07:23:03 -05:00
ret = lf_encode_string(tmplog, dst + maxsize,
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
'#', hdr_encode_map, s->res_cap[hdr], ctx);
CLEANUP: log: remove unused checks for encode_{chunk,string} Thanks to 8226e92eb ("BUG/MINOR: tools/log: invalid encode_{chunk,string} usage"), we only need to check for NULL return value from encode_{chunk,string}() and escape_string() to know if the call failed.
2024-04-09 11:54:41 -04:00
if (ret == NULL)
BUG/MAJOR: log: possible segfault with logformat Possible zero-pointer deference in sess_log(). Checks of return values in sess_log() fix the issue. Fix bad computation in logformat_write_string(). This issue is 1.5-specific and was introduced just before 1.5-dev8. No backport is needed.
2012-03-26 11:52:55 -04:00
goto out;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
tmplog = ret;
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
} else if (!(ctx->options & LOG_OPT_QUOTE))
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
LOGCHAR('-');
MINOR: log: use LOG_VARTEXT_{START,END} to enclose text strings Rename LOGQUOTE_{START,END} macros to more generic LOG_VARTEXT_{START,END} in order to prepare for new encoding types that rely on specific treatment for variable-length texts. No functional change should be expected.
2024-04-16 05:17:55 -04:00
/* Manually end variable text as we're emitting multiple
* texts at once
MINOR: log: simplify quotes handling in sess_build_logline() quotes building for some log formats is directly performed under each switch case statement so it would become painful to add other conditions to prevent the quotes from being generated when it's not supported by the the data encoding format for instance (ie: JSON). Let's centralize and simplify quotes handling by adding LOGQUOTE_START() and LOGQUOTE_END() helper macros. If a quotation is started and not explicitly ended, it will be automatically ended at the end of the current logformat node: LOGQUOTE_START() sets 'quote' variable to 1, this way LOGQUOTE_END() only prints the ending quote when needed. LOGQUOTE_END() is systematically called after each node switch-case (after each value). LOGQUOTE_START() does nothing if LOG_OPT_QUOTE isn't set, so does LOGQUOTE_END(). Some rare cases such as %hsl (list of captured headers) required special handling: in this case multiple quoted texts are generated for the same field value so explicit LOGQUOTE_START() + LOGQUOTE_END() combination was needed.
2024-01-10 13:14:25 -05:00
*/
MINOR: log: use LOG_VARTEXT_{START,END} to enclose text strings Rename LOGQUOTE_{START,END} macros to more generic LOG_VARTEXT_{START,END} in order to prepare for new encoding types that rely on specific treatment for variable-length texts. No functional change should be expected.
2024-04-16 05:17:55 -04:00
LOG_VARTEXT_END();
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
}
MINOR: log: automate string array construction in sess_build_logline() make it so string array construction is performed by dedicated macro helpers instead of manual char insertion between string members. The goal is to easily be able to support multiple forms of array construction depending on the data encoding format (raw, json..). Only %hrl and %hsl logformats are concerned.
2024-01-31 08:37:07 -05:00
LOG_STRARRAY_END();
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
}
break;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
case LOG_FMT_REQ: // %r
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
/* Request */
MINOR: log: use LOG_VARTEXT_{START,END} to enclose text strings Rename LOGQUOTE_{START,END} macros to more generic LOG_VARTEXT_{START,END} in order to prepare for new encoding types that rely on specific treatment for variable-length texts. No functional change should be expected.
2024-04-16 05:17:55 -04:00
LOG_VARTEXT_START();
BUG/MEDIUM: log: fix risk of segfault when logging HTTP fields in TCP mode David Torgerson faced an issue when using HTTP fields in log-format in TCP sections. The txn is dereferenced while it's null, resulting in a crash of the process. Such configurations are invalid and a warning is emitted, but nevertheless the process must not crash. As found by Lukas Tribus, this is a side effect of the split between the stream and the HTTP transaction that happened in 1.6, making it possible to have txn==NULL there. The fix consists in checking that txn is valid before using it. Fortunately it's easy since almost all places already used to check for the existence of a field (eg: txn->uri). This patch should be backported to 1.6.
2016-04-25 11:09:40 -04:00
uri = txn && txn->uri ? txn->uri : "<BADREQ>";
MEDIUM: log: add a new log format flag "E" The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It mostly makes sense to use it in the RFC5424 structured-data log formats. Example: log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 07:23:03 -05:00
ret = lf_encode_string(tmplog, dst + maxsize,
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
'#', url_encode_map, uri, ctx);
CLEANUP: log: remove unused checks for encode_{chunk,string} Thanks to 8226e92eb ("BUG/MINOR: tools/log: invalid encode_{chunk,string} usage"), we only need to check for NULL return value from encode_{chunk,string}() and escape_string() to know if the call failed.
2024-04-09 11:54:41 -04:00
if (ret == NULL)
BUG/MAJOR: log: possible segfault with logformat Possible zero-pointer deference in sess_log(). Checks of return values in sess_log() fix the issue. Fix bad computation in logformat_write_string(). This issue is 1.5-specific and was introduced just before 1.5-dev8. No backport is needed.
2012-03-26 11:52:55 -04:00
goto out;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
tmplog = ret;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
break;
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
MEDIUM: logs: Add HTTP request-line log format directives This commit adds 4 new log format variables that parse the HTTP Request-Line for more specific logging than "%r" provides. For example, we can parse the following HTTP Request-Line with these new variables: "GET /foo?bar=baz HTTP/1.1" - %HM: HTTP Method ("GET") - %HV: HTTP Version ("HTTP/1.1") - %HU: HTTP Request-URI ("/foo?bar=baz") - %HP: HTTP Request-URI without query string ("/foo")
2015-04-27 17:37:03 -04:00
case LOG_FMT_HTTP_PATH: // %HP
BUG/MEDIUM: log: fix risk of segfault when logging HTTP fields in TCP mode David Torgerson faced an issue when using HTTP fields in log-format in TCP sections. The txn is dereferenced while it's null, resulting in a crash of the process. Such configurations are invalid and a warning is emitted, but nevertheless the process must not crash. As found by Lukas Tribus, this is a side effect of the split between the stream and the HTTP transaction that happened in 1.6, making it possible to have txn==NULL there. The fix consists in checking that txn is valid before using it. Fortunately it's easy since almost all places already used to check for the existence of a field (eg: txn->uri). This patch should be backported to 1.6.
2016-04-25 11:09:40 -04:00
uri = txn && txn->uri ? txn->uri : "<BADREQ>";
MEDIUM: logs: Add HTTP request-line log format directives This commit adds 4 new log format variables that parse the HTTP Request-Line for more specific logging than "%r" provides. For example, we can parse the following HTTP Request-Line with these new variables: "GET /foo?bar=baz HTTP/1.1" - %HM: HTTP Method ("GET") - %HV: HTTP Version ("HTTP/1.1") - %HU: HTTP Request-URI ("/foo?bar=baz") - %HP: HTTP Request-URI without query string ("/foo")
2015-04-27 17:37:03 -04:00
MINOR: log: use LOG_VARTEXT_{START,END} to enclose text strings Rename LOGQUOTE_{START,END} macros to more generic LOG_VARTEXT_{START,END} in order to prepare for new encoding types that rely on specific treatment for variable-length texts. No functional change should be expected.
2024-04-16 05:17:55 -04:00
LOG_VARTEXT_START();
MEDIUM: logs: Add HTTP request-line log format directives This commit adds 4 new log format variables that parse the HTTP Request-Line for more specific logging than "%r" provides. For example, we can parse the following HTTP Request-Line with these new variables: "GET /foo?bar=baz HTTP/1.1" - %HM: HTTP Method ("GET") - %HV: HTTP Version ("HTTP/1.1") - %HU: HTTP Request-URI ("/foo?bar=baz") - %HP: HTTP Request-URI without query string ("/foo")
2015-04-27 17:37:03 -04:00
end = uri + strlen(uri);
// look for the first whitespace character
while (uri < end && !HTTP_IS_SPHT(*uri))
uri++;
// keep advancing past multiple spaces
while (uri < end && HTTP_IS_SPHT(*uri)) {
uri++; nspaces++;
}
// look for first space or question mark after url
spc = uri;
while (spc < end && *spc != '?' && !HTTP_IS_SPHT(*spc))
spc++;
BUG/MINOR: log: fix a typo that would cause %HP to log <BADREQ> Typo was introduced in 57bc891 ("BUG/MEDIUM: log: fix risk of segfault when logging HTTP fields in TCP mode") which inverted the condition in the test and caused <BADREQ> to be logged when using %HP. Signed-off-by: Nenad Merdanovic <nmerdan@anine.io>
2016-04-25 19:39:02 -04:00
if (!txn || !txn->uri || nspaces == 0) {
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
chunk.area = "<BADREQ>";
chunk.data = strlen("<BADREQ>");
MEDIUM: logs: Add HTTP request-line log format directives This commit adds 4 new log format variables that parse the HTTP Request-Line for more specific logging than "%r" provides. For example, we can parse the following HTTP Request-Line with these new variables: "GET /foo?bar=baz HTTP/1.1" - %HM: HTTP Method ("GET") - %HV: HTTP Version ("HTTP/1.1") - %HU: HTTP Request-URI ("/foo?bar=baz") - %HP: HTTP Request-URI without query string ("/foo")
2015-04-27 17:37:03 -04:00
} else {
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
chunk.area = uri;
chunk.data = spc - uri;
MEDIUM: logs: Add HTTP request-line log format directives This commit adds 4 new log format variables that parse the HTTP Request-Line for more specific logging than "%r" provides. For example, we can parse the following HTTP Request-Line with these new variables: "GET /foo?bar=baz HTTP/1.1" - %HM: HTTP Method ("GET") - %HV: HTTP Version ("HTTP/1.1") - %HU: HTTP Request-URI ("/foo?bar=baz") - %HP: HTTP Request-URI without query string ("/foo")
2015-04-27 17:37:03 -04:00
}
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_encode_chunk(tmplog, dst + maxsize, '#', url_encode_map, &chunk, ctx);
CLEANUP: log: remove unused checks for encode_{chunk,string} Thanks to 8226e92eb ("BUG/MINOR: tools/log: invalid encode_{chunk,string} usage"), we only need to check for NULL return value from encode_{chunk,string}() and escape_string() to know if the call failed.
2024-04-09 11:54:41 -04:00
if (ret == NULL)
MEDIUM: logs: Add HTTP request-line log format directives This commit adds 4 new log format variables that parse the HTTP Request-Line for more specific logging than "%r" provides. For example, we can parse the following HTTP Request-Line with these new variables: "GET /foo?bar=baz HTTP/1.1" - %HM: HTTP Method ("GET") - %HV: HTTP Version ("HTTP/1.1") - %HU: HTTP Request-URI ("/foo?bar=baz") - %HP: HTTP Request-URI without query string ("/foo")
2015-04-27 17:37:03 -04:00
goto out;
tmplog = ret;
break;
MINOR: log: Logging HTTP path only with %HPO This patch adds a new logging variable '%HPO' for logging HTTP path only (without query string) from relative or absolute URI. For example: log-format "hpo=%HPO hp=%HP hu=%HU hq=%HQ" GET /r/1 HTTP/1.1 => hpo=/r/1 hp=/r/1 hu=/r/1 hq= GET /r/2?q=2 HTTP/1.1 => hpo=/r/2 hp=/r/2 hu=/r/2?q=2 hq=?q=2 GET http://host/r/3 HTTP/1.1 => hpo=/r/3 hp=http://host/r/3 hu=http://host/r/3 hq= GET http://host/r/4?q=4 HTTP/1.1 => hpo=/r/4 hp=http://host/r/4 hu=http://host/r/4?q=4 hq=?q=4
2020-11-30 13:27:47 -05:00
case LOG_FMT_HTTP_PATH_ONLY: // %HPO
uri = txn && txn->uri ? txn->uri : "<BADREQ>";
MINOR: log: use LOG_VARTEXT_{START,END} to enclose text strings Rename LOGQUOTE_{START,END} macros to more generic LOG_VARTEXT_{START,END} in order to prepare for new encoding types that rely on specific treatment for variable-length texts. No functional change should be expected.
2024-04-16 05:17:55 -04:00
LOG_VARTEXT_START();
MINOR: log: Logging HTTP path only with %HPO This patch adds a new logging variable '%HPO' for logging HTTP path only (without query string) from relative or absolute URI. For example: log-format "hpo=%HPO hp=%HP hu=%HU hq=%HQ" GET /r/1 HTTP/1.1 => hpo=/r/1 hp=/r/1 hu=/r/1 hq= GET /r/2?q=2 HTTP/1.1 => hpo=/r/2 hp=/r/2 hu=/r/2?q=2 hq=?q=2 GET http://host/r/3 HTTP/1.1 => hpo=/r/3 hp=http://host/r/3 hu=http://host/r/3 hq= GET http://host/r/4?q=4 HTTP/1.1 => hpo=/r/4 hp=http://host/r/4 hu=http://host/r/4?q=4 hq=?q=4
2020-11-30 13:27:47 -05:00
end = uri + strlen(uri);
// look for the first whitespace character
while (uri < end && !HTTP_IS_SPHT(*uri))
uri++;
// keep advancing past multiple spaces
while (uri < end && HTTP_IS_SPHT(*uri)) {
uri++; nspaces++;
}
// look for first space after url
spc = uri;
while (spc < end && !HTTP_IS_SPHT(*spc))
spc++;
CLEANUP: Use ist2(const void*, size_t) whenever possible Refactoring performed with the following Coccinelle patch: @@ struct ist i; expression p, l; @@ - i.ptr = p; - i.len = l; + i = ist2(p, l);
2021-02-28 10:11:36 -05:00
path = ist2(uri, spc - uri);
MINOR: log: Logging HTTP path only with %HPO This patch adds a new logging variable '%HPO' for logging HTTP path only (without query string) from relative or absolute URI. For example: log-format "hpo=%HPO hp=%HP hu=%HU hq=%HQ" GET /r/1 HTTP/1.1 => hpo=/r/1 hp=/r/1 hu=/r/1 hq= GET /r/2?q=2 HTTP/1.1 => hpo=/r/2 hp=/r/2 hu=/r/2?q=2 hq=?q=2 GET http://host/r/3 HTTP/1.1 => hpo=/r/3 hp=http://host/r/3 hu=http://host/r/3 hq= GET http://host/r/4?q=4 HTTP/1.1 => hpo=/r/4 hp=http://host/r/4 hu=http://host/r/4?q=4 hq=?q=4
2020-11-30 13:27:47 -05:00
// extract relative path without query params from url
MINOR: http: use http uri parser for path Replace http_get_path by the http_uri_parser API. The new functions is renamed http_parse_path. Replace duplicated code for scheme and authority parsing by invocations to http_parse_scheme/authority. If no scheme is found for an URI detected as an absolute-uri/authority, consider it to be an authority format : no path will be found. For an absolute-uri or absolute-path, use the remaining of the string as the path. A new http_uri_parser state is declared to mark the path parsing as done.
2021-07-06 05:40:12 -04:00
parser = http_uri_parser_init(path);
path = iststop(http_parse_path(&parser), '?');
MINOR: log: Logging HTTP path only with %HPO This patch adds a new logging variable '%HPO' for logging HTTP path only (without query string) from relative or absolute URI. For example: log-format "hpo=%HPO hp=%HP hu=%HU hq=%HQ" GET /r/1 HTTP/1.1 => hpo=/r/1 hp=/r/1 hu=/r/1 hq= GET /r/2?q=2 HTTP/1.1 => hpo=/r/2 hp=/r/2 hu=/r/2?q=2 hq=?q=2 GET http://host/r/3 HTTP/1.1 => hpo=/r/3 hp=http://host/r/3 hu=http://host/r/3 hq= GET http://host/r/4?q=4 HTTP/1.1 => hpo=/r/4 hp=http://host/r/4 hu=http://host/r/4?q=4 hq=?q=4
2020-11-30 13:27:47 -05:00
if (!txn || !txn->uri || nspaces == 0) {
chunk.area = "<BADREQ>";
chunk.data = strlen("<BADREQ>");
} else {
chunk.area = path.ptr;
chunk.data = path.len;
}
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_encode_chunk(tmplog, dst + maxsize, '#', url_encode_map, &chunk, ctx);
CLEANUP: log: remove unused checks for encode_{chunk,string} Thanks to 8226e92eb ("BUG/MINOR: tools/log: invalid encode_{chunk,string} usage"), we only need to check for NULL return value from encode_{chunk,string}() and escape_string() to know if the call failed.
2024-04-09 11:54:41 -04:00
if (ret == NULL)
MINOR: log: Logging HTTP path only with %HPO This patch adds a new logging variable '%HPO' for logging HTTP path only (without query string) from relative or absolute URI. For example: log-format "hpo=%HPO hp=%HP hu=%HU hq=%HQ" GET /r/1 HTTP/1.1 => hpo=/r/1 hp=/r/1 hu=/r/1 hq= GET /r/2?q=2 HTTP/1.1 => hpo=/r/2 hp=/r/2 hu=/r/2?q=2 hq=?q=2 GET http://host/r/3 HTTP/1.1 => hpo=/r/3 hp=http://host/r/3 hu=http://host/r/3 hq= GET http://host/r/4?q=4 HTTP/1.1 => hpo=/r/4 hp=http://host/r/4 hu=http://host/r/4?q=4 hq=?q=4
2020-11-30 13:27:47 -05:00
goto out;
tmplog = ret;
break;
MINOR: log: Add log-format variable %HQ, to log HTTP query strings Since sample fetches are not always available in the response phase, this patch implements %HQ such that: GET /foo?bar=baz HTTP/1.0 ...would be logged as: ?bar=baz
2015-07-31 12:14:16 -04:00
case LOG_FMT_HTTP_QUERY: // %HQ
MINOR: log: use LOG_VARTEXT_{START,END} to enclose text strings Rename LOGQUOTE_{START,END} macros to more generic LOG_VARTEXT_{START,END} in order to prepare for new encoding types that rely on specific treatment for variable-length texts. No functional change should be expected.
2024-04-16 05:17:55 -04:00
LOG_VARTEXT_START();
MINOR: log: Add log-format variable %HQ, to log HTTP query strings Since sample fetches are not always available in the response phase, this patch implements %HQ such that: GET /foo?bar=baz HTTP/1.0 ...would be logged as: ?bar=baz
2015-07-31 12:14:16 -04:00
BUG/MEDIUM: log: fix risk of segfault when logging HTTP fields in TCP mode David Torgerson faced an issue when using HTTP fields in log-format in TCP sections. The txn is dereferenced while it's null, resulting in a crash of the process. Such configurations are invalid and a warning is emitted, but nevertheless the process must not crash. As found by Lukas Tribus, this is a side effect of the split between the stream and the HTTP transaction that happened in 1.6, making it possible to have txn==NULL there. The fix consists in checking that txn is valid before using it. Fortunately it's easy since almost all places already used to check for the existence of a field (eg: txn->uri). This patch should be backported to 1.6.
2016-04-25 11:09:40 -04:00
if (!txn || !txn->uri) {
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
chunk.area = "<BADREQ>";
chunk.data = strlen("<BADREQ>");
MINOR: log: Add log-format variable %HQ, to log HTTP query strings Since sample fetches are not always available in the response phase, this patch implements %HQ such that: GET /foo?bar=baz HTTP/1.0 ...would be logged as: ?bar=baz
2015-07-31 12:14:16 -04:00
} else {
uri = txn->uri;
end = uri + strlen(uri);
// look for the first question mark
while (uri < end && *uri != '?')
uri++;
qmark = uri;
// look for first space or question mark after url
while (uri < end && !HTTP_IS_SPHT(*uri))
uri++;
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
chunk.area = qmark;
chunk.data = uri - qmark;
MINOR: log: Add log-format variable %HQ, to log HTTP query strings Since sample fetches are not always available in the response phase, this patch implements %HQ such that: GET /foo?bar=baz HTTP/1.0 ...would be logged as: ?bar=baz
2015-07-31 12:14:16 -04:00
}
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_encode_chunk(tmplog, dst + maxsize, '#', url_encode_map, &chunk, ctx);
CLEANUP: log: remove unused checks for encode_{chunk,string} Thanks to 8226e92eb ("BUG/MINOR: tools/log: invalid encode_{chunk,string} usage"), we only need to check for NULL return value from encode_{chunk,string}() and escape_string() to know if the call failed.
2024-04-09 11:54:41 -04:00
if (ret == NULL)
MINOR: log: Add log-format variable %HQ, to log HTTP query strings Since sample fetches are not always available in the response phase, this patch implements %HQ such that: GET /foo?bar=baz HTTP/1.0 ...would be logged as: ?bar=baz
2015-07-31 12:14:16 -04:00
goto out;
tmplog = ret;
break;
MEDIUM: logs: Add HTTP request-line log format directives This commit adds 4 new log format variables that parse the HTTP Request-Line for more specific logging than "%r" provides. For example, we can parse the following HTTP Request-Line with these new variables: "GET /foo?bar=baz HTTP/1.1" - %HM: HTTP Method ("GET") - %HV: HTTP Version ("HTTP/1.1") - %HU: HTTP Request-URI ("/foo?bar=baz") - %HP: HTTP Request-URI without query string ("/foo")
2015-04-27 17:37:03 -04:00
case LOG_FMT_HTTP_URI: // %HU
BUG/MEDIUM: log: fix risk of segfault when logging HTTP fields in TCP mode David Torgerson faced an issue when using HTTP fields in log-format in TCP sections. The txn is dereferenced while it's null, resulting in a crash of the process. Such configurations are invalid and a warning is emitted, but nevertheless the process must not crash. As found by Lukas Tribus, this is a side effect of the split between the stream and the HTTP transaction that happened in 1.6, making it possible to have txn==NULL there. The fix consists in checking that txn is valid before using it. Fortunately it's easy since almost all places already used to check for the existence of a field (eg: txn->uri). This patch should be backported to 1.6.
2016-04-25 11:09:40 -04:00
uri = txn && txn->uri ? txn->uri : "<BADREQ>";
MEDIUM: logs: Add HTTP request-line log format directives This commit adds 4 new log format variables that parse the HTTP Request-Line for more specific logging than "%r" provides. For example, we can parse the following HTTP Request-Line with these new variables: "GET /foo?bar=baz HTTP/1.1" - %HM: HTTP Method ("GET") - %HV: HTTP Version ("HTTP/1.1") - %HU: HTTP Request-URI ("/foo?bar=baz") - %HP: HTTP Request-URI without query string ("/foo")
2015-04-27 17:37:03 -04:00
MINOR: log: use LOG_VARTEXT_{START,END} to enclose text strings Rename LOGQUOTE_{START,END} macros to more generic LOG_VARTEXT_{START,END} in order to prepare for new encoding types that rely on specific treatment for variable-length texts. No functional change should be expected.
2024-04-16 05:17:55 -04:00
LOG_VARTEXT_START();
MEDIUM: logs: Add HTTP request-line log format directives This commit adds 4 new log format variables that parse the HTTP Request-Line for more specific logging than "%r" provides. For example, we can parse the following HTTP Request-Line with these new variables: "GET /foo?bar=baz HTTP/1.1" - %HM: HTTP Method ("GET") - %HV: HTTP Version ("HTTP/1.1") - %HU: HTTP Request-URI ("/foo?bar=baz") - %HP: HTTP Request-URI without query string ("/foo")
2015-04-27 17:37:03 -04:00
end = uri + strlen(uri);
// look for the first whitespace character
while (uri < end && !HTTP_IS_SPHT(*uri))
uri++;
// keep advancing past multiple spaces
while (uri < end && HTTP_IS_SPHT(*uri)) {
uri++; nspaces++;
}
// look for first space after url
spc = uri;
while (spc < end && !HTTP_IS_SPHT(*spc))
spc++;
BUG/MEDIUM: log: fix risk of segfault when logging HTTP fields in TCP mode David Torgerson faced an issue when using HTTP fields in log-format in TCP sections. The txn is dereferenced while it's null, resulting in a crash of the process. Such configurations are invalid and a warning is emitted, but nevertheless the process must not crash. As found by Lukas Tribus, this is a side effect of the split between the stream and the HTTP transaction that happened in 1.6, making it possible to have txn==NULL there. The fix consists in checking that txn is valid before using it. Fortunately it's easy since almost all places already used to check for the existence of a field (eg: txn->uri). This patch should be backported to 1.6.
2016-04-25 11:09:40 -04:00
if (!txn || !txn->uri || nspaces == 0) {
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
chunk.area = "<BADREQ>";
chunk.data = strlen("<BADREQ>");
MEDIUM: logs: Add HTTP request-line log format directives This commit adds 4 new log format variables that parse the HTTP Request-Line for more specific logging than "%r" provides. For example, we can parse the following HTTP Request-Line with these new variables: "GET /foo?bar=baz HTTP/1.1" - %HM: HTTP Method ("GET") - %HV: HTTP Version ("HTTP/1.1") - %HU: HTTP Request-URI ("/foo?bar=baz") - %HP: HTTP Request-URI without query string ("/foo")
2015-04-27 17:37:03 -04:00
} else {
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
chunk.area = uri;
chunk.data = spc - uri;
MEDIUM: logs: Add HTTP request-line log format directives This commit adds 4 new log format variables that parse the HTTP Request-Line for more specific logging than "%r" provides. For example, we can parse the following HTTP Request-Line with these new variables: "GET /foo?bar=baz HTTP/1.1" - %HM: HTTP Method ("GET") - %HV: HTTP Version ("HTTP/1.1") - %HU: HTTP Request-URI ("/foo?bar=baz") - %HP: HTTP Request-URI without query string ("/foo")
2015-04-27 17:37:03 -04:00
}
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_encode_chunk(tmplog, dst + maxsize, '#', url_encode_map, &chunk, ctx);
CLEANUP: log: remove unused checks for encode_{chunk,string} Thanks to 8226e92eb ("BUG/MINOR: tools/log: invalid encode_{chunk,string} usage"), we only need to check for NULL return value from encode_{chunk,string}() and escape_string() to know if the call failed.
2024-04-09 11:54:41 -04:00
if (ret == NULL)
MEDIUM: logs: Add HTTP request-line log format directives This commit adds 4 new log format variables that parse the HTTP Request-Line for more specific logging than "%r" provides. For example, we can parse the following HTTP Request-Line with these new variables: "GET /foo?bar=baz HTTP/1.1" - %HM: HTTP Method ("GET") - %HV: HTTP Version ("HTTP/1.1") - %HU: HTTP Request-URI ("/foo?bar=baz") - %HP: HTTP Request-URI without query string ("/foo")
2015-04-27 17:37:03 -04:00
goto out;
tmplog = ret;
break;
case LOG_FMT_HTTP_METHOD: // %HM
BUG/MEDIUM: log: fix risk of segfault when logging HTTP fields in TCP mode David Torgerson faced an issue when using HTTP fields in log-format in TCP sections. The txn is dereferenced while it's null, resulting in a crash of the process. Such configurations are invalid and a warning is emitted, but nevertheless the process must not crash. As found by Lukas Tribus, this is a side effect of the split between the stream and the HTTP transaction that happened in 1.6, making it possible to have txn==NULL there. The fix consists in checking that txn is valid before using it. Fortunately it's easy since almost all places already used to check for the existence of a field (eg: txn->uri). This patch should be backported to 1.6.
2016-04-25 11:09:40 -04:00
uri = txn && txn->uri ? txn->uri : "<BADREQ>";
MINOR: log: use LOG_VARTEXT_{START,END} to enclose text strings Rename LOGQUOTE_{START,END} macros to more generic LOG_VARTEXT_{START,END} in order to prepare for new encoding types that rely on specific treatment for variable-length texts. No functional change should be expected.
2024-04-16 05:17:55 -04:00
LOG_VARTEXT_START();
MEDIUM: logs: Add HTTP request-line log format directives This commit adds 4 new log format variables that parse the HTTP Request-Line for more specific logging than "%r" provides. For example, we can parse the following HTTP Request-Line with these new variables: "GET /foo?bar=baz HTTP/1.1" - %HM: HTTP Method ("GET") - %HV: HTTP Version ("HTTP/1.1") - %HU: HTTP Request-URI ("/foo?bar=baz") - %HP: HTTP Request-URI without query string ("/foo")
2015-04-27 17:37:03 -04:00
end = uri + strlen(uri);
// look for the first whitespace character
spc = uri;
while (spc < end && !HTTP_IS_SPHT(*spc))
spc++;
if (spc == end) { // odd case, we have txn->uri, but we only got a verb
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
chunk.area = "<BADREQ>";
chunk.data = strlen("<BADREQ>");
MEDIUM: logs: Add HTTP request-line log format directives This commit adds 4 new log format variables that parse the HTTP Request-Line for more specific logging than "%r" provides. For example, we can parse the following HTTP Request-Line with these new variables: "GET /foo?bar=baz HTTP/1.1" - %HM: HTTP Method ("GET") - %HV: HTTP Version ("HTTP/1.1") - %HU: HTTP Request-URI ("/foo?bar=baz") - %HP: HTTP Request-URI without query string ("/foo")
2015-04-27 17:37:03 -04:00
} else {
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
chunk.area = uri;
chunk.data = spc - uri;
MEDIUM: logs: Add HTTP request-line log format directives This commit adds 4 new log format variables that parse the HTTP Request-Line for more specific logging than "%r" provides. For example, we can parse the following HTTP Request-Line with these new variables: "GET /foo?bar=baz HTTP/1.1" - %HM: HTTP Method ("GET") - %HV: HTTP Version ("HTTP/1.1") - %HU: HTTP Request-URI ("/foo?bar=baz") - %HP: HTTP Request-URI without query string ("/foo")
2015-04-27 17:37:03 -04:00
}
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_encode_chunk(tmplog, dst + maxsize, '#', url_encode_map, &chunk, ctx);
CLEANUP: log: remove unused checks for encode_{chunk,string} Thanks to 8226e92eb ("BUG/MINOR: tools/log: invalid encode_{chunk,string} usage"), we only need to check for NULL return value from encode_{chunk,string}() and escape_string() to know if the call failed.
2024-04-09 11:54:41 -04:00
if (ret == NULL)
MEDIUM: logs: Add HTTP request-line log format directives This commit adds 4 new log format variables that parse the HTTP Request-Line for more specific logging than "%r" provides. For example, we can parse the following HTTP Request-Line with these new variables: "GET /foo?bar=baz HTTP/1.1" - %HM: HTTP Method ("GET") - %HV: HTTP Version ("HTTP/1.1") - %HU: HTTP Request-URI ("/foo?bar=baz") - %HP: HTTP Request-URI without query string ("/foo")
2015-04-27 17:37:03 -04:00
goto out;
tmplog = ret;
break;
case LOG_FMT_HTTP_VERSION: // %HV
BUG/MEDIUM: log: fix risk of segfault when logging HTTP fields in TCP mode David Torgerson faced an issue when using HTTP fields in log-format in TCP sections. The txn is dereferenced while it's null, resulting in a crash of the process. Such configurations are invalid and a warning is emitted, but nevertheless the process must not crash. As found by Lukas Tribus, this is a side effect of the split between the stream and the HTTP transaction that happened in 1.6, making it possible to have txn==NULL there. The fix consists in checking that txn is valid before using it. Fortunately it's easy since almost all places already used to check for the existence of a field (eg: txn->uri). This patch should be backported to 1.6.
2016-04-25 11:09:40 -04:00
uri = txn && txn->uri ? txn->uri : "<BADREQ>";
MINOR: log: use LOG_VARTEXT_{START,END} to enclose text strings Rename LOGQUOTE_{START,END} macros to more generic LOG_VARTEXT_{START,END} in order to prepare for new encoding types that rely on specific treatment for variable-length texts. No functional change should be expected.
2024-04-16 05:17:55 -04:00
LOG_VARTEXT_START();
MEDIUM: logs: Add HTTP request-line log format directives This commit adds 4 new log format variables that parse the HTTP Request-Line for more specific logging than "%r" provides. For example, we can parse the following HTTP Request-Line with these new variables: "GET /foo?bar=baz HTTP/1.1" - %HM: HTTP Method ("GET") - %HV: HTTP Version ("HTTP/1.1") - %HU: HTTP Request-URI ("/foo?bar=baz") - %HP: HTTP Request-URI without query string ("/foo")
2015-04-27 17:37:03 -04:00
end = uri + strlen(uri);
// look for the first whitespace character
while (uri < end && !HTTP_IS_SPHT(*uri))
uri++;
// keep advancing past multiple spaces
while (uri < end && HTTP_IS_SPHT(*uri)) {
uri++; nspaces++;
}
// look for the next whitespace character
while (uri < end && !HTTP_IS_SPHT(*uri))
uri++;
// keep advancing past multiple spaces
while (uri < end && HTTP_IS_SPHT(*uri))
uri++;
BUG/MEDIUM: log: fix risk of segfault when logging HTTP fields in TCP mode David Torgerson faced an issue when using HTTP fields in log-format in TCP sections. The txn is dereferenced while it's null, resulting in a crash of the process. Such configurations are invalid and a warning is emitted, but nevertheless the process must not crash. As found by Lukas Tribus, this is a side effect of the split between the stream and the HTTP transaction that happened in 1.6, making it possible to have txn==NULL there. The fix consists in checking that txn is valid before using it. Fortunately it's easy since almost all places already used to check for the existence of a field (eg: txn->uri). This patch should be backported to 1.6.
2016-04-25 11:09:40 -04:00
if (!txn || !txn->uri || nspaces == 0) {
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
chunk.area = "<BADREQ>";
chunk.data = strlen("<BADREQ>");
MEDIUM: logs: Add HTTP request-line log format directives This commit adds 4 new log format variables that parse the HTTP Request-Line for more specific logging than "%r" provides. For example, we can parse the following HTTP Request-Line with these new variables: "GET /foo?bar=baz HTTP/1.1" - %HM: HTTP Method ("GET") - %HV: HTTP Version ("HTTP/1.1") - %HU: HTTP Request-URI ("/foo?bar=baz") - %HP: HTTP Request-URI without query string ("/foo")
2015-04-27 17:37:03 -04:00
} else if (uri == end) {
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
chunk.area = "HTTP/0.9";
chunk.data = strlen("HTTP/0.9");
MEDIUM: logs: Add HTTP request-line log format directives This commit adds 4 new log format variables that parse the HTTP Request-Line for more specific logging than "%r" provides. For example, we can parse the following HTTP Request-Line with these new variables: "GET /foo?bar=baz HTTP/1.1" - %HM: HTTP Method ("GET") - %HV: HTTP Version ("HTTP/1.1") - %HU: HTTP Request-URI ("/foo?bar=baz") - %HP: HTTP Request-URI without query string ("/foo")
2015-04-27 17:37:03 -04:00
} else {
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
chunk.area = uri;
chunk.data = end - uri;
MEDIUM: logs: Add HTTP request-line log format directives This commit adds 4 new log format variables that parse the HTTP Request-Line for more specific logging than "%r" provides. For example, we can parse the following HTTP Request-Line with these new variables: "GET /foo?bar=baz HTTP/1.1" - %HM: HTTP Method ("GET") - %HV: HTTP Version ("HTTP/1.1") - %HU: HTTP Request-URI ("/foo?bar=baz") - %HP: HTTP Request-URI without query string ("/foo")
2015-04-27 17:37:03 -04:00
}
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_encode_chunk(tmplog, dst + maxsize, '#', url_encode_map, &chunk, ctx);
CLEANUP: log: remove unused checks for encode_{chunk,string} Thanks to 8226e92eb ("BUG/MINOR: tools/log: invalid encode_{chunk,string} usage"), we only need to check for NULL return value from encode_{chunk,string}() and escape_string() to know if the call failed.
2024-04-09 11:54:41 -04:00
if (ret == NULL)
MEDIUM: logs: Add HTTP request-line log format directives This commit adds 4 new log format variables that parse the HTTP Request-Line for more specific logging than "%r" provides. For example, we can parse the following HTTP Request-Line with these new variables: "GET /foo?bar=baz HTTP/1.1" - %HM: HTTP Method ("GET") - %HV: HTTP Version ("HTTP/1.1") - %HU: HTTP Request-URI ("/foo?bar=baz") - %HP: HTTP Request-URI without query string ("/foo")
2015-04-27 17:37:03 -04:00
goto out;
tmplog = ret;
break;
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
case LOG_FMT_COUNTER: // %rt
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
if (ctx->options & LOG_OPT_HEXA) {
OPTIM: log: use lf_buildctx's buffer instead of temporary stack buffers Now that lf_buildctx isn't pushed on the stack anymore, let's take this opportunity to store a small buffer of 256 bytes within it, and then use this buffer as general purpose buffer to build fixed-length strings that are then printed using lf_{raw}text() function. By doing so we stop relying on temporary stack buffers.
2024-05-02 10:48:56 -04:00
iret = snprintf(ctx->_buf, sizeof(ctx->_buf), "%04X", uniq_id);
BUG/MINOR: log: invalid snprintf() usage in sess_build_logline() According to snprintf() man page: The functions snprintf() and vsnprintf() do not write more than size bytes (including the terminating null byte ('\0')). If the output was truncated due to this limit, then the return value is the number of characters (excluding the terminating null byte) which would have been written to the final string if enough space had been available. Thus, a return value of size or more means that the output was truncated. However, in sess_build_logline(), each time we need to check the return value of snprintf(), here is how we proceed: iret = snprintf(tmplog, max, ...); if (iret < 0 || iret > max) // error // success tmplog += iret; Here is the issue: if snprintf() lacks 1 byte space to write the terminating NULL byte, it will return max. Which means in this case that we fail to know that snprintf() truncated the output in reality, and we still add iret to tmplog pointer. Considering sess_build_logline() should NOT write more than <maxsize> bytes (including the terminating NULL byte) as per the function description, in this case the function would write <maxsize>+1 byte (to write the terminating NULL byte upon return), which may lead to invalid write if <dst> was meant to hold <maxsize> bytes at maximum. Hopefully, this bug wasn't triggered so far because sess_build_logline() is called with logline as <dst> argument and <global.max_syslog_len> as <maxsize> argument, logline being initialized with 1 extra byte upon startup. But we better fix this to comply with the function description and prevent any side-effect since some sess_build_logline() helpers may assume that 'tmplog-dst < maxsize' is always true. Also sess_build_logline() users probably don't expect NULL-byte to be accounted for in the produced logline length. This should be backported to all stable versions. [ada: for backports, the patch needs to be applied manually because of c6a713842 ("MINOR: log: simplify last_isspace in sess_build_logline()")]
2024-04-09 09:59:42 -04:00
if (iret < 0 || iret >= dst + maxsize - tmplog)
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
goto out;
OPTIM: log: use lf_buildctx's buffer instead of temporary stack buffers Now that lf_buildctx isn't pushed on the stack anymore, let's take this opportunity to store a small buffer of 256 bytes within it, and then use this buffer as general purpose buffer to build fixed-length strings that are then printed using lf_{raw}text() function. By doing so we stop relying on temporary stack buffers.
2024-05-02 10:48:56 -04:00
ret = lf_rawtext(tmplog, ctx->_buf, dst + maxsize - tmplog, ctx);
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
} else {
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_int(tmplog, dst + maxsize - tmplog, uniq_id, ctx, LF_INT_LTOA);
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
}
MEDIUM: log: write raw strings using lf_rawtext() Make use of the previous commit to print strings that should not be modified. For instance, when +X option is provided, we have to print numerical values in ASCII HEX form. For that, we used snprintf() to output the result to the log output buffer directly, but now we build the string in a temporary buffer of fixed-size and then print it using lf_rawtext() which will take care of encoding options. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice, we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-03-27 05:28:59 -04:00
if (ret == NULL)
goto out;
tmplog = ret;
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
break;
MINOR: log: add a new field "%lc" to implement a per-frontend log counter Sometimes it would be convenient to have a log counter so that from a log server we know whether some logs were lost or not. The frontend's log counter serves exactly this purpose. It's incremented each time a traffic log is produced. If a log is disabled using "http-request set-log-level silent", the counter will not be incremented. However, admin logs are not accounted for. Also, if logs are filtered out before being sent to the server because of a minimum level set on the log line, the counter will be increased anyway. The counter is 32-bit, so it will wrap, but that's not an issue considering that 4 billion logs are rarely in the same file, let alone close to each other.
2014-08-28 09:03:15 -04:00
case LOG_FMT_LOGCNT: // %lc
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
if (ctx->options & LOG_OPT_HEXA) {
OPTIM: log: use lf_buildctx's buffer instead of temporary stack buffers Now that lf_buildctx isn't pushed on the stack anymore, let's take this opportunity to store a small buffer of 256 bytes within it, and then use this buffer as general purpose buffer to build fixed-length strings that are then printed using lf_{raw}text() function. By doing so we stop relying on temporary stack buffers.
2024-05-02 10:48:56 -04:00
iret = snprintf(ctx->_buf, sizeof(ctx->_buf), "%04X", fe->log_count);
BUG/MINOR: log: invalid snprintf() usage in sess_build_logline() According to snprintf() man page: The functions snprintf() and vsnprintf() do not write more than size bytes (including the terminating null byte ('\0')). If the output was truncated due to this limit, then the return value is the number of characters (excluding the terminating null byte) which would have been written to the final string if enough space had been available. Thus, a return value of size or more means that the output was truncated. However, in sess_build_logline(), each time we need to check the return value of snprintf(), here is how we proceed: iret = snprintf(tmplog, max, ...); if (iret < 0 || iret > max) // error // success tmplog += iret; Here is the issue: if snprintf() lacks 1 byte space to write the terminating NULL byte, it will return max. Which means in this case that we fail to know that snprintf() truncated the output in reality, and we still add iret to tmplog pointer. Considering sess_build_logline() should NOT write more than <maxsize> bytes (including the terminating NULL byte) as per the function description, in this case the function would write <maxsize>+1 byte (to write the terminating NULL byte upon return), which may lead to invalid write if <dst> was meant to hold <maxsize> bytes at maximum. Hopefully, this bug wasn't triggered so far because sess_build_logline() is called with logline as <dst> argument and <global.max_syslog_len> as <maxsize> argument, logline being initialized with 1 extra byte upon startup. But we better fix this to comply with the function description and prevent any side-effect since some sess_build_logline() helpers may assume that 'tmplog-dst < maxsize' is always true. Also sess_build_logline() users probably don't expect NULL-byte to be accounted for in the produced logline length. This should be backported to all stable versions. [ada: for backports, the patch needs to be applied manually because of c6a713842 ("MINOR: log: simplify last_isspace in sess_build_logline()")]
2024-04-09 09:59:42 -04:00
if (iret < 0 || iret >= dst + maxsize - tmplog)
MINOR: log: add a new field "%lc" to implement a per-frontend log counter Sometimes it would be convenient to have a log counter so that from a log server we know whether some logs were lost or not. The frontend's log counter serves exactly this purpose. It's incremented each time a traffic log is produced. If a log is disabled using "http-request set-log-level silent", the counter will not be incremented. However, admin logs are not accounted for. Also, if logs are filtered out before being sent to the server because of a minimum level set on the log line, the counter will be increased anyway. The counter is 32-bit, so it will wrap, but that's not an issue considering that 4 billion logs are rarely in the same file, let alone close to each other.
2014-08-28 09:03:15 -04:00
goto out;
OPTIM: log: use lf_buildctx's buffer instead of temporary stack buffers Now that lf_buildctx isn't pushed on the stack anymore, let's take this opportunity to store a small buffer of 256 bytes within it, and then use this buffer as general purpose buffer to build fixed-length strings that are then printed using lf_{raw}text() function. By doing so we stop relying on temporary stack buffers.
2024-05-02 10:48:56 -04:00
ret = lf_rawtext(tmplog, ctx->_buf, dst + maxsize - tmplog, ctx);
MINOR: log: add a new field "%lc" to implement a per-frontend log counter Sometimes it would be convenient to have a log counter so that from a log server we know whether some logs were lost or not. The frontend's log counter serves exactly this purpose. It's incremented each time a traffic log is produced. If a log is disabled using "http-request set-log-level silent", the counter will not be incremented. However, admin logs are not accounted for. Also, if logs are filtered out before being sent to the server because of a minimum level set on the log line, the counter will be increased anyway. The counter is 32-bit, so it will wrap, but that's not an issue considering that 4 billion logs are rarely in the same file, let alone close to each other.
2014-08-28 09:03:15 -04:00
} else {
MINOR: log: add lf_int() wrapper to print integers Wrap ltoa(), lltoa(), ultoa() and utoa_pad() functions that are used by sess_build_logline() to print numerical values by implementing a dedicated helper named lf_int() that takes <dft_hld> as argument to know how to write the integer by default (when no encoding is specified). LF_INT_UTOA_PAD_4 is used to emulate utoa_pad(x, 4) since it's found only once under sess_build_logline(), thus there is no need to pass an extra parameter to lf_int() function.
2024-03-27 05:17:11 -04:00
ret = lf_int(tmplog, dst + maxsize - tmplog, fe->log_count,
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ctx, LF_INT_ULTOA);
MINOR: log: add a new field "%lc" to implement a per-frontend log counter Sometimes it would be convenient to have a log counter so that from a log server we know whether some logs were lost or not. The frontend's log counter serves exactly this purpose. It's incremented each time a traffic log is produced. If a log is disabled using "http-request set-log-level silent", the counter will not be incremented. However, admin logs are not accounted for. Also, if logs are filtered out before being sent to the server because of a minimum level set on the log line, the counter will be increased anyway. The counter is 32-bit, so it will wrap, but that's not an issue considering that 4 billion logs are rarely in the same file, let alone close to each other.
2014-08-28 09:03:15 -04:00
}
MEDIUM: log: write raw strings using lf_rawtext() Make use of the previous commit to print strings that should not be modified. For instance, when +X option is provided, we have to print numerical values in ASCII HEX form. For that, we used snprintf() to output the result to the log output buffer directly, but now we build the string in a temporary buffer of fixed-size and then print it using lf_rawtext() which will take care of encoding options. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice, we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-03-27 05:28:59 -04:00
if (ret == NULL)
goto out;
tmplog = ret;
MINOR: log: add a new field "%lc" to implement a per-frontend log counter Sometimes it would be convenient to have a log counter so that from a log server we know whether some logs were lost or not. The frontend's log counter serves exactly this purpose. It's incremented each time a traffic log is produced. If a log is disabled using "http-request set-log-level silent", the counter will not be incremented. However, admin logs are not accounted for. Also, if logs are filtered out before being sent to the server because of a minimum level set on the log line, the counter will be increased anyway. The counter is 32-bit, so it will wrap, but that's not an issue considering that 4 billion logs are rarely in the same file, let alone close to each other.
2014-08-28 09:03:15 -04:00
break;
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
case LOG_FMT_HOSTNAME: // %H
src = hostname;
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_text(tmplog, src, dst + maxsize - tmplog, ctx);
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
if (ret == NULL)
goto out;
tmplog = ret;
break;
case LOG_FMT_PID: // %pid
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
if (ctx->options & LOG_OPT_HEXA) {
OPTIM: log: use lf_buildctx's buffer instead of temporary stack buffers Now that lf_buildctx isn't pushed on the stack anymore, let's take this opportunity to store a small buffer of 256 bytes within it, and then use this buffer as general purpose buffer to build fixed-length strings that are then printed using lf_{raw}text() function. By doing so we stop relying on temporary stack buffers.
2024-05-02 10:48:56 -04:00
iret = snprintf(ctx->_buf, sizeof(ctx->_buf), "%04X", pid);
BUG/MINOR: log: invalid snprintf() usage in sess_build_logline() According to snprintf() man page: The functions snprintf() and vsnprintf() do not write more than size bytes (including the terminating null byte ('\0')). If the output was truncated due to this limit, then the return value is the number of characters (excluding the terminating null byte) which would have been written to the final string if enough space had been available. Thus, a return value of size or more means that the output was truncated. However, in sess_build_logline(), each time we need to check the return value of snprintf(), here is how we proceed: iret = snprintf(tmplog, max, ...); if (iret < 0 || iret > max) // error // success tmplog += iret; Here is the issue: if snprintf() lacks 1 byte space to write the terminating NULL byte, it will return max. Which means in this case that we fail to know that snprintf() truncated the output in reality, and we still add iret to tmplog pointer. Considering sess_build_logline() should NOT write more than <maxsize> bytes (including the terminating NULL byte) as per the function description, in this case the function would write <maxsize>+1 byte (to write the terminating NULL byte upon return), which may lead to invalid write if <dst> was meant to hold <maxsize> bytes at maximum. Hopefully, this bug wasn't triggered so far because sess_build_logline() is called with logline as <dst> argument and <global.max_syslog_len> as <maxsize> argument, logline being initialized with 1 extra byte upon startup. But we better fix this to comply with the function description and prevent any side-effect since some sess_build_logline() helpers may assume that 'tmplog-dst < maxsize' is always true. Also sess_build_logline() users probably don't expect NULL-byte to be accounted for in the produced logline length. This should be backported to all stable versions. [ada: for backports, the patch needs to be applied manually because of c6a713842 ("MINOR: log: simplify last_isspace in sess_build_logline()")]
2024-04-09 09:59:42 -04:00
if (iret < 0 || iret >= dst + maxsize - tmplog)
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
goto out;
OPTIM: log: use lf_buildctx's buffer instead of temporary stack buffers Now that lf_buildctx isn't pushed on the stack anymore, let's take this opportunity to store a small buffer of 256 bytes within it, and then use this buffer as general purpose buffer to build fixed-length strings that are then printed using lf_{raw}text() function. By doing so we stop relying on temporary stack buffers.
2024-05-02 10:48:56 -04:00
ret = lf_rawtext(tmplog, ctx->_buf, dst + maxsize - tmplog, ctx);
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
} else {
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_int(tmplog, dst + maxsize - tmplog, pid, ctx, LF_INT_LTOA);
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
}
MEDIUM: log: write raw strings using lf_rawtext() Make use of the previous commit to print strings that should not be modified. For instance, when +X option is provided, we have to print numerical values in ASCII HEX form. For that, we used snprintf() to output the result to the log output buffer directly, but now we build the string in a temporary buffer of fixed-size and then print it using lf_rawtext() which will take care of encoding options. Because of this patch, we add a little overhead because we first generate the text into a temporary variable and then use lf_rawtext() to print it. Thus we have a double-copy, and this could have some performance implications that were not yet evaluated. Due to the small number of bytes that can end up being copied twice, we could be lucky and have no visible performance impact, but if we happen to see a significant impact, it could be useful to add a passthrough mechanism (to keep historical behavior) when no encoding is involved.
2024-03-27 05:28:59 -04:00
if (ret == NULL)
goto out;
tmplog = ret;
MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid %Fi: Frontend IP %Fp: Frontend Port %Si: Server IP %Sp: Server Port %Ts: Timestamp %rt: HTTP request counter %H: hostname %pid: PID +X: Hexadecimal represenation The +X mode in logformat displays hexadecimal for the following flags %Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid rename logformat_write_string() to lf_text() Optimize size computation
2012-04-05 12:02:55 -04:00
break;
MEDIUM: log: Unique ID The Unique ID, is an ID generated with several informations. You can use a log-format string to customize it, with the "unique-id-format" keyword, and insert it in the request header, with the "unique-id-header" keyword.
2012-03-12 07:48:57 -04:00
case LOG_FMT_UNIQUEID: // %ID
BUG/MEDIUM: unique_id: junk in log on empty unique_id When a request fail, the unique_id was allocated but not generated. The string was not initialized and junk was printed in the log with %ID. This patch changes the behavior of the unique_id. The unique_id is now generated when a request failed. This bug was reported by Patrick Hemmer.
2013-08-28 09:44:19 -04:00
ret = NULL;
MEDIUM: stream: Make the `unique_id` member of `struct stream` a `struct ist` The `unique_id` member of `struct stream` now is a `struct ist`.
2020-03-05 14:19:02 -05:00
if (s)
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_text_len(tmplog, s->unique_id.ptr, s->unique_id.len, maxsize - (tmplog - dst), ctx);
MEDIUM: stream: Make the `unique_id` member of `struct stream` a `struct ist` The `unique_id` member of `struct stream` now is a `struct ist`.
2020-03-05 14:19:02 -05:00
else
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
ret = lf_text_len(tmplog, NULL, 0, maxsize - (tmplog - dst), ctx);
MEDIUM: log: Unique ID The Unique ID, is an ID generated with several informations. You can use a log-format string to customize it, with the "unique-id-format" keyword, and insert it in the request header, with the "unique-id-header" keyword.
2012-03-12 07:48:57 -04:00
if (ret == NULL)
goto out;
tmplog = ret;
break;
MINOR: log: provide log origin in logformat expressions using '%OG' '%OG' logformat alias may be used to report the log origin (when/where) that triggered log generation using sess_build_logline(). Possible values are: - "sess_error": log was generated during session error handling - "sess_killed": log was generated during session abortion (killed embryonic session) - "txn_accept": log was generated right after frontend conn was accepted - "txn_request": log was generated after client request was received - "txn_connect": log was generated after backend connection establishment - "txn_response": log was generated during server response handling - "txn_close": log was generated at the final txn step, before closing - "unspec": unknown or not specified Documentation was updated.
2024-05-06 08:13:11 -04:00
case LOG_FMT_ORIGIN: // %OG
ret = lf_text(tmplog, log_orig_to_str(log_orig),
dst + maxsize - tmplog, ctx);
if (ret == NULL)
goto out;
tmplog = ret;
break;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
}
MEDIUM: log: carry tag context in logformat node This is a pretty simple patch despite requiring to make some visible changes in the code: When parsing a logformat string, log tags (ie: '%tag', AKA log tags) are turned into logformat nodes with their type set to the type of the corresponding logformat_tag element which was matched by name. Thus, when "compiling" a logformat tag, we only keep a reference to the tag type from the original logformat_tag. For example, for "%B" log tag, we have the following logformat_tag element: { .name = "B", .type = LOG_FMT_BYTES, .mode = PR_MODE_TCP, .lw = LW_BYTES, .config_callback = NULL } When parsing "%B" string, we search for a matching logformat tag inside logformat_tags[] array using the provided name, once we find a matching element, we craft a logformat node whose type will be LOG_FMT_BYTES, but from the node itself, we no longer have access to other informations that are set in the logformat_tag struct element. Thus from a logformat_node resulting from a log tag, with current implementation, we cannot easily get back to matching logformat_tag struct element as it would require us to scan the whole logformat_tags array at runtime using node->type to find the matching element. Let's take a simpler path and consider all tag-specific LOG_FMT_* subtypes as being part of the same logformat node type: LOG_FMT_TAG. Thanks to that, we're now able to distinguish logformat nodes made from logformat tag from other logformat nodes, and link them to their corresponding logformat_tag element from logformat_tags[] array. All it costs is a simple indirection and an extra pointer in logformat_node struct. While at it, all LOG_FMT_* types related to logformat tags were moved inside log.c as they have no use outside of it since they are simply lookup indexes for sess_build_logline() and could even be replaced by function pointers some day...
2024-02-22 14:20:41 -05:00
next_fmt:
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
if (value_beg == tmplog) {
/* handle the case where no data was generated for the value after
* the key was already announced
*/
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
if (ctx->options & LOG_OPT_ENCODE_JSON) {
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
/* for JSON, we simply output 'null' */
iret = snprintf(tmplog, dst + maxsize - tmplog, "null");
if (iret < 0 || iret >= dst + maxsize - tmplog)
goto out;
tmplog += iret;
}
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
if (ctx->options & LOG_OPT_ENCODE_CBOR) {
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
/* for CBOR, we have the '22' primitive which is known as
* NULL
*/
LOG_CBOR_BYTE(0xF6);
}
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
}
MINOR: log: use LOG_VARTEXT_{START,END} to enclose text strings Rename LOGQUOTE_{START,END} macros to more generic LOG_VARTEXT_{START,END} in order to prepare for new encoding types that rely on specific treatment for variable-length texts. No functional change should be expected.
2024-04-16 05:17:55 -04:00
/* if variable text was started for the current node data, we need
* to end it
MINOR: log: simplify quotes handling in sess_build_logline() quotes building for some log formats is directly performed under each switch case statement so it would become painful to add other conditions to prevent the quotes from being generated when it's not supported by the the data encoding format for instance (ie: JSON). Let's centralize and simplify quotes handling by adding LOGQUOTE_START() and LOGQUOTE_END() helper macros. If a quotation is started and not explicitly ended, it will be automatically ended at the end of the current logformat node: LOGQUOTE_START() sets 'quote' variable to 1, this way LOGQUOTE_END() only prints the ending quote when needed. LOGQUOTE_END() is systematically called after each node switch-case (after each value). LOGQUOTE_START() does nothing if LOG_OPT_QUOTE isn't set, so does LOGQUOTE_END(). Some rare cases such as %hsl (list of captured headers) required special handling: in this case multiple quoted texts are generated for the same field value so explicit LOGQUOTE_START() + LOGQUOTE_END() combination was needed.
2024-01-10 13:14:25 -05:00
*/
MINOR: log: use LOG_VARTEXT_{START,END} to enclose text strings Rename LOGQUOTE_{START,END} macros to more generic LOG_VARTEXT_{START,END} in order to prepare for new encoding types that rely on specific treatment for variable-length texts. No functional change should be expected.
2024-04-16 05:17:55 -04:00
LOG_VARTEXT_END();
BUG/MINOR: log: prevent double spaces emission in sess_build_logline() Christian reported in GH #2556 that since 3.0-dev double spaces may be found in log messages on some cases where it was not the case before. As we were able to easily reproduce, a quick bisect led us to c6a7138 ("MINOR: log: simplify last_isspace in sess_build_logline()"). While it is true that all switch cases set the last_isspace variable to 0, there was a subtelty for some fields such as '%hr', '%hrl', '%hs' or '%hsl' and I overlooked it. Indeed, for '%hr', last_isspace was only set to 0 if data was emitted, else the assignment didn't occur. But with c6a7138, last_isspace is always set to 0 as long as the current node type is not a separator. Because of that, if no data is emitted for the current node value, and a space was already emitted prior to the current node, then an extra space could be emitted after the node, resulting in two spaces being emitted. Note that while c6a7138 introduces a slight behavior regression regarding last_isspace logic with the specific fields mentionned above, this behavior could already be triggered with a failing or empty logformat node sample expression. Consider this logformat expression: log-format "%{-M}o | %[str()] |" str() will not print anything, and since we disabled mandatory option with '-M', nothing gets printed for the node sample expression. As a result, we have the following output: "| |" Instead of (when mandatory option is enabled): "| - |" Thus in order to stick to the historical behavior, systematically set last_isspace to 0 for EXPR nodes, and only set last_isspace to 0 when data was written for TAG nodes. This way, '%hr', '%hrl', '%hs' or '%hsl' should behave as before. No backport needed.
2024-05-02 03:30:28 -04:00
if (tmplog != value_beg) {
/* data was actually generated for the current dynamic
* node, reset the space hint so that a new space may
* now be emitted when relevant.
*/
last_isspace = 0;
}
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
}
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
/* back to global ctx (some encoding types may need to output
* ending closure)
*/
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
lf_buildctx_prepare(ctx, g_options, NULL);
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
if (ctx->options & LOG_OPT_ENCODE_JSON)
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
LOGCHAR('}');
OPTIM: log: use thread local lf_buildctx to stop pushing it on the stack Following previous commit's logic, let's move lf_buildctx ctx away from sess_build_logline() to stop abusing from the stack to push large structure each time sess_build_logline() is called. Also, don't memset the structure for each invokation, but only reset members explicitly when required. For that we now declare one static lf_buildctx per thread (using THREAD_LOCAL) and make sess_build_logline() refer to it using a pointer.
2024-05-02 09:30:17 -04:00
else if (ctx->options & LOG_OPT_ENCODE_CBOR) {
MINOR: log: add +cbor encoding option In this patch, we make use of the CBOR (RFC8949) encode helper functions from the previous commit to implement '+cbor' encoding option for log- formats. The logic behind it is pretty similar to '+json' encoding option, except that the produced output is a CBOR payload written in HEX format so that it remains compatible to use this with regular syslog endpoints. Example: log-format "%{+cbor}o %[int(4)] test %(named_field)[str(ok)]" Will produce: BF6B6E616D65645F6669656C64626F6BFF Detailed view (from cbor.me): BF # map(*) 6B # text(11) 6E616D65645F6669656C64 # "named_field" 62 # text(2) 6F6B # "ok" FF # primitive(*) If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to CBOR specification. Example: log-format "test cbor bool: %{+cbor}[bool(true)]" Will produce: test cbor bool: F5
2024-04-23 04:12:46 -04:00
/* end indefinite-length map */
LOG_CBOR_BYTE(0xFF);
}
MINOR: log: add +json encoding option In this patch, we add the "+json" log format option that can be set globally or per log format node. What it does, it that it sets the LOG_OPT_ENCODE_JSON flag for the current context which is provided to all lf_* log building function. This way, all lf_* are now aware of this option and try to comply with JSON specification when the option is set. If the option is set globally, then sess_build_logline() will produce a map-like object with key=val pairs for named logformat nodes. (logformat nodes that don't have a name are simply ignored). Example: log-format "%{+json}o %[int(4)] test %(named_field)[str(ok)]" Will produce: {"named_field": "ok"} If the option isn't set globally, but on a specific node instead, then only the value will be encoded according to JSON specification. Example: log-format "{ \"manual_key\": %(named_field){+json}[bool(true)] }" Will produce: {"manual_key": true} When the option is set, +E option will be ignored, and partial numerical values (ie: because of logasap) will be encoded as-is.
2024-04-22 08:40:04 -04:00
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
out:
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
/* *tmplog is a unused character */
*tmplog = '\0';
BUG/MINOR: http: http-request add-header emits a corrupted header David BERARD reported that http-request add-header passes a \0 along with the header field, which of course is not appropriate. This is caused by build_logline() which sometimes returns the size with the trailing zero and sometimes can return an empty string. Let's fix this function instead of fixing the places where it's used.
2012-12-27 20:44:01 -05:00
return tmplog - dst;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-25 20:48:02 -04:00
}
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
/*
REORG/MAJOR: session: rename the "session" entity to "stream" With HTTP/2, we'll have to support multiplexed streams. A stream is in fact the largest part of what we currently call a session, it has buffers, logs, etc. In order to catch any error, this commit removes any reference to the struct session and tries to rename most "session" occurrences in function names to "stream" and "sess" to "strm" when that's related to a session. The files stream.{c,h} were added and session.{c,h} removed. The session will be reintroduced later and a few parts of the stream will progressively be moved overthere. It will more or less contain only what we need in an embryonic session. Sample fetch functions and converters will have to change a bit so that they'll use an L5 (session) instead of what's currently called "L4" which is in fact L6 for now. Once all changes are completed, we should see approximately this : L7 - http_txn L6 - stream L5 - session L4 - connection | applet There will be at most one http_txn per stream, and a same session will possibly be referenced by multiple streams. A connection will point to a session and to a stream. The session will hold all the information we need to keep even when we don't yet have a stream. Some more cleanup is needed because some code was already far from being clean. The server queue management still refers to sessions at many places while comments talk about connections. This will have to be cleaned up once we have a server-side connection pool manager. Stream flags "SN_*" still need to be renamed, it doesn't seem like any of them will need to move to the session.
2015-04-02 18:22:06 -04:00
* send a log for the stream when we have enough info about it.
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
* Will not log if the frontend has no log defined.
*/
MINOR: log: provide sending log context to process_send_log() when available This is another prerequisite work in preparation for log-profiles: in this patch we make process_send_log() aware of the log origin, primarily aiming for sess and txn logging steps such as error, accept, connect, close, as well as relevant sess and stream pointers.
2024-02-22 05:29:20 -05:00
void strm_log(struct stream *s, int origin)
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
{
MEDIUM: stream: move the frontend's pointer to the session Just like for the listener, the frontend is session-wide so let's move it to the session. There are a lot of places which were changed but the changes are minimal in fact.
2015-04-03 09:40:56 -04:00
struct session *sess = s->sess;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
int size, err, level;
MEDIUM: logs: add a new RFC5424 log-format for the structured-data This patch adds a new RFC5424-specific log-format for the structured-data that is automatically send by __send_log() when the sender is in RFC5424 mode. A new statement "log-format-sd" should be used in order to set log-format for the structured-data part in RFC5424 formatted syslog messages. Example: log-format-sd [exampleSDID@1234\ bytes=\"%B\"\ status=\"%ST\"]
2015-09-25 13:17:44 -04:00
int sd_size = 0;
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
/* if we don't want to log normal traffic, return now */
REORG/MEDIUM: stream: rename stream flags from SN_* to SF_* This is in order to keep things consistent.
2015-04-02 19:14:29 -04:00
err = (s->flags & SF_REDISP) ||
((s->flags & SF_ERR_MASK) > SF_ERR_LOCAL) ||
BUG/MINOR: log: Properly test connection retries to fix dontlog-normal option The commit 731c8e6cf ("MINOR: stream: Simplify retries counter calculation") introduced a regression. It broke the dontlog-normal option because the test on the connection retries counter was not updated accordingly. This patch should fix the issue #1754. It must be backported to 2.6.
2022-06-17 08:53:20 -04:00
(((s->flags & SF_ERR_MASK) == SF_ERR_NONE) && s->conn_retries) ||
MAJOR: http: move http_txn out of struct stream Now this one is dynamically allocated. It means that 280 bytes of memory are saved per TCP stream, but more importantly that it will become possible to remove the l7 pointer from fetches and converters since it will be deduced from the stream and will support being null. A lot of care was taken because it's easy to forget a test somewhere, and the previous code used to always trust s->txn for being valid, but all places seem to have been visited. All HTTP fetch functions check the txn first so we shouldn't have any issue there even when called from TCP. When branching from a TCP frontend to an HTTP backend, the txn is properly allocated at the same time as the hdr_idx.
2015-04-03 17:46:31 -04:00
((sess->fe->mode == PR_MODE_HTTP) && s->txn && s->txn->status >= 500);
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-25 20:48:02 -04:00
MEDIUM: stream: move the frontend's pointer to the session Just like for the listener, the frontend is session-wide so let's move it to the session. There are a lot of places which were changed but the changes are minimal in fact.
2015-04-03 09:40:56 -04:00
if (!err && (sess->fe->options2 & PR_O2_NOLOGNORM))
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
return;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
if (LIST_ISEMPTY(&sess->fe->loggers))
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
return;
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
MEDIUM: log: add a log level override value in struct session This log level will be used in a further patch to change the log level depending on the request or response.
2013-06-11 11:18:02 -04:00
if (s->logs.level) { /* loglevel was overridden */
if (s->logs.level == -1) {
s->logs.logwait = 0; /* logs disabled */
return;
}
level = s->logs.level - 1;
}
else {
level = LOG_INFO;
MEDIUM: stream: move the frontend's pointer to the session Just like for the listener, the frontend is session-wide so let's move it to the session. There are a lot of places which were changed but the changes are minimal in fact.
2015-04-03 09:40:56 -04:00
if (err && (sess->fe->options2 & PR_O2_LOGERRORS))
MEDIUM: log: add a log level override value in struct session This log level will be used in a further patch to change the log level depending on the request or response.
2013-06-11 11:18:02 -04:00
level = LOG_ERR;
}
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
BUG/MEDIUM: unique_id: junk in log on empty unique_id When a request fail, the unique_id was allocated but not generated. The string was not initialized and junk was printed in the log with %ID. This patch changes the behavior of the unique_id. The unique_id is now generated when a request failed. This bug was reported by Patrick Hemmer.
2013-08-28 09:44:19 -04:00
/* if unique-id was not generated */
MEDIUM: tree-wide: add logformat expressions wrapper log format expressions are broadly used within the code: once they are parsed from input string, they are converted to a linked list of logformat nodes. We're starting to face some limitations because we're simply storing the converted expression as a generic logformat_node list. The first issue we're facing is that storing logformat expressions that way doesn't allow us to add metadata alongside the list, which is part of the prerequites for implementing log-profiles. Another issue with storing logformat expressions as generic lists of logformat_node elements is that it's starting to become really hard to tell when we rely on logformat expressions or not in the code given that there isn't always a comment near the list declaration or manipulation to indicate that it's relying on logformat expressions under the hood, so this adds some complexity for code maintenance. This patch looks quite impressive due to changes in a lot of header and source files (since logformat expressions are broadly used), but it does a simple thing: it defines the lf_expr structure which itself holds a generic list of logformat nodes, and then declares some helpers to manipulate lf_expr elements and fixes the code so that we now exclusively manipulate logformat_node lists as lf_expr elements outside of log.c. For now, lf_expr struct only contains the list of logformat nodes (no additional metadata), but now that we have dedicated type and helpers, doing so in the future won't be problematic at all and won't require extensive code changes.
2024-02-23 09:57:21 -05:00
if (!isttest(s->unique_id) && !lf_expr_isempty(&sess->fe->format_unique_id)) {
MINOR: stream: Use stream_generate_unique_id This patch replaces the ad-hoc generation of stream's `unique_id` values by calls to `stream_generate_unique_id`.
2020-02-28 09:13:34 -05:00
stream_generate_unique_id(s, &sess->fe->format_unique_id);
BUG/MEDIUM: unique_id: junk in log on empty unique_id When a request fail, the unique_id was allocated but not generated. The string was not initialized and junk was printed in the log with %ID. This patch changes the behavior of the unique_id. The unique_id is now generated when a request failed. This bug was reported by Patrick Hemmer.
2013-08-28 09:44:19 -04:00
}
MEDIUM: tree-wide: add logformat expressions wrapper log format expressions are broadly used within the code: once they are parsed from input string, they are converted to a linked list of logformat nodes. We're starting to face some limitations because we're simply storing the converted expression as a generic logformat_node list. The first issue we're facing is that storing logformat expressions that way doesn't allow us to add metadata alongside the list, which is part of the prerequites for implementing log-profiles. Another issue with storing logformat expressions as generic lists of logformat_node elements is that it's starting to become really hard to tell when we rely on logformat expressions or not in the code given that there isn't always a comment near the list declaration or manipulation to indicate that it's relying on logformat expressions under the hood, so this adds some complexity for code maintenance. This patch looks quite impressive due to changes in a lot of header and source files (since logformat expressions are broadly used), but it does a simple thing: it defines the lf_expr structure which itself holds a generic list of logformat nodes, and then declares some helpers to manipulate lf_expr elements and fixes the code so that we now exclusively manipulate logformat_node lists as lf_expr elements outside of log.c. For now, lf_expr struct only contains the list of logformat nodes (no additional metadata), but now that we have dedicated type and helpers, doing so in the future won't be problematic at all and won't require extensive code changes.
2024-02-23 09:57:21 -05:00
if (!lf_expr_isempty(&sess->fe->logformat_sd)) {
MINOR: log: provide log origin in logformat expressions using '%OG' '%OG' logformat alias may be used to report the log origin (when/where) that triggered log generation using sess_build_logline(). Possible values are: - "sess_error": log was generated during session error handling - "sess_killed": log was generated during session abortion (killed embryonic session) - "txn_accept": log was generated right after frontend conn was accepted - "txn_request": log was generated after client request was received - "txn_connect": log was generated after backend connection establishment - "txn_response": log was generated during server response handling - "txn_close": log was generated at the final txn step, before closing - "unspec": unknown or not specified Documentation was updated.
2024-05-06 08:13:11 -04:00
sd_size = build_logline_orig(s, logline_rfc5424, global.max_syslog_len,
&sess->fe->logformat_sd, origin);
MEDIUM: logs: add a new RFC5424 log-format for the structured-data This patch adds a new RFC5424-specific log-format for the structured-data that is automatically send by __send_log() when the sender is in RFC5424 mode. A new statement "log-format-sd" should be used in order to set log-format for the structured-data part in RFC5424 formatted syslog messages. Example: log-format-sd [exampleSDID@1234\ bytes=\"%B\"\ status=\"%ST\"]
2015-09-25 13:17:44 -04:00
}
MINOR: log: provide log origin in logformat expressions using '%OG' '%OG' logformat alias may be used to report the log origin (when/where) that triggered log generation using sess_build_logline(). Possible values are: - "sess_error": log was generated during session error handling - "sess_killed": log was generated during session abortion (killed embryonic session) - "txn_accept": log was generated right after frontend conn was accepted - "txn_request": log was generated after client request was received - "txn_connect": log was generated after backend connection establishment - "txn_response": log was generated during server response handling - "txn_close": log was generated at the final txn step, before closing - "unspec": unknown or not specified Documentation was updated.
2024-05-06 08:13:11 -04:00
size = build_logline_orig(s, logline, global.max_syslog_len, &sess->fe->logformat, origin);
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
if (size > 0) {
MINOR: log: provide sending log context to process_send_log() when available This is another prerequisite work in preparation for log-profiles: in this patch we make process_send_log() aware of the log origin, primarily aiming for sess and txn logging steps such as error, accept, connect, close, as well as relevant sess and stream pointers.
2024-02-22 05:29:20 -05:00
struct process_send_log_ctx ctx;
CLEANUP: atomic/tree-wide: replace single increments/decrements with inc/dec This patch replaces roughly all occurrences of an HA_ATOMIC_ADD(&foo, 1) or HA_ATOMIC_SUB(&foo, 1) with the equivalent HA_ATOMIC_INC(&foo) and HA_ATOMIC_DEC(&foo) respectively. These are 507 changes over 45 files.
2021-04-06 07:53:36 -04:00
_HA_ATOMIC_INC(&sess->fe->log_count);
MINOR: log: provide sending log context to process_send_log() when available This is another prerequisite work in preparation for log-profiles: in this patch we make process_send_log() aware of the log origin, primarily aiming for sess and txn logging steps such as error, accept, connect, close, as well as relevant sess and stream pointers.
2024-02-22 05:29:20 -05:00
ctx.origin = origin;
ctx.sess = sess;
ctx.stream = s;
__send_log(&ctx, &sess->fe->loggers, &sess->fe->log_tag, level,
MINOR: log: Provide a function to emit a log for an application Application is a generic term here. It is a modules which handle its own log server list, with no dependency on a proxy. Such applications can now call the function app_log() to log messages, passing a log server list and a tag as parameters. Internally, the function __send_log() has been adapted accordingly.
2019-08-11 13:40:12 -04:00
logline, size + 1, logline_rfc5424, sd_size);
MEDIUM: log: split of log_format generation * logformat functions now take a format linked list as argument * build_logline() build a logline using a format linked list * rename LOG_* by LOG_FMT_* in enum * improve error management in build_logline()
2012-03-12 07:46:41 -04:00
s->logs.logwait = 0;
}
}
MEDIUM: log: use log_format for mode tcplog Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
2012-02-27 05:23:10 -05:00
MINOR: log: provide a function to emit a log for a session The new function sess_log() only needs a session to emit a log. It will ignore the parts that depend on the stream. It is usable to emit a log to report early errors in muxes. These ones will typically mention "<BADREQ>" for the request and 0 for the HTTP status code.
2018-09-05 13:51:10 -04:00
/*
* send a minimalist log for the session. Will not log if the frontend has no
* log defined. It is assumed that this is only used to report anomalies that
* cannot lead to the creation of a regular stream. Because of this the log
* level is LOG_INFO or LOG_ERR depending on the "log-separate-error" setting
* in the frontend. The caller must simply know that it should not call this
MINOR: log: make sess_log() support sess=NULL At many places in muxes we'll have to add tests to check if the connection is front or back before deciding to log. Instead let's centralize this test in sess_log() to simply do nothing when sess=NULL.
2018-10-05 04:22:27 -04:00
* function to report unimportant events. It is safe to call this function with
* sess==NULL (will not do anything).
MEDIUM: log/session: handle embryonic session log within sess_log() Move the embryonic session logging logic down to sess_log() in preparation for log-profiles because then log preferences will be set per logger and not per proxy. Indeed, as each logger may come with its own log-profile that possibly overrides proxy logformat preferences, the check will need to be performed at a central place by lower sending functions. To ensure the change doesn't break existing behavior, a dedicated sess_log_embryonic() wrapper was added and is exclusively used by session_kill_embryonic() to indicate that a special logging logic must be performed under sess_log(). Also, thanks to this change, log-format-sd will now be taken into account for legacy embryonic session logging.
2024-02-21 11:26:52 -05:00
*
* if <embryonic> is set, then legacy error log payload will be generated unless
* logformat_error is specified (ie: normal logformat is ignored in this case).
*
MINOR: log: provide a function to emit a log for a session The new function sess_log() only needs a session to emit a log. It will ignore the parts that depend on the stream. It is usable to emit a log to report early errors in muxes. These ones will typically mention "<BADREQ>" for the request and 0 for the HTTP status code.
2018-09-05 13:51:10 -04:00
*/
MEDIUM: log/session: handle embryonic session log within sess_log() Move the embryonic session logging logic down to sess_log() in preparation for log-profiles because then log preferences will be set per logger and not per proxy. Indeed, as each logger may come with its own log-profile that possibly overrides proxy logformat preferences, the check will need to be performed at a central place by lower sending functions. To ensure the change doesn't break existing behavior, a dedicated sess_log_embryonic() wrapper was added and is exclusively used by session_kill_embryonic() to indicate that a special logging logic must be performed under sess_log(). Also, thanks to this change, log-format-sd will now be taken into account for legacy embryonic session logging.
2024-02-21 11:26:52 -05:00
void _sess_log(struct session *sess, int embryonic)
MINOR: log: provide a function to emit a log for a session The new function sess_log() only needs a session to emit a log. It will ignore the parts that depend on the stream. It is usable to emit a log to report early errors in muxes. These ones will typically mention "<BADREQ>" for the request and 0 for the HTTP status code.
2018-09-05 13:51:10 -04:00
{
int size, level;
int sd_size = 0;
MINOR: log: provide log origin in logformat expressions using '%OG' '%OG' logformat alias may be used to report the log origin (when/where) that triggered log generation using sess_build_logline(). Possible values are: - "sess_error": log was generated during session error handling - "sess_killed": log was generated during session abortion (killed embryonic session) - "txn_accept": log was generated right after frontend conn was accepted - "txn_request": log was generated after client request was received - "txn_connect": log was generated after backend connection establishment - "txn_response": log was generated during server response handling - "txn_close": log was generated at the final txn step, before closing - "unspec": unknown or not specified Documentation was updated.
2024-05-06 08:13:11 -04:00
int orig = (embryonic) ? LOG_ORIG_SESS_KILL : LOG_ORIG_SESS_ERROR;
MINOR: log: provide a function to emit a log for a session The new function sess_log() only needs a session to emit a log. It will ignore the parts that depend on the stream. It is usable to emit a log to report early errors in muxes. These ones will typically mention "<BADREQ>" for the request and 0 for the HTTP status code.
2018-09-05 13:51:10 -04:00
MINOR: log: make sess_log() support sess=NULL At many places in muxes we'll have to add tests to check if the connection is front or back before deciding to log. Instead let's centralize this test in sess_log() to simply do nothing when sess=NULL.
2018-10-05 04:22:27 -04:00
if (!sess)
return;
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
if (LIST_ISEMPTY(&sess->fe->loggers))
MINOR: log: provide a function to emit a log for a session The new function sess_log() only needs a session to emit a log. It will ignore the parts that depend on the stream. It is usable to emit a log to report early errors in muxes. These ones will typically mention "<BADREQ>" for the request and 0 for the HTTP status code.
2018-09-05 13:51:10 -04:00
return;
level = LOG_INFO;
if (sess->fe->options2 & PR_O2_LOGERRORS)
level = LOG_ERR;
MEDIUM: tree-wide: add logformat expressions wrapper log format expressions are broadly used within the code: once they are parsed from input string, they are converted to a linked list of logformat nodes. We're starting to face some limitations because we're simply storing the converted expression as a generic logformat_node list. The first issue we're facing is that storing logformat expressions that way doesn't allow us to add metadata alongside the list, which is part of the prerequites for implementing log-profiles. Another issue with storing logformat expressions as generic lists of logformat_node elements is that it's starting to become really hard to tell when we rely on logformat expressions or not in the code given that there isn't always a comment near the list declaration or manipulation to indicate that it's relying on logformat expressions under the hood, so this adds some complexity for code maintenance. This patch looks quite impressive due to changes in a lot of header and source files (since logformat expressions are broadly used), but it does a simple thing: it defines the lf_expr structure which itself holds a generic list of logformat nodes, and then declares some helpers to manipulate lf_expr elements and fixes the code so that we now exclusively manipulate logformat_node lists as lf_expr elements outside of log.c. For now, lf_expr struct only contains the list of logformat nodes (no additional metadata), but now that we have dedicated type and helpers, doing so in the future won't be problematic at all and won't require extensive code changes.
2024-02-23 09:57:21 -05:00
if (!lf_expr_isempty(&sess->fe->logformat_sd)) {
MINOR: log: provide log origin in logformat expressions using '%OG' '%OG' logformat alias may be used to report the log origin (when/where) that triggered log generation using sess_build_logline(). Possible values are: - "sess_error": log was generated during session error handling - "sess_killed": log was generated during session abortion (killed embryonic session) - "txn_accept": log was generated right after frontend conn was accepted - "txn_request": log was generated after client request was received - "txn_connect": log was generated after backend connection establishment - "txn_response": log was generated during server response handling - "txn_close": log was generated at the final txn step, before closing - "unspec": unknown or not specified Documentation was updated.
2024-05-06 08:13:11 -04:00
sd_size = sess_build_logline_orig(sess, NULL,
logline_rfc5424, global.max_syslog_len,
&sess->fe->logformat_sd,
orig);
MINOR: log: provide a function to emit a log for a session The new function sess_log() only needs a session to emit a log. It will ignore the parts that depend on the stream. It is usable to emit a log to report early errors in muxes. These ones will typically mention "<BADREQ>" for the request and 0 for the HTTP status code.
2018-09-05 13:51:10 -04:00
}
MEDIUM: tree-wide: add logformat expressions wrapper log format expressions are broadly used within the code: once they are parsed from input string, they are converted to a linked list of logformat nodes. We're starting to face some limitations because we're simply storing the converted expression as a generic logformat_node list. The first issue we're facing is that storing logformat expressions that way doesn't allow us to add metadata alongside the list, which is part of the prerequites for implementing log-profiles. Another issue with storing logformat expressions as generic lists of logformat_node elements is that it's starting to become really hard to tell when we rely on logformat expressions or not in the code given that there isn't always a comment near the list declaration or manipulation to indicate that it's relying on logformat expressions under the hood, so this adds some complexity for code maintenance. This patch looks quite impressive due to changes in a lot of header and source files (since logformat expressions are broadly used), but it does a simple thing: it defines the lf_expr structure which itself holds a generic list of logformat nodes, and then declares some helpers to manipulate lf_expr elements and fixes the code so that we now exclusively manipulate logformat_node lists as lf_expr elements outside of log.c. For now, lf_expr struct only contains the list of logformat nodes (no additional metadata), but now that we have dedicated type and helpers, doing so in the future won't be problematic at all and won't require extensive code changes.
2024-02-23 09:57:21 -05:00
if (!lf_expr_isempty(&sess->fe->logformat_error))
MINOR: log: provide log origin in logformat expressions using '%OG' '%OG' logformat alias may be used to report the log origin (when/where) that triggered log generation using sess_build_logline(). Possible values are: - "sess_error": log was generated during session error handling - "sess_killed": log was generated during session abortion (killed embryonic session) - "txn_accept": log was generated right after frontend conn was accepted - "txn_request": log was generated after client request was received - "txn_connect": log was generated after backend connection establishment - "txn_response": log was generated during server response handling - "txn_close": log was generated at the final txn step, before closing - "unspec": unknown or not specified Documentation was updated.
2024-05-06 08:13:11 -04:00
size = sess_build_logline_orig(sess, NULL, logline,
global.max_syslog_len, &sess->fe->logformat_error,
orig);
MEDIUM: log/session: handle embryonic session log within sess_log() Move the embryonic session logging logic down to sess_log() in preparation for log-profiles because then log preferences will be set per logger and not per proxy. Indeed, as each logger may come with its own log-profile that possibly overrides proxy logformat preferences, the check will need to be performed at a central place by lower sending functions. To ensure the change doesn't break existing behavior, a dedicated sess_log_embryonic() wrapper was added and is exclusively used by session_kill_embryonic() to indicate that a special logging logic must be performed under sess_log(). Also, thanks to this change, log-format-sd will now be taken into account for legacy embryonic session logging.
2024-02-21 11:26:52 -05:00
else if (!embryonic)
MINOR: log: provide log origin in logformat expressions using '%OG' '%OG' logformat alias may be used to report the log origin (when/where) that triggered log generation using sess_build_logline(). Possible values are: - "sess_error": log was generated during session error handling - "sess_killed": log was generated during session abortion (killed embryonic session) - "txn_accept": log was generated right after frontend conn was accepted - "txn_request": log was generated after client request was received - "txn_connect": log was generated after backend connection establishment - "txn_response": log was generated during server response handling - "txn_close": log was generated at the final txn step, before closing - "unspec": unknown or not specified Documentation was updated.
2024-05-06 08:13:11 -04:00
size = sess_build_logline_orig(sess, NULL, logline,
global.max_syslog_len, &sess->fe->logformat,
orig);
MEDIUM: log/session: handle embryonic session log within sess_log() Move the embryonic session logging logic down to sess_log() in preparation for log-profiles because then log preferences will be set per logger and not per proxy. Indeed, as each logger may come with its own log-profile that possibly overrides proxy logformat preferences, the check will need to be performed at a central place by lower sending functions. To ensure the change doesn't break existing behavior, a dedicated sess_log_embryonic() wrapper was added and is exclusively used by session_kill_embryonic() to indicate that a special logging logic must be performed under sess_log(). Also, thanks to this change, log-format-sd will now be taken into account for legacy embryonic session logging.
2024-02-21 11:26:52 -05:00
else { /* no logformat_error and embryonic==1 */
struct buffer buf;
buf = b_make(logline, global.max_syslog_len, 0, 0);
session_embryonic_build_legacy_err(sess, &buf);
size = buf.data;
}
MINOR: log: provide a function to emit a log for a session The new function sess_log() only needs a session to emit a log. It will ignore the parts that depend on the stream. It is usable to emit a log to report early errors in muxes. These ones will typically mention "<BADREQ>" for the request and 0 for the HTTP status code.
2018-09-05 13:51:10 -04:00
if (size > 0) {
MINOR: log: provide sending log context to process_send_log() when available This is another prerequisite work in preparation for log-profiles: in this patch we make process_send_log() aware of the log origin, primarily aiming for sess and txn logging steps such as error, accept, connect, close, as well as relevant sess and stream pointers.
2024-02-22 05:29:20 -05:00
struct process_send_log_ctx ctx;
CLEANUP: atomic/tree-wide: replace single increments/decrements with inc/dec This patch replaces roughly all occurrences of an HA_ATOMIC_ADD(&foo, 1) or HA_ATOMIC_SUB(&foo, 1) with the equivalent HA_ATOMIC_INC(&foo) and HA_ATOMIC_DEC(&foo) respectively. These are 507 changes over 45 files.
2021-04-06 07:53:36 -04:00
_HA_ATOMIC_INC(&sess->fe->log_count);
MINOR: log: provide log origin in logformat expressions using '%OG' '%OG' logformat alias may be used to report the log origin (when/where) that triggered log generation using sess_build_logline(). Possible values are: - "sess_error": log was generated during session error handling - "sess_killed": log was generated during session abortion (killed embryonic session) - "txn_accept": log was generated right after frontend conn was accepted - "txn_request": log was generated after client request was received - "txn_connect": log was generated after backend connection establishment - "txn_response": log was generated during server response handling - "txn_close": log was generated at the final txn step, before closing - "unspec": unknown or not specified Documentation was updated.
2024-05-06 08:13:11 -04:00
ctx.origin = orig;
MINOR: log: provide sending log context to process_send_log() when available This is another prerequisite work in preparation for log-profiles: in this patch we make process_send_log() aware of the log origin, primarily aiming for sess and txn logging steps such as error, accept, connect, close, as well as relevant sess and stream pointers.
2024-02-22 05:29:20 -05:00
ctx.sess = sess;
ctx.stream = NULL;
__send_log(&ctx, &sess->fe->loggers,
&sess->fe->log_tag, level,
MINOR: log: Provide a function to emit a log for an application Application is a generic term here. It is a modules which handle its own log server list, with no dependency on a proxy. Such applications can now call the function app_log() to log messages, passing a log server list and a tag as parameters. Internally, the function __send_log() has been adapted accordingly.
2019-08-11 13:40:12 -04:00
logline, size + 1, logline_rfc5424, sd_size);
MINOR: log: provide a function to emit a log for a session The new function sess_log() only needs a session to emit a log. It will ignore the parts that depend on the stream. It is usable to emit a log to report early errors in muxes. These ones will typically mention "<BADREQ>" for the request and 0 for the HTTP status code.
2018-09-05 13:51:10 -04:00
}
}
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
void app_log(struct list *loggers, struct buffer *tag, int level, const char *format, ...)
MINOR: log: Provide a function to emit a log for an application Application is a generic term here. It is a modules which handle its own log server list, with no dependency on a proxy. Such applications can now call the function app_log() to log messages, passing a log server list and a tag as parameters. Internally, the function __send_log() has been adapted accordingly.
2019-08-11 13:40:12 -04:00
{
va_list argp;
int data_len;
if (level < 0 || format == NULL || logline == NULL)
return;
va_start(argp, format);
data_len = vsnprintf(logline, global.max_syslog_len, format, argp);
if (data_len < 0 || data_len > global.max_syslog_len)
data_len = global.max_syslog_len;
va_end(argp);
MINOR: log: provide sending log context to process_send_log() when available This is another prerequisite work in preparation for log-profiles: in this patch we make process_send_log() aware of the log origin, primarily aiming for sess and txn logging steps such as error, accept, connect, close, as well as relevant sess and stream pointers.
2024-02-22 05:29:20 -05:00
__send_log(NULL, loggers, tag, level, logline, data_len, default_rfc5424_sd_log_format, 2);
MINOR: log: Provide a function to emit a log for an application Application is a generic term here. It is a modules which handle its own log server list, with no dependency on a proxy. Such applications can now call the function app_log() to log messages, passing a log server list and a tag as parameters. Internally, the function __send_log() has been adapted accordingly.
2019-08-11 13:40:12 -04:00
}
MINOR: log: adds syslog udp message handler and parsing. This patch introduce a new fd handler used to parse syslog message on udp. The parsing function returns level, facility and metadata that can be immediatly reused to forward message to a log server. This handler is enabled on udp listeners if proxy is internally set to mode PR_MODE_SYSLOG
2020-07-07 03:43:24 -04:00
/*
* This function parse a received log message <buf>, of size <buflen>
* it fills <level>, <facility> and <metadata> depending of the detected
* header format and message will point on remaining payload of <size>
*
* <metadata> must point on a preallocated array of LOG_META_FIELDS*sizeof(struct ist)
* struct ist len will be set to 0 if field is not found
* <level> and <facility> will be set to -1 if not found.
*/
void parse_log_message(char *buf, size_t buflen, int *level, int *facility,
struct ist *metadata, char **message, size_t *size)
{
char *p;
int fac_level = 0;
*level = *facility = -1;
*message = buf;
*size = buflen;
memset(metadata, 0, LOG_META_FIELDS*sizeof(struct ist));
p = buf;
if (*size < 2 || *p != '<')
return;
p++;
while (*p != '>') {
if (*p > '9' || *p < '0')
return;
fac_level = 10*fac_level + (*p - '0');
p++;
if ((p - buf) > buflen)
return;
}
*facility = fac_level >> 3;
*level = fac_level & 0x7;
p++;
metadata[LOG_META_PRIO] = ist2(buf, p - buf);
buflen -= p - buf;
buf = p;
*size = buflen;
*message = buf;
/* for rfc5424, prio is always followed by '1' and ' ' */
if ((*size > 2) && (p[0] == '1') && (p[1] == ' ')) {
/* format is always '1 TIMESTAMP HOSTNAME TAG PID MSGID STDATA '
* followed by message.
* Each header field can present NILVALUE: '-'
*/
p += 2;
BUG/MINOR: log: fix parse_log_message rfc5424 size check In parse_log_message(), if log is rfc5424 compliant, p pointer is incremented and size is not. However size is still used in further checks as if p pointer was not incremented. This could lead to logic error or buffer overflow if input buf is not null-terminated. Fixing this by making sure size is up to date where it is needed. It could be backported up to 2.4.
2022-11-22 05:17:11 -05:00
*size -= 2;
MINOR: log: adds syslog udp message handler and parsing. This patch introduce a new fd handler used to parse syslog message on udp. The parsing function returns level, facility and metadata that can be immediatly reused to forward message to a log server. This handler is enabled on udp listeners if proxy is internally set to mode PR_MODE_SYSLOG
2020-07-07 03:43:24 -04:00
/* timestamp is NILVALUE '-' */
if (*size > 2 && (p[0] == '-') && p[1] == ' ') {
metadata[LOG_META_TIME] = ist2(p, 1);
p++;
}
else if (*size > LOG_ISOTIME_MINLEN) {
metadata[LOG_META_TIME].ptr = p;
CLEANUP: assorted typo fixes in the code and comments This is 12th iteration of typo fixes
2020-07-22 15:32:55 -04:00
/* check if optional secfrac is present
MINOR: log: adds syslog udp message handler and parsing. This patch introduce a new fd handler used to parse syslog message on udp. The parsing function returns level, facility and metadata that can be immediatly reused to forward message to a log server. This handler is enabled on udp listeners if proxy is internally set to mode PR_MODE_SYSLOG
2020-07-07 03:43:24 -04:00
* in timestamp.
* possible format are:
* ex: '1970-01-01T00:00:00.000000Z'
* '1970-01-01T00:00:00.000000+00:00'
* '1970-01-01T00:00:00.000000-00:00'
* '1970-01-01T00:00:00Z'
* '1970-01-01T00:00:00+00:00'
* '1970-01-01T00:00:00-00:00'
*/
p += 19;
if (*p == '.') {
p++;
if ((p - buf) >= buflen)
goto bad_format;
while (*p != 'Z' && *p != '+' && *p != '-') {
if ((unsigned char)(*p - '0') > 9)
goto bad_format;
p++;
if ((p - buf) >= buflen)
goto bad_format;
}
}
if (*p == 'Z')
p++;
else
p += 6; /* case of '+00:00 or '-00:00' */
if ((p - buf) >= buflen || *p != ' ')
goto bad_format;
metadata[LOG_META_TIME].len = p - metadata[LOG_META_TIME].ptr;
}
else
goto bad_format;
p++;
if ((p - buf) >= buflen || *p == ' ')
goto bad_format;
metadata[LOG_META_HOST].ptr = p;
while (*p != ' ') {
p++;
if ((p - buf) >= buflen)
goto bad_format;
}
metadata[LOG_META_HOST].len = p - metadata[LOG_META_HOST].ptr;
if (metadata[LOG_META_HOST].len == 1 && metadata[LOG_META_HOST].ptr[0] == '-')
metadata[LOG_META_HOST].len = 0;
p++;
if ((p - buf) >= buflen || *p == ' ')
goto bad_format;
metadata[LOG_META_TAG].ptr = p;
while (*p != ' ') {
p++;
if ((p - buf) >= buflen)
goto bad_format;
}
metadata[LOG_META_TAG].len = p - metadata[LOG_META_TAG].ptr;
if (metadata[LOG_META_TAG].len == 1 && metadata[LOG_META_TAG].ptr[0] == '-')
metadata[LOG_META_TAG].len = 0;
p++;
if ((p - buf) >= buflen || *p == ' ')
goto bad_format;
metadata[LOG_META_PID].ptr = p;
while (*p != ' ') {
p++;
if ((p - buf) >= buflen)
goto bad_format;
}
metadata[LOG_META_PID].len = p - metadata[LOG_META_PID].ptr;
if (metadata[LOG_META_PID].len == 1 && metadata[LOG_META_PID].ptr[0] == '-')
metadata[LOG_META_PID].len = 0;
p++;
if ((p - buf) >= buflen || *p == ' ')
goto bad_format;
metadata[LOG_META_MSGID].ptr = p;
while (*p != ' ') {
p++;
if ((p - buf) >= buflen)
goto bad_format;
}
metadata[LOG_META_MSGID].len = p - metadata[LOG_META_MSGID].ptr;
if (metadata[LOG_META_MSGID].len == 1 && metadata[LOG_META_MSGID].ptr[0] == '-')
metadata[LOG_META_MSGID].len = 0;
p++;
if ((p - buf) >= buflen || *p == ' ')
goto bad_format;
/* structured data format is:
* ex:
* '[key1=value1 key2=value2][key3=value3]'
*
* space is invalid outside [] because
* considered as the end of structured data field
*/
metadata[LOG_META_STDATA].ptr = p;
if (*p == '[') {
int elem = 0;
while (1) {
if (elem) {
/* according to rfc this char is escaped in param values */
if (*p == ']' && *(p-1) != '\\')
elem = 0;
}
else {
if (*p == '[')
elem = 1;
else if (*p == ' ')
break;
else
goto bad_format;
}
p++;
if ((p - buf) >= buflen)
goto bad_format;
}
}
else if (*p == '-') {
/* case of NILVALUE */
p++;
if ((p - buf) >= buflen || *p != ' ')
goto bad_format;
}
else
goto bad_format;
metadata[LOG_META_STDATA].len = p - metadata[LOG_META_STDATA].ptr;
if (metadata[LOG_META_STDATA].len == 1 && metadata[LOG_META_STDATA].ptr[0] == '-')
metadata[LOG_META_STDATA].len = 0;
p++;
buflen -= p - buf;
buf = p;
*size = buflen;
*message = p;
}
else if (*size > LOG_LEGACYTIME_LEN) {
int m;
/* supported header format according to rfc3164.
* ex:
* 'Jan 1 00:00:00 HOSTNAME TAG[PID]: '
* or 'Jan 1 00:00:00 HOSTNAME TAG: '
* or 'Jan 1 00:00:00 HOSTNAME '
* Note: HOSTNAME is mandatory, and day
* of month uses a single space prefix if
* less than 10 to ensure hour offset is
* always the same.
*/
/* Check month to see if it correspond to a rfc3164
* header ex 'Jan 1 00:00:00' */
for (m = 0; m < 12; m++)
if (!memcmp(monthname[m], p, 3))
break;
/* Month not found */
if (m == 12)
goto bad_format;
metadata[LOG_META_TIME] = ist2(p, LOG_LEGACYTIME_LEN);
p += LOG_LEGACYTIME_LEN;
if ((p - buf) >= buflen || *p != ' ')
goto bad_format;
p++;
if ((p - buf) >= buflen || *p == ' ')
goto bad_format;
metadata[LOG_META_HOST].ptr = p;
while (*p != ' ') {
p++;
if ((p - buf) >= buflen)
goto bad_format;
}
metadata[LOG_META_HOST].len = p - metadata[LOG_META_HOST].ptr;
/* TAG seems to no be mandatory */
p++;
buflen -= p - buf;
buf = p;
*size = buflen;
*message = buf;
if (!buflen)
return;
while (((p - buf) < buflen) && *p != ' ' && *p != ':')
p++;
/* a tag must present a trailing ':' */
if (((p - buf) >= buflen) || *p != ':')
return;
p++;
/* followed by a space */
if (((p - buf) >= buflen) || *p != ' ')
return;
/* rewind to parse tag and pid */
p = buf;
metadata[LOG_META_TAG].ptr = p;
/* we have the guarantee that ':' will be reach before size limit */
while (*p != ':') {
if (*p == '[') {
metadata[LOG_META_TAG].len = p - metadata[LOG_META_TAG].ptr;
metadata[LOG_META_PID].ptr = p + 1;
}
CLEANUP: Use isttest(const struct ist) whenever possible Refactoring performed with the following Coccinelle patch: @@ struct ist i; @@ - i.ptr != NULL + isttest(i)
2021-03-02 12:57:28 -05:00
else if (*p == ']' && isttest(metadata[LOG_META_PID])) {
MINOR: log: adds syslog udp message handler and parsing. This patch introduce a new fd handler used to parse syslog message on udp. The parsing function returns level, facility and metadata that can be immediatly reused to forward message to a log server. This handler is enabled on udp listeners if proxy is internally set to mode PR_MODE_SYSLOG
2020-07-07 03:43:24 -04:00
if (p[1] != ':')
return;
metadata[LOG_META_PID].len = p - metadata[LOG_META_PID].ptr;
}
p++;
}
if (!metadata[LOG_META_TAG].len)
metadata[LOG_META_TAG].len = p - metadata[LOG_META_TAG].ptr;
CLEANUP: assorted typo fixes in the code and comments This is 12th iteration of typo fixes
2020-07-22 15:32:55 -04:00
/* let pass ':' and ' ', we still have warranty size is large enough */
MINOR: log: adds syslog udp message handler and parsing. This patch introduce a new fd handler used to parse syslog message on udp. The parsing function returns level, facility and metadata that can be immediatly reused to forward message to a log server. This handler is enabled on udp listeners if proxy is internally set to mode PR_MODE_SYSLOG
2020-07-07 03:43:24 -04:00
p += 2;
buflen -= p - buf;
buf = p;
*size = buflen;
*message = buf;
}
return;
bad_format:
/* bad syslog format, we reset all parsed syslog fields
* but priority is kept because we are able to re-build
* this message using LOF_FORMAT_PRIO.
*/
metadata[LOG_META_TIME].len = 0;
metadata[LOG_META_HOST].len = 0;
metadata[LOG_META_TAG].len = 0;
metadata[LOG_META_PID].len = 0;
metadata[LOG_META_MSGID].len = 0;
metadata[LOG_META_STDATA].len = 0;
return;
}
/*
* UDP syslog fd handler
*/
void syslog_fd_handler(int fd)
{
static THREAD_LOCAL struct ist metadata[LOG_META_FIELDS];
ssize_t ret = 0;
struct buffer *buf = get_trash_chunk();
size_t size;
char *message;
int level;
int facility;
struct listener *l = objt_listener(fdtab[fd].owner);
int max_accept;
CLEANUP: Apply bug_on.cocci The changes look safe to me, even if `DEBUG_STRICT` is not enabled.
2021-09-15 07:58:49 -04:00
BUG_ON(!l);
MINOR: log: adds syslog udp message handler and parsing. This patch introduce a new fd handler used to parse syslog message on udp. The parsing function returns level, facility and metadata that can be immediatly reused to forward message to a log server. This handler is enabled on udp listeners if proxy is internally set to mode PR_MODE_SYSLOG
2020-07-07 03:43:24 -04:00
MEDIUM: fd: merge fdtab[].ev and state for FD_EV_* and FD_POLL_* into state For a long time we've had fdtab[].ev and fdtab[].state which contain two arbitrary sets of information, one is mostly the configuration plus some shutdown reports and the other one is the latest polling status report which also contains some sticky error and shutdown reports. These ones used to be stored into distinct chars, complicating certain operations and not even allowing to clearly see concurrent accesses (e.g. fd_delete_orphan() would set the state to zero while fd_insert() would only set the event to zero). This patch creates a single uint with the two sets in it, still delimited at the byte level for better readability. The original FD_EV_* values remained at the lowest bit levels as they are also known by their bit value. The next step will consist in merging the remaining bits into it. The whole bits are now cleared both in fd_insert() and _fd_delete_orphan() because after a complete check, it is certain that in both cases these functions are the only ones touching these areas. Indeed, for _fd_delete_orphan(), the thread_mask has already been zeroed before a poller can call fd_update_event() which would touch the state, so it is certain that _fd_delete_orphan() is alone. Regarding fd_insert(), only one thread will get an FD at any moment, and it as this FD has already been released by _fd_delete_orphan() by definition it is certain that previous users have definitely stopped touching it. Strictly speaking there's no need for clearing the state again in fd_insert() but it's cheap and will remove some doubts during some troubleshooting sessions.
2021-04-06 11:23:40 -04:00
if (fdtab[fd].state & FD_POLL_IN) {
MINOR: log: adds syslog udp message handler and parsing. This patch introduce a new fd handler used to parse syslog message on udp. The parsing function returns level, facility and metadata that can be immediatly reused to forward message to a log server. This handler is enabled on udp listeners if proxy is internally set to mode PR_MODE_SYSLOG
2020-07-07 03:43:24 -04:00
if (!fd_recv_ready(fd))
return;
MINOR: listener: move maxaccept from listener to bind_conf Like for previous values, maxaccept is really per-bind_conf, so let's move it there. Some frontends (peers, log) set it to 1 so the assignment was slightly moved.
2023-01-12 12:52:23 -05:00
max_accept = l->bind_conf->maxaccept ? l->bind_conf->maxaccept : 1;
MINOR: log: adds syslog udp message handler and parsing. This patch introduce a new fd handler used to parse syslog message on udp. The parsing function returns level, facility and metadata that can be immediatly reused to forward message to a log server. This handler is enabled on udp listeners if proxy is internally set to mode PR_MODE_SYSLOG
2020-07-07 03:43:24 -04:00
do {
/* Source address */
struct sockaddr_storage saddr = {0};
socklen_t saddrlen;
saddrlen = sizeof(saddr);
ret = recvfrom(fd, buf->area, buf->size, 0, (struct sockaddr *)&saddr, &saddrlen);
if (ret < 0) {
if (errno == EINTR)
continue;
MINOR: tree-wide: always consider EWOULDBLOCK in addition to EAGAIN Some older systems may routinely return EWOULDBLOCK for some syscalls while we tend to check only for EAGAIN nowadays. Modern systems define EWOULDBLOCK as EAGAIN so that solves it, but on a few older ones (AIX, VMS etc) both are different, and for portability we'd need to test for both or we never know if we risk to confuse some status codes with plain errors. There were few entries, the most annoying ones are the switch/case because they require to only add the entry when it differs, but the other ones are really trivial.
2022-04-25 14:32:15 -04:00
if (errno == EAGAIN || errno == EWOULDBLOCK)
MINOR: log: adds syslog udp message handler and parsing. This patch introduce a new fd handler used to parse syslog message on udp. The parsing function returns level, facility and metadata that can be immediatly reused to forward message to a log server. This handler is enabled on udp listeners if proxy is internally set to mode PR_MODE_SYSLOG
2020-07-07 03:43:24 -04:00
fd_cant_recv(fd);
goto out;
}
buf->data = ret;
MINOR: log: adds counters on received syslog messages. This patch adds a global counter of received syslog messages and this one is exported on CLI "show info" as "CumRecvLogs". This patch also updates internal conn counter and freq of the listener and the proxy for each received log message to prepare a further export on the "show stats".
2020-07-09 17:23:34 -04:00
/* update counters */
CLEANUP: atomic/tree-wide: replace single increments/decrements with inc/dec This patch replaces roughly all occurrences of an HA_ATOMIC_ADD(&foo, 1) or HA_ATOMIC_SUB(&foo, 1) with the equivalent HA_ATOMIC_INC(&foo) and HA_ATOMIC_DEC(&foo) respectively. These are 507 changes over 45 files.
2021-04-06 07:53:36 -04:00
_HA_ATOMIC_INC(&cum_log_messages);
MINOR: stats: add by HTTP version cumulated number of sessions and requests Add cum_sess_ver[] new array of counters to count the number of cumulated HTTP sessions by version (h1, h2 or h3). Implement proxy_inc_fe_cum_sess_ver_ctr() to increment these counter. This function is called each a HTTP mux is correctly initialized. The QUIC must before verify the application operations for the mux is for h3 before calling proxy_inc_fe_cum_sess_ver_ctr(). ST_F_SESS_OTHER stat field for the cumulated of sessions others than HTTP sessions is deduced from ->cum_sess_ver counter (for all the session, not only HTTP sessions) from which the HTTP sessions counters are substracted. Add cum_req[] new array of counters to count the number of cumulated HTTP requests by version and others than HTTP requests. This new member replace ->cum_req. Modify proxy_inc_fe_req_ctr() which increments these counters to pass an HTTP version, 0 special values meaning "other than an HTTP request". This is the case for instance for syslog.c from which proxy_inc_fe_req_ctr() is called with 0 as version parameter. ST_F_REQ_TOT stat field compputing for the cumulated number of requests is modified to count the sum of all the cum_req[] counters. As this patch is useful for QUIC, it must be backported to 2.7.
2023-01-18 05:52:21 -05:00
proxy_inc_fe_req_ctr(l, l->bind_conf->frontend, 0);
MINOR: log: adds counters on received syslog messages. This patch adds a global counter of received syslog messages and this one is exported on CLI "show info" as "CumRecvLogs". This patch also updates internal conn counter and freq of the listener and the proxy for each received log message to prepare a further export on the "show stats".
2020-07-09 17:23:34 -04:00
MINOR: log: adds syslog udp message handler and parsing. This patch introduce a new fd handler used to parse syslog message on udp. The parsing function returns level, facility and metadata that can be immediatly reused to forward message to a log server. This handler is enabled on udp listeners if proxy is internally set to mode PR_MODE_SYSLOG
2020-07-07 03:43:24 -04:00
parse_log_message(buf->area, buf->data, &level, &facility, metadata, &message, &size);
MINOR: log: provide sending log context to process_send_log() when available This is another prerequisite work in preparation for log-profiles: in this patch we make process_send_log() aware of the log origin, primarily aiming for sess and txn logging steps such as error, accept, connect, close, as well as relevant sess and stream pointers.
2024-02-22 05:29:20 -05:00
process_send_log(NULL, &l->bind_conf->frontend->loggers, level, facility, metadata, message, size);
MINOR: log: adds syslog udp message handler and parsing. This patch introduce a new fd handler used to parse syslog message on udp. The parsing function returns level, facility and metadata that can be immediatly reused to forward message to a log server. This handler is enabled on udp listeners if proxy is internally set to mode PR_MODE_SYSLOG
2020-07-07 03:43:24 -04:00
} while (--max_accept);
}
out:
return;
}
MINOR: log: Provide a function to emit a log for an application Application is a generic term here. It is a modules which handle its own log server list, with no dependency on a proxy. Such applications can now call the function app_log() to log messages, passing a log server list and a tag as parameters. Internally, the function __send_log() has been adapted accordingly.
2019-08-11 13:40:12 -04:00
MEDIUM: log: syslog TCP support on log forward section. This patch re-introduce the "bind" statement on log forward sections to handle syslog TCP listeners as defined in rfc-6587. As complement it introduce "maxconn", "backlog" and "timeout client" statements to parameter those listeners.
2020-10-05 08:39:35 -04:00
/*
* IO Handler to handle message exchange with a syslog tcp client
*/
static void syslog_io_handler(struct appctx *appctx)
{
static THREAD_LOCAL struct ist metadata[LOG_META_FIELDS];
CLEANUP: applet: rename appctx_cs() to appctx_sc() It returns a stream connector, not a conn_stream anymore, so let's fix its name.
2022-05-27 05:08:15 -04:00
struct stconn *sc = appctx_sc(appctx);
CLEANUP: log-forward: rename all occurrences of stconn "cs" to "sc" In the log-forwarding applet, function arguments and local variables called "cs" were renamed to "sc" to avoid future confusion.
2022-05-27 04:36:36 -04:00
struct stream *s = __sc_strm(sc);
MEDIUM: log: syslog TCP support on log forward section. This patch re-introduce the "bind" statement on log forward sections to handle syslog TCP listeners as defined in rfc-6587. As complement it introduce "maxconn", "backlog" and "timeout client" statements to parameter those listeners.
2020-10-05 08:39:35 -04:00
struct proxy *frontend = strm_fe(s);
struct listener *l = strm_li(s);
struct buffer *buf = get_trash_chunk();
int max_accept;
int to_skip;
int facility;
int level;
char *message;
size_t size;
MINOR: applet: Remove uselelss test on SE_FL_SHR/SHW flags These both flags are set after releasing the applet, in appctx_shut(). Concretly, it means the applet is shutdown for reads and writes. Once set, the applet's I/O handler was no longer called. Tests on these flags are useless. There is no chance to match them.
2024-02-14 01:56:50 -05:00
if (unlikely(se_fl_test(appctx->sedesc, (SE_FL_EOS|SE_FL_ERROR)))) {
BUG/MEDIUM: log: Eat output data when waiting for appctx shutdown When the log applet is executed while a shut is pending, the remaining output data must always be consumed. Otherwise, this can prevent the stream to exit, leading to a spinning loop on the applet. It is 2.8-specific. No backport needed.
2023-04-11 01:56:50 -04:00
co_skip(sc_oc(sc), co_data(sc_oc(sc)));
MEDIUM: log: Use the sedesc to report and detect end of processing Just like for other applets, we now use the SE descriptor instead of the channel to report error and end-of-stream. Here, the refactoring only reports errors by setting SE_FL_ERROR flag.
2023-03-31 05:17:13 -04:00
goto out;
BUG/MEDIUM: log: Eat output data when waiting for appctx shutdown When the log applet is executed while a shut is pending, the remaining output data must always be consumed. Otherwise, this can prevent the stream to exit, leading to a spinning loop on the applet. It is 2.8-specific. No backport needed.
2023-04-11 01:56:50 -04:00
}
MEDIUM: log: Use the sedesc to report and detect end of processing Just like for other applets, we now use the SE descriptor instead of the channel to report error and end-of-stream. Here, the refactoring only reports errors by setting SE_FL_ERROR flag.
2023-03-31 05:17:13 -04:00
MINOR: listener: move maxaccept from listener to bind_conf Like for previous values, maxaccept is really per-bind_conf, so let's move it there. Some frontends (peers, log) set it to 1 so the assignment was slightly moved.
2023-01-12 12:52:23 -05:00
max_accept = l->bind_conf->maxaccept ? l->bind_conf->maxaccept : 1;
BUG/MEDIUM: log: Properly handle client aborts in syslog applet In the syslog applet, when there is no output data, nothing is performed and the applet leaves by requesting more data. But it is an issue because a client abort is only handled if it reported with the last bytes of the message. If the abort occurs after the message was handled, it is ignored. The session remains opened and inactive until the client timeout is being triggered. It no such timeout is configured, given that the default maxconn is 10, all slots can be quickly busy and make the applet unresponsive. To fix the issue, the best is to always try to read a message when the I/O handle is called. This way, the abort can be handled. And if there is no data, we leave as usual. This patch should fix the issue #2112. It must be backported as far as 2.4.
2023-04-17 10:34:29 -04:00
while (1) {
MEDIUM: log: syslog TCP support on log forward section. This patch re-introduce the "bind" statement on log forward sections to handle syslog TCP listeners as defined in rfc-6587. As complement it introduce "maxconn", "backlog" and "timeout client" statements to parameter those listeners.
2020-10-05 08:39:35 -04:00
char c;
if (max_accept <= 0)
goto missing_budget;
max_accept--;
CLEANUP: log-forward: rename all occurrences of stconn "cs" to "sc" In the log-forwarding applet, function arguments and local variables called "cs" were renamed to "sc" to avoid future confusion.
2022-05-27 04:36:36 -04:00
to_skip = co_getchar(sc_oc(sc), &c);
MEDIUM: log: syslog TCP support on log forward section. This patch re-introduce the "bind" statement on log forward sections to handle syslog TCP listeners as defined in rfc-6587. As complement it introduce "maxconn", "backlog" and "timeout client" statements to parameter those listeners.
2020-10-05 08:39:35 -04:00
if (!to_skip)
goto missing_data;
else if (to_skip < 0)
goto cli_abort;
if (c == '<') {
/* rfc-6587, Non-Transparent-Framing: messages separated by
* a trailing LF or CR LF
*/
CLEANUP: log-forward: rename all occurrences of stconn "cs" to "sc" In the log-forwarding applet, function arguments and local variables called "cs" were renamed to "sc" to avoid future confusion.
2022-05-27 04:36:36 -04:00
to_skip = co_getline(sc_oc(sc), buf->area, buf->size);
MEDIUM: log: syslog TCP support on log forward section. This patch re-introduce the "bind" statement on log forward sections to handle syslog TCP listeners as defined in rfc-6587. As complement it introduce "maxconn", "backlog" and "timeout client" statements to parameter those listeners.
2020-10-05 08:39:35 -04:00
if (!to_skip)
goto missing_data;
else if (to_skip < 0)
goto cli_abort;
if (buf->area[to_skip - 1] != '\n')
goto parse_error;
buf->data = to_skip - 1;
/* according to rfc-6587, some devices adds CR before LF */
if (buf->data && buf->area[buf->data - 1] == '\r')
buf->data--;
}
else if ((unsigned char)(c - '1') <= 8) {
/* rfc-6587, Octet-Counting: message length in ASCII
* (first digit can not be ZERO), followed by a space
* and message length
*/
char *p = NULL;
int msglen;
CLEANUP: log-forward: rename all occurrences of stconn "cs" to "sc" In the log-forwarding applet, function arguments and local variables called "cs" were renamed to "sc" to avoid future confusion.
2022-05-27 04:36:36 -04:00
to_skip = co_getword(sc_oc(sc), buf->area, buf->size, ' ');
MEDIUM: log: syslog TCP support on log forward section. This patch re-introduce the "bind" statement on log forward sections to handle syslog TCP listeners as defined in rfc-6587. As complement it introduce "maxconn", "backlog" and "timeout client" statements to parameter those listeners.
2020-10-05 08:39:35 -04:00
if (!to_skip)
goto missing_data;
else if (to_skip < 0)
goto cli_abort;
if (buf->area[to_skip - 1] != ' ')
goto parse_error;
BUG/MINOR: log: fixing bug in tcp syslog_io_handler Octet-Counting syslog_io_handler does specific treatment to handle syslog tcp octet counting: Logic was good, but a sneaky mistake prevented rfc-6587 octet counting from working properly. trash.area was used as an input buffer. It does not make sense here since it is uninitialized. Compilation was unaffected because trash is a thread local "global" variable. buf->area should definitely be used instead. This should be backported as far as 2.4.
2022-10-26 17:40:08 -04:00
msglen = strtol(buf->area, &p, 10);
MEDIUM: log: syslog TCP support on log forward section. This patch re-introduce the "bind" statement on log forward sections to handle syslog TCP listeners as defined in rfc-6587. As complement it introduce "maxconn", "backlog" and "timeout client" statements to parameter those listeners.
2020-10-05 08:39:35 -04:00
if (!msglen || p != &buf->area[to_skip - 1])
goto parse_error;
/* message seems too large */
if (msglen > buf->size)
goto parse_error;
CLEANUP: log-forward: rename all occurrences of stconn "cs" to "sc" In the log-forwarding applet, function arguments and local variables called "cs" were renamed to "sc" to avoid future confusion.
2022-05-27 04:36:36 -04:00
msglen = co_getblk(sc_oc(sc), buf->area, msglen, to_skip);
MEDIUM: log: syslog TCP support on log forward section. This patch re-introduce the "bind" statement on log forward sections to handle syslog TCP listeners as defined in rfc-6587. As complement it introduce "maxconn", "backlog" and "timeout client" statements to parameter those listeners.
2020-10-05 08:39:35 -04:00
if (!msglen)
goto missing_data;
else if (msglen < 0)
goto cli_abort;
buf->data = msglen;
to_skip += msglen;
}
else
goto parse_error;
CLEANUP: log-forward: rename all occurrences of stconn "cs" to "sc" In the log-forwarding applet, function arguments and local variables called "cs" were renamed to "sc" to avoid future confusion.
2022-05-27 04:36:36 -04:00
co_skip(sc_oc(sc), to_skip);
MEDIUM: log: syslog TCP support on log forward section. This patch re-introduce the "bind" statement on log forward sections to handle syslog TCP listeners as defined in rfc-6587. As complement it introduce "maxconn", "backlog" and "timeout client" statements to parameter those listeners.
2020-10-05 08:39:35 -04:00
/* update counters */
CLEANUP: atomic/tree-wide: replace single increments/decrements with inc/dec This patch replaces roughly all occurrences of an HA_ATOMIC_ADD(&foo, 1) or HA_ATOMIC_SUB(&foo, 1) with the equivalent HA_ATOMIC_INC(&foo) and HA_ATOMIC_DEC(&foo) respectively. These are 507 changes over 45 files.
2021-04-06 07:53:36 -04:00
_HA_ATOMIC_INC(&cum_log_messages);
MINOR: stats: add by HTTP version cumulated number of sessions and requests Add cum_sess_ver[] new array of counters to count the number of cumulated HTTP sessions by version (h1, h2 or h3). Implement proxy_inc_fe_cum_sess_ver_ctr() to increment these counter. This function is called each a HTTP mux is correctly initialized. The QUIC must before verify the application operations for the mux is for h3 before calling proxy_inc_fe_cum_sess_ver_ctr(). ST_F_SESS_OTHER stat field for the cumulated of sessions others than HTTP sessions is deduced from ->cum_sess_ver counter (for all the session, not only HTTP sessions) from which the HTTP sessions counters are substracted. Add cum_req[] new array of counters to count the number of cumulated HTTP requests by version and others than HTTP requests. This new member replace ->cum_req. Modify proxy_inc_fe_req_ctr() which increments these counters to pass an HTTP version, 0 special values meaning "other than an HTTP request". This is the case for instance for syslog.c from which proxy_inc_fe_req_ctr() is called with 0 as version parameter. ST_F_REQ_TOT stat field compputing for the cumulated number of requests is modified to count the sum of all the cum_req[] counters. As this patch is useful for QUIC, it must be backported to 2.7.
2023-01-18 05:52:21 -05:00
proxy_inc_fe_req_ctr(l, frontend, 0);
MEDIUM: log: syslog TCP support on log forward section. This patch re-introduce the "bind" statement on log forward sections to handle syslog TCP listeners as defined in rfc-6587. As complement it introduce "maxconn", "backlog" and "timeout client" statements to parameter those listeners.
2020-10-05 08:39:35 -04:00
parse_log_message(buf->area, buf->data, &level, &facility, metadata, &message, &size);
MINOR: log: provide sending log context to process_send_log() when available This is another prerequisite work in preparation for log-profiles: in this patch we make process_send_log() aware of the log origin, primarily aiming for sess and txn logging steps such as error, accept, connect, close, as well as relevant sess and stream pointers.
2024-02-22 05:29:20 -05:00
process_send_log(NULL, &frontend->loggers, level, facility, metadata, message, size);
MEDIUM: log: syslog TCP support on log forward section. This patch re-introduce the "bind" statement on log forward sections to handle syslog TCP listeners as defined in rfc-6587. As complement it introduce "maxconn", "backlog" and "timeout client" statements to parameter those listeners.
2020-10-05 08:39:35 -04:00
}
missing_data:
/* we need more data to read */
BUG/MINOR: syslog: Request for more data if message was not fully received In the syslog applet, when a message was not fully received, we must request for more data by calling appctx_need_more_data() and not by setting CF_READ_DONTWAIT flag on the request channel. Indeed, this flag is only used to only try a read at once. This patch could be backported as far as 2.4. On 2.5 and 2.4, applet_need_more_data() must be replaced by si_cant_get().
2023-03-16 09:27:29 -04:00
applet_need_more_data(appctx);
MEDIUM: log: syslog TCP support on log forward section. This patch re-introduce the "bind" statement on log forward sections to handle syslog TCP listeners as defined in rfc-6587. As complement it introduce "maxconn", "backlog" and "timeout client" statements to parameter those listeners.
2020-10-05 08:39:35 -04:00
return;
missing_budget:
/* it may remain some stuff to do, let's retry later */
appctx_wakeup(appctx);
return;
parse_error:
if (l->counters)
CLEANUP: atomic/tree-wide: replace single increments/decrements with inc/dec This patch replaces roughly all occurrences of an HA_ATOMIC_ADD(&foo, 1) or HA_ATOMIC_SUB(&foo, 1) with the equivalent HA_ATOMIC_INC(&foo) and HA_ATOMIC_DEC(&foo) respectively. These are 507 changes over 45 files.
2021-04-06 07:53:36 -04:00
_HA_ATOMIC_INC(&l->counters->failed_req);
_HA_ATOMIC_INC(&frontend->fe_counters.failed_req);
MEDIUM: log: syslog TCP support on log forward section. This patch re-introduce the "bind" statement on log forward sections to handle syslog TCP listeners as defined in rfc-6587. As complement it introduce "maxconn", "backlog" and "timeout client" statements to parameter those listeners.
2020-10-05 08:39:35 -04:00
MEDIUM: log: Use the sedesc to report and detect end of processing Just like for other applets, we now use the SE descriptor instead of the channel to report error and end-of-stream. Here, the refactoring only reports errors by setting SE_FL_ERROR flag.
2023-03-31 05:17:13 -04:00
goto error;
MEDIUM: log: syslog TCP support on log forward section. This patch re-introduce the "bind" statement on log forward sections to handle syslog TCP listeners as defined in rfc-6587. As complement it introduce "maxconn", "backlog" and "timeout client" statements to parameter those listeners.
2020-10-05 08:39:35 -04:00
cli_abort:
if (l->counters)
CLEANUP: atomic/tree-wide: replace single increments/decrements with inc/dec This patch replaces roughly all occurrences of an HA_ATOMIC_ADD(&foo, 1) or HA_ATOMIC_SUB(&foo, 1) with the equivalent HA_ATOMIC_INC(&foo) and HA_ATOMIC_DEC(&foo) respectively. These are 507 changes over 45 files.
2021-04-06 07:53:36 -04:00
_HA_ATOMIC_INC(&l->counters->cli_aborts);
_HA_ATOMIC_INC(&frontend->fe_counters.cli_aborts);
MEDIUM: log: syslog TCP support on log forward section. This patch re-introduce the "bind" statement on log forward sections to handle syslog TCP listeners as defined in rfc-6587. As complement it introduce "maxconn", "backlog" and "timeout client" statements to parameter those listeners.
2020-10-05 08:39:35 -04:00
MEDIUM: log: Use the sedesc to report and detect end of processing Just like for other applets, we now use the SE descriptor instead of the channel to report error and end-of-stream. Here, the refactoring only reports errors by setting SE_FL_ERROR flag.
2023-03-31 05:17:13 -04:00
error:
se_fl_set(appctx->sedesc, SE_FL_ERROR);
out:
MEDIUM: log: syslog TCP support on log forward section. This patch re-introduce the "bind" statement on log forward sections to handle syslog TCP listeners as defined in rfc-6587. As complement it introduce "maxconn", "backlog" and "timeout client" statements to parameter those listeners.
2020-10-05 08:39:35 -04:00
return;
}
static struct applet syslog_applet = {
.obj_type = OBJ_TYPE_APPLET,
.name = "<SYSLOG>", /* used for logging */
.fct = syslog_io_handler,
.release = NULL,
};
MINOR: log/applet: add new function syslog_applet_append_event() This function takes a buffer on input, and offset and a length, and consumes the block from that buffer to send it to the appctx's output buffer. Contrary to its sibling applet_append_line(), instead of just appending an LF at the end of the line, it prepends the message size in decimal and a space before the message, as expected by syslog TCP implementaions. This will be used to simplify the ring reader code.
2024-02-27 10:17:42 -05:00
/* Atomically append an event to applet >ctx>'s output, prepending it with its
MEDIUM: ring: change the ring reader to use the new vector-based API now The code now looks cleaner and more easily shows what still needs to be addressed. There are not that many changes in practice, these are mostly mechanical, essentially hiding the buffer from the callers.
2024-02-27 01:58:26 -05:00
* size in decimal followed by a space. The line is read from vectors <v1> and
* <v2> at offset <ofs> relative to the area's origin, for <len> bytes. It
* returns the number of bytes consumed from the input vectors on success, -1
* if it temporarily cannot (buffer full), -2 if it will never be able to (too
* large msg). The input vectors are not modified. The caller is responsible for
* making sure that there are at least ofs+len bytes in the input buffer.
MINOR: log/applet: add new function syslog_applet_append_event() This function takes a buffer on input, and offset and a length, and consumes the block from that buffer to send it to the appctx's output buffer. Contrary to its sibling applet_append_line(), instead of just appending an LF at the end of the line, it prepends the message size in decimal and a space before the message, as expected by syslog TCP implementaions. This will be used to simplify the ring reader code.
2024-02-27 10:17:42 -05:00
*/
MEDIUM: ring: change the ring reader to use the new vector-based API now The code now looks cleaner and more easily shows what still needs to be addressed. There are not that many changes in practice, these are mostly mechanical, essentially hiding the buffer from the callers.
2024-02-27 01:58:26 -05:00
ssize_t syslog_applet_append_event(void *ctx, struct ist v1, struct ist v2, size_t ofs, size_t len)
MINOR: log/applet: add new function syslog_applet_append_event() This function takes a buffer on input, and offset and a length, and consumes the block from that buffer to send it to the appctx's output buffer. Contrary to its sibling applet_append_line(), instead of just appending an LF at the end of the line, it prepends the message size in decimal and a space before the message, as expected by syslog TCP implementaions. This will be used to simplify the ring reader code.
2024-02-27 10:17:42 -05:00
{
struct appctx *appctx = ctx;
char *p;
/* first, encode the message's size */
chunk_reset(&trash);
p = ulltoa(len, trash.area, b_size(&trash));
if (p) {
trash.data = p - trash.area;
trash.area[trash.data++] = ' ';
}
/* check if the message has a chance to fit */
if (unlikely(!p || trash.data + len > b_size(&trash)))
return -2;
/* try to transfer it or report full */
BUG/MEDIUM: log/ring: broken syslog octet counting As reported by Tristan in GH #2561, syslog messages sent over rings are malformed since commit 01aa0a05 ("MEDIUM: ring: change the ring reader to use the new vector-based API now"). Indeed, take a look at the following log message produced prior to 01aa0a05: 181 <134>1 2024-05-07T09:45:21.543263+02:00 - haproxy 113700 - - 127.0.0.1:56136 [07/May/2024:09:45:21.491] front front/s1 0/0/21/30/51 404 369 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1" Starting with 01aa0a05, here's the equivalent log message: <134>1 2024-05-07T09:45:21.543263+02:00 - haproxy 112729 - - 127.0.0.1:56136 [07/May/2024:09:45:21.491] front front/s1 0/0/66/39/105 404 369 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1"-fwr -> Message is missing octet counting header, and garbage bytes are found at the end of the payload. This bug is caused by a small mistake in syslog_applet_append_event(): when the function was refactored to use vector API instead of buffer API, we used 'trash.area' as starting pointer to write the event instead of 'trash.area + trash.data', causing existing octet counting prefix (already written in trash) to be overwritten and trash.data to be wrongly incremented. No backport needed (01aa0a05 was introduced during 3.0 development)
2024-05-07 10:46:35 -04:00
trash.data += vp_peek_ofs(v1, v2, ofs, trash.area + trash.data, len);
MINOR: log/applet: add new function syslog_applet_append_event() This function takes a buffer on input, and offset and a length, and consumes the block from that buffer to send it to the appctx's output buffer. Contrary to its sibling applet_append_line(), instead of just appending an LF at the end of the line, it prepends the message size in decimal and a space before the message, as expected by syslog TCP implementaions. This will be used to simplify the ring reader code.
2024-02-27 10:17:42 -05:00
if (applet_putchk(appctx, &trash) == -1)
return -1;
/* OK done */
return len;
}
MEDIUM: log: adds log forwarding section. Log forwarding: It is possible to declare one or multiple log forwarding section, haproxy will forward all received log messages to a log servers list. log-forward <name> Creates a new log forwarder proxy identified as <name>. bind <addr> [param*] Used to configure a log udp listener to receive messages to forward. Only udp listeners are allowed, address must be prefixed using 'udp@', 'udp4@' or 'udp6@'. This supports for all "bind" parameters found in 5.1 paragraph but most of them are irrelevant for udp/syslog case. log global log <address> [len <length>] [format <format>] [sample <ranges>:<smp_size>] <facility> [<level> [<minlevel>]] Used to configure target log servers. See more details on proxies documentation. If no format specified, haproxy tries to keep the incoming log format. Configured facility is ignored, except if incoming message does not present a facility but one is mandatory on the outgoing format. If there is no timestamp available in the input format, but the field exists in output format, haproxy will use the local date. Example: global log stderr format iso local7 ring myring description "My local buffer" format rfc5424 maxlen 1200 size 32764 timeout connect 5s timeout server 10s # syslog tcp server server mysyslogsrv 127.0.0.1:514 log-proto octet-count log-forward sylog-loadb bind udp4@127.0.0.1:1514 # all messages on stderr log global # all messages on local tcp syslog server log ring@myring local0 # load balance messages on 4 udp syslog servers log 127.0.0.1:10001 sample 1:4 local0 log 127.0.0.1:10002 sample 2:4 local0 log 127.0.0.1:10003 sample 3:4 local0 log 127.0.0.1:10004 sample 4:4 local0
2020-07-07 08:19:42 -04:00
/*
* Parse "log-forward" section and create corresponding sink buffer.
*
* The function returns 0 in success case, otherwise, it returns error
* flags.
*/
int cfg_parse_log_forward(const char *file, int linenum, char **args, int kwm)
{
BUG/MINOR: log: fix multiple error paths in cfg_parse_log_forward() Multiple error paths were badly handled in cfg_parse_log_forward(): some errors were raised without interrupting the function execution, resulting in undefined behavior. Instead of fixing issues separately, let's fix the whole function at once. This should be backported as far as 2.4.
2023-07-03 12:33:18 -04:00
int err_code = ERR_NONE;
MEDIUM: log: adds log forwarding section. Log forwarding: It is possible to declare one or multiple log forwarding section, haproxy will forward all received log messages to a log servers list. log-forward <name> Creates a new log forwarder proxy identified as <name>. bind <addr> [param*] Used to configure a log udp listener to receive messages to forward. Only udp listeners are allowed, address must be prefixed using 'udp@', 'udp4@' or 'udp6@'. This supports for all "bind" parameters found in 5.1 paragraph but most of them are irrelevant for udp/syslog case. log global log <address> [len <length>] [format <format>] [sample <ranges>:<smp_size>] <facility> [<level> [<minlevel>]] Used to configure target log servers. See more details on proxies documentation. If no format specified, haproxy tries to keep the incoming log format. Configured facility is ignored, except if incoming message does not present a facility but one is mandatory on the outgoing format. If there is no timestamp available in the input format, but the field exists in output format, haproxy will use the local date. Example: global log stderr format iso local7 ring myring description "My local buffer" format rfc5424 maxlen 1200 size 32764 timeout connect 5s timeout server 10s # syslog tcp server server mysyslogsrv 127.0.0.1:514 log-proto octet-count log-forward sylog-loadb bind udp4@127.0.0.1:1514 # all messages on stderr log global # all messages on local tcp syslog server log ring@myring local0 # load balance messages on 4 udp syslog servers log 127.0.0.1:10001 sample 1:4 local0 log 127.0.0.1:10002 sample 2:4 local0 log 127.0.0.1:10003 sample 3:4 local0 log 127.0.0.1:10004 sample 4:4 local0
2020-07-07 08:19:42 -04:00
struct proxy *px;
char *errmsg = NULL;
const char *err = NULL;
if (strcmp(args[0], "log-forward") == 0) {
if (!*args[1]) {
BUG/MINOR: log: fix missing name error message in cfg_parse_log_forward() "missing name for ip-forward section" is generated instead of "missing name name for log-forward section" in cfg_parse_log_forward(). This may be backported up to 2.4.
2023-07-03 11:35:54 -04:00
ha_alert("parsing [%s:%d] : missing name for log-forward section.\n", file, linenum);
MEDIUM: log: adds log forwarding section. Log forwarding: It is possible to declare one or multiple log forwarding section, haproxy will forward all received log messages to a log servers list. log-forward <name> Creates a new log forwarder proxy identified as <name>. bind <addr> [param*] Used to configure a log udp listener to receive messages to forward. Only udp listeners are allowed, address must be prefixed using 'udp@', 'udp4@' or 'udp6@'. This supports for all "bind" parameters found in 5.1 paragraph but most of them are irrelevant for udp/syslog case. log global log <address> [len <length>] [format <format>] [sample <ranges>:<smp_size>] <facility> [<level> [<minlevel>]] Used to configure target log servers. See more details on proxies documentation. If no format specified, haproxy tries to keep the incoming log format. Configured facility is ignored, except if incoming message does not present a facility but one is mandatory on the outgoing format. If there is no timestamp available in the input format, but the field exists in output format, haproxy will use the local date. Example: global log stderr format iso local7 ring myring description "My local buffer" format rfc5424 maxlen 1200 size 32764 timeout connect 5s timeout server 10s # syslog tcp server server mysyslogsrv 127.0.0.1:514 log-proto octet-count log-forward sylog-loadb bind udp4@127.0.0.1:1514 # all messages on stderr log global # all messages on local tcp syslog server log ring@myring local0 # load balance messages on 4 udp syslog servers log 127.0.0.1:10001 sample 1:4 local0 log 127.0.0.1:10002 sample 2:4 local0 log 127.0.0.1:10003 sample 3:4 local0 log 127.0.0.1:10004 sample 4:4 local0
2020-07-07 08:19:42 -04:00
err_code |= ERR_ALERT | ERR_ABORT;
goto out;
}
if (alertif_too_many_args(1, file, linenum, args, &err_code))
goto out;
err = invalid_char(args[1]);
if (err) {
ha_alert("parsing [%s:%d] : character '%c' is not permitted in '%s' name '%s'.\n",
file, linenum, *err, args[0], args[1]);
err_code |= ERR_ALERT | ERR_ABORT;
goto out;
}
BUG/MINOR: proxy/log: frontend/backend and log forward names must differ This patch disallow to use same name for a log forward section and a frontend/backend section.
2020-10-07 11:05:59 -04:00
px = log_forward_by_name(args[1]);
if (px) {
ha_alert("Parsing [%s:%d]: log-forward section '%s' has the same name as another log-forward section declared at %s:%d.\n",
file, linenum, args[1], px->conf.file, px->conf.line);
err_code |= ERR_ALERT | ERR_FATAL;
BUG/MINOR: log: fix multiple error paths in cfg_parse_log_forward() Multiple error paths were badly handled in cfg_parse_log_forward(): some errors were raised without interrupting the function execution, resulting in undefined behavior. Instead of fixing issues separately, let's fix the whole function at once. This should be backported as far as 2.4.
2023-07-03 12:33:18 -04:00
goto out;
BUG/MINOR: proxy/log: frontend/backend and log forward names must differ This patch disallow to use same name for a log forward section and a frontend/backend section.
2020-10-07 11:05:59 -04:00
}
px = proxy_find_by_name(args[1], 0, 0);
if (px) {
ha_alert("Parsing [%s:%d]: log forward section '%s' has the same name as %s '%s' declared at %s:%d.\n",
file, linenum, args[1], proxy_type_str(px),
px->id, px->conf.file, px->conf.line);
err_code |= ERR_ALERT | ERR_FATAL;
BUG/MINOR: log: fix multiple error paths in cfg_parse_log_forward() Multiple error paths were badly handled in cfg_parse_log_forward(): some errors were raised without interrupting the function execution, resulting in undefined behavior. Instead of fixing issues separately, let's fix the whole function at once. This should be backported as far as 2.4.
2023-07-03 12:33:18 -04:00
goto out;
MEDIUM: log: adds log forwarding section. Log forwarding: It is possible to declare one or multiple log forwarding section, haproxy will forward all received log messages to a log servers list. log-forward <name> Creates a new log forwarder proxy identified as <name>. bind <addr> [param*] Used to configure a log udp listener to receive messages to forward. Only udp listeners are allowed, address must be prefixed using 'udp@', 'udp4@' or 'udp6@'. This supports for all "bind" parameters found in 5.1 paragraph but most of them are irrelevant for udp/syslog case. log global log <address> [len <length>] [format <format>] [sample <ranges>:<smp_size>] <facility> [<level> [<minlevel>]] Used to configure target log servers. See more details on proxies documentation. If no format specified, haproxy tries to keep the incoming log format. Configured facility is ignored, except if incoming message does not present a facility but one is mandatory on the outgoing format. If there is no timestamp available in the input format, but the field exists in output format, haproxy will use the local date. Example: global log stderr format iso local7 ring myring description "My local buffer" format rfc5424 maxlen 1200 size 32764 timeout connect 5s timeout server 10s # syslog tcp server server mysyslogsrv 127.0.0.1:514 log-proto octet-count log-forward sylog-loadb bind udp4@127.0.0.1:1514 # all messages on stderr log global # all messages on local tcp syslog server log ring@myring local0 # load balance messages on 4 udp syslog servers log 127.0.0.1:10001 sample 1:4 local0 log 127.0.0.1:10002 sample 2:4 local0 log 127.0.0.1:10003 sample 3:4 local0 log 127.0.0.1:10004 sample 4:4 local0
2020-07-07 08:19:42 -04:00
}
px = calloc(1, sizeof *px);
if (!px) {
err_code |= ERR_ALERT | ERR_FATAL;
goto out;
}
BUG/MAJOR: segfault using multiple log forward sections. For each new log forward section, the proxy was added to the log forward proxy list but the ref on the previous log forward section's proxy was scratched using "init_new_proxy" which performs a memset. After configuration parsing this list contains only the last section's proxy. The post processing walk through this list to resolve "ring" names. Since some section's proxies are missing in this list, the resolving is not done for those ones and the pointer on the ring is kept to null causing a segfault at runtime trying to write a log message into the ring. This patch shift the "init_new_proxy" before adding the ref on the previous log forward section's proxy on currently parsed one. This patch shoud fix github issue #1464 This patch should be backported to 2.3
2021-12-01 06:08:42 -05:00
init_new_proxy(px);
MEDIUM: log: adds log forwarding section. Log forwarding: It is possible to declare one or multiple log forwarding section, haproxy will forward all received log messages to a log servers list. log-forward <name> Creates a new log forwarder proxy identified as <name>. bind <addr> [param*] Used to configure a log udp listener to receive messages to forward. Only udp listeners are allowed, address must be prefixed using 'udp@', 'udp4@' or 'udp6@'. This supports for all "bind" parameters found in 5.1 paragraph but most of them are irrelevant for udp/syslog case. log global log <address> [len <length>] [format <format>] [sample <ranges>:<smp_size>] <facility> [<level> [<minlevel>]] Used to configure target log servers. See more details on proxies documentation. If no format specified, haproxy tries to keep the incoming log format. Configured facility is ignored, except if incoming message does not present a facility but one is mandatory on the outgoing format. If there is no timestamp available in the input format, but the field exists in output format, haproxy will use the local date. Example: global log stderr format iso local7 ring myring description "My local buffer" format rfc5424 maxlen 1200 size 32764 timeout connect 5s timeout server 10s # syslog tcp server server mysyslogsrv 127.0.0.1:514 log-proto octet-count log-forward sylog-loadb bind udp4@127.0.0.1:1514 # all messages on stderr log global # all messages on local tcp syslog server log ring@myring local0 # load balance messages on 4 udp syslog servers log 127.0.0.1:10001 sample 1:4 local0 log 127.0.0.1:10002 sample 2:4 local0 log 127.0.0.1:10003 sample 3:4 local0 log 127.0.0.1:10004 sample 4:4 local0
2020-07-07 08:19:42 -04:00
px->next = cfg_log_forward;
cfg_log_forward = px;
px->conf.file = strdup(file);
px->conf.line = linenum;
px->mode = PR_MODE_SYSLOG;
MINOR: counters: move last_change into counters struct last_change was a member present in both proxy and server struct. It is used as an age statistics to report the last update of the object. Move last_change into fe_counters/be_counters. This is necessary to be able to manipulate it through generic stat column and report it into stats-file. Note that there is a change for proxy structure with now 2 different last_change values, on frontend and backend side. Special care was taken to ensure that the value is initialized only on the proxy side. The other value is set to 0 unless a listen proxy is instantiated. For the moment, only backend counter is reported in stats. However, with now two distinct values, stats could be extended to report it on both side.
2024-04-30 06:04:57 -04:00
px->fe_counters.last_change = ns_to_sec(now_ns);
MEDIUM: log: syslog TCP support on log forward section. This patch re-introduce the "bind" statement on log forward sections to handle syslog TCP listeners as defined in rfc-6587. As complement it introduce "maxconn", "backlog" and "timeout client" statements to parameter those listeners.
2020-10-05 08:39:35 -04:00
px->cap = PR_CAP_FE;
px->maxconn = 10;
px->timeout.client = TICK_ETERNITY;
px->accept = frontend_accept;
px->default_target = &syslog_applet.obj_type;
MEDIUM: log: adds log forwarding section. Log forwarding: It is possible to declare one or multiple log forwarding section, haproxy will forward all received log messages to a log servers list. log-forward <name> Creates a new log forwarder proxy identified as <name>. bind <addr> [param*] Used to configure a log udp listener to receive messages to forward. Only udp listeners are allowed, address must be prefixed using 'udp@', 'udp4@' or 'udp6@'. This supports for all "bind" parameters found in 5.1 paragraph but most of them are irrelevant for udp/syslog case. log global log <address> [len <length>] [format <format>] [sample <ranges>:<smp_size>] <facility> [<level> [<minlevel>]] Used to configure target log servers. See more details on proxies documentation. If no format specified, haproxy tries to keep the incoming log format. Configured facility is ignored, except if incoming message does not present a facility but one is mandatory on the outgoing format. If there is no timestamp available in the input format, but the field exists in output format, haproxy will use the local date. Example: global log stderr format iso local7 ring myring description "My local buffer" format rfc5424 maxlen 1200 size 32764 timeout connect 5s timeout server 10s # syslog tcp server server mysyslogsrv 127.0.0.1:514 log-proto octet-count log-forward sylog-loadb bind udp4@127.0.0.1:1514 # all messages on stderr log global # all messages on local tcp syslog server log ring@myring local0 # load balance messages on 4 udp syslog servers log 127.0.0.1:10001 sample 1:4 local0 log 127.0.0.1:10002 sample 2:4 local0 log 127.0.0.1:10003 sample 3:4 local0 log 127.0.0.1:10004 sample 4:4 local0
2020-07-07 08:19:42 -04:00
px->id = strdup(args[1]);
}
CLEANUP: Compare the return value of `XXXcmp()` functions with zero According to coding-style.txt it is recommended to use: `strcmp(a, b) == 0` instead of `!strcmp(a, b)` So let's do this. The change was performed by running the following (very long) coccinelle patch on src/: @@ statement S; expression E; expression F; @@ if ( ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) != 0 ) ( S | { ... } ) @@ statement S; expression E; expression F; @@ if ( - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 ) ( S | { ... } ) @@ expression E; expression F; expression G; @@ ( G && ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) != 0 ) @@ expression E; expression F; expression G; @@ ( G || ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) != 0 ) @@ expression E; expression F; expression G; @@ ( ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) != 0 && G ) @@ expression E; expression F; expression G; @@ ( ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) != 0 || G ) @@ expression E; expression F; expression G; @@ ( G && - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 ) @@ expression E; expression F; expression G; @@ ( G || - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 ) @@ expression E; expression F; expression G; @@ ( - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 && G ) @@ expression E; expression F; expression G; @@ ( - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 || G ) @@ expression E; expression F; expression G; @@ ( - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 )
2021-01-02 16:31:53 -05:00
else if (strcmp(args[0], "maxconn") == 0) { /* maxconn */
MEDIUM: log: syslog TCP support on log forward section. This patch re-introduce the "bind" statement on log forward sections to handle syslog TCP listeners as defined in rfc-6587. As complement it introduce "maxconn", "backlog" and "timeout client" statements to parameter those listeners.
2020-10-05 08:39:35 -04:00
if (warnifnotcap(cfg_log_forward, PR_CAP_FE, file, linenum, args[0], " Maybe you want 'fullconn' instead ?"))
err_code |= ERR_WARN;
if (*(args[1]) == 0) {
ha_alert("parsing [%s:%d] : '%s' expects an integer argument.\n", file, linenum, args[0]);
err_code |= ERR_ALERT | ERR_FATAL;
goto out;
}
cfg_log_forward->maxconn = atol(args[1]);
if (alertif_too_many_args(1, file, linenum, args, &err_code))
goto out;
}
CLEANUP: Compare the return value of `XXXcmp()` functions with zero According to coding-style.txt it is recommended to use: `strcmp(a, b) == 0` instead of `!strcmp(a, b)` So let's do this. The change was performed by running the following (very long) coccinelle patch on src/: @@ statement S; expression E; expression F; @@ if ( ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) != 0 ) ( S | { ... } ) @@ statement S; expression E; expression F; @@ if ( - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 ) ( S | { ... } ) @@ expression E; expression F; expression G; @@ ( G && ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) != 0 ) @@ expression E; expression F; expression G; @@ ( G || ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) != 0 ) @@ expression E; expression F; expression G; @@ ( ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) != 0 && G ) @@ expression E; expression F; expression G; @@ ( ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) != 0 || G ) @@ expression E; expression F; expression G; @@ ( G && - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 ) @@ expression E; expression F; expression G; @@ ( G || - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 ) @@ expression E; expression F; expression G; @@ ( - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 && G ) @@ expression E; expression F; expression G; @@ ( - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 || G ) @@ expression E; expression F; expression G; @@ ( - ! ( dns_hostname_cmp | eb_memcmp | memcmp | strcasecmp | strcmp | strncasecmp | strncmp ) - (E, F) + (E, F) == 0 )
2021-01-02 16:31:53 -05:00
else if (strcmp(args[0], "backlog") == 0) { /* backlog */
MEDIUM: log: syslog TCP support on log forward section. This patch re-introduce the "bind" statement on log forward sections to handle syslog TCP listeners as defined in rfc-6587. As complement it introduce "maxconn", "backlog" and "timeout client" statements to parameter those listeners.
2020-10-05 08:39:35 -04:00
if (warnifnotcap(cfg_log_forward, PR_CAP_FE, file, linenum, args[0], NULL))
err_code |= ERR_WARN;
if (*(args[1]) == 0) {
ha_alert("parsing [%s:%d] : '%s' expects an integer argument.\n", file, linenum, args[0]);
err_code |= ERR_ALERT | ERR_FATAL;
goto out;
}
cfg_log_forward->backlog = atol(args[1]);
if (alertif_too_many_args(1, file, linenum, args, &err_code))
goto out;
}
else if (strcmp(args[0], "bind") == 0) {
int cur_arg;
struct bind_conf *bind_conf;
struct listener *l;
BUG/MAJOR: log-forward: Fix ssl layer not initialized on bind even if configured Since commit 2071a99df ("MINOR: listener/ssl: set the SSL xprt layer only once the whole config is known") the xprt is initialized for ssl directly from a generic funtion used to parse bind args. But the 'bind' lines from 'log-forward' sections were forgotten in commit 55f0f7bb5 ("MINOR: config: use the new bind_parse_args_list() to parse a "bind" line"). This patch re-works 'log-forward' section parsing to use the generic function to parse bind args and fix the issue. Since the generic way to parse was introduced in 2.6, this patch should be backported as far as this version.
2022-08-19 09:16:26 -04:00
int ret;
MEDIUM: log: syslog TCP support on log forward section. This patch re-introduce the "bind" statement on log forward sections to handle syslog TCP listeners as defined in rfc-6587. As complement it introduce "maxconn", "backlog" and "timeout client" statements to parameter those listeners.
2020-10-05 08:39:35 -04:00
cur_arg = 1;
bind_conf = bind_conf_alloc(cfg_log_forward, file, linenum,
NULL, xprt_get(XPRT_RAW));
if (!bind_conf) {
ha_alert("parsing [%s:%d] : out of memory error.", file, linenum);
err_code |= ERR_ALERT | ERR_FATAL;
goto out;
}
MINOR: listener: move maxaccept from listener to bind_conf Like for previous values, maxaccept is really per-bind_conf, so let's move it there. Some frontends (peers, log) set it to 1 so the assignment was slightly moved.
2023-01-12 12:52:23 -05:00
bind_conf->maxaccept = global.tune.maxaccept ? global.tune.maxaccept : MAX_ACCEPT;
MINOR: listener: move the ->accept callback to the bind_conf The accept callback directly derives from the upper layer, generally it's session_accept_fd(). As such it's also defined per bind line so it makes sense to move it there.
2023-01-12 13:10:17 -05:00
bind_conf->accept = session_accept_fd;
MINOR: listener: move maxaccept from listener to bind_conf Like for previous values, maxaccept is really per-bind_conf, so let's move it there. Some frontends (peers, log) set it to 1 so the assignment was slightly moved.
2023-01-12 12:52:23 -05:00
MEDIUM: log: syslog TCP support on log forward section. This patch re-introduce the "bind" statement on log forward sections to handle syslog TCP listeners as defined in rfc-6587. As complement it introduce "maxconn", "backlog" and "timeout client" statements to parameter those listeners.
2020-10-05 08:39:35 -04:00
if (!str2listener(args[1], cfg_log_forward, bind_conf, file, linenum, &errmsg)) {
if (errmsg && *errmsg) {
indent_msg(&errmsg, 2);
ha_alert("parsing [%s:%d] : '%s %s' : %s\n", file, linenum, args[0], args[1], errmsg);
}
else {
ha_alert("parsing [%s:%d] : '%s %s' : error encountered while parsing listening address %s.\n",
file, linenum, args[0], args[1], args[2]);
}
BUG/MINOR: log: fix multiple error paths in cfg_parse_log_forward() Multiple error paths were badly handled in cfg_parse_log_forward(): some errors were raised without interrupting the function execution, resulting in undefined behavior. Instead of fixing issues separately, let's fix the whole function at once. This should be backported as far as 2.4.
2023-07-03 12:33:18 -04:00
err_code |= ERR_ALERT | ERR_FATAL;
goto out;
MEDIUM: log: syslog TCP support on log forward section. This patch re-introduce the "bind" statement on log forward sections to handle syslog TCP listeners as defined in rfc-6587. As complement it introduce "maxconn", "backlog" and "timeout client" statements to parameter those listeners.
2020-10-05 08:39:35 -04:00
}
list_for_each_entry(l, &bind_conf->listeners, by_bind) {
global.maxsock++;
}
cur_arg++;
BUG/MAJOR: log-forward: Fix ssl layer not initialized on bind even if configured Since commit 2071a99df ("MINOR: listener/ssl: set the SSL xprt layer only once the whole config is known") the xprt is initialized for ssl directly from a generic funtion used to parse bind args. But the 'bind' lines from 'log-forward' sections were forgotten in commit 55f0f7bb5 ("MINOR: config: use the new bind_parse_args_list() to parse a "bind" line"). This patch re-works 'log-forward' section parsing to use the generic function to parse bind args and fix the issue. Since the generic way to parse was introduced in 2.6, this patch should be backported as far as this version.
2022-08-19 09:16:26 -04:00
ret = bind_parse_args_list(bind_conf, args, cur_arg, cursection, file, linenum);
err_code |= ret;
if (ret != 0) {
MEDIUM: log: syslog TCP support on log forward section. This patch re-introduce the "bind" statement on log forward sections to handle syslog TCP listeners as defined in rfc-6587. As complement it introduce "maxconn", "backlog" and "timeout client" statements to parameter those listeners.
2020-10-05 08:39:35 -04:00
err_code |= ERR_ALERT | ERR_FATAL;
goto out;
}
}
MEDIUM: log-forward: use "dgram-bind" instead of "bind" for the listener The use of "bind" wasn't that wise but was temporary. The problem is that it will not allow to coexist with tcp. Let's explicitly call it "dgram-bind" so that datagram listeners are expected here, leaving some room for stream listeners later. This is the only change.
2020-09-16 09:07:22 -04:00
else if (strcmp(args[0], "dgram-bind") == 0) {
MEDIUM: log: adds log forwarding section. Log forwarding: It is possible to declare one or multiple log forwarding section, haproxy will forward all received log messages to a log servers list. log-forward <name> Creates a new log forwarder proxy identified as <name>. bind <addr> [param*] Used to configure a log udp listener to receive messages to forward. Only udp listeners are allowed, address must be prefixed using 'udp@', 'udp4@' or 'udp6@'. This supports for all "bind" parameters found in 5.1 paragraph but most of them are irrelevant for udp/syslog case. log global log <address> [len <length>] [format <format>] [sample <ranges>:<smp_size>] <facility> [<level> [<minlevel>]] Used to configure target log servers. See more details on proxies documentation. If no format specified, haproxy tries to keep the incoming log format. Configured facility is ignored, except if incoming message does not present a facility but one is mandatory on the outgoing format. If there is no timestamp available in the input format, but the field exists in output format, haproxy will use the local date. Example: global log stderr format iso local7 ring myring description "My local buffer" format rfc5424 maxlen 1200 size 32764 timeout connect 5s timeout server 10s # syslog tcp server server mysyslogsrv 127.0.0.1:514 log-proto octet-count log-forward sylog-loadb bind udp4@127.0.0.1:1514 # all messages on stderr log global # all messages on local tcp syslog server log ring@myring local0 # load balance messages on 4 udp syslog servers log 127.0.0.1:10001 sample 1:4 local0 log 127.0.0.1:10002 sample 2:4 local0 log 127.0.0.1:10003 sample 3:4 local0 log 127.0.0.1:10004 sample 4:4 local0
2020-07-07 08:19:42 -04:00
int cur_arg;
struct bind_conf *bind_conf;
struct bind_kw *kw;
struct listener *l;
cur_arg = 1;
bind_conf = bind_conf_alloc(cfg_log_forward, file, linenum,
NULL, xprt_get(XPRT_RAW));
BUG/MINOR: listener: Handle allocation error when allocating a new bind_conf Allocation error are now handled in bind_conf_alloc() functions. Thus callers, when not already done, are also updated to catch NULL return value. This patch may be backported (at least partially) to all stable versions. However, it only fix errors durung configuration parsing. Thus it is not mandatory.
2021-04-12 10:56:37 -04:00
if (!bind_conf) {
ha_alert("parsing [%s:%d] : out of memory error.", file, linenum);
err_code |= ERR_ALERT | ERR_FATAL;
goto out;
}
MEDIUM: log: adds log forwarding section. Log forwarding: It is possible to declare one or multiple log forwarding section, haproxy will forward all received log messages to a log servers list. log-forward <name> Creates a new log forwarder proxy identified as <name>. bind <addr> [param*] Used to configure a log udp listener to receive messages to forward. Only udp listeners are allowed, address must be prefixed using 'udp@', 'udp4@' or 'udp6@'. This supports for all "bind" parameters found in 5.1 paragraph but most of them are irrelevant for udp/syslog case. log global log <address> [len <length>] [format <format>] [sample <ranges>:<smp_size>] <facility> [<level> [<minlevel>]] Used to configure target log servers. See more details on proxies documentation. If no format specified, haproxy tries to keep the incoming log format. Configured facility is ignored, except if incoming message does not present a facility but one is mandatory on the outgoing format. If there is no timestamp available in the input format, but the field exists in output format, haproxy will use the local date. Example: global log stderr format iso local7 ring myring description "My local buffer" format rfc5424 maxlen 1200 size 32764 timeout connect 5s timeout server 10s # syslog tcp server server mysyslogsrv 127.0.0.1:514 log-proto octet-count log-forward sylog-loadb bind udp4@127.0.0.1:1514 # all messages on stderr log global # all messages on local tcp syslog server log ring@myring local0 # load balance messages on 4 udp syslog servers log 127.0.0.1:10001 sample 1:4 local0 log 127.0.0.1:10002 sample 2:4 local0 log 127.0.0.1:10003 sample 3:4 local0 log 127.0.0.1:10004 sample 4:4 local0
2020-07-07 08:19:42 -04:00
MINOR: listener: move maxaccept from listener to bind_conf Like for previous values, maxaccept is really per-bind_conf, so let's move it there. Some frontends (peers, log) set it to 1 so the assignment was slightly moved.
2023-01-12 12:52:23 -05:00
bind_conf->maxaccept = global.tune.maxaccept ? global.tune.maxaccept : MAX_ACCEPT;
MINOR: log-forward: use str2receiver() to parse the dgram-bind address Thanks to this we don't need to specify "udp@" as it's implicitly a datagram type listener that is expected, so any AF_INET/AF_INET4 address will work.
2020-09-16 09:22:19 -04:00
if (!str2receiver(args[1], cfg_log_forward, bind_conf, file, linenum, &errmsg)) {
MEDIUM: log: adds log forwarding section. Log forwarding: It is possible to declare one or multiple log forwarding section, haproxy will forward all received log messages to a log servers list. log-forward <name> Creates a new log forwarder proxy identified as <name>. bind <addr> [param*] Used to configure a log udp listener to receive messages to forward. Only udp listeners are allowed, address must be prefixed using 'udp@', 'udp4@' or 'udp6@'. This supports for all "bind" parameters found in 5.1 paragraph but most of them are irrelevant for udp/syslog case. log global log <address> [len <length>] [format <format>] [sample <ranges>:<smp_size>] <facility> [<level> [<minlevel>]] Used to configure target log servers. See more details on proxies documentation. If no format specified, haproxy tries to keep the incoming log format. Configured facility is ignored, except if incoming message does not present a facility but one is mandatory on the outgoing format. If there is no timestamp available in the input format, but the field exists in output format, haproxy will use the local date. Example: global log stderr format iso local7 ring myring description "My local buffer" format rfc5424 maxlen 1200 size 32764 timeout connect 5s timeout server 10s # syslog tcp server server mysyslogsrv 127.0.0.1:514 log-proto octet-count log-forward sylog-loadb bind udp4@127.0.0.1:1514 # all messages on stderr log global # all messages on local tcp syslog server log ring@myring local0 # load balance messages on 4 udp syslog servers log 127.0.0.1:10001 sample 1:4 local0 log 127.0.0.1:10002 sample 2:4 local0 log 127.0.0.1:10003 sample 3:4 local0 log 127.0.0.1:10004 sample 4:4 local0
2020-07-07 08:19:42 -04:00
if (errmsg && *errmsg) {
indent_msg(&errmsg, 2);
ha_alert("parsing [%s:%d] : '%s %s' : %s\n", file, linenum, args[0], args[1], errmsg);
}
else {
ha_alert("parsing [%s:%d] : '%s %s' : error encountered while parsing listening address %s.\n",
file, linenum, args[0], args[1], args[2]);
}
BUG/MEDIUM: log-forward: always quit on parsing errors The err_code and goto were misplaced, causing a fatal parse error to be ignored when parsing a UDP listener's address. No backport is needed.
2020-09-16 10:24:14 -04:00
err_code |= ERR_ALERT | ERR_FATAL;
goto out;
MEDIUM: log: adds log forwarding section. Log forwarding: It is possible to declare one or multiple log forwarding section, haproxy will forward all received log messages to a log servers list. log-forward <name> Creates a new log forwarder proxy identified as <name>. bind <addr> [param*] Used to configure a log udp listener to receive messages to forward. Only udp listeners are allowed, address must be prefixed using 'udp@', 'udp4@' or 'udp6@'. This supports for all "bind" parameters found in 5.1 paragraph but most of them are irrelevant for udp/syslog case. log global log <address> [len <length>] [format <format>] [sample <ranges>:<smp_size>] <facility> [<level> [<minlevel>]] Used to configure target log servers. See more details on proxies documentation. If no format specified, haproxy tries to keep the incoming log format. Configured facility is ignored, except if incoming message does not present a facility but one is mandatory on the outgoing format. If there is no timestamp available in the input format, but the field exists in output format, haproxy will use the local date. Example: global log stderr format iso local7 ring myring description "My local buffer" format rfc5424 maxlen 1200 size 32764 timeout connect 5s timeout server 10s # syslog tcp server server mysyslogsrv 127.0.0.1:514 log-proto octet-count log-forward sylog-loadb bind udp4@127.0.0.1:1514 # all messages on stderr log global # all messages on local tcp syslog server log ring@myring local0 # load balance messages on 4 udp syslog servers log 127.0.0.1:10001 sample 1:4 local0 log 127.0.0.1:10002 sample 2:4 local0 log 127.0.0.1:10003 sample 3:4 local0 log 127.0.0.1:10004 sample 4:4 local0
2020-07-07 08:19:42 -04:00
}
list_for_each_entry(l, &bind_conf->listeners, by_bind) {
MINOR: log-forward: use str2receiver() to parse the dgram-bind address Thanks to this we don't need to specify "udp@" as it's implicitly a datagram type listener that is expected, so any AF_INET/AF_INET4 address will work.
2020-09-16 09:22:19 -04:00
/* the fact that the sockets are of type dgram is guaranteed by str2receiver() */
MINOR: log: set the UDP receiver's I/O handler in the receiver The I/O handler is syslog_fd_handler(), let's set it when creating the receivers.
2020-10-15 15:25:32 -04:00
l->rx.iocb = syslog_fd_handler;
MEDIUM: log: adds log forwarding section. Log forwarding: It is possible to declare one or multiple log forwarding section, haproxy will forward all received log messages to a log servers list. log-forward <name> Creates a new log forwarder proxy identified as <name>. bind <addr> [param*] Used to configure a log udp listener to receive messages to forward. Only udp listeners are allowed, address must be prefixed using 'udp@', 'udp4@' or 'udp6@'. This supports for all "bind" parameters found in 5.1 paragraph but most of them are irrelevant for udp/syslog case. log global log <address> [len <length>] [format <format>] [sample <ranges>:<smp_size>] <facility> [<level> [<minlevel>]] Used to configure target log servers. See more details on proxies documentation. If no format specified, haproxy tries to keep the incoming log format. Configured facility is ignored, except if incoming message does not present a facility but one is mandatory on the outgoing format. If there is no timestamp available in the input format, but the field exists in output format, haproxy will use the local date. Example: global log stderr format iso local7 ring myring description "My local buffer" format rfc5424 maxlen 1200 size 32764 timeout connect 5s timeout server 10s # syslog tcp server server mysyslogsrv 127.0.0.1:514 log-proto octet-count log-forward sylog-loadb bind udp4@127.0.0.1:1514 # all messages on stderr log global # all messages on local tcp syslog server log ring@myring local0 # load balance messages on 4 udp syslog servers log 127.0.0.1:10001 sample 1:4 local0 log 127.0.0.1:10002 sample 2:4 local0 log 127.0.0.1:10003 sample 3:4 local0 log 127.0.0.1:10004 sample 4:4 local0
2020-07-07 08:19:42 -04:00
global.maxsock++;
}
cur_arg++;
while (*args[cur_arg] && (kw = bind_find_kw(args[cur_arg]))) {
int ret;
ret = kw->parse(args, cur_arg, cfg_log_forward, bind_conf, &errmsg);
err_code |= ret;
if (ret) {
if (errmsg && *errmsg) {
indent_msg(&errmsg, 2);
ha_alert("parsing [%s:%d] : %s\n", file, linenum, errmsg);
}
else
ha_alert("parsing [%s:%d]: error encountered while processing '%s'\n",
file, linenum, args[cur_arg]);
if (ret & ERR_FATAL)
goto out;
}
cur_arg += 1 + kw->skip;
}
if (*args[cur_arg] != 0) {
MINOR: cfgparse/bind: suggest correct spelling for unknown bind keywords Just like with the server keywords, now's the turn of "bind" keywords. The difference is that 100% of the bind keywords are registered, thus we do not need the list of extra keywords. There are multiple bind line parsers today, all were updated: - peers - log - dgram-bind - cli $ printf "listen f\nbind :8000 tcut\n" | ./haproxy -c -f /dev/stdin [NOTICE] 070/101358 (25146) : haproxy version is 2.4-dev11-7b8787-26 [NOTICE] 070/101358 (25146) : path to executable is ./haproxy [ALERT] 070/101358 (25146) : parsing [/dev/stdin:2] : 'bind :8000' unknown keyword 'tcut'; did you mean 'tcp-ut' maybe ? [ALERT] 070/101358 (25146) : Error(s) found in configuration file : /dev/stdin [ALERT] 070/101358 (25146) : Fatal errors found in configuration.
2021-03-12 04:14:07 -05:00
const char *best = bind_find_best_kw(args[cur_arg]);
if (best)
ha_alert("parsing [%s:%d] : unknown keyword '%s' in '%s' section; did you mean '%s' maybe ?\n",
file, linenum, args[cur_arg], cursection, best);
else
ha_alert("parsing [%s:%d] : unknown keyword '%s' in '%s' section.\n",
file, linenum, args[cur_arg], cursection);
MEDIUM: log: adds log forwarding section. Log forwarding: It is possible to declare one or multiple log forwarding section, haproxy will forward all received log messages to a log servers list. log-forward <name> Creates a new log forwarder proxy identified as <name>. bind <addr> [param*] Used to configure a log udp listener to receive messages to forward. Only udp listeners are allowed, address must be prefixed using 'udp@', 'udp4@' or 'udp6@'. This supports for all "bind" parameters found in 5.1 paragraph but most of them are irrelevant for udp/syslog case. log global log <address> [len <length>] [format <format>] [sample <ranges>:<smp_size>] <facility> [<level> [<minlevel>]] Used to configure target log servers. See more details on proxies documentation. If no format specified, haproxy tries to keep the incoming log format. Configured facility is ignored, except if incoming message does not present a facility but one is mandatory on the outgoing format. If there is no timestamp available in the input format, but the field exists in output format, haproxy will use the local date. Example: global log stderr format iso local7 ring myring description "My local buffer" format rfc5424 maxlen 1200 size 32764 timeout connect 5s timeout server 10s # syslog tcp server server mysyslogsrv 127.0.0.1:514 log-proto octet-count log-forward sylog-loadb bind udp4@127.0.0.1:1514 # all messages on stderr log global # all messages on local tcp syslog server log ring@myring local0 # load balance messages on 4 udp syslog servers log 127.0.0.1:10001 sample 1:4 local0 log 127.0.0.1:10002 sample 2:4 local0 log 127.0.0.1:10003 sample 3:4 local0 log 127.0.0.1:10004 sample 4:4 local0
2020-07-07 08:19:42 -04:00
err_code |= ERR_ALERT | ERR_FATAL;
goto out;
}
}
else if (strcmp(args[0], "log") == 0) {
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
if (!parse_logger(args, &cfg_log_forward->loggers, (kwm == KWM_NO), file, linenum, &errmsg)) {
MEDIUM: log: adds log forwarding section. Log forwarding: It is possible to declare one or multiple log forwarding section, haproxy will forward all received log messages to a log servers list. log-forward <name> Creates a new log forwarder proxy identified as <name>. bind <addr> [param*] Used to configure a log udp listener to receive messages to forward. Only udp listeners are allowed, address must be prefixed using 'udp@', 'udp4@' or 'udp6@'. This supports for all "bind" parameters found in 5.1 paragraph but most of them are irrelevant for udp/syslog case. log global log <address> [len <length>] [format <format>] [sample <ranges>:<smp_size>] <facility> [<level> [<minlevel>]] Used to configure target log servers. See more details on proxies documentation. If no format specified, haproxy tries to keep the incoming log format. Configured facility is ignored, except if incoming message does not present a facility but one is mandatory on the outgoing format. If there is no timestamp available in the input format, but the field exists in output format, haproxy will use the local date. Example: global log stderr format iso local7 ring myring description "My local buffer" format rfc5424 maxlen 1200 size 32764 timeout connect 5s timeout server 10s # syslog tcp server server mysyslogsrv 127.0.0.1:514 log-proto octet-count log-forward sylog-loadb bind udp4@127.0.0.1:1514 # all messages on stderr log global # all messages on local tcp syslog server log ring@myring local0 # load balance messages on 4 udp syslog servers log 127.0.0.1:10001 sample 1:4 local0 log 127.0.0.1:10002 sample 2:4 local0 log 127.0.0.1:10003 sample 3:4 local0 log 127.0.0.1:10004 sample 4:4 local0
2020-07-07 08:19:42 -04:00
ha_alert("parsing [%s:%d] : %s : %s\n", file, linenum, args[0], errmsg);
err_code |= ERR_ALERT | ERR_FATAL;
goto out;
}
}
MEDIUM: log: syslog TCP support on log forward section. This patch re-introduce the "bind" statement on log forward sections to handle syslog TCP listeners as defined in rfc-6587. As complement it introduce "maxconn", "backlog" and "timeout client" statements to parameter those listeners.
2020-10-05 08:39:35 -04:00
else if (strcmp(args[0], "timeout") == 0) {
const char *res;
unsigned timeout;
if (strcmp(args[1], "client") != 0) {
ha_alert("parsing [%s:%d] : unknown keyword '%s %s' in log-forward section.\n", file, linenum, args[0], args[1]);
err_code |= ERR_ALERT | ERR_FATAL;
goto out;
}
if (*args[2] == 0) {
ha_alert("parsing [%s:%d] : missing timeout client value.\n", file, linenum);
err_code |= ERR_ALERT | ERR_FATAL;
goto out;
}
res = parse_time_err(args[2], &timeout, TIME_UNIT_MS);
if (res == PARSE_TIME_OVER) {
memprintf(&errmsg, "timer overflow in argument '%s' to 'timeout client' (maximum value is 2147483647 ms or ~24.8 days)", args[2]);
}
else if (res == PARSE_TIME_UNDER) {
memprintf(&errmsg, "timer underflow in argument '%s' to 'timeout client' (minimum non-null value is 1 ms)", args[2]);
}
else if (res) {
memprintf(&errmsg, "unexpected character '%c' in 'timeout client'", *res);
}
if (res) {
ha_alert("parsing [%s:%d] : %s : %s\n", file, linenum, args[0], errmsg);
err_code |= ERR_ALERT | ERR_FATAL;
goto out;
}
cfg_log_forward->timeout.client = MS_TO_TICKS(timeout);
}
BUG/MINOR: log-forward: fail on unknown keywords The log-forward section silently ignores junk and unknown keywords, make it fail! No backport is needed.
2020-09-16 09:04:33 -04:00
else {
ha_alert("parsing [%s:%d] : unknown keyword '%s' in log-forward section.\n", file, linenum, args[0]);
err_code |= ERR_ALERT | ERR_ABORT;
goto out;
}
MEDIUM: log: adds log forwarding section. Log forwarding: It is possible to declare one or multiple log forwarding section, haproxy will forward all received log messages to a log servers list. log-forward <name> Creates a new log forwarder proxy identified as <name>. bind <addr> [param*] Used to configure a log udp listener to receive messages to forward. Only udp listeners are allowed, address must be prefixed using 'udp@', 'udp4@' or 'udp6@'. This supports for all "bind" parameters found in 5.1 paragraph but most of them are irrelevant for udp/syslog case. log global log <address> [len <length>] [format <format>] [sample <ranges>:<smp_size>] <facility> [<level> [<minlevel>]] Used to configure target log servers. See more details on proxies documentation. If no format specified, haproxy tries to keep the incoming log format. Configured facility is ignored, except if incoming message does not present a facility but one is mandatory on the outgoing format. If there is no timestamp available in the input format, but the field exists in output format, haproxy will use the local date. Example: global log stderr format iso local7 ring myring description "My local buffer" format rfc5424 maxlen 1200 size 32764 timeout connect 5s timeout server 10s # syslog tcp server server mysyslogsrv 127.0.0.1:514 log-proto octet-count log-forward sylog-loadb bind udp4@127.0.0.1:1514 # all messages on stderr log global # all messages on local tcp syslog server log ring@myring local0 # load balance messages on 4 udp syslog servers log 127.0.0.1:10001 sample 1:4 local0 log 127.0.0.1:10002 sample 2:4 local0 log 127.0.0.1:10003 sample 3:4 local0 log 127.0.0.1:10004 sample 4:4 local0
2020-07-07 08:19:42 -04:00
out:
BUG/MINOR: log: free errmsg on error in cfg_parse_log_forward() When leaving cfg_parse_log_forward() on error paths, errmsg which is local to the function could still point to valid data, and it's our responsibility to free it. Instead of freeing it everywhere it is invoved, we free it prior to leaving the function. This should be backported as far as 2.4.
2023-07-04 09:36:45 -04:00
ha_free(&errmsg);
MEDIUM: log: adds log forwarding section. Log forwarding: It is possible to declare one or multiple log forwarding section, haproxy will forward all received log messages to a log servers list. log-forward <name> Creates a new log forwarder proxy identified as <name>. bind <addr> [param*] Used to configure a log udp listener to receive messages to forward. Only udp listeners are allowed, address must be prefixed using 'udp@', 'udp4@' or 'udp6@'. This supports for all "bind" parameters found in 5.1 paragraph but most of them are irrelevant for udp/syslog case. log global log <address> [len <length>] [format <format>] [sample <ranges>:<smp_size>] <facility> [<level> [<minlevel>]] Used to configure target log servers. See more details on proxies documentation. If no format specified, haproxy tries to keep the incoming log format. Configured facility is ignored, except if incoming message does not present a facility but one is mandatory on the outgoing format. If there is no timestamp available in the input format, but the field exists in output format, haproxy will use the local date. Example: global log stderr format iso local7 ring myring description "My local buffer" format rfc5424 maxlen 1200 size 32764 timeout connect 5s timeout server 10s # syslog tcp server server mysyslogsrv 127.0.0.1:514 log-proto octet-count log-forward sylog-loadb bind udp4@127.0.0.1:1514 # all messages on stderr log global # all messages on local tcp syslog server log ring@myring local0 # load balance messages on 4 udp syslog servers log 127.0.0.1:10001 sample 1:4 local0 log 127.0.0.1:10002 sample 2:4 local0 log 127.0.0.1:10003 sample 3:4 local0 log 127.0.0.1:10004 sample 4:4 local0
2020-07-07 08:19:42 -04:00
return err_code;
}
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
/* function: post-resolve a single list of loggers
MEDIUM: log/sink: make logsrv postparsing more generic We previously had postparsing logic but only for logsrv sinks, but now we need to make this operation on logsrv directly instead of sinks to prepare for additional postparsing logic that is not sink-specific. To do this, we migrated post_sink_resolve() and sink_postresolve_logsrvs() to their postresolve_logsrvs() and postresolve_logsrv_list() equivalents. Then, we split postresolve_logsrv_list() so that the sink-only logic stays in sink.c (sink_resolve_logsrv_buffer() function), and the "generic" target part stays in log.c as resolve_logsrv(). Error messages formatting was preserved as far as possible but some slight variations are to be expected. As for the functional aspect, no change should be expected.
2023-08-17 11:38:30 -04:00
*
* Returns err_code which defaults to ERR_NONE and can be set to a combination
* of ERR_WARN, ERR_ALERT, ERR_FATAL and ERR_ABORT in case of errors.
*/
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
int postresolve_logger_list(struct list *loggers, const char *section, const char *section_name)
MEDIUM: log/sink: make logsrv postparsing more generic We previously had postparsing logic but only for logsrv sinks, but now we need to make this operation on logsrv directly instead of sinks to prepare for additional postparsing logic that is not sink-specific. To do this, we migrated post_sink_resolve() and sink_postresolve_logsrvs() to their postresolve_logsrvs() and postresolve_logsrv_list() equivalents. Then, we split postresolve_logsrv_list() so that the sink-only logic stays in sink.c (sink_resolve_logsrv_buffer() function), and the "generic" target part stays in log.c as resolve_logsrv(). Error messages formatting was preserved as far as possible but some slight variations are to be expected. As for the functional aspect, no change should be expected.
2023-08-17 11:38:30 -04:00
{
int err_code = ERR_NONE;
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
struct logger *logger;
MEDIUM: log/sink: make logsrv postparsing more generic We previously had postparsing logic but only for logsrv sinks, but now we need to make this operation on logsrv directly instead of sinks to prepare for additional postparsing logic that is not sink-specific. To do this, we migrated post_sink_resolve() and sink_postresolve_logsrvs() to their postresolve_logsrvs() and postresolve_logsrv_list() equivalents. Then, we split postresolve_logsrv_list() so that the sink-only logic stays in sink.c (sink_resolve_logsrv_buffer() function), and the "generic" target part stays in log.c as resolve_logsrv(). Error messages formatting was preserved as far as possible but some slight variations are to be expected. As for the functional aspect, no change should be expected.
2023-08-17 11:38:30 -04:00
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
list_for_each_entry(logger, loggers, list) {
MEDIUM: log/sink: make logsrv postparsing more generic We previously had postparsing logic but only for logsrv sinks, but now we need to make this operation on logsrv directly instead of sinks to prepare for additional postparsing logic that is not sink-specific. To do this, we migrated post_sink_resolve() and sink_postresolve_logsrvs() to their postresolve_logsrvs() and postresolve_logsrv_list() equivalents. Then, we split postresolve_logsrv_list() so that the sink-only logic stays in sink.c (sink_resolve_logsrv_buffer() function), and the "generic" target part stays in log.c as resolve_logsrv(). Error messages formatting was preserved as far as possible but some slight variations are to be expected. As for the functional aspect, no change should be expected.
2023-08-17 11:38:30 -04:00
int cur_code;
char *msg = NULL;
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
cur_code = resolve_logger(logger, &msg);
MEDIUM: log/sink: make logsrv postparsing more generic We previously had postparsing logic but only for logsrv sinks, but now we need to make this operation on logsrv directly instead of sinks to prepare for additional postparsing logic that is not sink-specific. To do this, we migrated post_sink_resolve() and sink_postresolve_logsrvs() to their postresolve_logsrvs() and postresolve_logsrv_list() equivalents. Then, we split postresolve_logsrv_list() so that the sink-only logic stays in sink.c (sink_resolve_logsrv_buffer() function), and the "generic" target part stays in log.c as resolve_logsrv(). Error messages formatting was preserved as far as possible but some slight variations are to be expected. As for the functional aspect, no change should be expected.
2023-08-17 11:38:30 -04:00
if (msg) {
void (*e_func)(const char *fmt, ...) = NULL;
if (cur_code & ERR_ALERT)
e_func = ha_alert;
else if (cur_code & ERR_WARN)
e_func = ha_warning;
else
e_func = ha_diag_warning;
if (!section)
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
e_func("global log directive declared in file %s at line '%d' %s.\n",
logger->conf.file, logger->conf.line, msg);
MEDIUM: log/sink: make logsrv postparsing more generic We previously had postparsing logic but only for logsrv sinks, but now we need to make this operation on logsrv directly instead of sinks to prepare for additional postparsing logic that is not sink-specific. To do this, we migrated post_sink_resolve() and sink_postresolve_logsrvs() to their postresolve_logsrvs() and postresolve_logsrv_list() equivalents. Then, we split postresolve_logsrv_list() so that the sink-only logic stays in sink.c (sink_resolve_logsrv_buffer() function), and the "generic" target part stays in log.c as resolve_logsrv(). Error messages formatting was preserved as far as possible but some slight variations are to be expected. As for the functional aspect, no change should be expected.
2023-08-17 11:38:30 -04:00
else
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
e_func("log directive declared in %s section '%s' in file '%s' at line %d %s.\n",
section, section_name, logger->conf.file, logger->conf.line, msg);
MEDIUM: log/sink: make logsrv postparsing more generic We previously had postparsing logic but only for logsrv sinks, but now we need to make this operation on logsrv directly instead of sinks to prepare for additional postparsing logic that is not sink-specific. To do this, we migrated post_sink_resolve() and sink_postresolve_logsrvs() to their postresolve_logsrvs() and postresolve_logsrv_list() equivalents. Then, we split postresolve_logsrv_list() so that the sink-only logic stays in sink.c (sink_resolve_logsrv_buffer() function), and the "generic" target part stays in log.c as resolve_logsrv(). Error messages formatting was preserved as far as possible but some slight variations are to be expected. As for the functional aspect, no change should be expected.
2023-08-17 11:38:30 -04:00
ha_free(&msg);
}
err_code |= cur_code;
}
return err_code;
}
/* resolve default log directives at end of config. Returns 0 on success
* otherwise error flags.
*/
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
static int postresolve_loggers()
MEDIUM: log/sink: make logsrv postparsing more generic We previously had postparsing logic but only for logsrv sinks, but now we need to make this operation on logsrv directly instead of sinks to prepare for additional postparsing logic that is not sink-specific. To do this, we migrated post_sink_resolve() and sink_postresolve_logsrvs() to their postresolve_logsrvs() and postresolve_logsrv_list() equivalents. Then, we split postresolve_logsrv_list() so that the sink-only logic stays in sink.c (sink_resolve_logsrv_buffer() function), and the "generic" target part stays in log.c as resolve_logsrv(). Error messages formatting was preserved as far as possible but some slight variations are to be expected. As for the functional aspect, no change should be expected.
2023-08-17 11:38:30 -04:00
{
struct proxy *px;
int err_code = ERR_NONE;
/* global log directives */
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
err_code |= postresolve_logger_list(&global.loggers, NULL, NULL);
MEDIUM: log/sink: make logsrv postparsing more generic We previously had postparsing logic but only for logsrv sinks, but now we need to make this operation on logsrv directly instead of sinks to prepare for additional postparsing logic that is not sink-specific. To do this, we migrated post_sink_resolve() and sink_postresolve_logsrvs() to their postresolve_logsrvs() and postresolve_logsrv_list() equivalents. Then, we split postresolve_logsrv_list() so that the sink-only logic stays in sink.c (sink_resolve_logsrv_buffer() function), and the "generic" target part stays in log.c as resolve_logsrv(). Error messages formatting was preserved as far as possible but some slight variations are to be expected. As for the functional aspect, no change should be expected.
2023-08-17 11:38:30 -04:00
/* proxy log directives */
for (px = proxies_list; px; px = px->next)
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
err_code |= postresolve_logger_list(&px->loggers, "proxy", px->id);
MEDIUM: log/sink: make logsrv postparsing more generic We previously had postparsing logic but only for logsrv sinks, but now we need to make this operation on logsrv directly instead of sinks to prepare for additional postparsing logic that is not sink-specific. To do this, we migrated post_sink_resolve() and sink_postresolve_logsrvs() to their postresolve_logsrvs() and postresolve_logsrv_list() equivalents. Then, we split postresolve_logsrv_list() so that the sink-only logic stays in sink.c (sink_resolve_logsrv_buffer() function), and the "generic" target part stays in log.c as resolve_logsrv(). Error messages formatting was preserved as far as possible but some slight variations are to be expected. As for the functional aspect, no change should be expected.
2023-08-17 11:38:30 -04:00
/* log-forward log directives */
for (px = cfg_log_forward; px; px = px->next)
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
err_code |= postresolve_logger_list(&px->loggers, "log-forward", px->id);
MEDIUM: log/sink: make logsrv postparsing more generic We previously had postparsing logic but only for logsrv sinks, but now we need to make this operation on logsrv directly instead of sinks to prepare for additional postparsing logic that is not sink-specific. To do this, we migrated post_sink_resolve() and sink_postresolve_logsrvs() to their postresolve_logsrvs() and postresolve_logsrv_list() equivalents. Then, we split postresolve_logsrv_list() so that the sink-only logic stays in sink.c (sink_resolve_logsrv_buffer() function), and the "generic" target part stays in log.c as resolve_logsrv(). Error messages formatting was preserved as far as possible but some slight variations are to be expected. As for the functional aspect, no change should be expected.
2023-08-17 11:38:30 -04:00
return err_code;
}
MEDIUM: log: adds log forwarding section. Log forwarding: It is possible to declare one or multiple log forwarding section, haproxy will forward all received log messages to a log servers list. log-forward <name> Creates a new log forwarder proxy identified as <name>. bind <addr> [param*] Used to configure a log udp listener to receive messages to forward. Only udp listeners are allowed, address must be prefixed using 'udp@', 'udp4@' or 'udp6@'. This supports for all "bind" parameters found in 5.1 paragraph but most of them are irrelevant for udp/syslog case. log global log <address> [len <length>] [format <format>] [sample <ranges>:<smp_size>] <facility> [<level> [<minlevel>]] Used to configure target log servers. See more details on proxies documentation. If no format specified, haproxy tries to keep the incoming log format. Configured facility is ignored, except if incoming message does not present a facility but one is mandatory on the outgoing format. If there is no timestamp available in the input format, but the field exists in output format, haproxy will use the local date. Example: global log stderr format iso local7 ring myring description "My local buffer" format rfc5424 maxlen 1200 size 32764 timeout connect 5s timeout server 10s # syslog tcp server server mysyslogsrv 127.0.0.1:514 log-proto octet-count log-forward sylog-loadb bind udp4@127.0.0.1:1514 # all messages on stderr log global # all messages on local tcp syslog server log ring@myring local0 # load balance messages on 4 udp syslog servers log 127.0.0.1:10001 sample 1:4 local0 log 127.0.0.1:10002 sample 2:4 local0 log 127.0.0.1:10003 sample 3:4 local0 log 127.0.0.1:10004 sample 4:4 local0
2020-07-07 08:19:42 -04:00
/* config parsers for this section */
REGISTER_CONFIG_SECTION("log-forward", cfg_parse_log_forward, NULL);
MEDIUM: tree-wide: logsrv struct becomes logger When 'log' directive was implemented, the internal representation was named 'struct logsrv', because the 'log' directive would directly point to the log target, which used to be a (UDP) log server exclusively at that time, hence the name. But things have become more complex, since today 'log' directive can point to ring targets (implicit, or named) for example. Indeed, a 'log' directive does no longer reference the "final" server to which the log will be sent, but instead it describes which log API and parameters to use for transporting the log messages to the proper log destination. So now the term 'logsrv' is rather confusing and prevents us from introducing a new level of abstraction because they would be mixed with logsrv. So in order to better designate this 'log' directive, and make it more generic, we chose the word 'logger' which now replaces logsrv everywhere it was used in the code (including related comments). This is internal rewording, so no functional change should be expected on user-side.
2023-09-11 09:06:53 -04:00
REGISTER_POST_CHECK(postresolve_loggers);
MAJOR: log: introduce log backends Using "mode log" in a backend section turns the proxy in a log backend which can be used to log-balance logs between multiple log targets (udp or tcp servers) log backends can be used as regular log targets using the log directive with "backend@be_name" prefix, like so: | log backend@mybackend local0 A log backend will distribute log messages to servers according to the log load-balancing algorithm that can be set using the "log-balance" option from the log backend section. For now, only the roundrobin algorithm is supported and set by default.
2023-09-13 05:52:31 -04:00
REGISTER_POST_PROXY_CHECK(postcheck_log_backend);
MAJOR: log: implement proper postparsing for logformat expressions This patch tries to address a design flaw with how logformat expressions are parsed from config. Indeed, some parse_logformat_string() calls are performed during config parsing when the proxy mode is not yet known. Here's a config example that illustrates the issue: defaults mode tcp listen test bind :8888 http-response set-header custom-hdr "%trl" # needs http mode http The above config should work, because the effective proxy mode is http, yet haproxy fails with this error: [ALERT] (99051) : config : parsing [repro.conf:6] : error detected in proxy 'test' while parsing 'http-response set-header' rule : format tag 'trl' is reserved for HTTP mode. To fix the issue once and for all, let's implement smart postparsing for logformat expressions encountered during config parsing: - split parse_logformat_string() (and subfonctions) in order to create a new lf_expr_postcheck() function that must be called to finish preparing and checking the logformat expression once the proxy type is known. - save some config hints info during parse_logformat_string() to generate more precise error messages during lf_expr_postcheck(), if needed, we rely on curpx->conf.args.{file,line} hints for that because parse_logformat_string() doesn't know about current file and line number. - lf_expr_postcheck() uses PR_FL_CHECKED proxy flag to know if the function may try to make the proxy compatible with the expression, or if it should simply fail as soon as an incompatibility is detected. - if parse_logformat_string() is called from an unchecked proxy, then schedule the expression for postparsing, else (ie: during runtime), run the postcheck right away. This change will also allow for some logformat expression error handling simplifications in the future.
2024-02-23 11:26:32 -05:00
REGISTER_POST_PROXY_CHECK(postcheck_logformat_proxy);
MEDIUM: log: adds log forwarding section. Log forwarding: It is possible to declare one or multiple log forwarding section, haproxy will forward all received log messages to a log servers list. log-forward <name> Creates a new log forwarder proxy identified as <name>. bind <addr> [param*] Used to configure a log udp listener to receive messages to forward. Only udp listeners are allowed, address must be prefixed using 'udp@', 'udp4@' or 'udp6@'. This supports for all "bind" parameters found in 5.1 paragraph but most of them are irrelevant for udp/syslog case. log global log <address> [len <length>] [format <format>] [sample <ranges>:<smp_size>] <facility> [<level> [<minlevel>]] Used to configure target log servers. See more details on proxies documentation. If no format specified, haproxy tries to keep the incoming log format. Configured facility is ignored, except if incoming message does not present a facility but one is mandatory on the outgoing format. If there is no timestamp available in the input format, but the field exists in output format, haproxy will use the local date. Example: global log stderr format iso local7 ring myring description "My local buffer" format rfc5424 maxlen 1200 size 32764 timeout connect 5s timeout server 10s # syslog tcp server server mysyslogsrv 127.0.0.1:514 log-proto octet-count log-forward sylog-loadb bind udp4@127.0.0.1:1514 # all messages on stderr log global # all messages on local tcp syslog server log ring@myring local0 # load balance messages on 4 udp syslog servers log 127.0.0.1:10001 sample 1:4 local0 log 127.0.0.1:10002 sample 2:4 local0 log 127.0.0.1:10003 sample 3:4 local0 log 127.0.0.1:10004 sample 4:4 local0
2020-07-07 08:19:42 -04:00
BUG/MEDIUM: init/threads: provide per-thread alloc/free function callbacks We currently have the ability to register functions to be called early on thread creation and at thread deinitialization. It turns out this is not sufficient because certain such functions may use resources that are being allocated by the other ones, thus creating a race condition depending only on the linking order. For example the mworker needs to register a file descriptor while the pollers will reallocate the fd_updt[] array. Similarly logs and trashes may be used by some init functions while it's unclear whether they have been deduplicated. The same issue happens on deinit, if the fd_updt[] or trash is released before some functions finish to use them, we'll get into trouble. This patch creates a couple of early and late callbacks for per-thread allocation/freeing of resources. A few init functions were moved there, and the fd init code was split between the two (since it used to both allocate and initialize at once). This way the init/deinit sequence is expected to be safe now. This patch should be backported to 1.9 as at least the trash/log issue seems to be present. The run_thread_poll_loop() code is a bit different there as the mworker is not a callback, but it will have no effect and it's enough to drop the mworker changes. This bug was reported by Ilya Shipitsin in github issue #104.
2019-05-22 08:42:12 -04:00
REGISTER_PER_THREAD_ALLOC(init_log_buffers);
REGISTER_PER_THREAD_FREE(deinit_log_buffers);
MINOR: initcall: use initcalls for most post_{check,deinit} and per_thread* Most calls to hap_register_post_check(), hap_register_post_deinit(), hap_register_per_thread_init(), hap_register_per_thread_deinit() can be done using initcalls and will not require a constructor anymore. Let's create a set of simplified macros for this, called respectively REGISTER_POST_CHECK, REGISTER_POST_DEINIT, REGISTER_PER_THREAD_INIT, and REGISTER_PER_THREAD_DEINIT. Some files were not modified because they wouldn't benefit from this or because they conditionally register (e.g. the pollers).
2018-11-26 05:21:50 -05:00
MINOR: log: move log-forwarders cleanup in log.c Move the log-forwarded proxies cleanup from global deinit() function into log dedicated deinit function. No backport needed.
2023-07-03 12:07:30 -04:00
REGISTER_POST_DEINIT(deinit_log_forward);
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-25 20:48:02 -04:00
/*
* Local variables:
* c-indent-level: 8
* c-basic-offset: 8
* End:
*/
Reference in a new issue Copy permalink