upstream/haproxy
1
0
Fork 0
mirror of https://github.com/haproxy/haproxy.git synced 2026-04-06 09:35:22 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions 3
haproxy/src/pattern.c

2874 lines
79 KiB
C
Raw Normal View History

REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
/*
* Pattern management functions.
*
* Copyright 2000-2013 Willy Tarreau <w@1wt.eu>
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version
* 2 of the License, or (at your option) any later version.
*
*/
#include <ctype.h>
#include <stdio.h>
BUILD: pattern: include errno.h Commit 3c79d4bdc introduced the use of errno in pattern.c without including errno.h. If we build haproxy without any option errno is not defined and the build fails.
2020-01-17 12:01:20 -05:00
#include <errno.h>
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
MEDIUM: migrate the patterns reference to cebs_tree cebs_tree are 24 bytes smaller than ebst_tree (16B vs 40B), and pattern references are only used during map/acl updates, so their storage is pure loss between updates (which most of the time never happen). By switching their indexing to compact trees, we can save 16 to 24 bytes per entry depending on alightment (here it's 24 per struct but 16 practical as malloc's alignment keeps 8 unused). Tested on core i7-8650U running at 3.0 GHz, with a file containing 17.7M IP addresses (16.7M different): $ time ./haproxy -c -f acl-ip.cfg Save 280 MB RAM for 17.7M IP addresses, and slightly speeds up the startup (5.8%, from 19.2s to 18.2s), a part of which possible being attributed to having to write less memory. Note that this is on small strings. On larger ones such as user-agents, ebtree doesn't reread the whole key and might be more efficient. Before: RAM (VSZ/RSS): 4443912 3912444 real 0m19.211s user 0m18.138s sys 0m1.068s Overhead Command Shared Object Symbol 44.79% haproxy haproxy [.] ebst_insert 25.07% haproxy haproxy [.] ebmb_insert_prefix 3.44% haproxy libc-2.33.so [.] __libc_calloc 2.71% haproxy libc-2.33.so [.] _int_malloc 2.33% haproxy haproxy [.] free_pattern_tree 1.78% haproxy libc-2.33.so [.] inet_pton4 1.62% haproxy libc-2.33.so [.] _IO_fgets 1.58% haproxy libc-2.33.so [.] _int_free 1.56% haproxy haproxy [.] pat_ref_push 1.35% haproxy libc-2.33.so [.] malloc_consolidate 1.16% haproxy libc-2.33.so [.] __strlen_avx2 0.79% haproxy haproxy [.] pat_idx_tree_ip 0.76% haproxy haproxy [.] pat_ref_read_from_file 0.60% haproxy libc-2.33.so [.] __strrchr_avx2 0.55% haproxy libc-2.33.so [.] unlink_chunk.constprop.0 0.54% haproxy libc-2.33.so [.] __memchr_avx2 0.46% haproxy haproxy [.] pat_ref_append After: RAM (VSZ/RSS): 4166108 3634768 real 0m18.114s user 0m17.113s sys 0m0.996s Overhead Command Shared Object Symbol 38.99% haproxy haproxy [.] cebs_insert 27.09% haproxy haproxy [.] ebmb_insert_prefix 3.63% haproxy libc-2.33.so [.] __libc_calloc 3.18% haproxy libc-2.33.so [.] _int_malloc 2.69% haproxy haproxy [.] free_pattern_tree 1.99% haproxy libc-2.33.so [.] inet_pton4 1.74% haproxy libc-2.33.so [.] _IO_fgets 1.73% haproxy libc-2.33.so [.] _int_free 1.57% haproxy haproxy [.] pat_ref_push 1.48% haproxy libc-2.33.so [.] malloc_consolidate 1.22% haproxy libc-2.33.so [.] __strlen_avx2 1.05% haproxy libc-2.33.so [.] __strcmp_avx2 0.80% haproxy haproxy [.] pat_idx_tree_ip 0.74% haproxy libc-2.33.so [.] __memchr_avx2 0.69% haproxy libc-2.33.so [.] __strrchr_avx2 0.69% haproxy libc-2.33.so [.] _IO_getline_info 0.62% haproxy haproxy [.] pat_ref_read_from_file 0.56% haproxy libc-2.33.so [.] unlink_chunk.constprop.0 0.56% haproxy libc-2.33.so [.] cfree@GLIBC_2.2.5 0.46% haproxy haproxy [.] pat_ref_append If the addresses are totally disordered (via "shuf" on the input file), we see both implementations reach exactly 68.0s (slower due to much higher cache miss ratio). On large strings such as user agents (1 million here), it's now slightly slower (+9%): Before: real 0m2.475s user 0m2.316s sys 0m0.155s After: real 0m2.696s user 0m2.544s sys 0m0.147s But such patterns are much less common than short ones, and the memory savings do still count. Note that while it could be tempting to get rid of the list that chains all these pat_ref_elt together and only enumerate them by walking along the tree to save 16 extra bytes per entry, that's not possible due to the problem that insertion ordering is critical (think overlapping regex such as /index.* and /index.html). Currently it's not possible to proceed differently because patterns are first pre-loaded into the pat_ref via pat_ref_read_from_file_smp() and later indexed by pattern_read_from_file(), which has to only redo the second part anyway for maps/acls declared multiple times.
2025-01-12 13:38:28 -05:00
#include <import/cebs_tree.h>
MINOR: patterns: preliminary changes for reorganization Safe and non-functional changes that only add currently unused structures, field, functions and macros, in preparation of larger changes that alter the way pattern reference elements are stored. This includes code to create and lookup generation objects, and macros to iterate over the generations of a pattern reference.
2025-12-17 23:31:29 -05:00
#include <import/ceb32_tree.h>
MEDIUM: map/acl: Improve pat_ref_set() efficiency (for "set-map", "add-acl" action perfs) Organize reference to pattern element of map (struct pat_ref_elt) into an ebtree: - add an eb_root member to the map (pat_ref struct) and an ebpt_node to its element (pat_ref_elt struct), - modify the code to insert these nodes into their ebtrees each time they are allocated. This is done in pat_ref_append(). Note that ->head member (struct list) of map (struct pat_ref) is not removed could have been removed. This is not the case because still necessary to dump the map contents from the CLI in the order the map elememnts have been inserted. This patch also modifies http_action_set_map() which is the callback at least used by "set-map" action. The pat_ref_elt element returned by pat_ref_find_elt() is no more ignored, but reused if not NULL by pat_ref_set() as first element to lookup from. This latter is also modified to use the ebtree attached to the map in place of the ->head list attached to each map element (pat_ref_elt struct). Also modify pat_ref_find_elt() to makes it use ->eb_root map ebtree added to the map by this patch in place of inspecting all the elements with a strcmp() call.
2023-08-22 10:52:47 -04:00
#include <import/ebistree.h>
#include <import/ebpttree.h>
CLEANUP: include: tree-wide alphabetical sort of include files This patch fixes all the leftovers from the include cleanup campaign. There were not that many (~400 entries in ~150 files) but it was definitely worth doing it as it revealed a few duplicates.
2020-06-09 03:07:15 -04:00
#include <import/ebsttree.h>
#include <import/lru.h>
REORG: include: update all files to use haproxy/api.h or api-t.h if needed All files that were including one of the following include files have been updated to only include haproxy/api.h or haproxy/api-t.h once instead: - common/config.h - common/compat.h - common/compiler.h - common/defaults.h - common/initcall.h - common/tools.h The choice is simple: if the file only requires type definitions, it includes api-t.h, otherwise it includes the full api.h. In addition, in these files, explicit includes for inttypes.h and limits.h were dropped since these are now covered by api.h and api-t.h. No other change was performed, given that this patch is large and affects 201 files. At least one (tools.h) was already freestanding and didn't get the new one added.
2020-05-27 06:58:42 -04:00
#include <haproxy/api.h>
REORG: include: split global.h into haproxy/global{,-t}.h global.h was one of the messiest files, it has accumulated tons of implicit dependencies and declares many globals that make almost all other file include it. It managed to silence a dependency loop between server.h and proxy.h by being well placed to pre-define the required structs, forcing struct proxy and struct server to be forward-declared in a significant number of files. It was split in to, one which is the global struct definition and the few macros and flags, and the rest containing the functions prototypes. The UNIX_MAX_PATH definition was moved to compat.h.
2020-06-04 11:05:57 -04:00
#include <haproxy/global.h>
REORG: include: move log.h to haproxy/log{,-t}.h The current state of the logging is a real mess. The main problem is that almost all files include log.h just in order to have access to the alert/warning functions like ha_alert() etc, and don't care about logs. But log.h also deals with real logging as well as log-format and depends on stream.h and various other things. As such it forces a few heavy files like stream.h to be loaded early and to hide missing dependencies depending where it's loaded. Among the missing ones is syslog.h which was often automatically included resulting in no less than 3 users missing it. Among 76 users, only 5 could be removed, and probably 70 don't need the full set of dependencies. A good approach would consist in splitting that file in 3 parts: - one for error output ("errors" ?). - one for log_format processing - and one for actual logging.
2020-06-04 16:01:04 -04:00
#include <haproxy/log.h>
REORG: include: move common/net_helper.h to haproxy/net_helper.h No change was necessary.
2020-06-02 10:48:09 -04:00
#include <haproxy/net_helper.h>
REORG: include: move pattern.h to haproxy/pattern{,-t}.h It was moved as-is, except for extern declaration of pattern_reference. A few C files used to include it but didn't need it anymore after having been split apart so this was cleaned.
2020-06-04 09:06:28 -04:00
#include <haproxy/pattern.h>
REORG: include: split common/regex.h into haproxy/regex{,-t}.h Regex are essentially included for myregex_t but it turns out that several of the C files didn't include it directly, relying on the one included by their own .h. This has been cleanly addressed so that only the type is included by H files which need it, and adding the missing includes for the other ones.
2020-06-02 11:32:26 -04:00
#include <haproxy/regex.h>
REORG: include: move sample.h to haproxy/sample{,-t}.h This one is particularly tricky to move because everyone uses it and it depends on a lot of other types. For example it cannot include arg-t.h and must absolutely only rely on forward declarations to avoid dependency loops between vars -> sample_data -> arg. In order to address this one, it would be nice to split the sample_data part out of sample.h.
2020-06-04 09:33:47 -04:00
#include <haproxy/sample.h>
CLEANUP: include: tree-wide alphabetical sort of include files This patch fixes all the leftovers from the include cleanup campaign. There were not that many (~400 entries in ~150 files) but it was definitely worth doing it as it revealed a few duplicates.
2020-06-09 03:07:15 -04:00
#include <haproxy/tools.h>
CLEANUP: Add haproxy/xxhash.h to avoid modifying import/xxhash.h This solves setting XXH_INLINE_ALL in a cleaner way, because the imported header is not modified, easing future updates. see 6f7cc11e6dd0f01b437fba893da2edd2362660a2
2021-09-11 11:51:13 -04:00
#include <haproxy/xxhash.h>
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
MINOR: patterns: preliminary changes for reorganization Safe and non-functional changes that only add currently unused structures, field, functions and macros, in preparation of larger changes that alter the way pattern reference elements are stored. This includes code to create and lookup generation objects, and macros to iterate over the generations of a pattern reference.
2025-12-17 23:31:29 -05:00
/* Convenience macros for iterating over generations. */
#define pat_ref_gen_foreach(gen, ref) \
for (gen = cebu32_item_first(&ref->gen_root, gen_node, gen_id, struct pat_ref_gen); \
gen; \
gen = cebu32_item_next(&ref->gen_root, gen_node, gen_id, gen))
/* Safe variant that allows deleting an entry in the body of the loop. */
#define pat_ref_gen_foreach_safe(gen, next, ref) \
for (gen = cebu32_item_first(&ref->gen_root, gen_node, gen_id, struct pat_ref_gen); \
gen && (next = cebu32_item_next(&ref->gen_root, gen_node, gen_id, gen), 1); \
gen = next)
CLEANUP: pattern: make all pattern tables read-only Interestingly, all arrays used to declare patterns were read-write while only hard-coded. Let's mark them const so that they move from data to rodata and don't risk to experience false sharing.
2021-04-10 11:44:27 -04:00
const char *const pat_match_names[PAT_MATCH_NUM] = {
MEDIUM: pattern: rename "acl" prefix to "pat" This patch just renames functions, types and enums. No code was changed. A significant number of files were touched, especially the ACL arrays, so it is likely that some external patches will not apply anymore. One important thing is that we had to split ACL_PAT_* into two groups : - ACL_TEST_{PASS|MISS|FAIL} - PAT_{MATCH|UNMATCH} A future patch will enforce enums on all these places to avoid confusion.
2013-11-28 12:22:00 -05:00
[PAT_MATCH_FOUND] = "found",
[PAT_MATCH_BOOL] = "bool",
[PAT_MATCH_INT] = "int",
[PAT_MATCH_IP] = "ip",
[PAT_MATCH_BIN] = "bin",
[PAT_MATCH_LEN] = "len",
[PAT_MATCH_STR] = "str",
[PAT_MATCH_BEG] = "beg",
[PAT_MATCH_SUB] = "sub",
[PAT_MATCH_DIR] = "dir",
[PAT_MATCH_DOM] = "dom",
[PAT_MATCH_END] = "end",
[PAT_MATCH_REG] = "reg",
MINOR: map: Add regex matching replacement This patch declares a new map which provides a string based on a string with back references replaced by the content matched by the regex.
2016-02-10 16:55:20 -05:00
[PAT_MATCH_REGM] = "regm",
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
};
CLEANUP: pattern: make all pattern tables read-only Interestingly, all arrays used to declare patterns were read-write while only hard-coded. Let's mark them const so that they move from data to rodata and don't risk to experience false sharing.
2021-04-10 11:44:27 -04:00
int (*const pat_parse_fcts[PAT_MATCH_NUM])(const char *, struct pattern *, int, char **) = {
MEDIUM: pattern: rename "acl" prefix to "pat" This patch just renames functions, types and enums. No code was changed. A significant number of files were touched, especially the ACL arrays, so it is likely that some external patches will not apply anymore. One important thing is that we had to split ACL_PAT_* into two groups : - ACL_TEST_{PASS|MISS|FAIL} - PAT_{MATCH|UNMATCH} A future patch will enforce enums on all these places to avoid confusion.
2013-11-28 12:22:00 -05:00
[PAT_MATCH_FOUND] = pat_parse_nothing,
[PAT_MATCH_BOOL] = pat_parse_nothing,
[PAT_MATCH_INT] = pat_parse_int,
[PAT_MATCH_IP] = pat_parse_ip,
[PAT_MATCH_BIN] = pat_parse_bin,
MEDIUM: pattern: The expected type is stored in the pattern head, and conversion is executed once. This patch extract the expect_type variable from the "struct pattern" to "struct pattern_head". This variable is set during the declaration of ACL and MAP. With this change, the function "pat_parse_len()" become useless and can be replaced by "pat_parse_int()". Implicit ACLs by default rely on the fetch's output type, so let's simply do the same for all other ones. It has been verified that they all match.
2014-01-27 08:19:53 -05:00
[PAT_MATCH_LEN] = pat_parse_int,
MEDIUM: pattern: rename "acl" prefix to "pat" This patch just renames functions, types and enums. No code was changed. A significant number of files were touched, especially the ACL arrays, so it is likely that some external patches will not apply anymore. One important thing is that we had to split ACL_PAT_* into two groups : - ACL_TEST_{PASS|MISS|FAIL} - PAT_{MATCH|UNMATCH} A future patch will enforce enums on all these places to avoid confusion.
2013-11-28 12:22:00 -05:00
[PAT_MATCH_STR] = pat_parse_str,
[PAT_MATCH_BEG] = pat_parse_str,
[PAT_MATCH_SUB] = pat_parse_str,
[PAT_MATCH_DIR] = pat_parse_str,
[PAT_MATCH_DOM] = pat_parse_str,
[PAT_MATCH_END] = pat_parse_str,
[PAT_MATCH_REG] = pat_parse_reg,
MINOR: map: Add regex matching replacement This patch declares a new map which provides a string based on a string with back references replaced by the content matched by the regex.
2016-02-10 16:55:20 -05:00
[PAT_MATCH_REGM] = pat_parse_reg,
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
};
CLEANUP: pattern: make all pattern tables read-only Interestingly, all arrays used to declare patterns were read-write while only hard-coded. Let's mark them const so that they move from data to rodata and don't risk to experience false sharing.
2021-04-10 11:44:27 -04:00
int (*const pat_index_fcts[PAT_MATCH_NUM])(struct pattern_expr *, struct pattern *, char **) = {
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
[PAT_MATCH_FOUND] = pat_idx_list_val,
[PAT_MATCH_BOOL] = pat_idx_list_val,
[PAT_MATCH_INT] = pat_idx_list_val,
[PAT_MATCH_IP] = pat_idx_tree_ip,
[PAT_MATCH_BIN] = pat_idx_list_ptr,
[PAT_MATCH_LEN] = pat_idx_list_val,
[PAT_MATCH_STR] = pat_idx_tree_str,
MEDIUM: pattern: use ebtree's longest match to index/lookup string beginning Being able to map prefixes to values is already used for IPv4/IPv6 but was not yet used with strings. It can be very convenient to map directories to server farms but large lists may be slow. By using ebmb_insert_prefix() and ebmb_lookup_longest(), we can insert strings with their own length as a prefix, and lookup candidate strings and ensure that the longest matching one will be returned, which is the longest string matching the entry.
2014-05-10 02:53:48 -04:00
[PAT_MATCH_BEG] = pat_idx_tree_pfx,
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
[PAT_MATCH_SUB] = pat_idx_list_str,
[PAT_MATCH_DIR] = pat_idx_list_str,
[PAT_MATCH_DOM] = pat_idx_list_str,
[PAT_MATCH_END] = pat_idx_list_str,
[PAT_MATCH_REG] = pat_idx_list_reg,
MINOR: map: Add regex matching replacement This patch declares a new map which provides a string based on a string with back references replaced by the content matched by the regex.
2016-02-10 16:55:20 -05:00
[PAT_MATCH_REGM] = pat_idx_list_regm,
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
};
CLEANUP: pattern: make all pattern tables read-only Interestingly, all arrays used to declare patterns were read-write while only hard-coded. Let's mark them const so that they move from data to rodata and don't risk to experience false sharing.
2021-04-10 11:44:27 -04:00
void (*const pat_prune_fcts[PAT_MATCH_NUM])(struct pattern_expr *) = {
MINOR: pattern: make the delete and prune functions more generic Now we have a single prune() function to act on an expression, and one delete function for the lists and one for the trees. The presence of a pointer in the lists is enough to warrant a free, and we rely on the PAT_SF_REGFREE flag to decide whether to free using free() or regfree().
2020-11-02 13:26:02 -05:00
[PAT_MATCH_FOUND] = pat_prune_gen,
[PAT_MATCH_BOOL] = pat_prune_gen,
[PAT_MATCH_INT] = pat_prune_gen,
[PAT_MATCH_IP] = pat_prune_gen,
[PAT_MATCH_BIN] = pat_prune_gen,
[PAT_MATCH_LEN] = pat_prune_gen,
[PAT_MATCH_STR] = pat_prune_gen,
[PAT_MATCH_BEG] = pat_prune_gen,
[PAT_MATCH_SUB] = pat_prune_gen,
[PAT_MATCH_DIR] = pat_prune_gen,
[PAT_MATCH_DOM] = pat_prune_gen,
[PAT_MATCH_END] = pat_prune_gen,
[PAT_MATCH_REG] = pat_prune_gen,
[PAT_MATCH_REGM] = pat_prune_gen,
MEDIUM: pattern: add prune function This path add specific pointer to each expression to point on prune function. Now, each pattern expression embed his own prune function.
2014-01-14 10:24:51 -05:00
};
CLEANUP: pattern: make all pattern tables read-only Interestingly, all arrays used to declare patterns were read-write while only hard-coded. Let's mark them const so that they move from data to rodata and don't risk to experience false sharing.
2021-04-10 11:44:27 -04:00
struct pattern *(*const pat_match_fcts[PAT_MATCH_NUM])(struct sample *, struct pattern_expr *, int) = {
MEDIUM: pattern: rename "acl" prefix to "pat" This patch just renames functions, types and enums. No code was changed. A significant number of files were touched, especially the ACL arrays, so it is likely that some external patches will not apply anymore. One important thing is that we had to split ACL_PAT_* into two groups : - ACL_TEST_{PASS|MISS|FAIL} - PAT_{MATCH|UNMATCH} A future patch will enforce enums on all these places to avoid confusion.
2013-11-28 12:22:00 -05:00
[PAT_MATCH_FOUND] = NULL,
[PAT_MATCH_BOOL] = pat_match_nothing,
[PAT_MATCH_INT] = pat_match_int,
[PAT_MATCH_IP] = pat_match_ip,
[PAT_MATCH_BIN] = pat_match_bin,
[PAT_MATCH_LEN] = pat_match_len,
[PAT_MATCH_STR] = pat_match_str,
[PAT_MATCH_BEG] = pat_match_beg,
[PAT_MATCH_SUB] = pat_match_sub,
[PAT_MATCH_DIR] = pat_match_dir,
[PAT_MATCH_DOM] = pat_match_dom,
[PAT_MATCH_END] = pat_match_end,
[PAT_MATCH_REG] = pat_match_reg,
MINOR: map: Add regex matching replacement This patch declares a new map which provides a string based on a string with back references replaced by the content matched by the regex.
2016-02-10 16:55:20 -05:00
[PAT_MATCH_REGM] = pat_match_regm,
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
};
MEDIUM: acl: Last patch change the output type This patch remove the compatibility check from the input type and the match method. Now, it checks if a casts from the input type to output type exists and the pattern_exec_match() function apply casts before each pattern matching.
2013-12-06 09:36:54 -05:00
/* Just used for checking configuration compatibility */
CLEANUP: pattern: make all pattern tables read-only Interestingly, all arrays used to declare patterns were read-write while only hard-coded. Let's mark them const so that they move from data to rodata and don't risk to experience false sharing.
2021-04-10 11:44:27 -04:00
int const pat_match_types[PAT_MATCH_NUM] = {
MAJOR: sample: converts uint and sint in 64 bits signed integer This patch removes the 32 bits unsigned integer and the 32 bit signed integer. It replaces these types by a unique type 64 bit signed. This makes easy the usage of integer and clarify signed and unsigned use. With the previous version, signed and unsigned are used ones in place of others, and sometimes the converter loose the sign. For example, divisions are processed with "unsigned", if one entry is negative, the result is wrong. Note that the integer pattern matching and dotted version pattern matching are already working with signed 64 bits integer values. There is one user-visible change : the "uint()" and "sint()" sample fetch functions which used to return a constant integer have been replaced with a new more natural, unified "int()" function. These functions were only introduced in the latest 1.6-dev2 so there's no impact on regular deployments.
2015-07-06 17:43:03 -04:00
[PAT_MATCH_FOUND] = SMP_T_SINT,
[PAT_MATCH_BOOL] = SMP_T_SINT,
[PAT_MATCH_INT] = SMP_T_SINT,
MEDIUM: acl: Last patch change the output type This patch remove the compatibility check from the input type and the match method. Now, it checks if a casts from the input type to output type exists and the pattern_exec_match() function apply casts before each pattern matching.
2013-12-06 09:36:54 -05:00
[PAT_MATCH_IP] = SMP_T_ADDR,
MEDIUM: sample: Remove types SMP_T_CSTR and SMP_T_CBIN, replace it by SMP_F_CONST flags The operations applied on types SMP_T_CSTR and SMP_T_STR are the same, but the check code and the declarations are double, because it must declare action for SMP_T_C* and SMP_T_*. The declared actions and checks are the same. this complexify the code. Only the "conv" functions can change from "C*" to "*" Now, if a function needs to modify input string, it can call the new function smp_dup(). This one duplicate data in a trash buffer.
2013-12-16 18:20:33 -05:00
[PAT_MATCH_BIN] = SMP_T_BIN,
[PAT_MATCH_LEN] = SMP_T_STR,
[PAT_MATCH_STR] = SMP_T_STR,
[PAT_MATCH_BEG] = SMP_T_STR,
[PAT_MATCH_SUB] = SMP_T_STR,
[PAT_MATCH_DIR] = SMP_T_STR,
[PAT_MATCH_DOM] = SMP_T_STR,
[PAT_MATCH_END] = SMP_T_STR,
[PAT_MATCH_REG] = SMP_T_STR,
MINOR: map: Add regex matching replacement This patch declares a new map which provides a string based on a string with back references replaced by the content matched by the regex.
2016-02-10 16:55:20 -05:00
[PAT_MATCH_REGM] = SMP_T_STR,
MEDIUM: acl: Last patch change the output type This patch remove the compatibility check from the input type and the match method. Now, it checks if a casts from the input type to output type exists and the pattern_exec_match() function apply casts before each pattern matching.
2013-12-06 09:36:54 -05:00
};
MEDIUM: pattern: The function pattern_exec_match() returns "struct pattern" if the patten match. Before this commit, the pattern_exec_match() function returns the associate sample, the associate struct pattern or the associate struct pattern_tree. This is complex to use, because we can check the type of information returned. Now the function return always a "struct pattern". If <fill> is not set, only the value of the pointer can be used as boolean (NULL or other). If <fill> is set, you can use the <smp> pointer and the pattern information. If information must be duplicated, it is stored in trash buffer. Otherwise, the pattern can point on existing strings.
2014-01-17 09:25:13 -05:00
/* this struct is used to return information */
MAJOR: threads/map: Make acls/maps thread safe locks have been added in pat_ref and pattern_expr structures to protect all accesses to an instance of on of them. Moreover, a global lock has been added to protect the LRU cache used for pattern matching. Patterns are now duplicated after a successfull matching, to avoid modification by other threads when the result is used. Finally, the function reloading a pattern list has been modified to be thread-safe.
2017-07-03 05:34:05 -04:00
static THREAD_LOCAL struct pattern static_pattern;
static THREAD_LOCAL struct sample_data static_sample_data;
MEDIUM: pattern: The function pattern_exec_match() returns "struct pattern" if the patten match. Before this commit, the pattern_exec_match() function returns the associate sample, the associate struct pattern or the associate struct pattern_tree. This is complex to use, because we can check the type of information returned. Now the function return always a "struct pattern". If <fill> is not set, only the value of the pointer can be used as boolean (NULL or other). If <fill> is set, you can use the <smp> pointer and the pattern information. If information must be duplicated, it is stored in trash buffer. Otherwise, the pattern can point on existing strings.
2014-01-17 09:25:13 -05:00
CLEANUP: assorted typo fixes in the code, commits and doc
2025-04-02 14:44:22 -04:00
/* This is the root of the list of all available pattern_ref values. */
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
struct list pattern_reference = LIST_HEAD_INIT(pattern_reference);
BUG/MEDIUM: pattern: make the pattern LRU cache thread-local and lockless As reported in issue #335, a lot of contention happens on the PATLRU lock when performing expensive regex lookups. This is absurd since the purpose of the LRU cache was to have a fast cache for expressions, thus the cache must not be shared between threads and must remain lockless. This commit makes the LRU cache thread-local and gets rid of the PATLRU lock. A test with 7 threads on 4 cores climbed from 67kH/s to 369kH/s, or a scalability factor of 5.5. Given the huge performance difference and the regression caused to users migrating from processes to threads, this should be backported at least to 2.0. Thanks to Brian Diekelman for his detailed report about this regression.
2019-10-23 00:59:31 -04:00
static THREAD_LOCAL struct lru64_head *pat_lru_tree;
MINOR: pattern: make the pat_lru_seed read_mostly This seed is created once at boot and is used in every LRU hash when caching results. Let's mark it read_mostly.
2021-04-10 11:42:04 -04:00
static unsigned long long pat_lru_seed __read_mostly;
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
MINOR: pattern: add a counter of added/freed patterns Patterns are allocated when loading maps/acls from a file or dynamically via the CLI, and are released only from the CLI (e.g. "clear map xxx"). These ones do not use pools and are much harder to monitor, e.g. in case a script adds many and forgets to clear them, etc. Let's add a new pair of metrics "PatternsAdded" and "PatternsFreed" that will report the number of added and freed patterns respectively. This can allow to simply graph both. The difference between the two normally represents the number of allocated patterns. If Added grows without Freed following, it can indicate a faulty script that doesn't perform the needed cleanup. The metrics are also made available to Prometheus as patterns_added_total and patterns_freed_total respectively.
2025-07-04 18:07:25 -04:00
unsigned long long patterns_added = 0;
unsigned long long patterns_freed = 0;
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
/*
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
*
* The following functions are not exported and are used by internals process
* of pattern matching
*
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
*/
/* Background: Fast way to find a zero byte in a word
* http://graphics.stanford.edu/~seander/bithacks.html#ZeroInWord
* hasZeroByte = (v - 0x01010101UL) & ~v & 0x80808080UL;
*
* To look for 4 different byte values, xor the word with those bytes and
* then check for zero bytes:
*
* v = (((unsigned char)c * 0x1010101U) ^ delimiter)
* where <delimiter> is the 4 byte values to look for (as an uint)
* and <c> is the character that is being tested
*/
static inline unsigned int is_delimiter(unsigned char c, unsigned int mask)
{
mask ^= (c * 0x01010101); /* propagate the char to all 4 bytes */
return (mask - 0x01010101) & ~mask & 0x80808080U;
}
static inline unsigned int make_4delim(unsigned char d1, unsigned char d2, unsigned char d3, unsigned char d4)
{
return d1 << 24 | d2 << 16 | d3 << 8 | d4;
}
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
/*
*
* These functions are exported and may be used by any other component.
*
BUG/MINOR: pattern: remove useless allocation of unused trash in pat_parse_reg() Just like previous patch, this is a remains of an early implementation. Also fix the outdated comments above. The fix may be backported to 1.5 though the bug cannot be triggerred, thus it's just a matter of keeping the code clean.
2014-08-29 09:19:33 -04:00
* The following functions are used for parsing pattern matching input value.
* The <text> contain the string to be parsed. <pattern> must be a preallocated
* pattern. The pat_parse_* functions fill this structure with the parsed value.
* <err> is filled with an error message built with memprintf() function. It is
* allowed to use a trash as a temporary storage for the returned pattern, as
* the next call after these functions will be pat_idx_*.
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
*
BUG/MINOR: pattern: remove useless allocation of unused trash in pat_parse_reg() Just like previous patch, this is a remains of an early implementation. Also fix the outdated comments above. The fix may be backported to 1.5 though the bug cannot be triggerred, thus it's just a matter of keeping the code clean.
2014-08-29 09:19:33 -04:00
* In success case, the pat_parse_* function returns 1. If the function
* fails, it returns 0 and <err> is filled.
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
*/
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
/* ignore the current line */
BUG/MEDIUM: patterns: last fix was still not enough Last fix did address the issue for inlined patterns, but it was not enough because the flags are lost as well when updating patterns dynamically over the CLI. Also if the same file was used once with -i and another time without -i, their references would have been merged and both would have used the same matching method. It's appear that the patterns have two types of flags. The first ones are relative to the pattern matching, and the second are relative to the pattern storage. The pattern matching flags are the same for all the patterns of one expression. Now they are stored in the expression. The storage flags are information returned by the pattern mathing function. This information is relative to each entry and is stored in the "struct pattern". Now, the expression matching flags are forwarded to the parse and index functions. These flags are stored during the configuration parsing, and they are used during the parse and index actions. This issue was introduced in dev23 with the major pattern rework, and is a continuation of commit a631fc8 ("BUG/MAJOR: patterns: -i and -n are ignored for inlined patterns"). No backport is needed.
2014-04-28 05:18:57 -04:00
int pat_parse_nothing(const char *text, struct pattern *pattern, int mflags, char **err)
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
{
return 1;
}
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
/* Parse a string. It is allocated and duplicated. */
BUG/MEDIUM: patterns: last fix was still not enough Last fix did address the issue for inlined patterns, but it was not enough because the flags are lost as well when updating patterns dynamically over the CLI. Also if the same file was used once with -i and another time without -i, their references would have been merged and both would have used the same matching method. It's appear that the patterns have two types of flags. The first ones are relative to the pattern matching, and the second are relative to the pattern storage. The pattern matching flags are the same for all the patterns of one expression. Now they are stored in the expression. The storage flags are information returned by the pattern mathing function. This information is relative to each entry and is stored in the "struct pattern". Now, the expression matching flags are forwarded to the parse and index functions. These flags are stored during the configuration parsing, and they are used during the parse and index actions. This issue was introduced in dev23 with the major pattern rework, and is a continuation of commit a631fc8 ("BUG/MAJOR: patterns: -i and -n are ignored for inlined patterns"). No backport is needed.
2014-04-28 05:18:57 -04:00
int pat_parse_str(const char *text, struct pattern *pattern, int mflags, char **err)
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
{
MEDIUM: sample: Remove types SMP_T_CSTR and SMP_T_CBIN, replace it by SMP_F_CONST flags The operations applied on types SMP_T_CSTR and SMP_T_STR are the same, but the check code and the declarations are double, because it must declare action for SMP_T_C* and SMP_T_*. The declared actions and checks are the same. this complexify the code. Only the "conv" functions can change from "C*" to "*" Now, if a function needs to modify input string, it can call the new function smp_dup(). This one duplicate data in a trash buffer.
2013-12-16 18:20:33 -05:00
pattern->type = SMP_T_STR;
MEDIUM: pattern: The parse functions just return "struct pattern" without memory allocation The pattern parse functions put the parsed result in a "struct pattern" without memory allocation. If the pattern must reference the input data without changes, the pattern point to the parsed string. If buffers are needed to store translated data, it use th trash buffer. The indexation function that allocate the memory later if it is needed.
2013-12-13 09:36:59 -05:00
pattern->ptr.str = (char *)text;
MEDIUM: pattern: The pattern parser no more uses <opaque> and just takes one string. After the previous patches, the "pat_parse_strcat()" function disappear, and the "pat_parse_int()" and "pat_parse_dotted_ver()" functions dont use anymore the "opaque" argument, and take only one string on his input. So, after this patch, each pattern parser no longer use the opaque variable and take only one string as input. This patch change the prototype of the pattern parsing functions. Now, the "char **args" is replaced by a "char *arg", the "int *opaque" is removed and these functions return 1 in succes case, and 0 if fail.
2014-01-24 04:58:12 -05:00
pattern->len = strlen(text);
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
return 1;
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
}
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
/* Parse a binary written in hexa. It is allocated. */
BUG/MEDIUM: patterns: last fix was still not enough Last fix did address the issue for inlined patterns, but it was not enough because the flags are lost as well when updating patterns dynamically over the CLI. Also if the same file was used once with -i and another time without -i, their references would have been merged and both would have used the same matching method. It's appear that the patterns have two types of flags. The first ones are relative to the pattern matching, and the second are relative to the pattern storage. The pattern matching flags are the same for all the patterns of one expression. Now they are stored in the expression. The storage flags are information returned by the pattern mathing function. This information is relative to each entry and is stored in the "struct pattern". Now, the expression matching flags are forwarded to the parse and index functions. These flags are stored during the configuration parsing, and they are used during the parse and index actions. This issue was introduced in dev23 with the major pattern rework, and is a continuation of commit a631fc8 ("BUG/MAJOR: patterns: -i and -n are ignored for inlined patterns"). No backport is needed.
2014-04-28 05:18:57 -04:00
int pat_parse_bin(const char *text, struct pattern *pattern, int mflags, char **err)
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
{
MAJOR: chunks: replace struct chunk with struct buffer Now all the code used to manipulate chunks uses a struct buffer instead. The functions are still called "chunk*", and some of them will progressively move to the generic buffer handling code as they are cleaned up.
2018-07-13 05:56:34 -04:00
struct buffer *trash;
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
MEDIUM: sample: Remove types SMP_T_CSTR and SMP_T_CBIN, replace it by SMP_F_CONST flags The operations applied on types SMP_T_CSTR and SMP_T_STR are the same, but the check code and the declarations are double, because it must declare action for SMP_T_C* and SMP_T_*. The declared actions and checks are the same. this complexify the code. Only the "conv" functions can change from "C*" to "*" Now, if a function needs to modify input string, it can call the new function smp_dup(). This one duplicate data in a trash buffer.
2013-12-16 18:20:33 -05:00
pattern->type = SMP_T_BIN;
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
trash = get_trash_chunk();
pattern->len = trash->size;
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
pattern->ptr.str = trash->area;
MEDIUM: pattern: The pattern parser no more uses <opaque> and just takes one string. After the previous patches, the "pat_parse_strcat()" function disappear, and the "pat_parse_int()" and "pat_parse_dotted_ver()" functions dont use anymore the "opaque" argument, and take only one string on his input. So, after this patch, each pattern parser no longer use the opaque variable and take only one string as input. This patch change the prototype of the pattern parsing functions. Now, the "char **args" is replaced by a "char *arg", the "int *opaque" is removed and these functions return 1 in succes case, and 0 if fail.
2014-01-24 04:58:12 -05:00
return !!parse_binary(text, &pattern->ptr.str, &pattern->len, err);
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
}
/* Parse a regex. It is allocated. */
BUG/MEDIUM: patterns: last fix was still not enough Last fix did address the issue for inlined patterns, but it was not enough because the flags are lost as well when updating patterns dynamically over the CLI. Also if the same file was used once with -i and another time without -i, their references would have been merged and both would have used the same matching method. It's appear that the patterns have two types of flags. The first ones are relative to the pattern matching, and the second are relative to the pattern storage. The pattern matching flags are the same for all the patterns of one expression. Now they are stored in the expression. The storage flags are information returned by the pattern mathing function. This information is relative to each entry and is stored in the "struct pattern". Now, the expression matching flags are forwarded to the parse and index functions. These flags are stored during the configuration parsing, and they are used during the parse and index actions. This issue was introduced in dev23 with the major pattern rework, and is a continuation of commit a631fc8 ("BUG/MAJOR: patterns: -i and -n are ignored for inlined patterns"). No backport is needed.
2014-04-28 05:18:57 -04:00
int pat_parse_reg(const char *text, struct pattern *pattern, int mflags, char **err)
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
{
MINOR: regex: The pointer regstr in the struc regex is no longer used. The pointer <regstr> is only used to compare and identify the original regex string with the patterns. Now the patterns have a reference map containing this original string. It is useless to store this value two times.
2014-01-29 13:35:16 -05:00
pattern->ptr.str = (char *)text;
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
return 1;
}
/* Parse a range of positive integers delimited by either ':' or '-'. If only
* one integer is read, it is set as both min and max. An operator may be
* specified as the prefix, among this list of 5 :
*
* 0:eq, 1:gt, 2:ge, 3:lt, 4:le
*
* The default operator is "eq". It supports range matching. Ranges are
* rejected for other operators. The operator may be changed at any time.
* The operator is stored in the 'opaque' argument.
*
* If err is non-NULL, an error message will be returned there on errors and
MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and "pat_parse_dotted_ver()" The goal of these patch is to simplify the prototype of "pat_pattern_*()" functions. I want to replace the argument "char **args" by a simple "char *arg" and remove the "opaque" argument. "pat_parse_int()" and "pat_parse_dotted_ver()" are the unique pattern parser using the "opaque" argument and using more than one string argument of the char **args. These specificities are only used with ACL. Other systems using this pattern parser (MAP and CLI) just use one string for describing a range. This two functions can read a range, but the min and the max must y specified. This patch extends the syntax to describe a range with implicit min and max. This is used for operators like "lt", "le", "gt", and "ge". the syntax is the following: ":x" -> no min to "x" "x:" -> "x" to no max This patch moves the parsing of the comparison operator from the functions "pat_parse_int()" and "pat_parse_dotted_ver()" to the acl parser. The acl parser read the operator and the values and build a volatile string readable by the functions "pat_parse_int()" and "pat_parse_dotted_ver()". The transformation is done with these rules: If the parser is "pat_parse_int()": "eq x" -> "x" "le x" -> ":x" "lt x" -> ":y" (with y = x - 1) "ge x" -> "x:" "gt x" -> "y:" (with y = x + 1) If the parser is "pat_parse_dotted_ver()": "eq x.y" -> "x.y" "le x.y" -> ":x.y" "lt x.y" -> ":w.z" (with w.z = x.y - 1) "ge x.y" -> "x.y:" "gt x.y" -> "w.z:" (with w.z = x.y + 1) Note that, if "y" is not present, assume that is "0". Now "pat_parse_int()" and "pat_parse_dotted_ver()" accept only one pattern and the variable "opaque" is no longer used. The prototype of the pattern parsers can be changed.
2014-01-23 11:40:34 -05:00
* the caller will have to free it. The function returns zero on error, and
* non-zero on success.
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
*
*/
BUG/MEDIUM: patterns: last fix was still not enough Last fix did address the issue for inlined patterns, but it was not enough because the flags are lost as well when updating patterns dynamically over the CLI. Also if the same file was used once with -i and another time without -i, their references would have been merged and both would have used the same matching method. It's appear that the patterns have two types of flags. The first ones are relative to the pattern matching, and the second are relative to the pattern storage. The pattern matching flags are the same for all the patterns of one expression. Now they are stored in the expression. The storage flags are information returned by the pattern mathing function. This information is relative to each entry and is stored in the "struct pattern". Now, the expression matching flags are forwarded to the parse and index functions. These flags are stored during the configuration parsing, and they are used during the parse and index actions. This issue was introduced in dev23 with the major pattern rework, and is a continuation of commit a631fc8 ("BUG/MAJOR: patterns: -i and -n are ignored for inlined patterns"). No backport is needed.
2014-04-28 05:18:57 -04:00
int pat_parse_int(const char *text, struct pattern *pattern, int mflags, char **err)
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
{
MEDIUM: pattern: The pattern parser no more uses <opaque> and just takes one string. After the previous patches, the "pat_parse_strcat()" function disappear, and the "pat_parse_int()" and "pat_parse_dotted_ver()" functions dont use anymore the "opaque" argument, and take only one string on his input. So, after this patch, each pattern parser no longer use the opaque variable and take only one string as input. This patch change the prototype of the pattern parsing functions. Now, the "char **args" is replaced by a "char *arg", the "int *opaque" is removed and these functions return 1 in succes case, and 0 if fail.
2014-01-24 04:58:12 -05:00
const char *ptr = text;
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
MAJOR: sample: converts uint and sint in 64 bits signed integer This patch removes the 32 bits unsigned integer and the 32 bit signed integer. It replaces these types by a unique type 64 bit signed. This makes easy the usage of integer and clarify signed and unsigned use. With the previous version, signed and unsigned are used ones in place of others, and sometimes the converter loose the sign. For example, divisions are processed with "unsigned", if one entry is negative, the result is wrong. Note that the integer pattern matching and dotted version pattern matching are already working with signed 64 bits integer values. There is one user-visible change : the "uint()" and "sint()" sample fetch functions which used to return a constant integer have been replaced with a new more natural, unified "int()" function. These functions were only introduced in the latest 1.6-dev2 so there's no impact on regular deployments.
2015-07-06 17:43:03 -04:00
pattern->type = SMP_T_SINT;
MEDIUM: pattern: Extract the index process from the pat_parse_*() functions Now, the pat_parse_*() functions parses the incoming data. The input "pattern" struct can be preallocated. If the parser needs to add some buffers, it allocates memory. The function pattern_register() runs the call to the parser, process the key indexation and associate the "sample_storage" used by maps.
2013-12-06 13:06:43 -05:00
MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and "pat_parse_dotted_ver()" The goal of these patch is to simplify the prototype of "pat_pattern_*()" functions. I want to replace the argument "char **args" by a simple "char *arg" and remove the "opaque" argument. "pat_parse_int()" and "pat_parse_dotted_ver()" are the unique pattern parser using the "opaque" argument and using more than one string argument of the char **args. These specificities are only used with ACL. Other systems using this pattern parser (MAP and CLI) just use one string for describing a range. This two functions can read a range, but the min and the max must y specified. This patch extends the syntax to describe a range with implicit min and max. This is used for operators like "lt", "le", "gt", and "ge". the syntax is the following: ":x" -> no min to "x" "x:" -> "x" to no max This patch moves the parsing of the comparison operator from the functions "pat_parse_int()" and "pat_parse_dotted_ver()" to the acl parser. The acl parser read the operator and the values and build a volatile string readable by the functions "pat_parse_int()" and "pat_parse_dotted_ver()". The transformation is done with these rules: If the parser is "pat_parse_int()": "eq x" -> "x" "le x" -> ":x" "lt x" -> ":y" (with y = x - 1) "ge x" -> "x:" "gt x" -> "y:" (with y = x + 1) If the parser is "pat_parse_dotted_ver()": "eq x.y" -> "x.y" "le x.y" -> ":x.y" "lt x.y" -> ":w.z" (with w.z = x.y - 1) "ge x.y" -> "x.y:" "gt x.y" -> "w.z:" (with w.z = x.y + 1) Note that, if "y" is not present, assume that is "0". Now "pat_parse_int()" and "pat_parse_dotted_ver()" accept only one pattern and the variable "opaque" is no longer used. The prototype of the pattern parsers can be changed.
2014-01-23 11:40:34 -05:00
/* Empty string is not valid */
MEDIUM: pattern: The pattern parser no more uses <opaque> and just takes one string. After the previous patches, the "pat_parse_strcat()" function disappear, and the "pat_parse_int()" and "pat_parse_dotted_ver()" functions dont use anymore the "opaque" argument, and take only one string on his input. So, after this patch, each pattern parser no longer use the opaque variable and take only one string as input. This patch change the prototype of the pattern parsing functions. Now, the "char **args" is replaced by a "char *arg", the "int *opaque" is removed and these functions return 1 in succes case, and 0 if fail.
2014-01-24 04:58:12 -05:00
if (!*text)
MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and "pat_parse_dotted_ver()" The goal of these patch is to simplify the prototype of "pat_pattern_*()" functions. I want to replace the argument "char **args" by a simple "char *arg" and remove the "opaque" argument. "pat_parse_int()" and "pat_parse_dotted_ver()" are the unique pattern parser using the "opaque" argument and using more than one string argument of the char **args. These specificities are only used with ACL. Other systems using this pattern parser (MAP and CLI) just use one string for describing a range. This two functions can read a range, but the min and the max must y specified. This patch extends the syntax to describe a range with implicit min and max. This is used for operators like "lt", "le", "gt", and "ge". the syntax is the following: ":x" -> no min to "x" "x:" -> "x" to no max This patch moves the parsing of the comparison operator from the functions "pat_parse_int()" and "pat_parse_dotted_ver()" to the acl parser. The acl parser read the operator and the values and build a volatile string readable by the functions "pat_parse_int()" and "pat_parse_dotted_ver()". The transformation is done with these rules: If the parser is "pat_parse_int()": "eq x" -> "x" "le x" -> ":x" "lt x" -> ":y" (with y = x - 1) "ge x" -> "x:" "gt x" -> "y:" (with y = x + 1) If the parser is "pat_parse_dotted_ver()": "eq x.y" -> "x.y" "le x.y" -> ":x.y" "lt x.y" -> ":w.z" (with w.z = x.y - 1) "ge x.y" -> "x.y:" "gt x.y" -> "w.z:" (with w.z = x.y + 1) Note that, if "y" is not present, assume that is "0". Now "pat_parse_int()" and "pat_parse_dotted_ver()" accept only one pattern and the variable "opaque" is no longer used. The prototype of the pattern parsers can be changed.
2014-01-23 11:40:34 -05:00
goto not_valid_range;
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and "pat_parse_dotted_ver()" The goal of these patch is to simplify the prototype of "pat_pattern_*()" functions. I want to replace the argument "char **args" by a simple "char *arg" and remove the "opaque" argument. "pat_parse_int()" and "pat_parse_dotted_ver()" are the unique pattern parser using the "opaque" argument and using more than one string argument of the char **args. These specificities are only used with ACL. Other systems using this pattern parser (MAP and CLI) just use one string for describing a range. This two functions can read a range, but the min and the max must y specified. This patch extends the syntax to describe a range with implicit min and max. This is used for operators like "lt", "le", "gt", and "ge". the syntax is the following: ":x" -> no min to "x" "x:" -> "x" to no max This patch moves the parsing of the comparison operator from the functions "pat_parse_int()" and "pat_parse_dotted_ver()" to the acl parser. The acl parser read the operator and the values and build a volatile string readable by the functions "pat_parse_int()" and "pat_parse_dotted_ver()". The transformation is done with these rules: If the parser is "pat_parse_int()": "eq x" -> "x" "le x" -> ":x" "lt x" -> ":y" (with y = x - 1) "ge x" -> "x:" "gt x" -> "y:" (with y = x + 1) If the parser is "pat_parse_dotted_ver()": "eq x.y" -> "x.y" "le x.y" -> ":x.y" "lt x.y" -> ":w.z" (with w.z = x.y - 1) "ge x.y" -> "x.y:" "gt x.y" -> "w.z:" (with w.z = x.y + 1) Note that, if "y" is not present, assume that is "0". Now "pat_parse_int()" and "pat_parse_dotted_ver()" accept only one pattern and the variable "opaque" is no longer used. The prototype of the pattern parsers can be changed.
2014-01-23 11:40:34 -05:00
/* Search ':' or '-' separator. */
while (*ptr != '\0' && *ptr != ':' && *ptr != '-')
ptr++;
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and "pat_parse_dotted_ver()" The goal of these patch is to simplify the prototype of "pat_pattern_*()" functions. I want to replace the argument "char **args" by a simple "char *arg" and remove the "opaque" argument. "pat_parse_int()" and "pat_parse_dotted_ver()" are the unique pattern parser using the "opaque" argument and using more than one string argument of the char **args. These specificities are only used with ACL. Other systems using this pattern parser (MAP and CLI) just use one string for describing a range. This two functions can read a range, but the min and the max must y specified. This patch extends the syntax to describe a range with implicit min and max. This is used for operators like "lt", "le", "gt", and "ge". the syntax is the following: ":x" -> no min to "x" "x:" -> "x" to no max This patch moves the parsing of the comparison operator from the functions "pat_parse_int()" and "pat_parse_dotted_ver()" to the acl parser. The acl parser read the operator and the values and build a volatile string readable by the functions "pat_parse_int()" and "pat_parse_dotted_ver()". The transformation is done with these rules: If the parser is "pat_parse_int()": "eq x" -> "x" "le x" -> ":x" "lt x" -> ":y" (with y = x - 1) "ge x" -> "x:" "gt x" -> "y:" (with y = x + 1) If the parser is "pat_parse_dotted_ver()": "eq x.y" -> "x.y" "le x.y" -> ":x.y" "lt x.y" -> ":w.z" (with w.z = x.y - 1) "ge x.y" -> "x.y:" "gt x.y" -> "w.z:" (with w.z = x.y + 1) Note that, if "y" is not present, assume that is "0". Now "pat_parse_int()" and "pat_parse_dotted_ver()" accept only one pattern and the variable "opaque" is no longer used. The prototype of the pattern parsers can be changed.
2014-01-23 11:40:34 -05:00
/* If separator not found. */
if (!*ptr) {
MEDIUM: pattern: The pattern parser no more uses <opaque> and just takes one string. After the previous patches, the "pat_parse_strcat()" function disappear, and the "pat_parse_int()" and "pat_parse_dotted_ver()" functions dont use anymore the "opaque" argument, and take only one string on his input. So, after this patch, each pattern parser no longer use the opaque variable and take only one string as input. This patch change the prototype of the pattern parsing functions. Now, the "char **args" is replaced by a "char *arg", the "int *opaque" is removed and these functions return 1 in succes case, and 0 if fail.
2014-01-24 04:58:12 -05:00
if (strl2llrc(text, ptr - text, &pattern->val.range.min) != 0) {
memprintf(err, "'%s' is not a number", text);
MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and "pat_parse_dotted_ver()" The goal of these patch is to simplify the prototype of "pat_pattern_*()" functions. I want to replace the argument "char **args" by a simple "char *arg" and remove the "opaque" argument. "pat_parse_int()" and "pat_parse_dotted_ver()" are the unique pattern parser using the "opaque" argument and using more than one string argument of the char **args. These specificities are only used with ACL. Other systems using this pattern parser (MAP and CLI) just use one string for describing a range. This two functions can read a range, but the min and the max must y specified. This patch extends the syntax to describe a range with implicit min and max. This is used for operators like "lt", "le", "gt", and "ge". the syntax is the following: ":x" -> no min to "x" "x:" -> "x" to no max This patch moves the parsing of the comparison operator from the functions "pat_parse_int()" and "pat_parse_dotted_ver()" to the acl parser. The acl parser read the operator and the values and build a volatile string readable by the functions "pat_parse_int()" and "pat_parse_dotted_ver()". The transformation is done with these rules: If the parser is "pat_parse_int()": "eq x" -> "x" "le x" -> ":x" "lt x" -> ":y" (with y = x - 1) "ge x" -> "x:" "gt x" -> "y:" (with y = x + 1) If the parser is "pat_parse_dotted_ver()": "eq x.y" -> "x.y" "le x.y" -> ":x.y" "lt x.y" -> ":w.z" (with w.z = x.y - 1) "ge x.y" -> "x.y:" "gt x.y" -> "w.z:" (with w.z = x.y + 1) Note that, if "y" is not present, assume that is "0". Now "pat_parse_int()" and "pat_parse_dotted_ver()" accept only one pattern and the variable "opaque" is no longer used. The prototype of the pattern parsers can be changed.
2014-01-23 11:40:34 -05:00
return 0;
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
}
MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and "pat_parse_dotted_ver()" The goal of these patch is to simplify the prototype of "pat_pattern_*()" functions. I want to replace the argument "char **args" by a simple "char *arg" and remove the "opaque" argument. "pat_parse_int()" and "pat_parse_dotted_ver()" are the unique pattern parser using the "opaque" argument and using more than one string argument of the char **args. These specificities are only used with ACL. Other systems using this pattern parser (MAP and CLI) just use one string for describing a range. This two functions can read a range, but the min and the max must y specified. This patch extends the syntax to describe a range with implicit min and max. This is used for operators like "lt", "le", "gt", and "ge". the syntax is the following: ":x" -> no min to "x" "x:" -> "x" to no max This patch moves the parsing of the comparison operator from the functions "pat_parse_int()" and "pat_parse_dotted_ver()" to the acl parser. The acl parser read the operator and the values and build a volatile string readable by the functions "pat_parse_int()" and "pat_parse_dotted_ver()". The transformation is done with these rules: If the parser is "pat_parse_int()": "eq x" -> "x" "le x" -> ":x" "lt x" -> ":y" (with y = x - 1) "ge x" -> "x:" "gt x" -> "y:" (with y = x + 1) If the parser is "pat_parse_dotted_ver()": "eq x.y" -> "x.y" "le x.y" -> ":x.y" "lt x.y" -> ":w.z" (with w.z = x.y - 1) "ge x.y" -> "x.y:" "gt x.y" -> "w.z:" (with w.z = x.y + 1) Note that, if "y" is not present, assume that is "0". Now "pat_parse_int()" and "pat_parse_dotted_ver()" accept only one pattern and the variable "opaque" is no longer used. The prototype of the pattern parsers can be changed.
2014-01-23 11:40:34 -05:00
pattern->val.range.max = pattern->val.range.min;
pattern->val.range.min_set = 1;
pattern->val.range.max_set = 1;
return 1;
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
}
MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and "pat_parse_dotted_ver()" The goal of these patch is to simplify the prototype of "pat_pattern_*()" functions. I want to replace the argument "char **args" by a simple "char *arg" and remove the "opaque" argument. "pat_parse_int()" and "pat_parse_dotted_ver()" are the unique pattern parser using the "opaque" argument and using more than one string argument of the char **args. These specificities are only used with ACL. Other systems using this pattern parser (MAP and CLI) just use one string for describing a range. This two functions can read a range, but the min and the max must y specified. This patch extends the syntax to describe a range with implicit min and max. This is used for operators like "lt", "le", "gt", and "ge". the syntax is the following: ":x" -> no min to "x" "x:" -> "x" to no max This patch moves the parsing of the comparison operator from the functions "pat_parse_int()" and "pat_parse_dotted_ver()" to the acl parser. The acl parser read the operator and the values and build a volatile string readable by the functions "pat_parse_int()" and "pat_parse_dotted_ver()". The transformation is done with these rules: If the parser is "pat_parse_int()": "eq x" -> "x" "le x" -> ":x" "lt x" -> ":y" (with y = x - 1) "ge x" -> "x:" "gt x" -> "y:" (with y = x + 1) If the parser is "pat_parse_dotted_ver()": "eq x.y" -> "x.y" "le x.y" -> ":x.y" "lt x.y" -> ":w.z" (with w.z = x.y - 1) "ge x.y" -> "x.y:" "gt x.y" -> "w.z:" (with w.z = x.y + 1) Note that, if "y" is not present, assume that is "0". Now "pat_parse_int()" and "pat_parse_dotted_ver()" accept only one pattern and the variable "opaque" is no longer used. The prototype of the pattern parsers can be changed.
2014-01-23 11:40:34 -05:00
/* If the separator is the first character. */
MEDIUM: pattern: The pattern parser no more uses <opaque> and just takes one string. After the previous patches, the "pat_parse_strcat()" function disappear, and the "pat_parse_int()" and "pat_parse_dotted_ver()" functions dont use anymore the "opaque" argument, and take only one string on his input. So, after this patch, each pattern parser no longer use the opaque variable and take only one string as input. This patch change the prototype of the pattern parsing functions. Now, the "char **args" is replaced by a "char *arg", the "int *opaque" is removed and these functions return 1 in succes case, and 0 if fail.
2014-01-24 04:58:12 -05:00
if (ptr == text && *(ptr + 1) != '\0') {
MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and "pat_parse_dotted_ver()" The goal of these patch is to simplify the prototype of "pat_pattern_*()" functions. I want to replace the argument "char **args" by a simple "char *arg" and remove the "opaque" argument. "pat_parse_int()" and "pat_parse_dotted_ver()" are the unique pattern parser using the "opaque" argument and using more than one string argument of the char **args. These specificities are only used with ACL. Other systems using this pattern parser (MAP and CLI) just use one string for describing a range. This two functions can read a range, but the min and the max must y specified. This patch extends the syntax to describe a range with implicit min and max. This is used for operators like "lt", "le", "gt", and "ge". the syntax is the following: ":x" -> no min to "x" "x:" -> "x" to no max This patch moves the parsing of the comparison operator from the functions "pat_parse_int()" and "pat_parse_dotted_ver()" to the acl parser. The acl parser read the operator and the values and build a volatile string readable by the functions "pat_parse_int()" and "pat_parse_dotted_ver()". The transformation is done with these rules: If the parser is "pat_parse_int()": "eq x" -> "x" "le x" -> ":x" "lt x" -> ":y" (with y = x - 1) "ge x" -> "x:" "gt x" -> "y:" (with y = x + 1) If the parser is "pat_parse_dotted_ver()": "eq x.y" -> "x.y" "le x.y" -> ":x.y" "lt x.y" -> ":w.z" (with w.z = x.y - 1) "ge x.y" -> "x.y:" "gt x.y" -> "w.z:" (with w.z = x.y + 1) Note that, if "y" is not present, assume that is "0". Now "pat_parse_int()" and "pat_parse_dotted_ver()" accept only one pattern and the variable "opaque" is no longer used. The prototype of the pattern parsers can be changed.
2014-01-23 11:40:34 -05:00
if (strl2llrc(ptr + 1, strlen(ptr + 1), &pattern->val.range.max) != 0)
goto not_valid_range;
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and "pat_parse_dotted_ver()" The goal of these patch is to simplify the prototype of "pat_pattern_*()" functions. I want to replace the argument "char **args" by a simple "char *arg" and remove the "opaque" argument. "pat_parse_int()" and "pat_parse_dotted_ver()" are the unique pattern parser using the "opaque" argument and using more than one string argument of the char **args. These specificities are only used with ACL. Other systems using this pattern parser (MAP and CLI) just use one string for describing a range. This two functions can read a range, but the min and the max must y specified. This patch extends the syntax to describe a range with implicit min and max. This is used for operators like "lt", "le", "gt", and "ge". the syntax is the following: ":x" -> no min to "x" "x:" -> "x" to no max This patch moves the parsing of the comparison operator from the functions "pat_parse_int()" and "pat_parse_dotted_ver()" to the acl parser. The acl parser read the operator and the values and build a volatile string readable by the functions "pat_parse_int()" and "pat_parse_dotted_ver()". The transformation is done with these rules: If the parser is "pat_parse_int()": "eq x" -> "x" "le x" -> ":x" "lt x" -> ":y" (with y = x - 1) "ge x" -> "x:" "gt x" -> "y:" (with y = x + 1) If the parser is "pat_parse_dotted_ver()": "eq x.y" -> "x.y" "le x.y" -> ":x.y" "lt x.y" -> ":w.z" (with w.z = x.y - 1) "ge x.y" -> "x.y:" "gt x.y" -> "w.z:" (with w.z = x.y + 1) Note that, if "y" is not present, assume that is "0". Now "pat_parse_int()" and "pat_parse_dotted_ver()" accept only one pattern and the variable "opaque" is no longer used. The prototype of the pattern parsers can be changed.
2014-01-23 11:40:34 -05:00
pattern->val.range.min_set = 0;
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
pattern->val.range.max_set = 1;
MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and "pat_parse_dotted_ver()" The goal of these patch is to simplify the prototype of "pat_pattern_*()" functions. I want to replace the argument "char **args" by a simple "char *arg" and remove the "opaque" argument. "pat_parse_int()" and "pat_parse_dotted_ver()" are the unique pattern parser using the "opaque" argument and using more than one string argument of the char **args. These specificities are only used with ACL. Other systems using this pattern parser (MAP and CLI) just use one string for describing a range. This two functions can read a range, but the min and the max must y specified. This patch extends the syntax to describe a range with implicit min and max. This is used for operators like "lt", "le", "gt", and "ge". the syntax is the following: ":x" -> no min to "x" "x:" -> "x" to no max This patch moves the parsing of the comparison operator from the functions "pat_parse_int()" and "pat_parse_dotted_ver()" to the acl parser. The acl parser read the operator and the values and build a volatile string readable by the functions "pat_parse_int()" and "pat_parse_dotted_ver()". The transformation is done with these rules: If the parser is "pat_parse_int()": "eq x" -> "x" "le x" -> ":x" "lt x" -> ":y" (with y = x - 1) "ge x" -> "x:" "gt x" -> "y:" (with y = x + 1) If the parser is "pat_parse_dotted_ver()": "eq x.y" -> "x.y" "le x.y" -> ":x.y" "lt x.y" -> ":w.z" (with w.z = x.y - 1) "ge x.y" -> "x.y:" "gt x.y" -> "w.z:" (with w.z = x.y + 1) Note that, if "y" is not present, assume that is "0". Now "pat_parse_int()" and "pat_parse_dotted_ver()" accept only one pattern and the variable "opaque" is no longer used. The prototype of the pattern parsers can be changed.
2014-01-23 11:40:34 -05:00
return 1;
}
/* If separator is the last character. */
if (*(ptr + 1) == '\0') {
MEDIUM: pattern: The pattern parser no more uses <opaque> and just takes one string. After the previous patches, the "pat_parse_strcat()" function disappear, and the "pat_parse_int()" and "pat_parse_dotted_ver()" functions dont use anymore the "opaque" argument, and take only one string on his input. So, after this patch, each pattern parser no longer use the opaque variable and take only one string as input. This patch change the prototype of the pattern parsing functions. Now, the "char **args" is replaced by a "char *arg", the "int *opaque" is removed and these functions return 1 in succes case, and 0 if fail.
2014-01-24 04:58:12 -05:00
if (strl2llrc(text, ptr - text, &pattern->val.range.min) != 0)
MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and "pat_parse_dotted_ver()" The goal of these patch is to simplify the prototype of "pat_pattern_*()" functions. I want to replace the argument "char **args" by a simple "char *arg" and remove the "opaque" argument. "pat_parse_int()" and "pat_parse_dotted_ver()" are the unique pattern parser using the "opaque" argument and using more than one string argument of the char **args. These specificities are only used with ACL. Other systems using this pattern parser (MAP and CLI) just use one string for describing a range. This two functions can read a range, but the min and the max must y specified. This patch extends the syntax to describe a range with implicit min and max. This is used for operators like "lt", "le", "gt", and "ge". the syntax is the following: ":x" -> no min to "x" "x:" -> "x" to no max This patch moves the parsing of the comparison operator from the functions "pat_parse_int()" and "pat_parse_dotted_ver()" to the acl parser. The acl parser read the operator and the values and build a volatile string readable by the functions "pat_parse_int()" and "pat_parse_dotted_ver()". The transformation is done with these rules: If the parser is "pat_parse_int()": "eq x" -> "x" "le x" -> ":x" "lt x" -> ":y" (with y = x - 1) "ge x" -> "x:" "gt x" -> "y:" (with y = x + 1) If the parser is "pat_parse_dotted_ver()": "eq x.y" -> "x.y" "le x.y" -> ":x.y" "lt x.y" -> ":w.z" (with w.z = x.y - 1) "ge x.y" -> "x.y:" "gt x.y" -> "w.z:" (with w.z = x.y + 1) Note that, if "y" is not present, assume that is "0". Now "pat_parse_int()" and "pat_parse_dotted_ver()" accept only one pattern and the variable "opaque" is no longer used. The prototype of the pattern parsers can be changed.
2014-01-23 11:40:34 -05:00
goto not_valid_range;
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
pattern->val.range.min_set = 1;
pattern->val.range.max_set = 0;
MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and "pat_parse_dotted_ver()" The goal of these patch is to simplify the prototype of "pat_pattern_*()" functions. I want to replace the argument "char **args" by a simple "char *arg" and remove the "opaque" argument. "pat_parse_int()" and "pat_parse_dotted_ver()" are the unique pattern parser using the "opaque" argument and using more than one string argument of the char **args. These specificities are only used with ACL. Other systems using this pattern parser (MAP and CLI) just use one string for describing a range. This two functions can read a range, but the min and the max must y specified. This patch extends the syntax to describe a range with implicit min and max. This is used for operators like "lt", "le", "gt", and "ge". the syntax is the following: ":x" -> no min to "x" "x:" -> "x" to no max This patch moves the parsing of the comparison operator from the functions "pat_parse_int()" and "pat_parse_dotted_ver()" to the acl parser. The acl parser read the operator and the values and build a volatile string readable by the functions "pat_parse_int()" and "pat_parse_dotted_ver()". The transformation is done with these rules: If the parser is "pat_parse_int()": "eq x" -> "x" "le x" -> ":x" "lt x" -> ":y" (with y = x - 1) "ge x" -> "x:" "gt x" -> "y:" (with y = x + 1) If the parser is "pat_parse_dotted_ver()": "eq x.y" -> "x.y" "le x.y" -> ":x.y" "lt x.y" -> ":w.z" (with w.z = x.y - 1) "ge x.y" -> "x.y:" "gt x.y" -> "w.z:" (with w.z = x.y + 1) Note that, if "y" is not present, assume that is "0". Now "pat_parse_int()" and "pat_parse_dotted_ver()" accept only one pattern and the variable "opaque" is no longer used. The prototype of the pattern parsers can be changed.
2014-01-23 11:40:34 -05:00
return 1;
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
}
MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and "pat_parse_dotted_ver()" The goal of these patch is to simplify the prototype of "pat_pattern_*()" functions. I want to replace the argument "char **args" by a simple "char *arg" and remove the "opaque" argument. "pat_parse_int()" and "pat_parse_dotted_ver()" are the unique pattern parser using the "opaque" argument and using more than one string argument of the char **args. These specificities are only used with ACL. Other systems using this pattern parser (MAP and CLI) just use one string for describing a range. This two functions can read a range, but the min and the max must y specified. This patch extends the syntax to describe a range with implicit min and max. This is used for operators like "lt", "le", "gt", and "ge". the syntax is the following: ":x" -> no min to "x" "x:" -> "x" to no max This patch moves the parsing of the comparison operator from the functions "pat_parse_int()" and "pat_parse_dotted_ver()" to the acl parser. The acl parser read the operator and the values and build a volatile string readable by the functions "pat_parse_int()" and "pat_parse_dotted_ver()". The transformation is done with these rules: If the parser is "pat_parse_int()": "eq x" -> "x" "le x" -> ":x" "lt x" -> ":y" (with y = x - 1) "ge x" -> "x:" "gt x" -> "y:" (with y = x + 1) If the parser is "pat_parse_dotted_ver()": "eq x.y" -> "x.y" "le x.y" -> ":x.y" "lt x.y" -> ":w.z" (with w.z = x.y - 1) "ge x.y" -> "x.y:" "gt x.y" -> "w.z:" (with w.z = x.y + 1) Note that, if "y" is not present, assume that is "0". Now "pat_parse_int()" and "pat_parse_dotted_ver()" accept only one pattern and the variable "opaque" is no longer used. The prototype of the pattern parsers can be changed.
2014-01-23 11:40:34 -05:00
/* Else, parse two numbers. */
MEDIUM: pattern: The pattern parser no more uses <opaque> and just takes one string. After the previous patches, the "pat_parse_strcat()" function disappear, and the "pat_parse_int()" and "pat_parse_dotted_ver()" functions dont use anymore the "opaque" argument, and take only one string on his input. So, after this patch, each pattern parser no longer use the opaque variable and take only one string as input. This patch change the prototype of the pattern parsing functions. Now, the "char **args" is replaced by a "char *arg", the "int *opaque" is removed and these functions return 1 in succes case, and 0 if fail.
2014-01-24 04:58:12 -05:00
if (strl2llrc(text, ptr - text, &pattern->val.range.min) != 0)
MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and "pat_parse_dotted_ver()" The goal of these patch is to simplify the prototype of "pat_pattern_*()" functions. I want to replace the argument "char **args" by a simple "char *arg" and remove the "opaque" argument. "pat_parse_int()" and "pat_parse_dotted_ver()" are the unique pattern parser using the "opaque" argument and using more than one string argument of the char **args. These specificities are only used with ACL. Other systems using this pattern parser (MAP and CLI) just use one string for describing a range. This two functions can read a range, but the min and the max must y specified. This patch extends the syntax to describe a range with implicit min and max. This is used for operators like "lt", "le", "gt", and "ge". the syntax is the following: ":x" -> no min to "x" "x:" -> "x" to no max This patch moves the parsing of the comparison operator from the functions "pat_parse_int()" and "pat_parse_dotted_ver()" to the acl parser. The acl parser read the operator and the values and build a volatile string readable by the functions "pat_parse_int()" and "pat_parse_dotted_ver()". The transformation is done with these rules: If the parser is "pat_parse_int()": "eq x" -> "x" "le x" -> ":x" "lt x" -> ":y" (with y = x - 1) "ge x" -> "x:" "gt x" -> "y:" (with y = x + 1) If the parser is "pat_parse_dotted_ver()": "eq x.y" -> "x.y" "le x.y" -> ":x.y" "lt x.y" -> ":w.z" (with w.z = x.y - 1) "ge x.y" -> "x.y:" "gt x.y" -> "w.z:" (with w.z = x.y + 1) Note that, if "y" is not present, assume that is "0". Now "pat_parse_int()" and "pat_parse_dotted_ver()" accept only one pattern and the variable "opaque" is no longer used. The prototype of the pattern parsers can be changed.
2014-01-23 11:40:34 -05:00
goto not_valid_range;
if (strl2llrc(ptr + 1, strlen(ptr + 1), &pattern->val.range.max) != 0)
goto not_valid_range;
if (pattern->val.range.min > pattern->val.range.max)
goto not_valid_range;
pattern->val.range.min_set = 1;
pattern->val.range.max_set = 1;
return 1;
not_valid_range:
MEDIUM: pattern: The pattern parser no more uses <opaque> and just takes one string. After the previous patches, the "pat_parse_strcat()" function disappear, and the "pat_parse_int()" and "pat_parse_dotted_ver()" functions dont use anymore the "opaque" argument, and take only one string on his input. So, after this patch, each pattern parser no longer use the opaque variable and take only one string as input. This patch change the prototype of the pattern parsing functions. Now, the "char **args" is replaced by a "char *arg", the "int *opaque" is removed and these functions return 1 in succes case, and 0 if fail.
2014-01-24 04:58:12 -05:00
memprintf(err, "'%s' is not a valid number range", text);
MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and "pat_parse_dotted_ver()" The goal of these patch is to simplify the prototype of "pat_pattern_*()" functions. I want to replace the argument "char **args" by a simple "char *arg" and remove the "opaque" argument. "pat_parse_int()" and "pat_parse_dotted_ver()" are the unique pattern parser using the "opaque" argument and using more than one string argument of the char **args. These specificities are only used with ACL. Other systems using this pattern parser (MAP and CLI) just use one string for describing a range. This two functions can read a range, but the min and the max must y specified. This patch extends the syntax to describe a range with implicit min and max. This is used for operators like "lt", "le", "gt", and "ge". the syntax is the following: ":x" -> no min to "x" "x:" -> "x" to no max This patch moves the parsing of the comparison operator from the functions "pat_parse_int()" and "pat_parse_dotted_ver()" to the acl parser. The acl parser read the operator and the values and build a volatile string readable by the functions "pat_parse_int()" and "pat_parse_dotted_ver()". The transformation is done with these rules: If the parser is "pat_parse_int()": "eq x" -> "x" "le x" -> ":x" "lt x" -> ":y" (with y = x - 1) "ge x" -> "x:" "gt x" -> "y:" (with y = x + 1) If the parser is "pat_parse_dotted_ver()": "eq x.y" -> "x.y" "le x.y" -> ":x.y" "lt x.y" -> ":w.z" (with w.z = x.y - 1) "ge x.y" -> "x.y:" "gt x.y" -> "w.z:" (with w.z = x.y + 1) Note that, if "y" is not present, assume that is "0". Now "pat_parse_int()" and "pat_parse_dotted_ver()" accept only one pattern and the variable "opaque" is no longer used. The prototype of the pattern parsers can be changed.
2014-01-23 11:40:34 -05:00
return 0;
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
}
/* Parse a range of positive 2-component versions delimited by either ':' or
* '-'. The version consists in a major and a minor, both of which must be
* smaller than 65536, because internally they will be represented as a 32-bit
* integer.
* If only one version is read, it is set as both min and max. Just like for
* pure integers, an operator may be specified as the prefix, among this list
* of 5 :
*
* 0:eq, 1:gt, 2:ge, 3:lt, 4:le
*
* The default operator is "eq". It supports range matching. Ranges are
* rejected for other operators. The operator may be changed at any time.
* The operator is stored in the 'opaque' argument. This allows constructs
* such as the following one :
*
* acl obsolete_ssl ssl_req_proto lt 3
* acl unsupported_ssl ssl_req_proto gt 3.1
* acl valid_ssl ssl_req_proto 3.0-3.1
*
*/
BUG/MEDIUM: patterns: last fix was still not enough Last fix did address the issue for inlined patterns, but it was not enough because the flags are lost as well when updating patterns dynamically over the CLI. Also if the same file was used once with -i and another time without -i, their references would have been merged and both would have used the same matching method. It's appear that the patterns have two types of flags. The first ones are relative to the pattern matching, and the second are relative to the pattern storage. The pattern matching flags are the same for all the patterns of one expression. Now they are stored in the expression. The storage flags are information returned by the pattern mathing function. This information is relative to each entry and is stored in the "struct pattern". Now, the expression matching flags are forwarded to the parse and index functions. These flags are stored during the configuration parsing, and they are used during the parse and index actions. This issue was introduced in dev23 with the major pattern rework, and is a continuation of commit a631fc8 ("BUG/MAJOR: patterns: -i and -n are ignored for inlined patterns"). No backport is needed.
2014-04-28 05:18:57 -04:00
int pat_parse_dotted_ver(const char *text, struct pattern *pattern, int mflags, char **err)
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
{
MEDIUM: pattern: The pattern parser no more uses <opaque> and just takes one string. After the previous patches, the "pat_parse_strcat()" function disappear, and the "pat_parse_int()" and "pat_parse_dotted_ver()" functions dont use anymore the "opaque" argument, and take only one string on his input. So, after this patch, each pattern parser no longer use the opaque variable and take only one string as input. This patch change the prototype of the pattern parsing functions. Now, the "char **args" is replaced by a "char *arg", the "int *opaque" is removed and these functions return 1 in succes case, and 0 if fail.
2014-01-24 04:58:12 -05:00
const char *ptr = text;
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
MAJOR: sample: converts uint and sint in 64 bits signed integer This patch removes the 32 bits unsigned integer and the 32 bit signed integer. It replaces these types by a unique type 64 bit signed. This makes easy the usage of integer and clarify signed and unsigned use. With the previous version, signed and unsigned are used ones in place of others, and sometimes the converter loose the sign. For example, divisions are processed with "unsigned", if one entry is negative, the result is wrong. Note that the integer pattern matching and dotted version pattern matching are already working with signed 64 bits integer values. There is one user-visible change : the "uint()" and "sint()" sample fetch functions which used to return a constant integer have been replaced with a new more natural, unified "int()" function. These functions were only introduced in the latest 1.6-dev2 so there's no impact on regular deployments.
2015-07-06 17:43:03 -04:00
pattern->type = SMP_T_SINT;
MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and "pat_parse_dotted_ver()" The goal of these patch is to simplify the prototype of "pat_pattern_*()" functions. I want to replace the argument "char **args" by a simple "char *arg" and remove the "opaque" argument. "pat_parse_int()" and "pat_parse_dotted_ver()" are the unique pattern parser using the "opaque" argument and using more than one string argument of the char **args. These specificities are only used with ACL. Other systems using this pattern parser (MAP and CLI) just use one string for describing a range. This two functions can read a range, but the min and the max must y specified. This patch extends the syntax to describe a range with implicit min and max. This is used for operators like "lt", "le", "gt", and "ge". the syntax is the following: ":x" -> no min to "x" "x:" -> "x" to no max This patch moves the parsing of the comparison operator from the functions "pat_parse_int()" and "pat_parse_dotted_ver()" to the acl parser. The acl parser read the operator and the values and build a volatile string readable by the functions "pat_parse_int()" and "pat_parse_dotted_ver()". The transformation is done with these rules: If the parser is "pat_parse_int()": "eq x" -> "x" "le x" -> ":x" "lt x" -> ":y" (with y = x - 1) "ge x" -> "x:" "gt x" -> "y:" (with y = x + 1) If the parser is "pat_parse_dotted_ver()": "eq x.y" -> "x.y" "le x.y" -> ":x.y" "lt x.y" -> ":w.z" (with w.z = x.y - 1) "ge x.y" -> "x.y:" "gt x.y" -> "w.z:" (with w.z = x.y + 1) Note that, if "y" is not present, assume that is "0". Now "pat_parse_int()" and "pat_parse_dotted_ver()" accept only one pattern and the variable "opaque" is no longer used. The prototype of the pattern parsers can be changed.
2014-01-23 11:40:34 -05:00
/* Search ':' or '-' separator. */
while (*ptr != '\0' && *ptr != ':' && *ptr != '-')
ptr++;
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and "pat_parse_dotted_ver()" The goal of these patch is to simplify the prototype of "pat_pattern_*()" functions. I want to replace the argument "char **args" by a simple "char *arg" and remove the "opaque" argument. "pat_parse_int()" and "pat_parse_dotted_ver()" are the unique pattern parser using the "opaque" argument and using more than one string argument of the char **args. These specificities are only used with ACL. Other systems using this pattern parser (MAP and CLI) just use one string for describing a range. This two functions can read a range, but the min and the max must y specified. This patch extends the syntax to describe a range with implicit min and max. This is used for operators like "lt", "le", "gt", and "ge". the syntax is the following: ":x" -> no min to "x" "x:" -> "x" to no max This patch moves the parsing of the comparison operator from the functions "pat_parse_int()" and "pat_parse_dotted_ver()" to the acl parser. The acl parser read the operator and the values and build a volatile string readable by the functions "pat_parse_int()" and "pat_parse_dotted_ver()". The transformation is done with these rules: If the parser is "pat_parse_int()": "eq x" -> "x" "le x" -> ":x" "lt x" -> ":y" (with y = x - 1) "ge x" -> "x:" "gt x" -> "y:" (with y = x + 1) If the parser is "pat_parse_dotted_ver()": "eq x.y" -> "x.y" "le x.y" -> ":x.y" "lt x.y" -> ":w.z" (with w.z = x.y - 1) "ge x.y" -> "x.y:" "gt x.y" -> "w.z:" (with w.z = x.y + 1) Note that, if "y" is not present, assume that is "0". Now "pat_parse_int()" and "pat_parse_dotted_ver()" accept only one pattern and the variable "opaque" is no longer used. The prototype of the pattern parsers can be changed.
2014-01-23 11:40:34 -05:00
/* If separator not found. */
MEDIUM: pattern: The pattern parser no more uses <opaque> and just takes one string. After the previous patches, the "pat_parse_strcat()" function disappear, and the "pat_parse_int()" and "pat_parse_dotted_ver()" functions dont use anymore the "opaque" argument, and take only one string on his input. So, after this patch, each pattern parser no longer use the opaque variable and take only one string as input. This patch change the prototype of the pattern parsing functions. Now, the "char **args" is replaced by a "char *arg", the "int *opaque" is removed and these functions return 1 in succes case, and 0 if fail.
2014-01-24 04:58:12 -05:00
if (*ptr == '\0' && ptr > text) {
if (strl2llrc_dotted(text, ptr-text, &pattern->val.range.min) != 0) {
memprintf(err, "'%s' is not a dotted number", text);
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
return 0;
}
MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and "pat_parse_dotted_ver()" The goal of these patch is to simplify the prototype of "pat_pattern_*()" functions. I want to replace the argument "char **args" by a simple "char *arg" and remove the "opaque" argument. "pat_parse_int()" and "pat_parse_dotted_ver()" are the unique pattern parser using the "opaque" argument and using more than one string argument of the char **args. These specificities are only used with ACL. Other systems using this pattern parser (MAP and CLI) just use one string for describing a range. This two functions can read a range, but the min and the max must y specified. This patch extends the syntax to describe a range with implicit min and max. This is used for operators like "lt", "le", "gt", and "ge". the syntax is the following: ":x" -> no min to "x" "x:" -> "x" to no max This patch moves the parsing of the comparison operator from the functions "pat_parse_int()" and "pat_parse_dotted_ver()" to the acl parser. The acl parser read the operator and the values and build a volatile string readable by the functions "pat_parse_int()" and "pat_parse_dotted_ver()". The transformation is done with these rules: If the parser is "pat_parse_int()": "eq x" -> "x" "le x" -> ":x" "lt x" -> ":y" (with y = x - 1) "ge x" -> "x:" "gt x" -> "y:" (with y = x + 1) If the parser is "pat_parse_dotted_ver()": "eq x.y" -> "x.y" "le x.y" -> ":x.y" "lt x.y" -> ":w.z" (with w.z = x.y - 1) "ge x.y" -> "x.y:" "gt x.y" -> "w.z:" (with w.z = x.y + 1) Note that, if "y" is not present, assume that is "0". Now "pat_parse_int()" and "pat_parse_dotted_ver()" accept only one pattern and the variable "opaque" is no longer used. The prototype of the pattern parsers can be changed.
2014-01-23 11:40:34 -05:00
pattern->val.range.max = pattern->val.range.min;
pattern->val.range.min_set = 1;
pattern->val.range.max_set = 1;
return 1;
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
}
MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and "pat_parse_dotted_ver()" The goal of these patch is to simplify the prototype of "pat_pattern_*()" functions. I want to replace the argument "char **args" by a simple "char *arg" and remove the "opaque" argument. "pat_parse_int()" and "pat_parse_dotted_ver()" are the unique pattern parser using the "opaque" argument and using more than one string argument of the char **args. These specificities are only used with ACL. Other systems using this pattern parser (MAP and CLI) just use one string for describing a range. This two functions can read a range, but the min and the max must y specified. This patch extends the syntax to describe a range with implicit min and max. This is used for operators like "lt", "le", "gt", and "ge". the syntax is the following: ":x" -> no min to "x" "x:" -> "x" to no max This patch moves the parsing of the comparison operator from the functions "pat_parse_int()" and "pat_parse_dotted_ver()" to the acl parser. The acl parser read the operator and the values and build a volatile string readable by the functions "pat_parse_int()" and "pat_parse_dotted_ver()". The transformation is done with these rules: If the parser is "pat_parse_int()": "eq x" -> "x" "le x" -> ":x" "lt x" -> ":y" (with y = x - 1) "ge x" -> "x:" "gt x" -> "y:" (with y = x + 1) If the parser is "pat_parse_dotted_ver()": "eq x.y" -> "x.y" "le x.y" -> ":x.y" "lt x.y" -> ":w.z" (with w.z = x.y - 1) "ge x.y" -> "x.y:" "gt x.y" -> "w.z:" (with w.z = x.y + 1) Note that, if "y" is not present, assume that is "0". Now "pat_parse_int()" and "pat_parse_dotted_ver()" accept only one pattern and the variable "opaque" is no longer used. The prototype of the pattern parsers can be changed.
2014-01-23 11:40:34 -05:00
/* If the separator is the first character. */
MEDIUM: pattern: The pattern parser no more uses <opaque> and just takes one string. After the previous patches, the "pat_parse_strcat()" function disappear, and the "pat_parse_int()" and "pat_parse_dotted_ver()" functions dont use anymore the "opaque" argument, and take only one string on his input. So, after this patch, each pattern parser no longer use the opaque variable and take only one string as input. This patch change the prototype of the pattern parsing functions. Now, the "char **args" is replaced by a "char *arg", the "int *opaque" is removed and these functions return 1 in succes case, and 0 if fail.
2014-01-24 04:58:12 -05:00
if (ptr == text && *(ptr+1) != '\0') {
MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and "pat_parse_dotted_ver()" The goal of these patch is to simplify the prototype of "pat_pattern_*()" functions. I want to replace the argument "char **args" by a simple "char *arg" and remove the "opaque" argument. "pat_parse_int()" and "pat_parse_dotted_ver()" are the unique pattern parser using the "opaque" argument and using more than one string argument of the char **args. These specificities are only used with ACL. Other systems using this pattern parser (MAP and CLI) just use one string for describing a range. This two functions can read a range, but the min and the max must y specified. This patch extends the syntax to describe a range with implicit min and max. This is used for operators like "lt", "le", "gt", and "ge". the syntax is the following: ":x" -> no min to "x" "x:" -> "x" to no max This patch moves the parsing of the comparison operator from the functions "pat_parse_int()" and "pat_parse_dotted_ver()" to the acl parser. The acl parser read the operator and the values and build a volatile string readable by the functions "pat_parse_int()" and "pat_parse_dotted_ver()". The transformation is done with these rules: If the parser is "pat_parse_int()": "eq x" -> "x" "le x" -> ":x" "lt x" -> ":y" (with y = x - 1) "ge x" -> "x:" "gt x" -> "y:" (with y = x + 1) If the parser is "pat_parse_dotted_ver()": "eq x.y" -> "x.y" "le x.y" -> ":x.y" "lt x.y" -> ":w.z" (with w.z = x.y - 1) "ge x.y" -> "x.y:" "gt x.y" -> "w.z:" (with w.z = x.y + 1) Note that, if "y" is not present, assume that is "0". Now "pat_parse_int()" and "pat_parse_dotted_ver()" accept only one pattern and the variable "opaque" is no longer used. The prototype of the pattern parsers can be changed.
2014-01-23 11:40:34 -05:00
if (strl2llrc_dotted(ptr+1, strlen(ptr+1), &pattern->val.range.max) != 0) {
MEDIUM: pattern: The pattern parser no more uses <opaque> and just takes one string. After the previous patches, the "pat_parse_strcat()" function disappear, and the "pat_parse_int()" and "pat_parse_dotted_ver()" functions dont use anymore the "opaque" argument, and take only one string on his input. So, after this patch, each pattern parser no longer use the opaque variable and take only one string as input. This patch change the prototype of the pattern parsing functions. Now, the "char **args" is replaced by a "char *arg", the "int *opaque" is removed and these functions return 1 in succes case, and 0 if fail.
2014-01-24 04:58:12 -05:00
memprintf(err, "'%s' is not a valid dotted number range", text);
MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and "pat_parse_dotted_ver()" The goal of these patch is to simplify the prototype of "pat_pattern_*()" functions. I want to replace the argument "char **args" by a simple "char *arg" and remove the "opaque" argument. "pat_parse_int()" and "pat_parse_dotted_ver()" are the unique pattern parser using the "opaque" argument and using more than one string argument of the char **args. These specificities are only used with ACL. Other systems using this pattern parser (MAP and CLI) just use one string for describing a range. This two functions can read a range, but the min and the max must y specified. This patch extends the syntax to describe a range with implicit min and max. This is used for operators like "lt", "le", "gt", and "ge". the syntax is the following: ":x" -> no min to "x" "x:" -> "x" to no max This patch moves the parsing of the comparison operator from the functions "pat_parse_int()" and "pat_parse_dotted_ver()" to the acl parser. The acl parser read the operator and the values and build a volatile string readable by the functions "pat_parse_int()" and "pat_parse_dotted_ver()". The transformation is done with these rules: If the parser is "pat_parse_int()": "eq x" -> "x" "le x" -> ":x" "lt x" -> ":y" (with y = x - 1) "ge x" -> "x:" "gt x" -> "y:" (with y = x + 1) If the parser is "pat_parse_dotted_ver()": "eq x.y" -> "x.y" "le x.y" -> ":x.y" "lt x.y" -> ":w.z" (with w.z = x.y - 1) "ge x.y" -> "x.y:" "gt x.y" -> "w.z:" (with w.z = x.y + 1) Note that, if "y" is not present, assume that is "0". Now "pat_parse_int()" and "pat_parse_dotted_ver()" accept only one pattern and the variable "opaque" is no longer used. The prototype of the pattern parsers can be changed.
2014-01-23 11:40:34 -05:00
return 0;
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
}
MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and "pat_parse_dotted_ver()" The goal of these patch is to simplify the prototype of "pat_pattern_*()" functions. I want to replace the argument "char **args" by a simple "char *arg" and remove the "opaque" argument. "pat_parse_int()" and "pat_parse_dotted_ver()" are the unique pattern parser using the "opaque" argument and using more than one string argument of the char **args. These specificities are only used with ACL. Other systems using this pattern parser (MAP and CLI) just use one string for describing a range. This two functions can read a range, but the min and the max must y specified. This patch extends the syntax to describe a range with implicit min and max. This is used for operators like "lt", "le", "gt", and "ge". the syntax is the following: ":x" -> no min to "x" "x:" -> "x" to no max This patch moves the parsing of the comparison operator from the functions "pat_parse_int()" and "pat_parse_dotted_ver()" to the acl parser. The acl parser read the operator and the values and build a volatile string readable by the functions "pat_parse_int()" and "pat_parse_dotted_ver()". The transformation is done with these rules: If the parser is "pat_parse_int()": "eq x" -> "x" "le x" -> ":x" "lt x" -> ":y" (with y = x - 1) "ge x" -> "x:" "gt x" -> "y:" (with y = x + 1) If the parser is "pat_parse_dotted_ver()": "eq x.y" -> "x.y" "le x.y" -> ":x.y" "lt x.y" -> ":w.z" (with w.z = x.y - 1) "ge x.y" -> "x.y:" "gt x.y" -> "w.z:" (with w.z = x.y + 1) Note that, if "y" is not present, assume that is "0". Now "pat_parse_int()" and "pat_parse_dotted_ver()" accept only one pattern and the variable "opaque" is no longer used. The prototype of the pattern parsers can be changed.
2014-01-23 11:40:34 -05:00
pattern->val.range.min_set = 0;
pattern->val.range.max_set = 1;
return 1;
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
}
MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and "pat_parse_dotted_ver()" The goal of these patch is to simplify the prototype of "pat_pattern_*()" functions. I want to replace the argument "char **args" by a simple "char *arg" and remove the "opaque" argument. "pat_parse_int()" and "pat_parse_dotted_ver()" are the unique pattern parser using the "opaque" argument and using more than one string argument of the char **args. These specificities are only used with ACL. Other systems using this pattern parser (MAP and CLI) just use one string for describing a range. This two functions can read a range, but the min and the max must y specified. This patch extends the syntax to describe a range with implicit min and max. This is used for operators like "lt", "le", "gt", and "ge". the syntax is the following: ":x" -> no min to "x" "x:" -> "x" to no max This patch moves the parsing of the comparison operator from the functions "pat_parse_int()" and "pat_parse_dotted_ver()" to the acl parser. The acl parser read the operator and the values and build a volatile string readable by the functions "pat_parse_int()" and "pat_parse_dotted_ver()". The transformation is done with these rules: If the parser is "pat_parse_int()": "eq x" -> "x" "le x" -> ":x" "lt x" -> ":y" (with y = x - 1) "ge x" -> "x:" "gt x" -> "y:" (with y = x + 1) If the parser is "pat_parse_dotted_ver()": "eq x.y" -> "x.y" "le x.y" -> ":x.y" "lt x.y" -> ":w.z" (with w.z = x.y - 1) "ge x.y" -> "x.y:" "gt x.y" -> "w.z:" (with w.z = x.y + 1) Note that, if "y" is not present, assume that is "0". Now "pat_parse_int()" and "pat_parse_dotted_ver()" accept only one pattern and the variable "opaque" is no longer used. The prototype of the pattern parsers can be changed.
2014-01-23 11:40:34 -05:00
/* If separator is the last character. */
MEDIUM: pattern: The pattern parser no more uses <opaque> and just takes one string. After the previous patches, the "pat_parse_strcat()" function disappear, and the "pat_parse_int()" and "pat_parse_dotted_ver()" functions dont use anymore the "opaque" argument, and take only one string on his input. So, after this patch, each pattern parser no longer use the opaque variable and take only one string as input. This patch change the prototype of the pattern parsing functions. Now, the "char **args" is replaced by a "char *arg", the "int *opaque" is removed and these functions return 1 in succes case, and 0 if fail.
2014-01-24 04:58:12 -05:00
if (ptr == &text[strlen(text)-1]) {
if (strl2llrc_dotted(text, ptr-text, &pattern->val.range.min) != 0) {
memprintf(err, "'%s' is not a valid dotted number range", text);
MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and "pat_parse_dotted_ver()" The goal of these patch is to simplify the prototype of "pat_pattern_*()" functions. I want to replace the argument "char **args" by a simple "char *arg" and remove the "opaque" argument. "pat_parse_int()" and "pat_parse_dotted_ver()" are the unique pattern parser using the "opaque" argument and using more than one string argument of the char **args. These specificities are only used with ACL. Other systems using this pattern parser (MAP and CLI) just use one string for describing a range. This two functions can read a range, but the min and the max must y specified. This patch extends the syntax to describe a range with implicit min and max. This is used for operators like "lt", "le", "gt", and "ge". the syntax is the following: ":x" -> no min to "x" "x:" -> "x" to no max This patch moves the parsing of the comparison operator from the functions "pat_parse_int()" and "pat_parse_dotted_ver()" to the acl parser. The acl parser read the operator and the values and build a volatile string readable by the functions "pat_parse_int()" and "pat_parse_dotted_ver()". The transformation is done with these rules: If the parser is "pat_parse_int()": "eq x" -> "x" "le x" -> ":x" "lt x" -> ":y" (with y = x - 1) "ge x" -> "x:" "gt x" -> "y:" (with y = x + 1) If the parser is "pat_parse_dotted_ver()": "eq x.y" -> "x.y" "le x.y" -> ":x.y" "lt x.y" -> ":w.z" (with w.z = x.y - 1) "ge x.y" -> "x.y:" "gt x.y" -> "w.z:" (with w.z = x.y + 1) Note that, if "y" is not present, assume that is "0". Now "pat_parse_int()" and "pat_parse_dotted_ver()" accept only one pattern and the variable "opaque" is no longer used. The prototype of the pattern parsers can be changed.
2014-01-23 11:40:34 -05:00
return 0;
}
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
pattern->val.range.min_set = 1;
pattern->val.range.max_set = 0;
MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and "pat_parse_dotted_ver()" The goal of these patch is to simplify the prototype of "pat_pattern_*()" functions. I want to replace the argument "char **args" by a simple "char *arg" and remove the "opaque" argument. "pat_parse_int()" and "pat_parse_dotted_ver()" are the unique pattern parser using the "opaque" argument and using more than one string argument of the char **args. These specificities are only used with ACL. Other systems using this pattern parser (MAP and CLI) just use one string for describing a range. This two functions can read a range, but the min and the max must y specified. This patch extends the syntax to describe a range with implicit min and max. This is used for operators like "lt", "le", "gt", and "ge". the syntax is the following: ":x" -> no min to "x" "x:" -> "x" to no max This patch moves the parsing of the comparison operator from the functions "pat_parse_int()" and "pat_parse_dotted_ver()" to the acl parser. The acl parser read the operator and the values and build a volatile string readable by the functions "pat_parse_int()" and "pat_parse_dotted_ver()". The transformation is done with these rules: If the parser is "pat_parse_int()": "eq x" -> "x" "le x" -> ":x" "lt x" -> ":y" (with y = x - 1) "ge x" -> "x:" "gt x" -> "y:" (with y = x + 1) If the parser is "pat_parse_dotted_ver()": "eq x.y" -> "x.y" "le x.y" -> ":x.y" "lt x.y" -> ":w.z" (with w.z = x.y - 1) "ge x.y" -> "x.y:" "gt x.y" -> "w.z:" (with w.z = x.y + 1) Note that, if "y" is not present, assume that is "0". Now "pat_parse_int()" and "pat_parse_dotted_ver()" accept only one pattern and the variable "opaque" is no longer used. The prototype of the pattern parsers can be changed.
2014-01-23 11:40:34 -05:00
return 1;
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
}
MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and "pat_parse_dotted_ver()" The goal of these patch is to simplify the prototype of "pat_pattern_*()" functions. I want to replace the argument "char **args" by a simple "char *arg" and remove the "opaque" argument. "pat_parse_int()" and "pat_parse_dotted_ver()" are the unique pattern parser using the "opaque" argument and using more than one string argument of the char **args. These specificities are only used with ACL. Other systems using this pattern parser (MAP and CLI) just use one string for describing a range. This two functions can read a range, but the min and the max must y specified. This patch extends the syntax to describe a range with implicit min and max. This is used for operators like "lt", "le", "gt", and "ge". the syntax is the following: ":x" -> no min to "x" "x:" -> "x" to no max This patch moves the parsing of the comparison operator from the functions "pat_parse_int()" and "pat_parse_dotted_ver()" to the acl parser. The acl parser read the operator and the values and build a volatile string readable by the functions "pat_parse_int()" and "pat_parse_dotted_ver()". The transformation is done with these rules: If the parser is "pat_parse_int()": "eq x" -> "x" "le x" -> ":x" "lt x" -> ":y" (with y = x - 1) "ge x" -> "x:" "gt x" -> "y:" (with y = x + 1) If the parser is "pat_parse_dotted_ver()": "eq x.y" -> "x.y" "le x.y" -> ":x.y" "lt x.y" -> ":w.z" (with w.z = x.y - 1) "ge x.y" -> "x.y:" "gt x.y" -> "w.z:" (with w.z = x.y + 1) Note that, if "y" is not present, assume that is "0". Now "pat_parse_int()" and "pat_parse_dotted_ver()" accept only one pattern and the variable "opaque" is no longer used. The prototype of the pattern parsers can be changed.
2014-01-23 11:40:34 -05:00
/* Else, parse two numbers. */
MEDIUM: pattern: The pattern parser no more uses <opaque> and just takes one string. After the previous patches, the "pat_parse_strcat()" function disappear, and the "pat_parse_int()" and "pat_parse_dotted_ver()" functions dont use anymore the "opaque" argument, and take only one string on his input. So, after this patch, each pattern parser no longer use the opaque variable and take only one string as input. This patch change the prototype of the pattern parsing functions. Now, the "char **args" is replaced by a "char *arg", the "int *opaque" is removed and these functions return 1 in succes case, and 0 if fail.
2014-01-24 04:58:12 -05:00
if (strl2llrc_dotted(text, ptr-text, &pattern->val.range.min) != 0) {
memprintf(err, "'%s' is not a valid dotted number range", text);
MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and "pat_parse_dotted_ver()" The goal of these patch is to simplify the prototype of "pat_pattern_*()" functions. I want to replace the argument "char **args" by a simple "char *arg" and remove the "opaque" argument. "pat_parse_int()" and "pat_parse_dotted_ver()" are the unique pattern parser using the "opaque" argument and using more than one string argument of the char **args. These specificities are only used with ACL. Other systems using this pattern parser (MAP and CLI) just use one string for describing a range. This two functions can read a range, but the min and the max must y specified. This patch extends the syntax to describe a range with implicit min and max. This is used for operators like "lt", "le", "gt", and "ge". the syntax is the following: ":x" -> no min to "x" "x:" -> "x" to no max This patch moves the parsing of the comparison operator from the functions "pat_parse_int()" and "pat_parse_dotted_ver()" to the acl parser. The acl parser read the operator and the values and build a volatile string readable by the functions "pat_parse_int()" and "pat_parse_dotted_ver()". The transformation is done with these rules: If the parser is "pat_parse_int()": "eq x" -> "x" "le x" -> ":x" "lt x" -> ":y" (with y = x - 1) "ge x" -> "x:" "gt x" -> "y:" (with y = x + 1) If the parser is "pat_parse_dotted_ver()": "eq x.y" -> "x.y" "le x.y" -> ":x.y" "lt x.y" -> ":w.z" (with w.z = x.y - 1) "ge x.y" -> "x.y:" "gt x.y" -> "w.z:" (with w.z = x.y + 1) Note that, if "y" is not present, assume that is "0". Now "pat_parse_int()" and "pat_parse_dotted_ver()" accept only one pattern and the variable "opaque" is no longer used. The prototype of the pattern parsers can be changed.
2014-01-23 11:40:34 -05:00
return 0;
}
if (strl2llrc_dotted(ptr+1, strlen(ptr+1), &pattern->val.range.max) != 0) {
MEDIUM: pattern: The pattern parser no more uses <opaque> and just takes one string. After the previous patches, the "pat_parse_strcat()" function disappear, and the "pat_parse_int()" and "pat_parse_dotted_ver()" functions dont use anymore the "opaque" argument, and take only one string on his input. So, after this patch, each pattern parser no longer use the opaque variable and take only one string as input. This patch change the prototype of the pattern parsing functions. Now, the "char **args" is replaced by a "char *arg", the "int *opaque" is removed and these functions return 1 in succes case, and 0 if fail.
2014-01-24 04:58:12 -05:00
memprintf(err, "'%s' is not a valid dotted number range", text);
MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and "pat_parse_dotted_ver()" The goal of these patch is to simplify the prototype of "pat_pattern_*()" functions. I want to replace the argument "char **args" by a simple "char *arg" and remove the "opaque" argument. "pat_parse_int()" and "pat_parse_dotted_ver()" are the unique pattern parser using the "opaque" argument and using more than one string argument of the char **args. These specificities are only used with ACL. Other systems using this pattern parser (MAP and CLI) just use one string for describing a range. This two functions can read a range, but the min and the max must y specified. This patch extends the syntax to describe a range with implicit min and max. This is used for operators like "lt", "le", "gt", and "ge". the syntax is the following: ":x" -> no min to "x" "x:" -> "x" to no max This patch moves the parsing of the comparison operator from the functions "pat_parse_int()" and "pat_parse_dotted_ver()" to the acl parser. The acl parser read the operator and the values and build a volatile string readable by the functions "pat_parse_int()" and "pat_parse_dotted_ver()". The transformation is done with these rules: If the parser is "pat_parse_int()": "eq x" -> "x" "le x" -> ":x" "lt x" -> ":y" (with y = x - 1) "ge x" -> "x:" "gt x" -> "y:" (with y = x + 1) If the parser is "pat_parse_dotted_ver()": "eq x.y" -> "x.y" "le x.y" -> ":x.y" "lt x.y" -> ":w.z" (with w.z = x.y - 1) "ge x.y" -> "x.y:" "gt x.y" -> "w.z:" (with w.z = x.y + 1) Note that, if "y" is not present, assume that is "0". Now "pat_parse_int()" and "pat_parse_dotted_ver()" accept only one pattern and the variable "opaque" is no longer used. The prototype of the pattern parsers can be changed.
2014-01-23 11:40:34 -05:00
return 0;
}
if (pattern->val.range.min > pattern->val.range.max) {
MEDIUM: pattern: The pattern parser no more uses <opaque> and just takes one string. After the previous patches, the "pat_parse_strcat()" function disappear, and the "pat_parse_int()" and "pat_parse_dotted_ver()" functions dont use anymore the "opaque" argument, and take only one string on his input. So, after this patch, each pattern parser no longer use the opaque variable and take only one string as input. This patch change the prototype of the pattern parsing functions. Now, the "char **args" is replaced by a "char *arg", the "int *opaque" is removed and these functions return 1 in succes case, and 0 if fail.
2014-01-24 04:58:12 -05:00
memprintf(err, "'%s' is not a valid dotted number range", text);
MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and "pat_parse_dotted_ver()" The goal of these patch is to simplify the prototype of "pat_pattern_*()" functions. I want to replace the argument "char **args" by a simple "char *arg" and remove the "opaque" argument. "pat_parse_int()" and "pat_parse_dotted_ver()" are the unique pattern parser using the "opaque" argument and using more than one string argument of the char **args. These specificities are only used with ACL. Other systems using this pattern parser (MAP and CLI) just use one string for describing a range. This two functions can read a range, but the min and the max must y specified. This patch extends the syntax to describe a range with implicit min and max. This is used for operators like "lt", "le", "gt", and "ge". the syntax is the following: ":x" -> no min to "x" "x:" -> "x" to no max This patch moves the parsing of the comparison operator from the functions "pat_parse_int()" and "pat_parse_dotted_ver()" to the acl parser. The acl parser read the operator and the values and build a volatile string readable by the functions "pat_parse_int()" and "pat_parse_dotted_ver()". The transformation is done with these rules: If the parser is "pat_parse_int()": "eq x" -> "x" "le x" -> ":x" "lt x" -> ":y" (with y = x - 1) "ge x" -> "x:" "gt x" -> "y:" (with y = x + 1) If the parser is "pat_parse_dotted_ver()": "eq x.y" -> "x.y" "le x.y" -> ":x.y" "lt x.y" -> ":w.z" (with w.z = x.y - 1) "ge x.y" -> "x.y:" "gt x.y" -> "w.z:" (with w.z = x.y + 1) Note that, if "y" is not present, assume that is "0". Now "pat_parse_int()" and "pat_parse_dotted_ver()" accept only one pattern and the variable "opaque" is no longer used. The prototype of the pattern parsers can be changed.
2014-01-23 11:40:34 -05:00
return 0;
}
pattern->val.range.min_set = 1;
pattern->val.range.max_set = 1;
return 1;
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
}
/* Parse an IP address and an optional mask in the form addr[/mask].
* The addr may either be an IPv4 address or a hostname. The mask
* may either be a dotted mask or a number of bits. Returns 1 if OK,
* otherwise 0. NOTE: IP address patterns are typed (IPV4/IPV6).
*/
BUG/MEDIUM: patterns: last fix was still not enough Last fix did address the issue for inlined patterns, but it was not enough because the flags are lost as well when updating patterns dynamically over the CLI. Also if the same file was used once with -i and another time without -i, their references would have been merged and both would have used the same matching method. It's appear that the patterns have two types of flags. The first ones are relative to the pattern matching, and the second are relative to the pattern storage. The pattern matching flags are the same for all the patterns of one expression. Now they are stored in the expression. The storage flags are information returned by the pattern mathing function. This information is relative to each entry and is stored in the "struct pattern". Now, the expression matching flags are forwarded to the parse and index functions. These flags are stored during the configuration parsing, and they are used during the parse and index actions. This issue was introduced in dev23 with the major pattern rework, and is a continuation of commit a631fc8 ("BUG/MAJOR: patterns: -i and -n are ignored for inlined patterns"). No backport is needed.
2014-04-28 05:18:57 -04:00
int pat_parse_ip(const char *text, struct pattern *pattern, int mflags, char **err)
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
{
BUG/MEDIUM: patterns: last fix was still not enough Last fix did address the issue for inlined patterns, but it was not enough because the flags are lost as well when updating patterns dynamically over the CLI. Also if the same file was used once with -i and another time without -i, their references would have been merged and both would have used the same matching method. It's appear that the patterns have two types of flags. The first ones are relative to the pattern matching, and the second are relative to the pattern storage. The pattern matching flags are the same for all the patterns of one expression. Now they are stored in the expression. The storage flags are information returned by the pattern mathing function. This information is relative to each entry and is stored in the "struct pattern". Now, the expression matching flags are forwarded to the parse and index functions. These flags are stored during the configuration parsing, and they are used during the parse and index actions. This issue was introduced in dev23 with the major pattern rework, and is a continuation of commit a631fc8 ("BUG/MAJOR: patterns: -i and -n are ignored for inlined patterns"). No backport is needed.
2014-04-28 05:18:57 -04:00
if (str2net(text, !(mflags & PAT_MF_NO_DNS) && (global.mode & MODE_STARTING),
MINOR: standard: Disable ip resolution during the runtime The function str2net runs DNS resolution if valid ip cannot be parsed. The DNS function used is the standard function of the libc and it performs asynchronous request. The asynchronous request is not compatible with the haproxy archictecture. str2net() is used during the runtime throught the "socket". This patch remove the DNS resolution during the runtime.
2014-02-11 09:23:04 -05:00
&pattern->val.ipv4.addr, &pattern->val.ipv4.mask)) {
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
pattern->type = SMP_T_IPV4;
return 1;
}
MEDIUM: pattern: The pattern parser no more uses <opaque> and just takes one string. After the previous patches, the "pat_parse_strcat()" function disappear, and the "pat_parse_int()" and "pat_parse_dotted_ver()" functions dont use anymore the "opaque" argument, and take only one string on his input. So, after this patch, each pattern parser no longer use the opaque variable and take only one string as input. This patch change the prototype of the pattern parsing functions. Now, the "char **args" is replaced by a "char *arg", the "int *opaque" is removed and these functions return 1 in succes case, and 0 if fail.
2014-01-24 04:58:12 -05:00
else if (str62net(text, &pattern->val.ipv6.addr, &pattern->val.ipv6.mask)) {
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
pattern->type = SMP_T_IPV6;
return 1;
}
else {
MEDIUM: pattern: The pattern parser no more uses <opaque> and just takes one string. After the previous patches, the "pat_parse_strcat()" function disappear, and the "pat_parse_int()" and "pat_parse_dotted_ver()" functions dont use anymore the "opaque" argument, and take only one string on his input. So, after this patch, each pattern parser no longer use the opaque variable and take only one string as input. This patch change the prototype of the pattern parsing functions. Now, the "char **args" is replaced by a "char *arg", the "int *opaque" is removed and these functions return 1 in succes case, and 0 if fail.
2014-01-24 04:58:12 -05:00
memprintf(err, "'%s' is not a valid IPv4 or IPv6 address", text);
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
return 0;
}
}
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
/*
*
* These functions are exported and may be used by any other component.
*
CLEANUP: Fix typos in the pattern subsystem Fixes typos in the code comments of the pattern subsystem.
2018-11-15 13:22:31 -05:00
* This function just takes a sample <smp> and checks if this sample matches
* with the pattern <pattern>. This function returns only PAT_MATCH or
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
* PAT_NOMATCH.
*
*/
/* always return false */
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
struct pattern *pat_match_nothing(struct sample *smp, struct pattern_expr *expr, int fill)
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
{
MINOR: samples: rename union from "data" to "u" The union name "data" is a little bit heavy while we read the source code because we can read "data.data.sint". The rename from "data" to "u" makes the read easiest like "data.u.sint".
2015-08-19 03:07:19 -04:00
if (smp->data.u.sint) {
BUG/MEDIUM: acl: boolean only matches were broken by recent changes The ACL changes made in the last patchset force the execution of each pattern matching function. The function pat_match_nothing was not provided to be excuted, it was just used as a flag that was checked by the ACL execution code. Now this function is executed and always returns false. This patch makes it work as expected. Now, it returns the boolean status of the received sample just as was done previously in the ACL code. This bug is a part of the patchset just merged. It does not need to be backported.
2014-03-17 14:53:10 -04:00
if (fill) {
MINOR: samples: rename some struct member from "smp" to "data" This members contains data and not sample.
2015-08-19 02:35:43 -04:00
static_pattern.data = NULL;
BUG/MEDIUM: acl: boolean only matches were broken by recent changes The ACL changes made in the last patchset force the execution of each pattern matching function. The function pat_match_nothing was not provided to be excuted, it was just used as a flag that was checked by the ACL execution code. Now this function is executed and always returns false. This patch makes it work as expected. Now, it returns the boolean status of the received sample just as was done previously in the ACL code. This bug is a part of the patchset just merged. It does not need to be backported.
2014-03-17 14:53:10 -04:00
static_pattern.ref = NULL;
static_pattern.type = 0;
static_pattern.ptr.str = NULL;
}
return &static_pattern;
}
else
return NULL;
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
}
BUG/MEDIUM: pattern: prevent uninitialized reads in pat_match_{str,beg} Using valgrind when running map_beg or map_str, the following error is reported: ==242644== Conditional jump or move depends on uninitialised value(s) ==242644== at 0x2E4AB1: pat_match_str (pattern.c:457) ==242644== by 0x2E81ED: pattern_exec_match (pattern.c:2560) ==242644== by 0x343176: sample_conv_map (map.c:211) ==242644== by 0x27522F: sample_process_cnv (sample.c:1330) ==242644== by 0x2752DB: sample_process (sample.c:1373) ==242644== by 0x319917: action_store (vars.c:814) ==242644== by 0x24D451: http_req_get_intercept_rule (http_ana.c:2697) In fact, the error is legit, because in pat_match_{beg,str}, we dereference the buffer on len+1 to check if a value was previously set, and then decide to force NULL-byte if it wasn't set. But the approach is no longer compatible with current architecture: data past str.data is not guaranteed to be initialized in the buffer. Thus we cannot dereference the value, else we expose us to uninitialized read errors. Moreover, the check is useless, because we systematically set the ending byte to 0 when the conditions are met. Finally, restoring the older value after the lookup is not relevant: indeed, either the sample is marked as const and in such case it is already duplicated, or the sample is not const and we forcefully add a terminating NULL byte outside from the actual string bytes (since we're past str.data), so as we didn't alter effective string data and that data past str.data cannot be dereferenced anyway as it isn't guaranteed to be initialized, there's no point in restoring previous uninitialized data. It could be backported in all stable versions. But since this was only detected by valgrind and isn't known to cause issues in existing deployments, it's probably better to wait a bit before backporting it to avoid any breakage.. although the fix should be theoretically harmless.
2024-09-06 10:33:15 -04:00
/* ensure the input sample can be read as a string without knowing its size,
* that is, ensure the terminating null byte is there
*
* The function may fail. Returns 1 on success and 0 on failure
*/
static inline int pat_match_ensure_str(struct sample *smp)
{
if (smp->data.u.str.data < smp->data.u.str.size) {
/* we have to force a trailing zero on the test pattern and
* the buffer is large enough to accommodate it. If the flag
* CONST is set, duplicate the string
*/
if (smp->flags & SMP_F_CONST) {
if (!smp_dup(smp))
return 0;
} else
smp->data.u.str.area[smp->data.u.str.data] = '\0';
}
else {
/* Otherwise, the sample is duplicated. A trailing zero
* is automatically added to the string.
*/
if (!smp_dup(smp))
return 0;
}
return 1;
}
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
CLEANUP: Fix typos in the pattern subsystem Fixes typos in the code comments of the pattern subsystem.
2018-11-15 13:22:31 -05:00
/* NB: For two strings to be identical, it is required that their length match */
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
struct pattern *pat_match_str(struct sample *smp, struct pattern_expr *expr, int fill)
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
{
int icase;
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
struct ebmb_node *node;
struct pattern_tree *elt;
struct pattern_list *lst;
struct pattern *pattern;
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
struct pattern *ret = NULL;
struct lru64 *lru = NULL;
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
/* Lookup a string in the expression's pattern tree. */
if (!eb_is_empty(&expr->pattern_tree)) {
BUG/MEDIUM: pattern: prevent uninitialized reads in pat_match_{str,beg} Using valgrind when running map_beg or map_str, the following error is reported: ==242644== Conditional jump or move depends on uninitialised value(s) ==242644== at 0x2E4AB1: pat_match_str (pattern.c:457) ==242644== by 0x2E81ED: pattern_exec_match (pattern.c:2560) ==242644== by 0x343176: sample_conv_map (map.c:211) ==242644== by 0x27522F: sample_process_cnv (sample.c:1330) ==242644== by 0x2752DB: sample_process (sample.c:1373) ==242644== by 0x319917: action_store (vars.c:814) ==242644== by 0x24D451: http_req_get_intercept_rule (http_ana.c:2697) In fact, the error is legit, because in pat_match_{beg,str}, we dereference the buffer on len+1 to check if a value was previously set, and then decide to force NULL-byte if it wasn't set. But the approach is no longer compatible with current architecture: data past str.data is not guaranteed to be initialized in the buffer. Thus we cannot dereference the value, else we expose us to uninitialized read errors. Moreover, the check is useless, because we systematically set the ending byte to 0 when the conditions are met. Finally, restoring the older value after the lookup is not relevant: indeed, either the sample is marked as const and in such case it is already duplicated, or the sample is not const and we forcefully add a terminating NULL byte outside from the actual string bytes (since we're past str.data), so as we didn't alter effective string data and that data past str.data cannot be dereferenced anyway as it isn't guaranteed to be initialized, there's no point in restoring previous uninitialized data. It could be backported in all stable versions. But since this was only detected by valgrind and isn't known to cause issues in existing deployments, it's probably better to wait a bit before backporting it to avoid any breakage.. although the fix should be theoretically harmless.
2024-09-06 10:33:15 -04:00
if (!pat_match_ensure_str(smp))
return NULL;
BUG/MEDIUM: pattern: Add a trailing \0 to match strings only if possible In pat_match_str() and pat_math_beg() functions, a trailing zero is systematically added at the end of the string, even if the buffer is not large enough to accommodate it. It is a possible buffer overflow. For instance, when the alpn is matched against a list of strings, the sample fetch is filled with a non-null terminated string returned by the SSL library. No trailing zero must be added at the end of this string, because it is outside the buffer. So, to fix the bug, a trailing zero is added only if the buffer is large enough to accommodate it. Otherwise, the sample fetch is duplicated. smp_dup() function adds a trailing zero to the duplicated string, truncating it if it is too long. This patch should fix the issue #718. It must be backported to all supported versions.
2020-06-30 12:52:32 -04:00
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
node = ebst_lookup(&expr->pattern_tree, smp->data.u.str.area);
MEDIUM: pattern: only match patterns that match the current generation Instead of matching any pattern found in the tree, only match those matching the current generation of entries. This will make sure that reloads are atomic, regardless of the time they take to complete, and that newly added data are not matched until the whole reference is committed. For consistency we proceed the same way on "show map" and "show acl". This will have no impact for now since generations are not used.
2020-10-29 04:41:34 -04:00
while (node) {
elt = ebmb_entry(node, struct pattern_tree, node);
if (elt->ref->gen_id != expr->ref->curr_gen) {
BUG/MEDIUM: pattern: only visit equivalent nodes when skipping versions Miroslav reported in issue #1802 a problem that affects atomic map/acl updates. During an update, incorrect versions are properly skipped, but in order to do so, we rely on ebmb_next() instead of ebmb_next_dup(). This means that if a new matching entry is in the process of being added and is the first one to succeed in the lookup, we'll skip it due to its version and use the next entry regardless of its value provided that it has the correct version. For IP addresses and string prefixes it's particularly visible because a lookup may match a new longer prefix that's not yet committed (e.g. 11.0.0.1 would match 11/8 when 10/7 was the only committed one), and skipping it could end up on 12/8 for example. As soon as a commit for the last version happens, the issue disappears. This problem only affects tree-based matches: the "str", "ip", and "beg" matches. Here we replace the ebmb_next() values with ebmb_next_dup() for exact string matches, and with ebmb_lookup_shorter() for longest matches, which will first visit duplicates, then look for shorter prefixes. This relies on previous commit: MINOR: ebtree: add ebmb_lookup_shorter() to pursue lookups Both need to be backported to 2.4, where the generation ID was added. Note that nowadays a simpler and more efficient approach might be employed, by having a single version in the current tree, and a list of trees per version. Manipulations would look up the tree version and work (and lock) only in the relevant trees, while normal operations would be performed on the current tree only. Committing would just be a matter of swapping tree roots and deleting old trees contents.
2022-08-01 05:46:27 -04:00
node = ebmb_next_dup(node);
MEDIUM: pattern: only match patterns that match the current generation Instead of matching any pattern found in the tree, only match those matching the current generation of entries. This will make sure that reloads are atomic, regardless of the time they take to complete, and that newly added data are not matched until the whole reference is committed. For consistency we proceed the same way on "show map" and "show acl". This will have no impact for now since generations are not used.
2020-10-29 04:41:34 -04:00
continue;
}
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
if (fill) {
MINOR: samples: rename some struct member from "smp" to "data" This members contains data and not sample.
2015-08-19 02:35:43 -04:00
static_pattern.data = elt->data;
MINOR: pattern: Each pattern expression element store the reference struct. Now, each pattern entry known the original "struct pat_ref_elt" from that was built. This patch permit to delete each pattern entry without confusion. After this patch, each reference can use his pointer to be targeted.
2014-01-28 09:54:36 -05:00
static_pattern.ref = elt->ref;
BUG/MEDIUM: patterns: last fix was still not enough Last fix did address the issue for inlined patterns, but it was not enough because the flags are lost as well when updating patterns dynamically over the CLI. Also if the same file was used once with -i and another time without -i, their references would have been merged and both would have used the same matching method. It's appear that the patterns have two types of flags. The first ones are relative to the pattern matching, and the second are relative to the pattern storage. The pattern matching flags are the same for all the patterns of one expression. Now they are stored in the expression. The storage flags are information returned by the pattern mathing function. This information is relative to each entry and is stored in the "struct pattern". Now, the expression matching flags are forwarded to the parse and index functions. These flags are stored during the configuration parsing, and they are used during the parse and index actions. This issue was introduced in dev23 with the major pattern rework, and is a continuation of commit a631fc8 ("BUG/MAJOR: patterns: -i and -n are ignored for inlined patterns"). No backport is needed.
2014-04-28 05:18:57 -04:00
static_pattern.sflags = PAT_SF_TREE;
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
static_pattern.type = SMP_T_STR;
static_pattern.ptr.str = (char *)elt->node.key;
}
return &static_pattern;
}
}
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
/* look in the list */
OPTIM: pattern: only apply LRU cache for large enough lists As shown in issue #1518, the LRU cache has a non-null cost that can sometimes be above the match cost it's trying to avoid. After a number of tests, it appears that: - "simple" match operations (sub, beg, end, int etc) reach a break-even after ~20 patterns in list - "heavy" match operations (reg) reach a break-even after ~5 patterns in list Let's only consult the LRU cache when the number of patterns in the expression is at least as large as this limit. Of course there will always be outliers but it already starts good. Another improvement consists in reducing the cache size to further speed up lookups, which makes sense if less expressions use the cache.
2024-11-03 12:42:26 -05:00
if (pat_lru_tree && !LIST_ISEMPTY(&expr->patterns) && expr->ref->entry_cnt >= 20) {
BUILD: pattern: fix build warnings introduced in the LRU cache They're caused by the cast to long long from ptr in 32-bit. src/pattern.c: In function 'pat_match_str': src/pattern.c:479:44: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
2015-05-04 11:18:42 -04:00
unsigned long long seed = pat_lru_seed ^ (long)expr;
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
MEDIUM: xxhash: use the XXH3 functions to generate 64-bit hashes Replace the XXH64() function calls with the XXH3 variant function XXH3_64bits_withSeed() where possible.
2020-12-22 07:22:34 -05:00
lru = lru64_get(XXH3(smp->data.u.str.area, smp->data.u.str.data, seed),
MINOR: pattern: move the update revision to the pat_ref, not the expression It's not possible to uniquely update a single expression without updating the pattern reference, I don't know why we've put the revision in the expression back then, given that it in fact provides an update for a full pattern. Let's move the revision into the reference's head instead.
2020-11-02 09:26:51 -05:00
pat_lru_tree, expr, expr->ref->revision);
BUG/MEDIUM: pattern: make the pattern LRU cache thread-local and lockless As reported in issue #335, a lot of contention happens on the PATLRU lock when performing expensive regex lookups. This is absurd since the purpose of the LRU cache was to have a fast cache for expressions, thus the cache must not be shared between threads and must remain lockless. This commit makes the LRU cache thread-local and gets rid of the PATLRU lock. A test with 7 threads on 4 cores climbed from 67kH/s to 369kH/s, or a scalability factor of 5.5. Given the huge performance difference and the regression caused to users migrating from processes to threads, this should be backported at least to 2.0. Thanks to Brian Diekelman for his detailed report about this regression.
2019-10-23 00:59:31 -04:00
if (lru && lru->domain) {
MAJOR: threads/map: Make acls/maps thread safe locks have been added in pat_ref and pattern_expr structures to protect all accesses to an instance of on of them. Moreover, a global lock has been added to protect the LRU cache used for pattern matching. Patterns are now duplicated after a successfull matching, to avoid modification by other threads when the result is used. Finally, the function reloading a pattern list has been modified to be thread-safe.
2017-07-03 05:34:05 -04:00
ret = lru->data;
return ret;
}
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
}
MAJOR: threads/map: Make acls/maps thread safe locks have been added in pat_ref and pattern_expr structures to protect all accesses to an instance of on of them. Moreover, a global lock has been added to protect the LRU cache used for pattern matching. Patterns are now duplicated after a successfull matching, to avoid modification by other threads when the result is used. Finally, the function reloading a pattern list has been modified to be thread-safe.
2017-07-03 05:34:05 -04:00
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
list_for_each_entry(lst, &expr->patterns, list) {
pattern = &lst->pat;
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
MEDIUM: pattern: only match patterns that match the current generation Instead of matching any pattern found in the tree, only match those matching the current generation of entries. This will make sure that reloads are atomic, regardless of the time they take to complete, and that newly added data are not matched until the whole reference is committed. For consistency we proceed the same way on "show map" and "show acl". This will have no impact for now since generations are not used.
2020-10-29 04:41:34 -04:00
if (pattern->ref->gen_id != expr->ref->curr_gen)
continue;
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
if (pattern->len != smp->data.u.str.data)
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
continue;
BUG/MEDIUM: patterns: last fix was still not enough Last fix did address the issue for inlined patterns, but it was not enough because the flags are lost as well when updating patterns dynamically over the CLI. Also if the same file was used once with -i and another time without -i, their references would have been merged and both would have used the same matching method. It's appear that the patterns have two types of flags. The first ones are relative to the pattern matching, and the second are relative to the pattern storage. The pattern matching flags are the same for all the patterns of one expression. Now they are stored in the expression. The storage flags are information returned by the pattern mathing function. This information is relative to each entry and is stored in the "struct pattern". Now, the expression matching flags are forwarded to the parse and index functions. These flags are stored during the configuration parsing, and they are used during the parse and index actions. This issue was introduced in dev23 with the major pattern rework, and is a continuation of commit a631fc8 ("BUG/MAJOR: patterns: -i and -n are ignored for inlined patterns"). No backport is needed.
2014-04-28 05:18:57 -04:00
icase = expr->mflags & PAT_MF_IGNORE_CASE;
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
if ((icase && strncasecmp(pattern->ptr.str, smp->data.u.str.area, smp->data.u.str.data) == 0) ||
(!icase && strncmp(pattern->ptr.str, smp->data.u.str.area, smp->data.u.str.data) == 0)) {
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
ret = pattern;
break;
}
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
}
BUG/MEDIUM: pattern: make the pattern LRU cache thread-local and lockless As reported in issue #335, a lot of contention happens on the PATLRU lock when performing expensive regex lookups. This is absurd since the purpose of the LRU cache was to have a fast cache for expressions, thus the cache must not be shared between threads and must remain lockless. This commit makes the LRU cache thread-local and gets rid of the PATLRU lock. A test with 7 threads on 4 cores climbed from 67kH/s to 369kH/s, or a scalability factor of 5.5. Given the huge performance difference and the regression caused to users migrating from processes to threads, this should be backported at least to 2.0. Thanks to Brian Diekelman for his detailed report about this regression.
2019-10-23 00:59:31 -04:00
if (lru)
MINOR: pattern: move the update revision to the pat_ref, not the expression It's not possible to uniquely update a single expression without updating the pattern reference, I don't know why we've put the revision in the expression back then, given that it in fact provides an update for a full pattern. Let's move the revision into the reference's head instead.
2020-11-02 09:26:51 -05:00
lru64_commit(lru, ret, expr, expr->ref->revision, NULL);
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
return ret;
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
}
/* NB: For two binaries buf to be identical, it is required that their lengths match */
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
struct pattern *pat_match_bin(struct sample *smp, struct pattern_expr *expr, int fill)
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
{
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
struct pattern_list *lst;
struct pattern *pattern;
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
struct pattern *ret = NULL;
struct lru64 *lru = NULL;
OPTIM: pattern: only apply LRU cache for large enough lists As shown in issue #1518, the LRU cache has a non-null cost that can sometimes be above the match cost it's trying to avoid. After a number of tests, it appears that: - "simple" match operations (sub, beg, end, int etc) reach a break-even after ~20 patterns in list - "heavy" match operations (reg) reach a break-even after ~5 patterns in list Let's only consult the LRU cache when the number of patterns in the expression is at least as large as this limit. Of course there will always be outliers but it already starts good. Another improvement consists in reducing the cache size to further speed up lookups, which makes sense if less expressions use the cache.
2024-11-03 12:42:26 -05:00
if (pat_lru_tree && !LIST_ISEMPTY(&expr->patterns) && expr->ref->entry_cnt >= 20) {
BUILD: pattern: fix build warnings introduced in the LRU cache They're caused by the cast to long long from ptr in 32-bit. src/pattern.c: In function 'pat_match_str': src/pattern.c:479:44: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
2015-05-04 11:18:42 -04:00
unsigned long long seed = pat_lru_seed ^ (long)expr;
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
MEDIUM: xxhash: use the XXH3 functions to generate 64-bit hashes Replace the XXH64() function calls with the XXH3 variant function XXH3_64bits_withSeed() where possible.
2020-12-22 07:22:34 -05:00
lru = lru64_get(XXH3(smp->data.u.str.area, smp->data.u.str.data, seed),
MINOR: pattern: move the update revision to the pat_ref, not the expression It's not possible to uniquely update a single expression without updating the pattern reference, I don't know why we've put the revision in the expression back then, given that it in fact provides an update for a full pattern. Let's move the revision into the reference's head instead.
2020-11-02 09:26:51 -05:00
pat_lru_tree, expr, expr->ref->revision);
BUG/MEDIUM: pattern: make the pattern LRU cache thread-local and lockless As reported in issue #335, a lot of contention happens on the PATLRU lock when performing expensive regex lookups. This is absurd since the purpose of the LRU cache was to have a fast cache for expressions, thus the cache must not be shared between threads and must remain lockless. This commit makes the LRU cache thread-local and gets rid of the PATLRU lock. A test with 7 threads on 4 cores climbed from 67kH/s to 369kH/s, or a scalability factor of 5.5. Given the huge performance difference and the regression caused to users migrating from processes to threads, this should be backported at least to 2.0. Thanks to Brian Diekelman for his detailed report about this regression.
2019-10-23 00:59:31 -04:00
if (lru && lru->domain) {
MAJOR: threads/map: Make acls/maps thread safe locks have been added in pat_ref and pattern_expr structures to protect all accesses to an instance of on of them. Moreover, a global lock has been added to protect the LRU cache used for pattern matching. Patterns are now duplicated after a successfull matching, to avoid modification by other threads when the result is used. Finally, the function reloading a pattern list has been modified to be thread-safe.
2017-07-03 05:34:05 -04:00
ret = lru->data;
return ret;
}
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
}
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
list_for_each_entry(lst, &expr->patterns, list) {
pattern = &lst->pat;
MEDIUM: pattern: only match patterns that match the current generation Instead of matching any pattern found in the tree, only match those matching the current generation of entries. This will make sure that reloads are atomic, regardless of the time they take to complete, and that newly added data are not matched until the whole reference is committed. For consistency we proceed the same way on "show map" and "show acl". This will have no impact for now since generations are not used.
2020-10-29 04:41:34 -04:00
if (pattern->ref->gen_id != expr->ref->curr_gen)
continue;
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
if (pattern->len != smp->data.u.str.data)
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
continue;
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
if (memcmp(pattern->ptr.str, smp->data.u.str.area, smp->data.u.str.data) == 0) {
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
ret = pattern;
break;
}
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
}
BUG/MEDIUM: pattern: make the pattern LRU cache thread-local and lockless As reported in issue #335, a lot of contention happens on the PATLRU lock when performing expensive regex lookups. This is absurd since the purpose of the LRU cache was to have a fast cache for expressions, thus the cache must not be shared between threads and must remain lockless. This commit makes the LRU cache thread-local and gets rid of the PATLRU lock. A test with 7 threads on 4 cores climbed from 67kH/s to 369kH/s, or a scalability factor of 5.5. Given the huge performance difference and the regression caused to users migrating from processes to threads, this should be backported at least to 2.0. Thanks to Brian Diekelman for his detailed report about this regression.
2019-10-23 00:59:31 -04:00
if (lru)
MINOR: pattern: move the update revision to the pat_ref, not the expression It's not possible to uniquely update a single expression without updating the pattern reference, I don't know why we've put the revision in the expression back then, given that it in fact provides an update for a full pattern. Let's move the revision into the reference's head instead.
2020-11-02 09:26:51 -05:00
lru64_commit(lru, ret, expr, expr->ref->revision, NULL);
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
return ret;
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
}
MINOR: map: Add regex matching replacement This patch declares a new map which provides a string based on a string with back references replaced by the content matched by the regex.
2016-02-10 16:55:20 -05:00
/* Executes a regex. It temporarily changes the data to add a trailing zero,
* and restores the previous character when leaving. This function fills
* a matching array.
*/
struct pattern *pat_match_regm(struct sample *smp, struct pattern_expr *expr, int fill)
{
struct pattern_list *lst;
struct pattern *pattern;
struct pattern *ret = NULL;
list_for_each_entry(lst, &expr->patterns, list) {
pattern = &lst->pat;
MEDIUM: pattern: only match patterns that match the current generation Instead of matching any pattern found in the tree, only match those matching the current generation of entries. This will make sure that reloads are atomic, regardless of the time they take to complete, and that newly added data are not matched until the whole reference is committed. For consistency we proceed the same way on "show map" and "show acl". This will have no impact for now since generations are not used.
2020-10-29 04:41:34 -04:00
if (pattern->ref->gen_id != expr->ref->curr_gen)
continue;
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
if (regex_exec_match2(pattern->ptr.reg, smp->data.u.str.area, smp->data.u.str.data,
MINOR: map: Add regex matching replacement This patch declares a new map which provides a string based on a string with back references replaced by the content matched by the regex.
2016-02-10 16:55:20 -05:00
MAX_MATCH, pmatch, 0)) {
ret = pattern;
smp->ctx.a[0] = pmatch;
break;
}
}
return ret;
}
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
/* Executes a regex. It temporarily changes the data to add a trailing zero,
* and restores the previous character when leaving.
*/
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
struct pattern *pat_match_reg(struct sample *smp, struct pattern_expr *expr, int fill)
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
{
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
struct pattern_list *lst;
struct pattern *pattern;
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
struct pattern *ret = NULL;
struct lru64 *lru = NULL;
OPTIM: pattern: only apply LRU cache for large enough lists As shown in issue #1518, the LRU cache has a non-null cost that can sometimes be above the match cost it's trying to avoid. After a number of tests, it appears that: - "simple" match operations (sub, beg, end, int etc) reach a break-even after ~20 patterns in list - "heavy" match operations (reg) reach a break-even after ~5 patterns in list Let's only consult the LRU cache when the number of patterns in the expression is at least as large as this limit. Of course there will always be outliers but it already starts good. Another improvement consists in reducing the cache size to further speed up lookups, which makes sense if less expressions use the cache.
2024-11-03 12:42:26 -05:00
if (pat_lru_tree && !LIST_ISEMPTY(&expr->patterns) && expr->ref->entry_cnt >= 5) {
BUILD: pattern: fix build warnings introduced in the LRU cache They're caused by the cast to long long from ptr in 32-bit. src/pattern.c: In function 'pat_match_str': src/pattern.c:479:44: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
2015-05-04 11:18:42 -04:00
unsigned long long seed = pat_lru_seed ^ (long)expr;
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
MEDIUM: xxhash: use the XXH3 functions to generate 64-bit hashes Replace the XXH64() function calls with the XXH3 variant function XXH3_64bits_withSeed() where possible.
2020-12-22 07:22:34 -05:00
lru = lru64_get(XXH3(smp->data.u.str.area, smp->data.u.str.data, seed),
MINOR: pattern: move the update revision to the pat_ref, not the expression It's not possible to uniquely update a single expression without updating the pattern reference, I don't know why we've put the revision in the expression back then, given that it in fact provides an update for a full pattern. Let's move the revision into the reference's head instead.
2020-11-02 09:26:51 -05:00
pat_lru_tree, expr, expr->ref->revision);
BUG/MEDIUM: pattern: make the pattern LRU cache thread-local and lockless As reported in issue #335, a lot of contention happens on the PATLRU lock when performing expensive regex lookups. This is absurd since the purpose of the LRU cache was to have a fast cache for expressions, thus the cache must not be shared between threads and must remain lockless. This commit makes the LRU cache thread-local and gets rid of the PATLRU lock. A test with 7 threads on 4 cores climbed from 67kH/s to 369kH/s, or a scalability factor of 5.5. Given the huge performance difference and the regression caused to users migrating from processes to threads, this should be backported at least to 2.0. Thanks to Brian Diekelman for his detailed report about this regression.
2019-10-23 00:59:31 -04:00
if (lru && lru->domain) {
MAJOR: threads/map: Make acls/maps thread safe locks have been added in pat_ref and pattern_expr structures to protect all accesses to an instance of on of them. Moreover, a global lock has been added to protect the LRU cache used for pattern matching. Patterns are now duplicated after a successfull matching, to avoid modification by other threads when the result is used. Finally, the function reloading a pattern list has been modified to be thread-safe.
2017-07-03 05:34:05 -04:00
ret = lru->data;
return ret;
}
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
}
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
list_for_each_entry(lst, &expr->patterns, list) {
pattern = &lst->pat;
MEDIUM: pattern: only match patterns that match the current generation Instead of matching any pattern found in the tree, only match those matching the current generation of entries. This will make sure that reloads are atomic, regardless of the time they take to complete, and that newly added data are not matched until the whole reference is committed. For consistency we proceed the same way on "show map" and "show acl". This will have no impact for now since generations are not used.
2020-10-29 04:41:34 -04:00
if (pattern->ref->gen_id != expr->ref->curr_gen)
continue;
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
if (regex_exec2(pattern->ptr.reg, smp->data.u.str.area, smp->data.u.str.data)) {
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
ret = pattern;
break;
}
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
}
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
BUG/MEDIUM: pattern: make the pattern LRU cache thread-local and lockless As reported in issue #335, a lot of contention happens on the PATLRU lock when performing expensive regex lookups. This is absurd since the purpose of the LRU cache was to have a fast cache for expressions, thus the cache must not be shared between threads and must remain lockless. This commit makes the LRU cache thread-local and gets rid of the PATLRU lock. A test with 7 threads on 4 cores climbed from 67kH/s to 369kH/s, or a scalability factor of 5.5. Given the huge performance difference and the regression caused to users migrating from processes to threads, this should be backported at least to 2.0. Thanks to Brian Diekelman for his detailed report about this regression.
2019-10-23 00:59:31 -04:00
if (lru)
MINOR: pattern: move the update revision to the pat_ref, not the expression It's not possible to uniquely update a single expression without updating the pattern reference, I don't know why we've put the revision in the expression back then, given that it in fact provides an update for a full pattern. Let's move the revision into the reference's head instead.
2020-11-02 09:26:51 -05:00
lru64_commit(lru, ret, expr, expr->ref->revision, NULL);
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
return ret;
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
}
/* Checks that the pattern matches the beginning of the tested string. */
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
struct pattern *pat_match_beg(struct sample *smp, struct pattern_expr *expr, int fill)
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
{
int icase;
MEDIUM: pattern: use ebtree's longest match to index/lookup string beginning Being able to map prefixes to values is already used for IPv4/IPv6 but was not yet used with strings. It can be very convenient to map directories to server farms but large lists may be slow. By using ebmb_insert_prefix() and ebmb_lookup_longest(), we can insert strings with their own length as a prefix, and lookup candidate strings and ensure that the longest matching one will be returned, which is the longest string matching the entry.
2014-05-10 02:53:48 -04:00
struct ebmb_node *node;
struct pattern_tree *elt;
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
struct pattern_list *lst;
struct pattern *pattern;
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
struct pattern *ret = NULL;
struct lru64 *lru = NULL;
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
MEDIUM: pattern: use ebtree's longest match to index/lookup string beginning Being able to map prefixes to values is already used for IPv4/IPv6 but was not yet used with strings. It can be very convenient to map directories to server farms but large lists may be slow. By using ebmb_insert_prefix() and ebmb_lookup_longest(), we can insert strings with their own length as a prefix, and lookup candidate strings and ensure that the longest matching one will be returned, which is the longest string matching the entry.
2014-05-10 02:53:48 -04:00
/* Lookup a string in the expression's pattern tree. */
if (!eb_is_empty(&expr->pattern_tree)) {
BUG/MEDIUM: pattern: prevent uninitialized reads in pat_match_{str,beg} Using valgrind when running map_beg or map_str, the following error is reported: ==242644== Conditional jump or move depends on uninitialised value(s) ==242644== at 0x2E4AB1: pat_match_str (pattern.c:457) ==242644== by 0x2E81ED: pattern_exec_match (pattern.c:2560) ==242644== by 0x343176: sample_conv_map (map.c:211) ==242644== by 0x27522F: sample_process_cnv (sample.c:1330) ==242644== by 0x2752DB: sample_process (sample.c:1373) ==242644== by 0x319917: action_store (vars.c:814) ==242644== by 0x24D451: http_req_get_intercept_rule (http_ana.c:2697) In fact, the error is legit, because in pat_match_{beg,str}, we dereference the buffer on len+1 to check if a value was previously set, and then decide to force NULL-byte if it wasn't set. But the approach is no longer compatible with current architecture: data past str.data is not guaranteed to be initialized in the buffer. Thus we cannot dereference the value, else we expose us to uninitialized read errors. Moreover, the check is useless, because we systematically set the ending byte to 0 when the conditions are met. Finally, restoring the older value after the lookup is not relevant: indeed, either the sample is marked as const and in such case it is already duplicated, or the sample is not const and we forcefully add a terminating NULL byte outside from the actual string bytes (since we're past str.data), so as we didn't alter effective string data and that data past str.data cannot be dereferenced anyway as it isn't guaranteed to be initialized, there's no point in restoring previous uninitialized data. It could be backported in all stable versions. But since this was only detected by valgrind and isn't known to cause issues in existing deployments, it's probably better to wait a bit before backporting it to avoid any breakage.. although the fix should be theoretically harmless.
2024-09-06 10:33:15 -04:00
if (!pat_match_ensure_str(smp))
return NULL;
BUG/MEDIUM: pattern: Add a trailing \0 to match strings only if possible In pat_match_str() and pat_math_beg() functions, a trailing zero is systematically added at the end of the string, even if the buffer is not large enough to accommodate it. It is a possible buffer overflow. For instance, when the alpn is matched against a list of strings, the sample fetch is filled with a non-null terminated string returned by the SSL library. No trailing zero must be added at the end of this string, because it is outside the buffer. So, to fix the bug, a trailing zero is added only if the buffer is large enough to accommodate it. Otherwise, the sample fetch is duplicated. smp_dup() function adds a trailing zero to the duplicated string, truncating it if it is too long. This patch should fix the issue #718. It must be backported to all supported versions.
2020-06-30 12:52:32 -04:00
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
node = ebmb_lookup_longest(&expr->pattern_tree,
smp->data.u.str.area);
MEDIUM: pattern: use ebtree's longest match to index/lookup string beginning Being able to map prefixes to values is already used for IPv4/IPv6 but was not yet used with strings. It can be very convenient to map directories to server farms but large lists may be slow. By using ebmb_insert_prefix() and ebmb_lookup_longest(), we can insert strings with their own length as a prefix, and lookup candidate strings and ensure that the longest matching one will be returned, which is the longest string matching the entry.
2014-05-10 02:53:48 -04:00
MEDIUM: pattern: only match patterns that match the current generation Instead of matching any pattern found in the tree, only match those matching the current generation of entries. This will make sure that reloads are atomic, regardless of the time they take to complete, and that newly added data are not matched until the whole reference is committed. For consistency we proceed the same way on "show map" and "show acl". This will have no impact for now since generations are not used.
2020-10-29 04:41:34 -04:00
while (node) {
elt = ebmb_entry(node, struct pattern_tree, node);
if (elt->ref->gen_id != expr->ref->curr_gen) {
BUG/MEDIUM: pattern: only visit equivalent nodes when skipping versions Miroslav reported in issue #1802 a problem that affects atomic map/acl updates. During an update, incorrect versions are properly skipped, but in order to do so, we rely on ebmb_next() instead of ebmb_next_dup(). This means that if a new matching entry is in the process of being added and is the first one to succeed in the lookup, we'll skip it due to its version and use the next entry regardless of its value provided that it has the correct version. For IP addresses and string prefixes it's particularly visible because a lookup may match a new longer prefix that's not yet committed (e.g. 11.0.0.1 would match 11/8 when 10/7 was the only committed one), and skipping it could end up on 12/8 for example. As soon as a commit for the last version happens, the issue disappears. This problem only affects tree-based matches: the "str", "ip", and "beg" matches. Here we replace the ebmb_next() values with ebmb_next_dup() for exact string matches, and with ebmb_lookup_shorter() for longest matches, which will first visit duplicates, then look for shorter prefixes. This relies on previous commit: MINOR: ebtree: add ebmb_lookup_shorter() to pursue lookups Both need to be backported to 2.4, where the generation ID was added. Note that nowadays a simpler and more efficient approach might be employed, by having a single version in the current tree, and a list of trees per version. Manipulations would look up the tree version and work (and lock) only in the relevant trees, while normal operations would be performed on the current tree only. Committing would just be a matter of swapping tree roots and deleting old trees contents.
2022-08-01 05:46:27 -04:00
node = ebmb_lookup_shorter(node);
MEDIUM: pattern: only match patterns that match the current generation Instead of matching any pattern found in the tree, only match those matching the current generation of entries. This will make sure that reloads are atomic, regardless of the time they take to complete, and that newly added data are not matched until the whole reference is committed. For consistency we proceed the same way on "show map" and "show acl". This will have no impact for now since generations are not used.
2020-10-29 04:41:34 -04:00
continue;
}
MEDIUM: pattern: use ebtree's longest match to index/lookup string beginning Being able to map prefixes to values is already used for IPv4/IPv6 but was not yet used with strings. It can be very convenient to map directories to server farms but large lists may be slow. By using ebmb_insert_prefix() and ebmb_lookup_longest(), we can insert strings with their own length as a prefix, and lookup candidate strings and ensure that the longest matching one will be returned, which is the longest string matching the entry.
2014-05-10 02:53:48 -04:00
if (fill) {
MINOR: samples: rename some struct member from "smp" to "data" This members contains data and not sample.
2015-08-19 02:35:43 -04:00
static_pattern.data = elt->data;
MEDIUM: pattern: use ebtree's longest match to index/lookup string beginning Being able to map prefixes to values is already used for IPv4/IPv6 but was not yet used with strings. It can be very convenient to map directories to server farms but large lists may be slow. By using ebmb_insert_prefix() and ebmb_lookup_longest(), we can insert strings with their own length as a prefix, and lookup candidate strings and ensure that the longest matching one will be returned, which is the longest string matching the entry.
2014-05-10 02:53:48 -04:00
static_pattern.ref = elt->ref;
static_pattern.sflags = PAT_SF_TREE;
static_pattern.type = SMP_T_STR;
static_pattern.ptr.str = (char *)elt->node.key;
}
return &static_pattern;
}
}
/* look in the list */
OPTIM: pattern: only apply LRU cache for large enough lists As shown in issue #1518, the LRU cache has a non-null cost that can sometimes be above the match cost it's trying to avoid. After a number of tests, it appears that: - "simple" match operations (sub, beg, end, int etc) reach a break-even after ~20 patterns in list - "heavy" match operations (reg) reach a break-even after ~5 patterns in list Let's only consult the LRU cache when the number of patterns in the expression is at least as large as this limit. Of course there will always be outliers but it already starts good. Another improvement consists in reducing the cache size to further speed up lookups, which makes sense if less expressions use the cache.
2024-11-03 12:42:26 -05:00
if (pat_lru_tree && !LIST_ISEMPTY(&expr->patterns) && expr->ref->entry_cnt >= 20) {
BUILD: pattern: fix build warnings introduced in the LRU cache They're caused by the cast to long long from ptr in 32-bit. src/pattern.c: In function 'pat_match_str': src/pattern.c:479:44: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
2015-05-04 11:18:42 -04:00
unsigned long long seed = pat_lru_seed ^ (long)expr;
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
MEDIUM: xxhash: use the XXH3 functions to generate 64-bit hashes Replace the XXH64() function calls with the XXH3 variant function XXH3_64bits_withSeed() where possible.
2020-12-22 07:22:34 -05:00
lru = lru64_get(XXH3(smp->data.u.str.area, smp->data.u.str.data, seed),
MINOR: pattern: move the update revision to the pat_ref, not the expression It's not possible to uniquely update a single expression without updating the pattern reference, I don't know why we've put the revision in the expression back then, given that it in fact provides an update for a full pattern. Let's move the revision into the reference's head instead.
2020-11-02 09:26:51 -05:00
pat_lru_tree, expr, expr->ref->revision);
BUG/MEDIUM: pattern: make the pattern LRU cache thread-local and lockless As reported in issue #335, a lot of contention happens on the PATLRU lock when performing expensive regex lookups. This is absurd since the purpose of the LRU cache was to have a fast cache for expressions, thus the cache must not be shared between threads and must remain lockless. This commit makes the LRU cache thread-local and gets rid of the PATLRU lock. A test with 7 threads on 4 cores climbed from 67kH/s to 369kH/s, or a scalability factor of 5.5. Given the huge performance difference and the regression caused to users migrating from processes to threads, this should be backported at least to 2.0. Thanks to Brian Diekelman for his detailed report about this regression.
2019-10-23 00:59:31 -04:00
if (lru && lru->domain) {
MAJOR: threads/map: Make acls/maps thread safe locks have been added in pat_ref and pattern_expr structures to protect all accesses to an instance of on of them. Moreover, a global lock has been added to protect the LRU cache used for pattern matching. Patterns are now duplicated after a successfull matching, to avoid modification by other threads when the result is used. Finally, the function reloading a pattern list has been modified to be thread-safe.
2017-07-03 05:34:05 -04:00
ret = lru->data;
return ret;
}
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
}
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
list_for_each_entry(lst, &expr->patterns, list) {
pattern = &lst->pat;
MEDIUM: pattern: only match patterns that match the current generation Instead of matching any pattern found in the tree, only match those matching the current generation of entries. This will make sure that reloads are atomic, regardless of the time they take to complete, and that newly added data are not matched until the whole reference is committed. For consistency we proceed the same way on "show map" and "show acl". This will have no impact for now since generations are not used.
2020-10-29 04:41:34 -04:00
if (pattern->ref->gen_id != expr->ref->curr_gen)
continue;
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
if (pattern->len > smp->data.u.str.data)
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
continue;
BUG/MEDIUM: patterns: last fix was still not enough Last fix did address the issue for inlined patterns, but it was not enough because the flags are lost as well when updating patterns dynamically over the CLI. Also if the same file was used once with -i and another time without -i, their references would have been merged and both would have used the same matching method. It's appear that the patterns have two types of flags. The first ones are relative to the pattern matching, and the second are relative to the pattern storage. The pattern matching flags are the same for all the patterns of one expression. Now they are stored in the expression. The storage flags are information returned by the pattern mathing function. This information is relative to each entry and is stored in the "struct pattern". Now, the expression matching flags are forwarded to the parse and index functions. These flags are stored during the configuration parsing, and they are used during the parse and index actions. This issue was introduced in dev23 with the major pattern rework, and is a continuation of commit a631fc8 ("BUG/MAJOR: patterns: -i and -n are ignored for inlined patterns"). No backport is needed.
2014-04-28 05:18:57 -04:00
icase = expr->mflags & PAT_MF_IGNORE_CASE;
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
if ((icase && strncasecmp(pattern->ptr.str, smp->data.u.str.area, pattern->len) != 0) ||
(!icase && strncmp(pattern->ptr.str, smp->data.u.str.area, pattern->len) != 0))
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
continue;
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
ret = pattern;
break;
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
}
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
BUG/MEDIUM: pattern: make the pattern LRU cache thread-local and lockless As reported in issue #335, a lot of contention happens on the PATLRU lock when performing expensive regex lookups. This is absurd since the purpose of the LRU cache was to have a fast cache for expressions, thus the cache must not be shared between threads and must remain lockless. This commit makes the LRU cache thread-local and gets rid of the PATLRU lock. A test with 7 threads on 4 cores climbed from 67kH/s to 369kH/s, or a scalability factor of 5.5. Given the huge performance difference and the regression caused to users migrating from processes to threads, this should be backported at least to 2.0. Thanks to Brian Diekelman for his detailed report about this regression.
2019-10-23 00:59:31 -04:00
if (lru)
MINOR: pattern: move the update revision to the pat_ref, not the expression It's not possible to uniquely update a single expression without updating the pattern reference, I don't know why we've put the revision in the expression back then, given that it in fact provides an update for a full pattern. Let's move the revision into the reference's head instead.
2020-11-02 09:26:51 -05:00
lru64_commit(lru, ret, expr, expr->ref->revision, NULL);
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
return ret;
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
}
/* Checks that the pattern matches the end of the tested string. */
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
struct pattern *pat_match_end(struct sample *smp, struct pattern_expr *expr, int fill)
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
{
int icase;
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
struct pattern_list *lst;
struct pattern *pattern;
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
struct pattern *ret = NULL;
struct lru64 *lru = NULL;
OPTIM: pattern: only apply LRU cache for large enough lists As shown in issue #1518, the LRU cache has a non-null cost that can sometimes be above the match cost it's trying to avoid. After a number of tests, it appears that: - "simple" match operations (sub, beg, end, int etc) reach a break-even after ~20 patterns in list - "heavy" match operations (reg) reach a break-even after ~5 patterns in list Let's only consult the LRU cache when the number of patterns in the expression is at least as large as this limit. Of course there will always be outliers but it already starts good. Another improvement consists in reducing the cache size to further speed up lookups, which makes sense if less expressions use the cache.
2024-11-03 12:42:26 -05:00
if (pat_lru_tree && !LIST_ISEMPTY(&expr->patterns) && expr->ref->entry_cnt >= 20) {
BUILD: pattern: fix build warnings introduced in the LRU cache They're caused by the cast to long long from ptr in 32-bit. src/pattern.c: In function 'pat_match_str': src/pattern.c:479:44: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
2015-05-04 11:18:42 -04:00
unsigned long long seed = pat_lru_seed ^ (long)expr;
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
MEDIUM: xxhash: use the XXH3 functions to generate 64-bit hashes Replace the XXH64() function calls with the XXH3 variant function XXH3_64bits_withSeed() where possible.
2020-12-22 07:22:34 -05:00
lru = lru64_get(XXH3(smp->data.u.str.area, smp->data.u.str.data, seed),
MINOR: pattern: move the update revision to the pat_ref, not the expression It's not possible to uniquely update a single expression without updating the pattern reference, I don't know why we've put the revision in the expression back then, given that it in fact provides an update for a full pattern. Let's move the revision into the reference's head instead.
2020-11-02 09:26:51 -05:00
pat_lru_tree, expr, expr->ref->revision);
BUG/MEDIUM: pattern: make the pattern LRU cache thread-local and lockless As reported in issue #335, a lot of contention happens on the PATLRU lock when performing expensive regex lookups. This is absurd since the purpose of the LRU cache was to have a fast cache for expressions, thus the cache must not be shared between threads and must remain lockless. This commit makes the LRU cache thread-local and gets rid of the PATLRU lock. A test with 7 threads on 4 cores climbed from 67kH/s to 369kH/s, or a scalability factor of 5.5. Given the huge performance difference and the regression caused to users migrating from processes to threads, this should be backported at least to 2.0. Thanks to Brian Diekelman for his detailed report about this regression.
2019-10-23 00:59:31 -04:00
if (lru && lru->domain) {
MAJOR: threads/map: Make acls/maps thread safe locks have been added in pat_ref and pattern_expr structures to protect all accesses to an instance of on of them. Moreover, a global lock has been added to protect the LRU cache used for pattern matching. Patterns are now duplicated after a successfull matching, to avoid modification by other threads when the result is used. Finally, the function reloading a pattern list has been modified to be thread-safe.
2017-07-03 05:34:05 -04:00
ret = lru->data;
return ret;
}
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
}
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
list_for_each_entry(lst, &expr->patterns, list) {
pattern = &lst->pat;
MEDIUM: pattern: only match patterns that match the current generation Instead of matching any pattern found in the tree, only match those matching the current generation of entries. This will make sure that reloads are atomic, regardless of the time they take to complete, and that newly added data are not matched until the whole reference is committed. For consistency we proceed the same way on "show map" and "show acl". This will have no impact for now since generations are not used.
2020-10-29 04:41:34 -04:00
if (pattern->ref->gen_id != expr->ref->curr_gen)
continue;
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
if (pattern->len > smp->data.u.str.data)
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
continue;
BUG/MEDIUM: patterns: last fix was still not enough Last fix did address the issue for inlined patterns, but it was not enough because the flags are lost as well when updating patterns dynamically over the CLI. Also if the same file was used once with -i and another time without -i, their references would have been merged and both would have used the same matching method. It's appear that the patterns have two types of flags. The first ones are relative to the pattern matching, and the second are relative to the pattern storage. The pattern matching flags are the same for all the patterns of one expression. Now they are stored in the expression. The storage flags are information returned by the pattern mathing function. This information is relative to each entry and is stored in the "struct pattern". Now, the expression matching flags are forwarded to the parse and index functions. These flags are stored during the configuration parsing, and they are used during the parse and index actions. This issue was introduced in dev23 with the major pattern rework, and is a continuation of commit a631fc8 ("BUG/MAJOR: patterns: -i and -n are ignored for inlined patterns"). No backport is needed.
2014-04-28 05:18:57 -04:00
icase = expr->mflags & PAT_MF_IGNORE_CASE;
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
if ((icase && strncasecmp(pattern->ptr.str, smp->data.u.str.area + smp->data.u.str.data - pattern->len, pattern->len) != 0) ||
(!icase && strncmp(pattern->ptr.str, smp->data.u.str.area + smp->data.u.str.data - pattern->len, pattern->len) != 0))
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
continue;
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
ret = pattern;
break;
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
}
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
BUG/MEDIUM: pattern: make the pattern LRU cache thread-local and lockless As reported in issue #335, a lot of contention happens on the PATLRU lock when performing expensive regex lookups. This is absurd since the purpose of the LRU cache was to have a fast cache for expressions, thus the cache must not be shared between threads and must remain lockless. This commit makes the LRU cache thread-local and gets rid of the PATLRU lock. A test with 7 threads on 4 cores climbed from 67kH/s to 369kH/s, or a scalability factor of 5.5. Given the huge performance difference and the regression caused to users migrating from processes to threads, this should be backported at least to 2.0. Thanks to Brian Diekelman for his detailed report about this regression.
2019-10-23 00:59:31 -04:00
if (lru)
MINOR: pattern: move the update revision to the pat_ref, not the expression It's not possible to uniquely update a single expression without updating the pattern reference, I don't know why we've put the revision in the expression back then, given that it in fact provides an update for a full pattern. Let's move the revision into the reference's head instead.
2020-11-02 09:26:51 -05:00
lru64_commit(lru, ret, expr, expr->ref->revision, NULL);
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
return ret;
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
}
/* Checks that the pattern is included inside the tested string.
* NB: Suboptimal, should be rewritten using a Boyer-Moore method.
*/
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
struct pattern *pat_match_sub(struct sample *smp, struct pattern_expr *expr, int fill)
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
{
int icase;
char *end;
char *c;
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
struct pattern_list *lst;
struct pattern *pattern;
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
struct pattern *ret = NULL;
struct lru64 *lru = NULL;
OPTIM: pattern: only apply LRU cache for large enough lists As shown in issue #1518, the LRU cache has a non-null cost that can sometimes be above the match cost it's trying to avoid. After a number of tests, it appears that: - "simple" match operations (sub, beg, end, int etc) reach a break-even after ~20 patterns in list - "heavy" match operations (reg) reach a break-even after ~5 patterns in list Let's only consult the LRU cache when the number of patterns in the expression is at least as large as this limit. Of course there will always be outliers but it already starts good. Another improvement consists in reducing the cache size to further speed up lookups, which makes sense if less expressions use the cache.
2024-11-03 12:42:26 -05:00
if (pat_lru_tree && !LIST_ISEMPTY(&expr->patterns) && expr->ref->entry_cnt >= 20) {
BUILD: pattern: fix build warnings introduced in the LRU cache They're caused by the cast to long long from ptr in 32-bit. src/pattern.c: In function 'pat_match_str': src/pattern.c:479:44: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
2015-05-04 11:18:42 -04:00
unsigned long long seed = pat_lru_seed ^ (long)expr;
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
MEDIUM: xxhash: use the XXH3 functions to generate 64-bit hashes Replace the XXH64() function calls with the XXH3 variant function XXH3_64bits_withSeed() where possible.
2020-12-22 07:22:34 -05:00
lru = lru64_get(XXH3(smp->data.u.str.area, smp->data.u.str.data, seed),
MINOR: pattern: move the update revision to the pat_ref, not the expression It's not possible to uniquely update a single expression without updating the pattern reference, I don't know why we've put the revision in the expression back then, given that it in fact provides an update for a full pattern. Let's move the revision into the reference's head instead.
2020-11-02 09:26:51 -05:00
pat_lru_tree, expr, expr->ref->revision);
BUG/MEDIUM: pattern: make the pattern LRU cache thread-local and lockless As reported in issue #335, a lot of contention happens on the PATLRU lock when performing expensive regex lookups. This is absurd since the purpose of the LRU cache was to have a fast cache for expressions, thus the cache must not be shared between threads and must remain lockless. This commit makes the LRU cache thread-local and gets rid of the PATLRU lock. A test with 7 threads on 4 cores climbed from 67kH/s to 369kH/s, or a scalability factor of 5.5. Given the huge performance difference and the regression caused to users migrating from processes to threads, this should be backported at least to 2.0. Thanks to Brian Diekelman for his detailed report about this regression.
2019-10-23 00:59:31 -04:00
if (lru && lru->domain) {
MAJOR: threads/map: Make acls/maps thread safe locks have been added in pat_ref and pattern_expr structures to protect all accesses to an instance of on of them. Moreover, a global lock has been added to protect the LRU cache used for pattern matching. Patterns are now duplicated after a successfull matching, to avoid modification by other threads when the result is used. Finally, the function reloading a pattern list has been modified to be thread-safe.
2017-07-03 05:34:05 -04:00
ret = lru->data;
return ret;
}
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
}
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
list_for_each_entry(lst, &expr->patterns, list) {
pattern = &lst->pat;
MEDIUM: pattern: only match patterns that match the current generation Instead of matching any pattern found in the tree, only match those matching the current generation of entries. This will make sure that reloads are atomic, regardless of the time they take to complete, and that newly added data are not matched until the whole reference is committed. For consistency we proceed the same way on "show map" and "show acl". This will have no impact for now since generations are not used.
2020-10-29 04:41:34 -04:00
if (pattern->ref->gen_id != expr->ref->curr_gen)
continue;
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
if (pattern->len > smp->data.u.str.data)
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
continue;
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
end = smp->data.u.str.area + smp->data.u.str.data - pattern->len;
BUG/MEDIUM: patterns: last fix was still not enough Last fix did address the issue for inlined patterns, but it was not enough because the flags are lost as well when updating patterns dynamically over the CLI. Also if the same file was used once with -i and another time without -i, their references would have been merged and both would have used the same matching method. It's appear that the patterns have two types of flags. The first ones are relative to the pattern matching, and the second are relative to the pattern storage. The pattern matching flags are the same for all the patterns of one expression. Now they are stored in the expression. The storage flags are information returned by the pattern mathing function. This information is relative to each entry and is stored in the "struct pattern". Now, the expression matching flags are forwarded to the parse and index functions. These flags are stored during the configuration parsing, and they are used during the parse and index actions. This issue was introduced in dev23 with the major pattern rework, and is a continuation of commit a631fc8 ("BUG/MAJOR: patterns: -i and -n are ignored for inlined patterns"). No backport is needed.
2014-04-28 05:18:57 -04:00
icase = expr->mflags & PAT_MF_IGNORE_CASE;
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
if (icase) {
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
for (c = smp->data.u.str.area; c <= end; c++) {
BUILD: tree-wide: cast arguments to tolower/toupper to unsigned char NetBSD apparently uses macros for tolower/toupper and complains about the use of char for array subscripts. Let's properly cast all of them to unsigned char where they are used. This is needed to fix issue #729.
2020-07-05 15:46:32 -04:00
if (tolower((unsigned char)*c) != tolower((unsigned char)*pattern->ptr.str))
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
continue;
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
if (strncasecmp(pattern->ptr.str, c, pattern->len) == 0) {
ret = pattern;
goto leave;
}
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
}
} else {
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
for (c = smp->data.u.str.area; c <= end; c++) {
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
if (*c != *pattern->ptr.str)
continue;
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
if (strncmp(pattern->ptr.str, c, pattern->len) == 0) {
ret = pattern;
goto leave;
}
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
}
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
}
}
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
leave:
BUG/MEDIUM: pattern: make the pattern LRU cache thread-local and lockless As reported in issue #335, a lot of contention happens on the PATLRU lock when performing expensive regex lookups. This is absurd since the purpose of the LRU cache was to have a fast cache for expressions, thus the cache must not be shared between threads and must remain lockless. This commit makes the LRU cache thread-local and gets rid of the PATLRU lock. A test with 7 threads on 4 cores climbed from 67kH/s to 369kH/s, or a scalability factor of 5.5. Given the huge performance difference and the regression caused to users migrating from processes to threads, this should be backported at least to 2.0. Thanks to Brian Diekelman for his detailed report about this regression.
2019-10-23 00:59:31 -04:00
if (lru)
MINOR: pattern: move the update revision to the pat_ref, not the expression It's not possible to uniquely update a single expression without updating the pattern reference, I don't know why we've put the revision in the expression back then, given that it in fact provides an update for a full pattern. Let's move the revision into the reference's head instead.
2020-11-02 09:26:51 -05:00
lru64_commit(lru, ret, expr, expr->ref->revision, NULL);
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
return ret;
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
}
/* This one is used by other real functions. It checks that the pattern is
* included inside the tested string, but enclosed between the specified
* delimiters or at the beginning or end of the string. The delimiters are
* provided as an unsigned int made by make_4delim() and match up to 4 different
* delimiters. Delimiters are stripped at the beginning and end of the pattern.
*/
BUG/MEDIUM: patterns: last fix was still not enough Last fix did address the issue for inlined patterns, but it was not enough because the flags are lost as well when updating patterns dynamically over the CLI. Also if the same file was used once with -i and another time without -i, their references would have been merged and both would have used the same matching method. It's appear that the patterns have two types of flags. The first ones are relative to the pattern matching, and the second are relative to the pattern storage. The pattern matching flags are the same for all the patterns of one expression. Now they are stored in the expression. The storage flags are information returned by the pattern mathing function. This information is relative to each entry and is stored in the "struct pattern". Now, the expression matching flags are forwarded to the parse and index functions. These flags are stored during the configuration parsing, and they are used during the parse and index actions. This issue was introduced in dev23 with the major pattern rework, and is a continuation of commit a631fc8 ("BUG/MAJOR: patterns: -i and -n are ignored for inlined patterns"). No backport is needed.
2014-04-28 05:18:57 -04:00
static int match_word(struct sample *smp, struct pattern *pattern, int mflags, unsigned int delimiters)
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
{
int may_match, icase;
char *c, *end;
char *ps;
int pl;
pl = pattern->len;
ps = pattern->ptr.str;
while (pl > 0 && is_delimiter(*ps, delimiters)) {
pl--;
ps++;
}
while (pl > 0 && is_delimiter(ps[pl - 1], delimiters))
pl--;
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
if (pl > smp->data.u.str.data)
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
return PAT_NOMATCH;
may_match = 1;
BUG/MEDIUM: patterns: last fix was still not enough Last fix did address the issue for inlined patterns, but it was not enough because the flags are lost as well when updating patterns dynamically over the CLI. Also if the same file was used once with -i and another time without -i, their references would have been merged and both would have used the same matching method. It's appear that the patterns have two types of flags. The first ones are relative to the pattern matching, and the second are relative to the pattern storage. The pattern matching flags are the same for all the patterns of one expression. Now they are stored in the expression. The storage flags are information returned by the pattern mathing function. This information is relative to each entry and is stored in the "struct pattern". Now, the expression matching flags are forwarded to the parse and index functions. These flags are stored during the configuration parsing, and they are used during the parse and index actions. This issue was introduced in dev23 with the major pattern rework, and is a continuation of commit a631fc8 ("BUG/MAJOR: patterns: -i and -n are ignored for inlined patterns"). No backport is needed.
2014-04-28 05:18:57 -04:00
icase = mflags & PAT_MF_IGNORE_CASE;
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
end = smp->data.u.str.area + smp->data.u.str.data - pl;
for (c = smp->data.u.str.area; c <= end; c++) {
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
if (is_delimiter(*c, delimiters)) {
may_match = 1;
continue;
}
if (!may_match)
continue;
if (icase) {
BUILD: tree-wide: cast arguments to tolower/toupper to unsigned char NetBSD apparently uses macros for tolower/toupper and complains about the use of char for array subscripts. Let's properly cast all of them to unsigned char where they are used. This is needed to fix issue #729.
2020-07-05 15:46:32 -04:00
if ((tolower((unsigned char)*c) == tolower((unsigned char)*ps)) &&
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
(strncasecmp(ps, c, pl) == 0) &&
(c == end || is_delimiter(c[pl], delimiters)))
return PAT_MATCH;
} else {
if ((*c == *ps) &&
(strncmp(ps, c, pl) == 0) &&
(c == end || is_delimiter(c[pl], delimiters)))
return PAT_MATCH;
}
may_match = 0;
}
return PAT_NOMATCH;
}
/* Checks that the pattern is included inside the tested string, but enclosed
* between the delimiters '?' or '/' or at the beginning or end of the string.
* Delimiters at the beginning or end of the pattern are ignored.
*/
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
struct pattern *pat_match_dir(struct sample *smp, struct pattern_expr *expr, int fill)
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
{
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
struct pattern_list *lst;
struct pattern *pattern;
list_for_each_entry(lst, &expr->patterns, list) {
pattern = &lst->pat;
MEDIUM: pattern: only match patterns that match the current generation Instead of matching any pattern found in the tree, only match those matching the current generation of entries. This will make sure that reloads are atomic, regardless of the time they take to complete, and that newly added data are not matched until the whole reference is committed. For consistency we proceed the same way on "show map" and "show acl". This will have no impact for now since generations are not used.
2020-10-29 04:41:34 -04:00
if (pattern->ref->gen_id != expr->ref->curr_gen)
continue;
BUG/MEDIUM: patterns: last fix was still not enough Last fix did address the issue for inlined patterns, but it was not enough because the flags are lost as well when updating patterns dynamically over the CLI. Also if the same file was used once with -i and another time without -i, their references would have been merged and both would have used the same matching method. It's appear that the patterns have two types of flags. The first ones are relative to the pattern matching, and the second are relative to the pattern storage. The pattern matching flags are the same for all the patterns of one expression. Now they are stored in the expression. The storage flags are information returned by the pattern mathing function. This information is relative to each entry and is stored in the "struct pattern". Now, the expression matching flags are forwarded to the parse and index functions. These flags are stored during the configuration parsing, and they are used during the parse and index actions. This issue was introduced in dev23 with the major pattern rework, and is a continuation of commit a631fc8 ("BUG/MAJOR: patterns: -i and -n are ignored for inlined patterns"). No backport is needed.
2014-04-28 05:18:57 -04:00
if (match_word(smp, pattern, expr->mflags, make_4delim('/', '?', '?', '?')))
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
return pattern;
}
return NULL;
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
}
/* Checks that the pattern is included inside the tested string, but enclosed
* between the delmiters '/', '?', '.' or ":" or at the beginning or end of
* the string. Delimiters at the beginning or end of the pattern are ignored.
*/
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
struct pattern *pat_match_dom(struct sample *smp, struct pattern_expr *expr, int fill)
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
{
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
struct pattern_list *lst;
struct pattern *pattern;
list_for_each_entry(lst, &expr->patterns, list) {
pattern = &lst->pat;
MEDIUM: pattern: only match patterns that match the current generation Instead of matching any pattern found in the tree, only match those matching the current generation of entries. This will make sure that reloads are atomic, regardless of the time they take to complete, and that newly added data are not matched until the whole reference is committed. For consistency we proceed the same way on "show map" and "show acl". This will have no impact for now since generations are not used.
2020-10-29 04:41:34 -04:00
if (pattern->ref->gen_id != expr->ref->curr_gen)
continue;
BUG/MEDIUM: patterns: last fix was still not enough Last fix did address the issue for inlined patterns, but it was not enough because the flags are lost as well when updating patterns dynamically over the CLI. Also if the same file was used once with -i and another time without -i, their references would have been merged and both would have used the same matching method. It's appear that the patterns have two types of flags. The first ones are relative to the pattern matching, and the second are relative to the pattern storage. The pattern matching flags are the same for all the patterns of one expression. Now they are stored in the expression. The storage flags are information returned by the pattern mathing function. This information is relative to each entry and is stored in the "struct pattern". Now, the expression matching flags are forwarded to the parse and index functions. These flags are stored during the configuration parsing, and they are used during the parse and index actions. This issue was introduced in dev23 with the major pattern rework, and is a continuation of commit a631fc8 ("BUG/MAJOR: patterns: -i and -n are ignored for inlined patterns"). No backport is needed.
2014-04-28 05:18:57 -04:00
if (match_word(smp, pattern, expr->mflags, make_4delim('/', '?', '.', ':')))
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
return pattern;
}
return NULL;
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
}
/* Checks that the integer in <test> is included between min and max */
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
struct pattern *pat_match_int(struct sample *smp, struct pattern_expr *expr, int fill)
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
{
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
struct pattern_list *lst;
struct pattern *pattern;
list_for_each_entry(lst, &expr->patterns, list) {
pattern = &lst->pat;
MEDIUM: pattern: only match patterns that match the current generation Instead of matching any pattern found in the tree, only match those matching the current generation of entries. This will make sure that reloads are atomic, regardless of the time they take to complete, and that newly added data are not matched until the whole reference is committed. For consistency we proceed the same way on "show map" and "show acl". This will have no impact for now since generations are not used.
2020-10-29 04:41:34 -04:00
if (pattern->ref->gen_id != expr->ref->curr_gen)
continue;
MINOR: samples: rename union from "data" to "u" The union name "data" is a little bit heavy while we read the source code because we can read "data.data.sint". The rename from "data" to "u" makes the read easiest like "data.u.sint".
2015-08-19 03:07:19 -04:00
if ((!pattern->val.range.min_set || pattern->val.range.min <= smp->data.u.sint) &&
(!pattern->val.range.max_set || smp->data.u.sint <= pattern->val.range.max))
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
return pattern;
}
return NULL;
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
}
/* Checks that the length of the pattern in <test> is included between min and max */
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
struct pattern *pat_match_len(struct sample *smp, struct pattern_expr *expr, int fill)
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
{
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
struct pattern_list *lst;
struct pattern *pattern;
list_for_each_entry(lst, &expr->patterns, list) {
pattern = &lst->pat;
MEDIUM: pattern: only match patterns that match the current generation Instead of matching any pattern found in the tree, only match those matching the current generation of entries. This will make sure that reloads are atomic, regardless of the time they take to complete, and that newly added data are not matched until the whole reference is committed. For consistency we proceed the same way on "show map" and "show acl". This will have no impact for now since generations are not used.
2020-10-29 04:41:34 -04:00
if (pattern->ref->gen_id != expr->ref->curr_gen)
continue;
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
if ((!pattern->val.range.min_set || pattern->val.range.min <= smp->data.u.str.data) &&
(!pattern->val.range.max_set || smp->data.u.str.data <= pattern->val.range.max))
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
return pattern;
}
return NULL;
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
}
MINOR: pattern/ip: simplify pat_match_ip() function pat_match_ip() has been updated several times over the last decade to introduce new features, but it was never cleaned up. The result is that the function is pretty hard to read, and there are multiple duplicated code blocks so it becomes error-prone to maintain it, plus it bloats the haproxy binary for nothing. In this patch, we move the tree search (ip4 / ip6) logic into 2 dedicated helper functions. This allows us to refactor pat_match_ip() without touching to the original behavior.
2023-09-06 05:32:54 -04:00
/* Performs ipv4 key lookup in <expr> ipv4 tree
* Returns NULL on failure
*/
static struct pattern *_pat_match_tree_ipv4(struct in_addr *key, struct pattern_expr *expr, int fill)
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
{
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
struct ebmb_node *node;
struct pattern_tree *elt;
MINOR: pattern/ip: simplify pat_match_ip() function pat_match_ip() has been updated several times over the last decade to introduce new features, but it was never cleaned up. The result is that the function is pretty hard to read, and there are multiple duplicated code blocks so it becomes error-prone to maintain it, plus it bloats the haproxy binary for nothing. In this patch, we move the tree search (ip4 / ip6) logic into 2 dedicated helper functions. This allows us to refactor pat_match_ip() without touching to the original behavior.
2023-09-06 05:32:54 -04:00
/* Lookup an IPv4 address in the expression's pattern tree using
* the longest match method.
*/
node = ebmb_lookup_longest(&expr->pattern_tree, key);
while (node) {
elt = ebmb_entry(node, struct pattern_tree, node);
if (elt->ref->gen_id != expr->ref->curr_gen) {
node = ebmb_lookup_shorter(node);
continue;
}
if (fill) {
static_pattern.data = elt->data;
static_pattern.ref = elt->ref;
static_pattern.sflags = PAT_SF_TREE;
static_pattern.type = SMP_T_IPV4;
static_pattern.val.ipv4.addr.s_addr = read_u32(elt->node.key);
if (!cidr2dotted(elt->node.node.pfx, &static_pattern.val.ipv4.mask))
return NULL;
}
return &static_pattern;
}
return NULL;
}
/* Performs ipv6 key lookup in <expr> ipv6 tree
* Returns NULL on failure
*/
static struct pattern *_pat_match_tree_ipv6(struct in6_addr *key, struct pattern_expr *expr, int fill)
{
struct ebmb_node *node;
struct pattern_tree *elt;
/* Lookup an IPv6 address in the expression's pattern tree using
* the longest match method.
*/
node = ebmb_lookup_longest(&expr->pattern_tree_2, key);
while (node) {
elt = ebmb_entry(node, struct pattern_tree, node);
if (elt->ref->gen_id != expr->ref->curr_gen) {
node = ebmb_lookup_shorter(node);
continue;
}
if (fill) {
static_pattern.data = elt->data;
static_pattern.ref = elt->ref;
static_pattern.sflags = PAT_SF_TREE;
static_pattern.type = SMP_T_IPV6;
memcpy(&static_pattern.val.ipv6.addr, elt->node.key, 16);
static_pattern.val.ipv6.mask = elt->node.node.pfx;
}
return &static_pattern;
}
return NULL;
}
struct pattern *pat_match_ip(struct sample *smp, struct pattern_expr *expr, int fill)
{
struct in_addr v4;
struct in6_addr v6;
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
struct pattern_list *lst;
struct pattern *pattern;
MEDIUM: pattern: Index IPv6 addresses in a tree. This commit adds second tree node in the pattern struct and use it to index IPv6 addresses. This commit report feature used in the list. If IPv4 not match the tree, try to convert the IPv4 address in IPv6 with prefixing the IPv4 address by "::ffff", after this operation, the match function try lookup in the IPv6 tree. If the IPv6 sample dont match the IPv6 tree, try to convert the IPv6 addresses prefixed by "2002:IPv4", "::ffff:IPv4" and "::0000:IPv4" in IPv4 address. after this operation, the match function try lookup in the IPv4 tree.
2013-12-19 17:54:54 -05:00
/* The input sample is IPv4. Try to match in the trees. */
MEDIUM: samples: Use the "struct sample_data" in the "struct sample" This patch remove the struct information stored both in the struct sample_data and in the striuct sample. Now, only thestruct sample_data contains data, and the struct sample use the struct sample_data for storing his own data.
2015-08-19 03:00:18 -04:00
if (smp->data.type == SMP_T_IPV4) {
MINOR: pattern/ip: simplify pat_match_ip() function pat_match_ip() has been updated several times over the last decade to introduce new features, but it was never cleaned up. The result is that the function is pretty hard to read, and there are multiple duplicated code blocks so it becomes error-prone to maintain it, plus it bloats the haproxy binary for nothing. In this patch, we move the tree search (ip4 / ip6) logic into 2 dedicated helper functions. This allows us to refactor pat_match_ip() without touching to the original behavior.
2023-09-06 05:32:54 -04:00
pattern = _pat_match_tree_ipv4(&smp->data.u.ipv4, expr, fill);
if (pattern)
return pattern;
CLEANUP: Fix spelling errors in comments This is from the output of codespell. It's done at once over a bunch of files and only affects comments, so there is nothing user-visible. No backport needed.
2021-01-07 23:35:52 -05:00
/* The IPv4 sample don't match the IPv4 tree. Convert the IPv4
MINOR: pattern/ip: offload ip conversion logic to helper functions Now that v4tov6() and v6tov4() were reworked to match behavior from pat_match_ip() function in ("MINOR: tools/ip: v4tov6() and v6tov4() rework"), we can remove code duplication in pat_match_ip() by directly using those dedicated functions where relevant.
2023-09-05 08:58:53 -04:00
* sample address to IPv6 and try to lookup in the IPv6 tree.
MEDIUM: pattern: Index IPv6 addresses in a tree. This commit adds second tree node in the pattern struct and use it to index IPv6 addresses. This commit report feature used in the list. If IPv4 not match the tree, try to convert the IPv4 address in IPv6 with prefixing the IPv4 address by "::ffff", after this operation, the match function try lookup in the IPv6 tree. If the IPv6 sample dont match the IPv6 tree, try to convert the IPv6 addresses prefixed by "2002:IPv4", "::ffff:IPv4" and "::0000:IPv4" in IPv4 address. after this operation, the match function try lookup in the IPv4 tree.
2013-12-19 17:54:54 -05:00
*/
MINOR: pattern/ip: simplify pat_match_ip() function pat_match_ip() has been updated several times over the last decade to introduce new features, but it was never cleaned up. The result is that the function is pretty hard to read, and there are multiple duplicated code blocks so it becomes error-prone to maintain it, plus it bloats the haproxy binary for nothing. In this patch, we move the tree search (ip4 / ip6) logic into 2 dedicated helper functions. This allows us to refactor pat_match_ip() without touching to the original behavior.
2023-09-06 05:32:54 -04:00
v4tov6(&v6, &smp->data.u.ipv4);
pattern = _pat_match_tree_ipv6(&v6, expr, fill);
if (pattern)
return pattern;
/* eligible for list lookup using IPv4 address */
v4 = smp->data.u.ipv4;
goto list_lookup;
MEDIUM: pattern: Index IPv6 addresses in a tree. This commit adds second tree node in the pattern struct and use it to index IPv6 addresses. This commit report feature used in the list. If IPv4 not match the tree, try to convert the IPv4 address in IPv6 with prefixing the IPv4 address by "::ffff", after this operation, the match function try lookup in the IPv6 tree. If the IPv6 sample dont match the IPv6 tree, try to convert the IPv6 addresses prefixed by "2002:IPv4", "::ffff:IPv4" and "::0000:IPv4" in IPv4 address. after this operation, the match function try lookup in the IPv4 tree.
2013-12-19 17:54:54 -05:00
}
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
MEDIUM: pattern: Index IPv6 addresses in a tree. This commit adds second tree node in the pattern struct and use it to index IPv6 addresses. This commit report feature used in the list. If IPv4 not match the tree, try to convert the IPv4 address in IPv6 with prefixing the IPv4 address by "::ffff", after this operation, the match function try lookup in the IPv6 tree. If the IPv6 sample dont match the IPv6 tree, try to convert the IPv6 addresses prefixed by "2002:IPv4", "::ffff:IPv4" and "::0000:IPv4" in IPv4 address. after this operation, the match function try lookup in the IPv4 tree.
2013-12-19 17:54:54 -05:00
/* The input sample is IPv6. Try to match in the trees. */
MEDIUM: samples: Use the "struct sample_data" in the "struct sample" This patch remove the struct information stored both in the struct sample_data and in the striuct sample. Now, only thestruct sample_data contains data, and the struct sample use the struct sample_data for storing his own data.
2015-08-19 03:00:18 -04:00
if (smp->data.type == SMP_T_IPV6) {
MINOR: pattern/ip: simplify pat_match_ip() function pat_match_ip() has been updated several times over the last decade to introduce new features, but it was never cleaned up. The result is that the function is pretty hard to read, and there are multiple duplicated code blocks so it becomes error-prone to maintain it, plus it bloats the haproxy binary for nothing. In this patch, we move the tree search (ip4 / ip6) logic into 2 dedicated helper functions. This allows us to refactor pat_match_ip() without touching to the original behavior.
2023-09-06 05:32:54 -04:00
pattern = _pat_match_tree_ipv6(&smp->data.u.ipv6, expr, fill);
if (pattern)
return pattern;
/* No match in the IPv6 tree. Try to convert 6 to 4 to lookup in
* the IPv4 tree
MEDIUM: pattern: Index IPv6 addresses in a tree. This commit adds second tree node in the pattern struct and use it to index IPv6 addresses. This commit report feature used in the list. If IPv4 not match the tree, try to convert the IPv4 address in IPv6 with prefixing the IPv4 address by "::ffff", after this operation, the match function try lookup in the IPv6 tree. If the IPv6 sample dont match the IPv6 tree, try to convert the IPv6 addresses prefixed by "2002:IPv4", "::ffff:IPv4" and "::0000:IPv4" in IPv4 address. after this operation, the match function try lookup in the IPv4 tree.
2013-12-19 17:54:54 -05:00
*/
MINOR: pattern/ip: offload ip conversion logic to helper functions Now that v4tov6() and v6tov4() were reworked to match behavior from pat_match_ip() function in ("MINOR: tools/ip: v4tov6() and v6tov4() rework"), we can remove code duplication in pat_match_ip() by directly using those dedicated functions where relevant.
2023-09-05 08:58:53 -04:00
if (v6tov4(&v4, &smp->data.u.ipv6)) {
MINOR: pattern/ip: simplify pat_match_ip() function pat_match_ip() has been updated several times over the last decade to introduce new features, but it was never cleaned up. The result is that the function is pretty hard to read, and there are multiple duplicated code blocks so it becomes error-prone to maintain it, plus it bloats the haproxy binary for nothing. In this patch, we move the tree search (ip4 / ip6) logic into 2 dedicated helper functions. This allows us to refactor pat_match_ip() without touching to the original behavior.
2023-09-06 05:32:54 -04:00
pattern = _pat_match_tree_ipv4(&v4, expr, fill);
if (pattern)
return pattern;
/* eligible for list lookup using IPv4 address */
goto list_lookup;
MEDIUM: pattern: Index IPv6 addresses in a tree. This commit adds second tree node in the pattern struct and use it to index IPv6 addresses. This commit report feature used in the list. If IPv4 not match the tree, try to convert the IPv4 address in IPv6 with prefixing the IPv4 address by "::ffff", after this operation, the match function try lookup in the IPv6 tree. If the IPv6 sample dont match the IPv6 tree, try to convert the IPv6 addresses prefixed by "2002:IPv4", "::ffff:IPv4" and "::0000:IPv4" in IPv4 address. after this operation, the match function try lookup in the IPv4 tree.
2013-12-19 17:54:54 -05:00
}
}
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
MINOR: pattern/ip: simplify pat_match_ip() function pat_match_ip() has been updated several times over the last decade to introduce new features, but it was never cleaned up. The result is that the function is pretty hard to read, and there are multiple duplicated code blocks so it becomes error-prone to maintain it, plus it bloats the haproxy binary for nothing. In this patch, we move the tree search (ip4 / ip6) logic into 2 dedicated helper functions. This allows us to refactor pat_match_ip() without touching to the original behavior.
2023-09-06 05:32:54 -04:00
not_found:
return NULL;
list_lookup:
/* No match in the trees, but we still have a valid IPv4 address: lookup
* in the IPv4 list (non-contiguous masks list). This is our last resort
*/
MEDIUM: pattern: Index IPv6 addresses in a tree. This commit adds second tree node in the pattern struct and use it to index IPv6 addresses. This commit report feature used in the list. If IPv4 not match the tree, try to convert the IPv4 address in IPv6 with prefixing the IPv4 address by "::ffff", after this operation, the match function try lookup in the IPv6 tree. If the IPv6 sample dont match the IPv6 tree, try to convert the IPv6 addresses prefixed by "2002:IPv4", "::ffff:IPv4" and "::0000:IPv4" in IPv4 address. after this operation, the match function try lookup in the IPv4 tree.
2013-12-19 17:54:54 -05:00
list_for_each_entry(lst, &expr->patterns, list) {
pattern = &lst->pat;
MEDIUM: pattern: only match patterns that match the current generation Instead of matching any pattern found in the tree, only match those matching the current generation of entries. This will make sure that reloads are atomic, regardless of the time they take to complete, and that newly added data are not matched until the whole reference is committed. For consistency we proceed the same way on "show map" and "show acl". This will have no impact for now since generations are not used.
2020-10-29 04:41:34 -04:00
if (pattern->ref->gen_id != expr->ref->curr_gen)
continue;
MEDIUM: pattern: Index IPv6 addresses in a tree. This commit adds second tree node in the pattern struct and use it to index IPv6 addresses. This commit report feature used in the list. If IPv4 not match the tree, try to convert the IPv4 address in IPv6 with prefixing the IPv4 address by "::ffff", after this operation, the match function try lookup in the IPv6 tree. If the IPv6 sample dont match the IPv6 tree, try to convert the IPv6 addresses prefixed by "2002:IPv4", "::ffff:IPv4" and "::0000:IPv4" in IPv4 address. after this operation, the match function try lookup in the IPv4 tree.
2013-12-19 17:54:54 -05:00
/* Check if the input sample match the current pattern. */
MINOR: pattern/ip: offload ip conversion logic to helper functions Now that v4tov6() and v6tov4() were reworked to match behavior from pat_match_ip() function in ("MINOR: tools/ip: v4tov6() and v6tov4() rework"), we can remove code duplication in pat_match_ip() by directly using those dedicated functions where relevant.
2023-09-05 08:58:53 -04:00
if (((v4.s_addr ^ pattern->val.ipv4.addr.s_addr) & pattern->val.ipv4.mask.s_addr) == 0)
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
return pattern;
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
}
MINOR: pattern/ip: simplify pat_match_ip() function pat_match_ip() has been updated several times over the last decade to introduce new features, but it was never cleaned up. The result is that the function is pretty hard to read, and there are multiple duplicated code blocks so it becomes error-prone to maintain it, plus it bloats the haproxy binary for nothing. In this patch, we move the tree search (ip4 / ip6) logic into 2 dedicated helper functions. This allows us to refactor pat_match_ip() without touching to the original behavior.
2023-09-06 05:32:54 -04:00
goto not_found;
MINOR: pattern: move functions for grouping pat_match_* and pat_parse_* and add documentation.
2014-01-21 05:25:41 -05:00
}
MINOR: pattern: prepare removal of a pattern from the list head Instead of using LIST_DEL() on the pattern itself inside an expression, we look it up from its head. The goal is to get rid of the double-linked list while this usage remains exclusively for freeing on startup error!
2020-11-03 05:22:04 -05:00
/* finds the pattern holding <list> from list head <head> and deletes it.
* This is made for use for pattern removal within an expression.
*/
MEDIUM: pattern: turn the pattern chaining to single-linked list It does not require heavy deletion from the expr anymore, so we can now turn this to a single-linked list since most of the time we want to delete all instances of a given pattern from the head. By doing so we save 32 bytes of memory per pattern. The pat_unlink_from_head() function was adjusted accordingly.
2020-11-03 08:50:29 -05:00
static void pat_unlink_from_head(void **head, void **list)
MINOR: pattern: prepare removal of a pattern from the list head Instead of using LIST_DEL() on the pattern itself inside an expression, we look it up from its head. The goal is to get rid of the double-linked list while this usage remains exclusively for freeing on startup error!
2020-11-03 05:22:04 -05:00
{
MEDIUM: pattern: turn the pattern chaining to single-linked list It does not require heavy deletion from the expr anymore, so we can now turn this to a single-linked list since most of the time we want to delete all instances of a given pattern from the head. By doing so we save 32 bytes of memory per pattern. The pat_unlink_from_head() function was adjusted accordingly.
2020-11-03 08:50:29 -05:00
while (*head) {
if (*head == list) {
*head = *list;
MINOR: pattern: prepare removal of a pattern from the list head Instead of using LIST_DEL() on the pattern itself inside an expression, we look it up from its head. The goal is to get rid of the double-linked list while this usage remains exclusively for freeing on startup error!
2020-11-03 05:22:04 -05:00
return;
}
MEDIUM: pattern: turn the pattern chaining to single-linked list It does not require heavy deletion from the expr anymore, so we can now turn this to a single-linked list since most of the time we want to delete all instances of a given pattern from the head. By doing so we save 32 bytes of memory per pattern. The pat_unlink_from_head() function was adjusted accordingly.
2020-11-03 08:50:29 -05:00
head = *head;
MINOR: pattern: prepare removal of a pattern from the list head Instead of using LIST_DEL() on the pattern itself inside an expression, we look it up from its head. The goal is to get rid of the double-linked list while this usage remains exclusively for freeing on startup error!
2020-11-03 05:22:04 -05:00
}
}
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
void free_pattern_tree(struct eb_root *root)
{
struct eb_node *node, *next;
MINOR: pattern: Rename "pat_idx_elt" to "pattern_tree" This is just for having coherent struct names.
2013-12-13 10:09:50 -05:00
struct pattern_tree *elt;
BUG/MEDIUM: pattern: Pattern node has type of "struct pat_idx_elt" in place of "struct eb_node" The free() function must free the "struct pat_idx_elt". This bug was introduced by commit ed66c29 (REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c), no backport is needed.
2013-12-09 05:29:46 -05:00
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
node = eb_first(root);
while (node) {
next = eb_next(node);
eb_delete(node);
MINOR: pattern: Rename "pat_idx_elt" to "pattern_tree" This is just for having coherent struct names.
2013-12-13 10:09:50 -05:00
elt = container_of(node, struct pattern_tree, node);
MINOR: pattern: prepare removal of a pattern from the list head Instead of using LIST_DEL() on the pattern itself inside an expression, we look it up from its head. The goal is to get rid of the double-linked list while this usage remains exclusively for freeing on startup error!
2020-11-03 05:22:04 -05:00
pat_unlink_from_head(&elt->ref->tree_head, &elt->from_ref);
MINOR: samples: rename some struct member from "smp" to "data" This members contains data and not sample.
2015-08-19 02:35:43 -04:00
free(elt->data);
BUG/MEDIUM: pattern: Pattern node has type of "struct pat_idx_elt" in place of "struct eb_node" The free() function must free the "struct pat_idx_elt". This bug was introduced by commit ed66c29 (REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c), no backport is needed.
2013-12-09 05:29:46 -05:00
free(elt);
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
node = next;
}
}
MINOR: pattern: make the delete and prune functions more generic Now we have a single prune() function to act on an expression, and one delete function for the lists and one for the trees. The presence of a pointer in the lists is enough to warrant a free, and we rely on the PAT_SF_REGFREE flag to decide whether to free using free() or regfree().
2020-11-02 13:26:02 -05:00
void pat_prune_gen(struct pattern_expr *expr)
MEDIUM: pattern: create pattern expression This new structure contains the data needed for pattern matching. It's the first step to the complete independance of the pattern matching.
2013-11-28 05:41:23 -05:00
{
MEDIUM: pattern: add prune function This path add specific pointer to each expression to point on prune function. Now, each pattern expression embed his own prune function.
2014-01-14 10:24:51 -05:00
struct pattern_list *pat, *tmp;
list_for_each_entry_safe(pat, tmp, &expr->patterns, list) {
CLEANUP: lists/tree-wide: rename some list operations to avoid some confusion The current "ADD" vs "ADDQ" is confusing because when thinking in terms of appending at the end of a list, "ADD" naturally comes to mind, but here it does the opposite, it inserts. Several times already it's been incorrectly used where ADDQ was expected, the latest of which was a fortunate accident explained in 6fa922562 ("CLEANUP: stream: explain why we queue the stream at the head of the server list"). Let's use more explicit (but slightly longer) names now: LIST_ADD -> LIST_INSERT LIST_ADDQ -> LIST_APPEND LIST_ADDED -> LIST_INLIST LIST_DEL -> LIST_DELETE The same is true for MT_LISTs, including their "TRY" variant. LIST_DEL_INIT keeps its short name to encourage to use it instead of the lazier LIST_DELETE which is often less safe. The change is large (~674 non-comment entries) but is mechanical enough to remain safe. No permutation was performed, so any out-of-tree code can easily map older names to new ones. The list doc was updated.
2021-04-21 01:32:39 -04:00
LIST_DELETE(&pat->list);
MINOR: pattern: prepare removal of a pattern from the list head Instead of using LIST_DEL() on the pattern itself inside an expression, we look it up from its head. The goal is to get rid of the double-linked list while this usage remains exclusively for freeing on startup error!
2020-11-03 05:22:04 -05:00
pat_unlink_from_head(&pat->pat.ref->list_head, &pat->from_ref);
MINOR: pattern: make the delete and prune functions more generic Now we have a single prune() function to act on an expression, and one delete function for the lists and one for the trees. The presence of a pointer in the lists is enough to warrant a free, and we rely on the PAT_SF_REGFREE flag to decide whether to free using free() or regfree().
2020-11-02 13:26:02 -05:00
if (pat->pat.sflags & PAT_SF_REGFREE)
regex_free(pat->pat.ptr.ptr);
else
free(pat->pat.ptr.ptr);
MINOR: samples: rename some struct member from "smp" to "data" This members contains data and not sample.
2015-08-19 02:35:43 -04:00
free(pat->pat.data);
MEDIUM: pattern: add prune function This path add specific pointer to each expression to point on prune function. Now, each pattern expression embed his own prune function.
2014-01-14 10:24:51 -05:00
free(pat);
}
MEDIUM: pattern: create pattern expression This new structure contains the data needed for pattern matching. It's the first step to the complete independance of the pattern matching.
2013-11-28 05:41:23 -05:00
free_pattern_tree(&expr->pattern_tree);
MEDIUM: pattern: Index IPv6 addresses in a tree. This commit adds second tree node in the pattern struct and use it to index IPv6 addresses. This commit report feature used in the list. If IPv4 not match the tree, try to convert the IPv4 address in IPv6 with prefixing the IPv4 address by "::ffff", after this operation, the match function try lookup in the IPv6 tree. If the IPv6 sample dont match the IPv6 tree, try to convert the IPv6 addresses prefixed by "2002:IPv4", "::ffff:IPv4" and "::0000:IPv4" in IPv4 address. after this operation, the match function try lookup in the IPv4 tree.
2013-12-19 17:54:54 -05:00
free_pattern_tree(&expr->pattern_tree_2);
MEDIUM: pattern: create pattern expression This new structure contains the data needed for pattern matching. It's the first step to the complete independance of the pattern matching.
2013-11-28 05:41:23 -05:00
LIST_INIT(&expr->patterns);
MINOR: pattern: move the update revision to the pat_ref, not the expression It's not possible to uniquely update a single expression without updating the pattern reference, I don't know why we've put the revision in the expression back then, given that it in fact provides an update for a full pattern. Let's move the revision into the reference's head instead.
2020-11-02 09:26:51 -05:00
expr->ref->revision = rdtsc();
MINOR: map/acl: print the count of all the map/acl entries in "show map/acl" The output of "show map/acl" now contains the 'entry_cnt' value that represents the count of all the entries for each map/acl, not just the active ones, which means that it also includes entries currently being added.
2021-05-21 10:59:15 -04:00
expr->ref->entry_cnt = 0;
MEDIUM: pattern: create pattern expression This new structure contains the data needed for pattern matching. It's the first step to the complete independance of the pattern matching.
2013-11-28 05:41:23 -05:00
}
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
/*
*
* The following functions are used for the pattern indexation
*
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
*/
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
int pat_idx_list_val(struct pattern_expr *expr, struct pattern *pat, char **err)
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
{
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
struct pattern_list *patl;
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
/* allocate pattern */
patl = calloc(1, sizeof(*patl));
if (!patl) {
memprintf(err, "out of memory while indexing pattern");
MEDIUM: pattern: Change the prototype of the function pattern_register(). Each pattern parser take only one string. This change is reported to the function prototype of the function "pattern_register()". Now, it is called with just one string and no need to browse the array of args.
2014-01-23 11:53:31 -05:00
return 0;
}
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
/* duplicate pattern */
memcpy(&patl->pat, pat, sizeof(*pat));
MEDIUM: pattern: Extract the index process from the pat_parse_*() functions Now, the pat_parse_*() functions parses the incoming data. The input "pattern" struct can be preallocated. If the parser needs to add some buffers, it allocates memory. The function pattern_register() runs the call to the parser, process the key indexation and associate the "sample_storage" used by maps.
2013-12-06 13:06:43 -05:00
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
/* chain pattern in the expression */
CLEANUP: lists/tree-wide: rename some list operations to avoid some confusion The current "ADD" vs "ADDQ" is confusing because when thinking in terms of appending at the end of a list, "ADD" naturally comes to mind, but here it does the opposite, it inserts. Several times already it's been incorrectly used where ADDQ was expected, the latest of which was a fortunate accident explained in 6fa922562 ("CLEANUP: stream: explain why we queue the stream at the head of the server list"). Let's use more explicit (but slightly longer) names now: LIST_ADD -> LIST_INSERT LIST_ADDQ -> LIST_APPEND LIST_ADDED -> LIST_INLIST LIST_DEL -> LIST_DELETE The same is true for MT_LISTs, including their "TRY" variant. LIST_DEL_INIT keeps its short name to encourage to use it instead of the lazier LIST_DELETE which is often less safe. The change is large (~674 non-comment entries) but is mechanical enough to remain safe. No permutation was performed, so any out-of-tree code can easily map older names to new ones. The list doc was updated.
2021-04-21 01:32:39 -04:00
LIST_APPEND(&expr->patterns, &patl->list);
MEDIUM: map/acl: Improve pat_ref_set_elt() efficiency (for "set-map", "add-acl"action perfs) Store a pointer to the expression (struct pattern_expr) into the data structure used to chain/store the map element references (struct pat_ref_elt) , e.g. the struct pattern_tree when stored into an ebtree or struct pattern_list when chained to a list. Modify pat_ref_set_elt() to stop inspecting all the expressions attached to a map and to look for the <elt> element passed as parameter to retrieve the sample data to be parsed. Indeed, thanks to the pointer added above to each pattern tree nodes or list elements, they all can be inspected directly from the <elt> passed as parameter and its ->tree_head and ->list_head member: the pattern tree nodes are stored into elt->tree_head, and the pattern list elements are chained to elt->list_head list. This inspection was also the job of pattern_find_smp() which is no more useful. This patch removes the code of this function.
2023-08-23 09:58:26 -04:00
patl->expr = expr;
MEDIUM: pattern: link all final elements from the reference There is a data model issue in the current pattern design that makes pattern deletion extremely expensive: there's no direct way from a reference to access all indexed occurrences. As such, the only way to remove all indexed entries corresponding to a reference update is to scan all expressions's lists and trees to find a link to the reference. While this was possibly OK when map removal was not common and most maps were small, this is not conceivable anymore with GeoIP maps containing 10M+ entries and del-map operations that are triggered from http-request rulesets. This patch introduces two list heads from the pattern reference, one for the objects linked by lists and one for those linked by tree node. Ideally a single list would be enough but the linked elements are too much unrelated to be distinguished at the moment, so we'll need two lists. However for the long term a single-linked list will suffice but for now it's not possible due to the way elements are removed from expressions. As such this patch adds 32 bytes of memory usage per reference plus 16 per indexed entry, but both will be cut in half later. The links are not yet used for deletion, this patch only ensures the list is always consistent.
2020-11-02 06:10:48 -05:00
/* and from the reference */
MEDIUM: pattern: turn the pattern chaining to single-linked list It does not require heavy deletion from the expr anymore, so we can now turn this to a single-linked list since most of the time we want to delete all instances of a given pattern from the head. By doing so we save 32 bytes of memory per pattern. The pat_unlink_from_head() function was adjusted accordingly.
2020-11-03 08:50:29 -05:00
patl->from_ref = pat->ref->list_head;
pat->ref->list_head = &patl->from_ref;
MINOR: pattern: move the update revision to the pat_ref, not the expression It's not possible to uniquely update a single expression without updating the pattern reference, I don't know why we've put the revision in the expression back then, given that it in fact provides an update for a full pattern. Let's move the revision into the reference's head instead.
2020-11-02 09:26:51 -05:00
expr->ref->revision = rdtsc();
MINOR: map/acl: print the count of all the map/acl entries in "show map/acl" The output of "show map/acl" now contains the 'entry_cnt' value that represents the count of all the entries for each map/acl, not just the active ones, which means that it also includes entries currently being added.
2021-05-21 10:59:15 -04:00
expr->ref->entry_cnt++;
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
/* that's ok */
return 1;
}
int pat_idx_list_ptr(struct pattern_expr *expr, struct pattern *pat, char **err)
{
struct pattern_list *patl;
/* allocate pattern */
patl = calloc(1, sizeof(*patl));
BUG/MINOR: pattern: error message missing This patch must be backported in 1.5 version.
2015-02-06 11:50:55 -05:00
if (!patl) {
memprintf(err, "out of memory while indexing pattern");
MEDIUM: pattern: Change the prototype of the function pattern_register(). Each pattern parser take only one string. This change is reported to the function prototype of the function "pattern_register()". Now, it is called with just one string and no need to browse the array of args.
2014-01-23 11:53:31 -05:00
return 0;
BUG/MINOR: pattern: error message missing This patch must be backported in 1.5 version.
2015-02-06 11:50:55 -05:00
}
MEDIUM: pattern: Extract the index process from the pat_parse_*() functions Now, the pat_parse_*() functions parses the incoming data. The input "pattern" struct can be preallocated. If the parser needs to add some buffers, it allocates memory. The function pattern_register() runs the call to the parser, process the key indexation and associate the "sample_storage" used by maps.
2013-12-06 13:06:43 -05:00
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
/* duplicate pattern */
memcpy(&patl->pat, pat, sizeof(*pat));
patl->pat.ptr.ptr = malloc(patl->pat.len);
if (!patl->pat.ptr.ptr) {
free(patl);
memprintf(err, "out of memory while indexing pattern");
return 0;
}
memcpy(patl->pat.ptr.ptr, pat->ptr.ptr, pat->len);
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
/* chain pattern in the expression */
CLEANUP: lists/tree-wide: rename some list operations to avoid some confusion The current "ADD" vs "ADDQ" is confusing because when thinking in terms of appending at the end of a list, "ADD" naturally comes to mind, but here it does the opposite, it inserts. Several times already it's been incorrectly used where ADDQ was expected, the latest of which was a fortunate accident explained in 6fa922562 ("CLEANUP: stream: explain why we queue the stream at the head of the server list"). Let's use more explicit (but slightly longer) names now: LIST_ADD -> LIST_INSERT LIST_ADDQ -> LIST_APPEND LIST_ADDED -> LIST_INLIST LIST_DEL -> LIST_DELETE The same is true for MT_LISTs, including their "TRY" variant. LIST_DEL_INIT keeps its short name to encourage to use it instead of the lazier LIST_DELETE which is often less safe. The change is large (~674 non-comment entries) but is mechanical enough to remain safe. No permutation was performed, so any out-of-tree code can easily map older names to new ones. The list doc was updated.
2021-04-21 01:32:39 -04:00
LIST_APPEND(&expr->patterns, &patl->list);
MEDIUM: map/acl: Improve pat_ref_set_elt() efficiency (for "set-map", "add-acl"action perfs) Store a pointer to the expression (struct pattern_expr) into the data structure used to chain/store the map element references (struct pat_ref_elt) , e.g. the struct pattern_tree when stored into an ebtree or struct pattern_list when chained to a list. Modify pat_ref_set_elt() to stop inspecting all the expressions attached to a map and to look for the <elt> element passed as parameter to retrieve the sample data to be parsed. Indeed, thanks to the pointer added above to each pattern tree nodes or list elements, they all can be inspected directly from the <elt> passed as parameter and its ->tree_head and ->list_head member: the pattern tree nodes are stored into elt->tree_head, and the pattern list elements are chained to elt->list_head list. This inspection was also the job of pattern_find_smp() which is no more useful. This patch removes the code of this function.
2023-08-23 09:58:26 -04:00
patl->expr = expr;
MEDIUM: pattern: link all final elements from the reference There is a data model issue in the current pattern design that makes pattern deletion extremely expensive: there's no direct way from a reference to access all indexed occurrences. As such, the only way to remove all indexed entries corresponding to a reference update is to scan all expressions's lists and trees to find a link to the reference. While this was possibly OK when map removal was not common and most maps were small, this is not conceivable anymore with GeoIP maps containing 10M+ entries and del-map operations that are triggered from http-request rulesets. This patch introduces two list heads from the pattern reference, one for the objects linked by lists and one for those linked by tree node. Ideally a single list would be enough but the linked elements are too much unrelated to be distinguished at the moment, so we'll need two lists. However for the long term a single-linked list will suffice but for now it's not possible due to the way elements are removed from expressions. As such this patch adds 32 bytes of memory usage per reference plus 16 per indexed entry, but both will be cut in half later. The links are not yet used for deletion, this patch only ensures the list is always consistent.
2020-11-02 06:10:48 -05:00
/* and from the reference */
MEDIUM: pattern: turn the pattern chaining to single-linked list It does not require heavy deletion from the expr anymore, so we can now turn this to a single-linked list since most of the time we want to delete all instances of a given pattern from the head. By doing so we save 32 bytes of memory per pattern. The pat_unlink_from_head() function was adjusted accordingly.
2020-11-03 08:50:29 -05:00
patl->from_ref = pat->ref->list_head;
pat->ref->list_head = &patl->from_ref;
MINOR: pattern: move the update revision to the pat_ref, not the expression It's not possible to uniquely update a single expression without updating the pattern reference, I don't know why we've put the revision in the expression back then, given that it in fact provides an update for a full pattern. Let's move the revision into the reference's head instead.
2020-11-02 09:26:51 -05:00
expr->ref->revision = rdtsc();
MINOR: map/acl: print the count of all the map/acl entries in "show map/acl" The output of "show map/acl" now contains the 'entry_cnt' value that represents the count of all the entries for each map/acl, not just the active ones, which means that it also includes entries currently being added.
2021-05-21 10:59:15 -04:00
expr->ref->entry_cnt++;
MEDIUM: pattern: Extract the index process from the pat_parse_*() functions Now, the pat_parse_*() functions parses the incoming data. The input "pattern" struct can be preallocated. If the parser needs to add some buffers, it allocates memory. The function pattern_register() runs the call to the parser, process the key indexation and associate the "sample_storage" used by maps.
2013-12-06 13:06:43 -05:00
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
/* that's ok */
return 1;
}
MEDIUM: pattern: Extract the index process from the pat_parse_*() functions Now, the pat_parse_*() functions parses the incoming data. The input "pattern" struct can be preallocated. If the parser needs to add some buffers, it allocates memory. The function pattern_register() runs the call to the parser, process the key indexation and associate the "sample_storage" used by maps.
2013-12-06 13:06:43 -05:00
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
int pat_idx_list_str(struct pattern_expr *expr, struct pattern *pat, char **err)
{
struct pattern_list *patl;
MEDIUM: pattern: Extract the index process from the pat_parse_*() functions Now, the pat_parse_*() functions parses the incoming data. The input "pattern" struct can be preallocated. If the parser needs to add some buffers, it allocates memory. The function pattern_register() runs the call to the parser, process the key indexation and associate the "sample_storage" used by maps.
2013-12-06 13:06:43 -05:00
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
/* allocate pattern */
patl = calloc(1, sizeof(*patl));
if (!patl) {
memprintf(err, "out of memory while indexing pattern");
return 0;
}
/* duplicate pattern */
memcpy(&patl->pat, pat, sizeof(*pat));
patl->pat.ptr.str = malloc(patl->pat.len + 1);
if (!patl->pat.ptr.str) {
free(patl);
memprintf(err, "out of memory while indexing pattern");
return 0;
}
memcpy(patl->pat.ptr.ptr, pat->ptr.ptr, pat->len);
patl->pat.ptr.str[patl->pat.len] = '\0';
MEDIUM: pattern: Extract the index process from the pat_parse_*() functions Now, the pat_parse_*() functions parses the incoming data. The input "pattern" struct can be preallocated. If the parser needs to add some buffers, it allocates memory. The function pattern_register() runs the call to the parser, process the key indexation and associate the "sample_storage" used by maps.
2013-12-06 13:06:43 -05:00
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
/* chain pattern in the expression */
CLEANUP: lists/tree-wide: rename some list operations to avoid some confusion The current "ADD" vs "ADDQ" is confusing because when thinking in terms of appending at the end of a list, "ADD" naturally comes to mind, but here it does the opposite, it inserts. Several times already it's been incorrectly used where ADDQ was expected, the latest of which was a fortunate accident explained in 6fa922562 ("CLEANUP: stream: explain why we queue the stream at the head of the server list"). Let's use more explicit (but slightly longer) names now: LIST_ADD -> LIST_INSERT LIST_ADDQ -> LIST_APPEND LIST_ADDED -> LIST_INLIST LIST_DEL -> LIST_DELETE The same is true for MT_LISTs, including their "TRY" variant. LIST_DEL_INIT keeps its short name to encourage to use it instead of the lazier LIST_DELETE which is often less safe. The change is large (~674 non-comment entries) but is mechanical enough to remain safe. No permutation was performed, so any out-of-tree code can easily map older names to new ones. The list doc was updated.
2021-04-21 01:32:39 -04:00
LIST_APPEND(&expr->patterns, &patl->list);
MEDIUM: map/acl: Improve pat_ref_set_elt() efficiency (for "set-map", "add-acl"action perfs) Store a pointer to the expression (struct pattern_expr) into the data structure used to chain/store the map element references (struct pat_ref_elt) , e.g. the struct pattern_tree when stored into an ebtree or struct pattern_list when chained to a list. Modify pat_ref_set_elt() to stop inspecting all the expressions attached to a map and to look for the <elt> element passed as parameter to retrieve the sample data to be parsed. Indeed, thanks to the pointer added above to each pattern tree nodes or list elements, they all can be inspected directly from the <elt> passed as parameter and its ->tree_head and ->list_head member: the pattern tree nodes are stored into elt->tree_head, and the pattern list elements are chained to elt->list_head list. This inspection was also the job of pattern_find_smp() which is no more useful. This patch removes the code of this function.
2023-08-23 09:58:26 -04:00
patl->expr = expr;
MEDIUM: pattern: link all final elements from the reference There is a data model issue in the current pattern design that makes pattern deletion extremely expensive: there's no direct way from a reference to access all indexed occurrences. As such, the only way to remove all indexed entries corresponding to a reference update is to scan all expressions's lists and trees to find a link to the reference. While this was possibly OK when map removal was not common and most maps were small, this is not conceivable anymore with GeoIP maps containing 10M+ entries and del-map operations that are triggered from http-request rulesets. This patch introduces two list heads from the pattern reference, one for the objects linked by lists and one for those linked by tree node. Ideally a single list would be enough but the linked elements are too much unrelated to be distinguished at the moment, so we'll need two lists. However for the long term a single-linked list will suffice but for now it's not possible due to the way elements are removed from expressions. As such this patch adds 32 bytes of memory usage per reference plus 16 per indexed entry, but both will be cut in half later. The links are not yet used for deletion, this patch only ensures the list is always consistent.
2020-11-02 06:10:48 -05:00
/* and from the reference */
MEDIUM: pattern: turn the pattern chaining to single-linked list It does not require heavy deletion from the expr anymore, so we can now turn this to a single-linked list since most of the time we want to delete all instances of a given pattern from the head. By doing so we save 32 bytes of memory per pattern. The pat_unlink_from_head() function was adjusted accordingly.
2020-11-03 08:50:29 -05:00
patl->from_ref = pat->ref->list_head;
pat->ref->list_head = &patl->from_ref;
MINOR: pattern: move the update revision to the pat_ref, not the expression It's not possible to uniquely update a single expression without updating the pattern reference, I don't know why we've put the revision in the expression back then, given that it in fact provides an update for a full pattern. Let's move the revision into the reference's head instead.
2020-11-02 09:26:51 -05:00
expr->ref->revision = rdtsc();
MINOR: map/acl: print the count of all the map/acl entries in "show map/acl" The output of "show map/acl" now contains the 'entry_cnt' value that represents the count of all the entries for each map/acl, not just the active ones, which means that it also includes entries currently being added.
2021-05-21 10:59:15 -04:00
expr->ref->entry_cnt++;
MEDIUM: pattern: Extract the index process from the pat_parse_*() functions Now, the pat_parse_*() functions parses the incoming data. The input "pattern" struct can be preallocated. If the parser needs to add some buffers, it allocates memory. The function pattern_register() runs the call to the parser, process the key indexation and associate the "sample_storage" used by maps.
2013-12-06 13:06:43 -05:00
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
/* that's ok */
return 1;
}
MINOR: map: Add regex matching replacement This patch declares a new map which provides a string based on a string with back references replaced by the content matched by the regex.
2016-02-10 16:55:20 -05:00
int pat_idx_list_reg_cap(struct pattern_expr *expr, struct pattern *pat, int cap, char **err)
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
{
struct pattern_list *patl;
MEDIUM: pattern: Extract the index process from the pat_parse_*() functions Now, the pat_parse_*() functions parses the incoming data. The input "pattern" struct can be preallocated. If the parser needs to add some buffers, it allocates memory. The function pattern_register() runs the call to the parser, process the key indexation and associate the "sample_storage" used by maps.
2013-12-06 13:06:43 -05:00
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
/* allocate pattern */
patl = calloc(1, sizeof(*patl));
if (!patl) {
memprintf(err, "out of memory while indexing pattern");
return 0;
MEDIUM: pattern: Change the prototype of the function pattern_register(). Each pattern parser take only one string. This change is reported to the function prototype of the function "pattern_register()". Now, it is called with just one string and no need to browse the array of args.
2014-01-23 11:53:31 -05:00
}
MEDIUM: pattern: Extract the index process from the pat_parse_*() functions Now, the pat_parse_*() functions parses the incoming data. The input "pattern" struct can be preallocated. If the parser needs to add some buffers, it allocates memory. The function pattern_register() runs the call to the parser, process the key indexation and associate the "sample_storage" used by maps.
2013-12-06 13:06:43 -05:00
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
/* duplicate pattern */
memcpy(&patl->pat, pat, sizeof(*pat));
/* compile regex */
MINOR: pattern: new sflag PAT_SF_REGFREE indicates regex_free() is needed Currently we have no way to know how to delete/prune a pattern in a generic way. A pattern doesn't contain its own type so we don't know what function to call. Tree nodes are roughly OK but not lists where regex are possible. Let's add one new bit for sflags at index time to indicate that regex_free() will be needed upon deletion. It's not used for now.
2020-11-02 13:16:23 -05:00
patl->pat.sflags |= PAT_SF_REGFREE;
MEDIUM: regex: modify regex_comp() to atomically allocate/free the my_regex struct Now we atomically allocate the my_regex struct within function regex_comp() and compile the regex or free both in case of failure. The pointer to the allocated my_regex struct is returned directly. The my_regex* argument to regex_comp() is removed. Function regex_free() was modified so that it systematically frees the my_regex entry. The function does nothing when called with a NULL as argument (like free()). It will avoid existing risk of not properly freeing the initialized area. Other structures are also updated in order to be compatible (the ones related to Lua and action rules).
2019-04-30 09:54:36 -04:00
if (!(patl->pat.ptr.reg = regex_comp(pat->ptr.str, !(expr->mflags & PAT_MF_IGNORE_CASE),
cap, err))) {
MINOR: fix a few memory usage errors These are either use after free errors or small leaks where memory is not free'd after some error state is detected.
2014-04-28 18:57:16 -04:00
free(patl);
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
return 0;
}
/* chain pattern in the expression */
CLEANUP: lists/tree-wide: rename some list operations to avoid some confusion The current "ADD" vs "ADDQ" is confusing because when thinking in terms of appending at the end of a list, "ADD" naturally comes to mind, but here it does the opposite, it inserts. Several times already it's been incorrectly used where ADDQ was expected, the latest of which was a fortunate accident explained in 6fa922562 ("CLEANUP: stream: explain why we queue the stream at the head of the server list"). Let's use more explicit (but slightly longer) names now: LIST_ADD -> LIST_INSERT LIST_ADDQ -> LIST_APPEND LIST_ADDED -> LIST_INLIST LIST_DEL -> LIST_DELETE The same is true for MT_LISTs, including their "TRY" variant. LIST_DEL_INIT keeps its short name to encourage to use it instead of the lazier LIST_DELETE which is often less safe. The change is large (~674 non-comment entries) but is mechanical enough to remain safe. No permutation was performed, so any out-of-tree code can easily map older names to new ones. The list doc was updated.
2021-04-21 01:32:39 -04:00
LIST_APPEND(&expr->patterns, &patl->list);
MEDIUM: map/acl: Improve pat_ref_set_elt() efficiency (for "set-map", "add-acl"action perfs) Store a pointer to the expression (struct pattern_expr) into the data structure used to chain/store the map element references (struct pat_ref_elt) , e.g. the struct pattern_tree when stored into an ebtree or struct pattern_list when chained to a list. Modify pat_ref_set_elt() to stop inspecting all the expressions attached to a map and to look for the <elt> element passed as parameter to retrieve the sample data to be parsed. Indeed, thanks to the pointer added above to each pattern tree nodes or list elements, they all can be inspected directly from the <elt> passed as parameter and its ->tree_head and ->list_head member: the pattern tree nodes are stored into elt->tree_head, and the pattern list elements are chained to elt->list_head list. This inspection was also the job of pattern_find_smp() which is no more useful. This patch removes the code of this function.
2023-08-23 09:58:26 -04:00
patl->expr = expr;
MEDIUM: pattern: link all final elements from the reference There is a data model issue in the current pattern design that makes pattern deletion extremely expensive: there's no direct way from a reference to access all indexed occurrences. As such, the only way to remove all indexed entries corresponding to a reference update is to scan all expressions's lists and trees to find a link to the reference. While this was possibly OK when map removal was not common and most maps were small, this is not conceivable anymore with GeoIP maps containing 10M+ entries and del-map operations that are triggered from http-request rulesets. This patch introduces two list heads from the pattern reference, one for the objects linked by lists and one for those linked by tree node. Ideally a single list would be enough but the linked elements are too much unrelated to be distinguished at the moment, so we'll need two lists. However for the long term a single-linked list will suffice but for now it's not possible due to the way elements are removed from expressions. As such this patch adds 32 bytes of memory usage per reference plus 16 per indexed entry, but both will be cut in half later. The links are not yet used for deletion, this patch only ensures the list is always consistent.
2020-11-02 06:10:48 -05:00
/* and from the reference */
MEDIUM: pattern: turn the pattern chaining to single-linked list It does not require heavy deletion from the expr anymore, so we can now turn this to a single-linked list since most of the time we want to delete all instances of a given pattern from the head. By doing so we save 32 bytes of memory per pattern. The pat_unlink_from_head() function was adjusted accordingly.
2020-11-03 08:50:29 -05:00
patl->from_ref = pat->ref->list_head;
pat->ref->list_head = &patl->from_ref;
MINOR: pattern: move the update revision to the pat_ref, not the expression It's not possible to uniquely update a single expression without updating the pattern reference, I don't know why we've put the revision in the expression back then, given that it in fact provides an update for a full pattern. Let's move the revision into the reference's head instead.
2020-11-02 09:26:51 -05:00
expr->ref->revision = rdtsc();
MINOR: map/acl: print the count of all the map/acl entries in "show map/acl" The output of "show map/acl" now contains the 'entry_cnt' value that represents the count of all the entries for each map/acl, not just the active ones, which means that it also includes entries currently being added.
2021-05-21 10:59:15 -04:00
expr->ref->entry_cnt++;
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
/* that's ok */
return 1;
}
MINOR: map: Add regex matching replacement This patch declares a new map which provides a string based on a string with back references replaced by the content matched by the regex.
2016-02-10 16:55:20 -05:00
int pat_idx_list_reg(struct pattern_expr *expr, struct pattern *pat, char **err)
{
return pat_idx_list_reg_cap(expr, pat, 0, err);
}
int pat_idx_list_regm(struct pattern_expr *expr, struct pattern *pat, char **err)
{
return pat_idx_list_reg_cap(expr, pat, 1, err);
}
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
int pat_idx_tree_ip(struct pattern_expr *expr, struct pattern *pat, char **err)
{
unsigned int mask;
MINOR: pattern: Rename "pat_idx_elt" to "pattern_tree" This is just for having coherent struct names.
2013-12-13 10:09:50 -05:00
struct pattern_tree *node;
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
/* Only IPv4 can be indexed */
if (pat->type == SMP_T_IPV4) {
MEDIUM: pattern: Change the prototype of the function pattern_register(). Each pattern parser take only one string. This change is reported to the function prototype of the function "pattern_register()". Now, it is called with just one string and no need to browse the array of args.
2014-01-23 11:53:31 -05:00
/* in IPv4 case, check if the mask is contiguous so that we can
* insert the network into the tree. A continuous mask has only
* ones on the left. This means that this mask + its lower bit
* added once again is null.
*/
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
mask = ntohl(pat->val.ipv4.mask.s_addr);
if (mask + (mask & -mask) == 0) {
mask = mask ? 33 - flsnz(mask & -mask) : 0; /* equals cidr value */
/* node memory allocation */
node = calloc(1, sizeof(*node) + 4);
if (!node) {
memprintf(err, "out of memory while loading pattern");
return 0;
}
MEDIUM: pattern: Extract the index process from the pat_parse_*() functions Now, the pat_parse_*() functions parses the incoming data. The input "pattern" struct can be preallocated. If the parser needs to add some buffers, it allocates memory. The function pattern_register() runs the call to the parser, process the key indexation and associate the "sample_storage" used by maps.
2013-12-06 13:06:43 -05:00
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
/* copy the pointer to sample associated to this node */
MINOR: samples: rename some struct member from "smp" to "data" This members contains data and not sample.
2015-08-19 02:35:43 -04:00
node->data = pat->data;
MINOR: pattern: Each pattern expression element store the reference struct. Now, each pattern entry known the original "struct pat_ref_elt" from that was built. This patch permit to delete each pattern entry without confusion. After this patch, each reference can use his pointer to be targeted.
2014-01-28 09:54:36 -05:00
node->ref = pat->ref;
MEDIUM: pattern: Extract the index process from the pat_parse_*() functions Now, the pat_parse_*() functions parses the incoming data. The input "pattern" struct can be preallocated. If the parser needs to add some buffers, it allocates memory. The function pattern_register() runs the call to the parser, process the key indexation and associate the "sample_storage" used by maps.
2013-12-06 13:06:43 -05:00
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
/* FIXME: insert <addr>/<mask> into the tree here */
memcpy(node->node.key, &pat->val.ipv4.addr, 4); /* network byte order */
node->node.node.pfx = mask;
MINOR: pattern: index duplicates The indexation functions now accept duplicates. This way it is possible to always have some consistency between lists and trees. The "add" command will always add regardless of any previous existence. The new entry will not be used because both trees and list retrieve keys in insertion order. Thus the "add" operation will always succeed (as long as there is enough memory).
2014-01-29 18:27:15 -05:00
/* Insert the entry. */
ebmb_insert_prefix(&expr->pattern_tree, &node->node, 4);
MEDIUM: map/acl: Improve pat_ref_set_elt() efficiency (for "set-map", "add-acl"action perfs) Store a pointer to the expression (struct pattern_expr) into the data structure used to chain/store the map element references (struct pat_ref_elt) , e.g. the struct pattern_tree when stored into an ebtree or struct pattern_list when chained to a list. Modify pat_ref_set_elt() to stop inspecting all the expressions attached to a map and to look for the <elt> element passed as parameter to retrieve the sample data to be parsed. Indeed, thanks to the pointer added above to each pattern tree nodes or list elements, they all can be inspected directly from the <elt> passed as parameter and its ->tree_head and ->list_head member: the pattern tree nodes are stored into elt->tree_head, and the pattern list elements are chained to elt->list_head list. This inspection was also the job of pattern_find_smp() which is no more useful. This patch removes the code of this function.
2023-08-23 09:58:26 -04:00
node->expr = expr;
MEDIUM: pattern: turn the pattern chaining to single-linked list It does not require heavy deletion from the expr anymore, so we can now turn this to a single-linked list since most of the time we want to delete all instances of a given pattern from the head. By doing so we save 32 bytes of memory per pattern. The pat_unlink_from_head() function was adjusted accordingly.
2020-11-03 08:50:29 -05:00
node->from_ref = pat->ref->tree_head;
pat->ref->tree_head = &node->from_ref;
MINOR: pattern: move the update revision to the pat_ref, not the expression It's not possible to uniquely update a single expression without updating the pattern reference, I don't know why we've put the revision in the expression back then, given that it in fact provides an update for a full pattern. Let's move the revision into the reference's head instead.
2020-11-02 09:26:51 -05:00
expr->ref->revision = rdtsc();
MINOR: map/acl: print the count of all the map/acl entries in "show map/acl" The output of "show map/acl" now contains the 'entry_cnt' value that represents the count of all the entries for each map/acl, not just the active ones, which means that it also includes entries currently being added.
2021-05-21 10:59:15 -04:00
expr->ref->entry_cnt++;
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
/* that's ok */
return 1;
}
MEDIUM: pattern: Index IPv6 addresses in a tree. This commit adds second tree node in the pattern struct and use it to index IPv6 addresses. This commit report feature used in the list. If IPv4 not match the tree, try to convert the IPv4 address in IPv6 with prefixing the IPv4 address by "::ffff", after this operation, the match function try lookup in the IPv6 tree. If the IPv6 sample dont match the IPv6 tree, try to convert the IPv6 addresses prefixed by "2002:IPv4", "::ffff:IPv4" and "::0000:IPv4" in IPv4 address. after this operation, the match function try lookup in the IPv4 tree.
2013-12-19 17:54:54 -05:00
else {
/* If the mask is not contiguous, just add the pattern to the list */
return pat_idx_list_val(expr, pat, err);
}
}
else if (pat->type == SMP_T_IPV6) {
/* IPv6 also can be indexed */
node = calloc(1, sizeof(*node) + 16);
if (!node) {
memprintf(err, "out of memory while loading pattern");
return 0;
}
/* copy the pointer to sample associated to this node */
MINOR: samples: rename some struct member from "smp" to "data" This members contains data and not sample.
2015-08-19 02:35:43 -04:00
node->data = pat->data;
MINOR: pattern: Each pattern expression element store the reference struct. Now, each pattern entry known the original "struct pat_ref_elt" from that was built. This patch permit to delete each pattern entry without confusion. After this patch, each reference can use his pointer to be targeted.
2014-01-28 09:54:36 -05:00
node->ref = pat->ref;
MEDIUM: pattern: Index IPv6 addresses in a tree. This commit adds second tree node in the pattern struct and use it to index IPv6 addresses. This commit report feature used in the list. If IPv4 not match the tree, try to convert the IPv4 address in IPv6 with prefixing the IPv4 address by "::ffff", after this operation, the match function try lookup in the IPv6 tree. If the IPv6 sample dont match the IPv6 tree, try to convert the IPv6 addresses prefixed by "2002:IPv4", "::ffff:IPv4" and "::0000:IPv4" in IPv4 address. after this operation, the match function try lookup in the IPv4 tree.
2013-12-19 17:54:54 -05:00
/* FIXME: insert <addr>/<mask> into the tree here */
memcpy(node->node.key, &pat->val.ipv6.addr, 16); /* network byte order */
node->node.node.pfx = pat->val.ipv6.mask;
MINOR: pattern: index duplicates The indexation functions now accept duplicates. This way it is possible to always have some consistency between lists and trees. The "add" command will always add regardless of any previous existence. The new entry will not be used because both trees and list retrieve keys in insertion order. Thus the "add" operation will always succeed (as long as there is enough memory).
2014-01-29 18:27:15 -05:00
/* Insert the entry. */
ebmb_insert_prefix(&expr->pattern_tree_2, &node->node, 16);
MEDIUM: map/acl: Improve pat_ref_set_elt() efficiency (for "set-map", "add-acl"action perfs) Store a pointer to the expression (struct pattern_expr) into the data structure used to chain/store the map element references (struct pat_ref_elt) , e.g. the struct pattern_tree when stored into an ebtree or struct pattern_list when chained to a list. Modify pat_ref_set_elt() to stop inspecting all the expressions attached to a map and to look for the <elt> element passed as parameter to retrieve the sample data to be parsed. Indeed, thanks to the pointer added above to each pattern tree nodes or list elements, they all can be inspected directly from the <elt> passed as parameter and its ->tree_head and ->list_head member: the pattern tree nodes are stored into elt->tree_head, and the pattern list elements are chained to elt->list_head list. This inspection was also the job of pattern_find_smp() which is no more useful. This patch removes the code of this function.
2023-08-23 09:58:26 -04:00
node->expr = expr;
MEDIUM: pattern: turn the pattern chaining to single-linked list It does not require heavy deletion from the expr anymore, so we can now turn this to a single-linked list since most of the time we want to delete all instances of a given pattern from the head. By doing so we save 32 bytes of memory per pattern. The pat_unlink_from_head() function was adjusted accordingly.
2020-11-03 08:50:29 -05:00
node->from_ref = pat->ref->tree_head;
pat->ref->tree_head = &node->from_ref;
MINOR: pattern: move the update revision to the pat_ref, not the expression It's not possible to uniquely update a single expression without updating the pattern reference, I don't know why we've put the revision in the expression back then, given that it in fact provides an update for a full pattern. Let's move the revision into the reference's head instead.
2020-11-02 09:26:51 -05:00
expr->ref->revision = rdtsc();
MINOR: map/acl: print the count of all the map/acl entries in "show map/acl" The output of "show map/acl" now contains the 'entry_cnt' value that represents the count of all the entries for each map/acl, not just the active ones, which means that it also includes entries currently being added.
2021-05-21 10:59:15 -04:00
expr->ref->entry_cnt++;
MEDIUM: pattern: Index IPv6 addresses in a tree. This commit adds second tree node in the pattern struct and use it to index IPv6 addresses. This commit report feature used in the list. If IPv4 not match the tree, try to convert the IPv4 address in IPv6 with prefixing the IPv4 address by "::ffff", after this operation, the match function try lookup in the IPv6 tree. If the IPv6 sample dont match the IPv6 tree, try to convert the IPv6 addresses prefixed by "2002:IPv4", "::ffff:IPv4" and "::0000:IPv4" in IPv4 address. after this operation, the match function try lookup in the IPv4 tree.
2013-12-19 17:54:54 -05:00
/* that's ok */
return 1;
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
}
MEDIUM: pattern: Extract the index process from the pat_parse_*() functions Now, the pat_parse_*() functions parses the incoming data. The input "pattern" struct can be preallocated. If the parser needs to add some buffers, it allocates memory. The function pattern_register() runs the call to the parser, process the key indexation and associate the "sample_storage" used by maps.
2013-12-06 13:06:43 -05:00
MEDIUM: pattern: Index IPv6 addresses in a tree. This commit adds second tree node in the pattern struct and use it to index IPv6 addresses. This commit report feature used in the list. If IPv4 not match the tree, try to convert the IPv4 address in IPv6 with prefixing the IPv4 address by "::ffff", after this operation, the match function try lookup in the IPv6 tree. If the IPv6 sample dont match the IPv6 tree, try to convert the IPv6 addresses prefixed by "2002:IPv4", "::ffff:IPv4" and "::0000:IPv4" in IPv4 address. after this operation, the match function try lookup in the IPv4 tree.
2013-12-19 17:54:54 -05:00
return 0;
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
}
int pat_idx_tree_str(struct pattern_expr *expr, struct pattern *pat, char **err)
{
int len;
MINOR: pattern: Rename "pat_idx_elt" to "pattern_tree" This is just for having coherent struct names.
2013-12-13 10:09:50 -05:00
struct pattern_tree *node;
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
/* Only string can be indexed */
MEDIUM: sample: Remove types SMP_T_CSTR and SMP_T_CBIN, replace it by SMP_F_CONST flags The operations applied on types SMP_T_CSTR and SMP_T_STR are the same, but the check code and the declarations are double, because it must declare action for SMP_T_C* and SMP_T_*. The declared actions and checks are the same. this complexify the code. Only the "conv" functions can change from "C*" to "*" Now, if a function needs to modify input string, it can call the new function smp_dup(). This one duplicate data in a trash buffer.
2013-12-16 18:20:33 -05:00
if (pat->type != SMP_T_STR) {
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
memprintf(err, "internal error: string expected, but the type is '%s'",
smp_to_type[pat->type]);
return 0;
MEDIUM: pattern: Change the prototype of the function pattern_register(). Each pattern parser take only one string. This change is reported to the function prototype of the function "pattern_register()". Now, it is called with just one string and no need to browse the array of args.
2014-01-23 11:53:31 -05:00
}
MEDIUM: pattern: Extract the index process from the pat_parse_*() functions Now, the pat_parse_*() functions parses the incoming data. The input "pattern" struct can be preallocated. If the parser needs to add some buffers, it allocates memory. The function pattern_register() runs the call to the parser, process the key indexation and associate the "sample_storage" used by maps.
2013-12-06 13:06:43 -05:00
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
/* If the flag PAT_F_IGNORE_CASE is set, we cannot use trees */
BUG/MEDIUM: patterns: last fix was still not enough Last fix did address the issue for inlined patterns, but it was not enough because the flags are lost as well when updating patterns dynamically over the CLI. Also if the same file was used once with -i and another time without -i, their references would have been merged and both would have used the same matching method. It's appear that the patterns have two types of flags. The first ones are relative to the pattern matching, and the second are relative to the pattern storage. The pattern matching flags are the same for all the patterns of one expression. Now they are stored in the expression. The storage flags are information returned by the pattern mathing function. This information is relative to each entry and is stored in the "struct pattern". Now, the expression matching flags are forwarded to the parse and index functions. These flags are stored during the configuration parsing, and they are used during the parse and index actions. This issue was introduced in dev23 with the major pattern rework, and is a continuation of commit a631fc8 ("BUG/MAJOR: patterns: -i and -n are ignored for inlined patterns"). No backport is needed.
2014-04-28 05:18:57 -04:00
if (expr->mflags & PAT_MF_IGNORE_CASE)
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
return pat_idx_list_str(expr, pat, err);
/* Process the key len */
len = strlen(pat->ptr.str) + 1;
MEDIUM: pattern: Extract the index process from the pat_parse_*() functions Now, the pat_parse_*() functions parses the incoming data. The input "pattern" struct can be preallocated. If the parser needs to add some buffers, it allocates memory. The function pattern_register() runs the call to the parser, process the key indexation and associate the "sample_storage" used by maps.
2013-12-06 13:06:43 -05:00
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
/* node memory allocation */
node = calloc(1, sizeof(*node) + len);
if (!node) {
memprintf(err, "out of memory while loading pattern");
return 0;
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
}
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
/* copy the pointer to sample associated to this node */
MINOR: samples: rename some struct member from "smp" to "data" This members contains data and not sample.
2015-08-19 02:35:43 -04:00
node->data = pat->data;
MINOR: pattern: Each pattern expression element store the reference struct. Now, each pattern entry known the original "struct pat_ref_elt" from that was built. This patch permit to delete each pattern entry without confusion. After this patch, each reference can use his pointer to be targeted.
2014-01-28 09:54:36 -05:00
node->ref = pat->ref;
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
/* copy the string */
memcpy(node->node.key, pat->ptr.str, len);
/* index the new node */
MINOR: pattern: index duplicates The indexation functions now accept duplicates. This way it is possible to always have some consistency between lists and trees. The "add" command will always add regardless of any previous existence. The new entry will not be used because both trees and list retrieve keys in insertion order. Thus the "add" operation will always succeed (as long as there is enough memory).
2014-01-29 18:27:15 -05:00
ebst_insert(&expr->pattern_tree, &node->node);
MEDIUM: map/acl: Improve pat_ref_set_elt() efficiency (for "set-map", "add-acl"action perfs) Store a pointer to the expression (struct pattern_expr) into the data structure used to chain/store the map element references (struct pat_ref_elt) , e.g. the struct pattern_tree when stored into an ebtree or struct pattern_list when chained to a list. Modify pat_ref_set_elt() to stop inspecting all the expressions attached to a map and to look for the <elt> element passed as parameter to retrieve the sample data to be parsed. Indeed, thanks to the pointer added above to each pattern tree nodes or list elements, they all can be inspected directly from the <elt> passed as parameter and its ->tree_head and ->list_head member: the pattern tree nodes are stored into elt->tree_head, and the pattern list elements are chained to elt->list_head list. This inspection was also the job of pattern_find_smp() which is no more useful. This patch removes the code of this function.
2023-08-23 09:58:26 -04:00
node->expr = expr;
MEDIUM: pattern: turn the pattern chaining to single-linked list It does not require heavy deletion from the expr anymore, so we can now turn this to a single-linked list since most of the time we want to delete all instances of a given pattern from the head. By doing so we save 32 bytes of memory per pattern. The pat_unlink_from_head() function was adjusted accordingly.
2020-11-03 08:50:29 -05:00
node->from_ref = pat->ref->tree_head;
pat->ref->tree_head = &node->from_ref;
MINOR: pattern: move the update revision to the pat_ref, not the expression It's not possible to uniquely update a single expression without updating the pattern reference, I don't know why we've put the revision in the expression back then, given that it in fact provides an update for a full pattern. Let's move the revision into the reference's head instead.
2020-11-02 09:26:51 -05:00
expr->ref->revision = rdtsc();
MINOR: map/acl: print the count of all the map/acl entries in "show map/acl" The output of "show map/acl" now contains the 'entry_cnt' value that represents the count of all the entries for each map/acl, not just the active ones, which means that it also includes entries currently being added.
2021-05-21 10:59:15 -04:00
expr->ref->entry_cnt++;
MEDIUM: pattern: add indexation function. Before this patch, the indexation function check the declared patttern matching function and index the data according with this function. This is not useful to add some indexation mode. This commit adds dedicated indexation function. Each struct pattern is associated with one indexation function. This function permit to index data according with the type of pattern and with the type of match.
2013-12-13 09:12:32 -05:00
/* that's ok */
return 1;
}
MEDIUM: pattern: use ebtree's longest match to index/lookup string beginning Being able to map prefixes to values is already used for IPv4/IPv6 but was not yet used with strings. It can be very convenient to map directories to server farms but large lists may be slow. By using ebmb_insert_prefix() and ebmb_lookup_longest(), we can insert strings with their own length as a prefix, and lookup candidate strings and ensure that the longest matching one will be returned, which is the longest string matching the entry.
2014-05-10 02:53:48 -04:00
int pat_idx_tree_pfx(struct pattern_expr *expr, struct pattern *pat, char **err)
{
int len;
struct pattern_tree *node;
/* Only string can be indexed */
if (pat->type != SMP_T_STR) {
memprintf(err, "internal error: string expected, but the type is '%s'",
smp_to_type[pat->type]);
return 0;
}
/* If the flag PAT_F_IGNORE_CASE is set, we cannot use trees */
if (expr->mflags & PAT_MF_IGNORE_CASE)
return pat_idx_list_str(expr, pat, err);
/* Process the key len */
len = strlen(pat->ptr.str);
/* node memory allocation */
node = calloc(1, sizeof(*node) + len + 1);
if (!node) {
memprintf(err, "out of memory while loading pattern");
return 0;
}
/* copy the pointer to sample associated to this node */
MINOR: samples: rename some struct member from "smp" to "data" This members contains data and not sample.
2015-08-19 02:35:43 -04:00
node->data = pat->data;
MEDIUM: pattern: use ebtree's longest match to index/lookup string beginning Being able to map prefixes to values is already used for IPv4/IPv6 but was not yet used with strings. It can be very convenient to map directories to server farms but large lists may be slow. By using ebmb_insert_prefix() and ebmb_lookup_longest(), we can insert strings with their own length as a prefix, and lookup candidate strings and ensure that the longest matching one will be returned, which is the longest string matching the entry.
2014-05-10 02:53:48 -04:00
node->ref = pat->ref;
/* copy the string and the trailing zero */
memcpy(node->node.key, pat->ptr.str, len + 1);
node->node.node.pfx = len * 8;
/* index the new node */
ebmb_insert_prefix(&expr->pattern_tree, &node->node, len);
MEDIUM: map/acl: Improve pat_ref_set_elt() efficiency (for "set-map", "add-acl"action perfs) Store a pointer to the expression (struct pattern_expr) into the data structure used to chain/store the map element references (struct pat_ref_elt) , e.g. the struct pattern_tree when stored into an ebtree or struct pattern_list when chained to a list. Modify pat_ref_set_elt() to stop inspecting all the expressions attached to a map and to look for the <elt> element passed as parameter to retrieve the sample data to be parsed. Indeed, thanks to the pointer added above to each pattern tree nodes or list elements, they all can be inspected directly from the <elt> passed as parameter and its ->tree_head and ->list_head member: the pattern tree nodes are stored into elt->tree_head, and the pattern list elements are chained to elt->list_head list. This inspection was also the job of pattern_find_smp() which is no more useful. This patch removes the code of this function.
2023-08-23 09:58:26 -04:00
node->expr = expr;
MEDIUM: pattern: turn the pattern chaining to single-linked list It does not require heavy deletion from the expr anymore, so we can now turn this to a single-linked list since most of the time we want to delete all instances of a given pattern from the head. By doing so we save 32 bytes of memory per pattern. The pat_unlink_from_head() function was adjusted accordingly.
2020-11-03 08:50:29 -05:00
node->from_ref = pat->ref->tree_head;
pat->ref->tree_head = &node->from_ref;
MINOR: pattern: move the update revision to the pat_ref, not the expression It's not possible to uniquely update a single expression without updating the pattern reference, I don't know why we've put the revision in the expression back then, given that it in fact provides an update for a full pattern. Let's move the revision into the reference's head instead.
2020-11-02 09:26:51 -05:00
expr->ref->revision = rdtsc();
MINOR: map/acl: print the count of all the map/acl entries in "show map/acl" The output of "show map/acl" now contains the 'entry_cnt' value that represents the count of all the entries for each map/acl, not just the active ones, which means that it also includes entries currently being added.
2021-05-21 10:59:15 -04:00
expr->ref->entry_cnt++;
MEDIUM: pattern: use ebtree's longest match to index/lookup string beginning Being able to map prefixes to values is already used for IPv4/IPv6 but was not yet used with strings. It can be very convenient to map directories to server farms but large lists may be slow. By using ebmb_insert_prefix() and ebmb_lookup_longest(), we can insert strings with their own length as a prefix, and lookup candidate strings and ensure that the longest matching one will be returned, which is the longest string matching the entry.
2014-05-10 02:53:48 -04:00
/* that's ok */
return 1;
}
MINOR: pattern: remerge the list and tree deletion functions pat_del_tree_gen() was already chained onto pat_del_list_gen() to deal with remaining cases, so let's complete the merge and have a generic pattern deletion function acting on the reference and taking care of reliably removing all elements.
2020-11-02 13:53:16 -05:00
/* Deletes all patterns from reference <elt>. Note that all of their
MEDIUM: pattern: change the pat_del_* functions to delete from the references This is the next step in speeding up entry removal. Now we don't scan the whole lists or trees for elements pointing to the target reference, instead we start from the reference and delete all linked patterns. This simplifies some delete functions since we don't need anymore to delete multiple times from an expression since all nodes appear after the reference element. We can now have one generic list and one generic tree deletion function. This required the replacement of pattern_delete() with an open-coded version since we now need to lock all expressions first before proceeding. This means there is a high risk of lock inversion here but given that the expressions are always scanned in the same order from the same head, this must not happen. Now deleting first entries is instantaneous, and it's still slow to delete the last ones when looking up their ID since it still requires to look them up by a full scan, but it's already way faster than previously. Typically removing the last 10 IP from a 20M entries ACL with a full-scan each took less than 2 seconds. It would be technically possible to make use of indexed entries to speed up most lookups for removal by value (e.g. IP addresses) but that's for later.
2020-11-02 07:55:22 -05:00
* expressions must be locked, and the pattern lock must be held as well.
*/
MINOR: pattern: remerge the list and tree deletion functions pat_del_tree_gen() was already chained onto pat_del_list_gen() to deal with remaining cases, so let's complete the merge and have a generic pattern deletion function acting on the reference and taking care of reliably removing all elements.
2020-11-02 13:53:16 -05:00
void pat_delete_gen(struct pat_ref *ref, struct pat_ref_elt *elt)
MEDIUM: pattern: add delete functions This commit adds a delete function for patterns. It looks up all instances of the pattern to delete and deletes them all. The fetch keyword declarations have been extended to point to the appropriate delete function.
2014-01-15 05:38:49 -05:00
{
MEDIUM: pattern: turn the pattern chaining to single-linked list It does not require heavy deletion from the expr anymore, so we can now turn this to a single-linked list since most of the time we want to delete all instances of a given pattern from the head. By doing so we save 32 bytes of memory per pattern. The pat_unlink_from_head() function was adjusted accordingly.
2020-11-03 08:50:29 -05:00
struct pattern_tree *tree;
struct pattern_list *pat;
void **node;
MINOR: pattern: remerge the list and tree deletion functions pat_del_tree_gen() was already chained onto pat_del_list_gen() to deal with remaining cases, so let's complete the merge and have a generic pattern deletion function acting on the reference and taking care of reliably removing all elements.
2020-11-02 13:53:16 -05:00
/* delete all known tree nodes. They are all allocated inline */
MEDIUM: pattern: turn the pattern chaining to single-linked list It does not require heavy deletion from the expr anymore, so we can now turn this to a single-linked list since most of the time we want to delete all instances of a given pattern from the head. By doing so we save 32 bytes of memory per pattern. The pat_unlink_from_head() function was adjusted accordingly.
2020-11-03 08:50:29 -05:00
for (node = elt->tree_head; node;) {
tree = container_of(node, struct pattern_tree, from_ref);
node = *node;
MINOR: pattern: remerge the list and tree deletion functions pat_del_tree_gen() was already chained onto pat_del_list_gen() to deal with remaining cases, so let's complete the merge and have a generic pattern deletion function acting on the reference and taking care of reliably removing all elements.
2020-11-02 13:53:16 -05:00
BUG_ON(tree->ref != elt);
ebmb_delete(&tree->node);
free(tree->data);
free(tree);
}
MEDIUM: pattern: add delete functions This commit adds a delete function for patterns. It looks up all instances of the pattern to delete and deletes them all. The fetch keyword declarations have been extended to point to the appropriate delete function.
2014-01-15 05:38:49 -05:00
MINOR: pattern: remerge the list and tree deletion functions pat_del_tree_gen() was already chained onto pat_del_list_gen() to deal with remaining cases, so let's complete the merge and have a generic pattern deletion function acting on the reference and taking care of reliably removing all elements.
2020-11-02 13:53:16 -05:00
/* delete all list nodes and free their pattern entries (str/reg) */
MEDIUM: pattern: turn the pattern chaining to single-linked list It does not require heavy deletion from the expr anymore, so we can now turn this to a single-linked list since most of the time we want to delete all instances of a given pattern from the head. By doing so we save 32 bytes of memory per pattern. The pat_unlink_from_head() function was adjusted accordingly.
2020-11-03 08:50:29 -05:00
for (node = elt->list_head; node;) {
pat = container_of(node, struct pattern_list, from_ref);
node = *node;
MEDIUM: pattern: change the pat_del_* functions to delete from the references This is the next step in speeding up entry removal. Now we don't scan the whole lists or trees for elements pointing to the target reference, instead we start from the reference and delete all linked patterns. This simplifies some delete functions since we don't need anymore to delete multiple times from an expression since all nodes appear after the reference element. We can now have one generic list and one generic tree deletion function. This required the replacement of pattern_delete() with an open-coded version since we now need to lock all expressions first before proceeding. This means there is a high risk of lock inversion here but given that the expressions are always scanned in the same order from the same head, this must not happen. Now deleting first entries is instantaneous, and it's still slow to delete the last ones when looking up their ID since it still requires to look them up by a full scan, but it's already way faster than previously. Typically removing the last 10 IP from a 20M entries ACL with a full-scan each took less than 2 seconds. It would be technically possible to make use of indexed entries to speed up most lookups for removal by value (e.g. IP addresses) but that's for later.
2020-11-02 07:55:22 -05:00
BUG_ON(pat->pat.ref != elt);
MEDIUM: pattern: add delete functions This commit adds a delete function for patterns. It looks up all instances of the pattern to delete and deletes them all. The fetch keyword declarations have been extended to point to the appropriate delete function.
2014-01-15 05:38:49 -05:00
/* Delete and free entry. */
CLEANUP: lists/tree-wide: rename some list operations to avoid some confusion The current "ADD" vs "ADDQ" is confusing because when thinking in terms of appending at the end of a list, "ADD" naturally comes to mind, but here it does the opposite, it inserts. Several times already it's been incorrectly used where ADDQ was expected, the latest of which was a fortunate accident explained in 6fa922562 ("CLEANUP: stream: explain why we queue the stream at the head of the server list"). Let's use more explicit (but slightly longer) names now: LIST_ADD -> LIST_INSERT LIST_ADDQ -> LIST_APPEND LIST_ADDED -> LIST_INLIST LIST_DEL -> LIST_DELETE The same is true for MT_LISTs, including their "TRY" variant. LIST_DEL_INIT keeps its short name to encourage to use it instead of the lazier LIST_DELETE which is often less safe. The change is large (~674 non-comment entries) but is mechanical enough to remain safe. No permutation was performed, so any out-of-tree code can easily map older names to new ones. The list doc was updated.
2021-04-21 01:32:39 -04:00
LIST_DELETE(&pat->list);
MINOR: pattern: make the delete and prune functions more generic Now we have a single prune() function to act on an expression, and one delete function for the lists and one for the trees. The presence of a pointer in the lists is enough to warrant a free, and we rely on the PAT_SF_REGFREE flag to decide whether to free using free() or regfree().
2020-11-02 13:26:02 -05:00
if (pat->pat.sflags & PAT_SF_REGFREE)
regex_free(pat->pat.ptr.reg);
else
free(pat->pat.ptr.ptr);
MINOR: samples: rename some struct member from "smp" to "data" This members contains data and not sample.
2015-08-19 02:35:43 -04:00
free(pat->pat.data);
MEDIUM: pattern: add delete functions This commit adds a delete function for patterns. It looks up all instances of the pattern to delete and deletes them all. The fetch keyword declarations have been extended to point to the appropriate delete function.
2014-01-15 05:38:49 -05:00
free(pat);
}
MINOR: pattern: remerge the list and tree deletion functions pat_del_tree_gen() was already chained onto pat_del_list_gen() to deal with remaining cases, so let's complete the merge and have a generic pattern deletion function acting on the reference and taking care of reliably removing all elements.
2020-11-02 13:53:16 -05:00
/* update revision number to refresh the cache */
ref->revision = rdtsc();
MINOR: map/acl: print the count of all the map/acl entries in "show map/acl" The output of "show map/acl" now contains the 'entry_cnt' value that represents the count of all the entries for each map/acl, not just the active ones, which means that it also includes entries currently being added.
2021-05-21 10:59:15 -04:00
ref->entry_cnt--;
MEDIUM: pattern: turn the pattern chaining to single-linked list It does not require heavy deletion from the expr anymore, so we can now turn this to a single-linked list since most of the time we want to delete all instances of a given pattern from the head. By doing so we save 32 bytes of memory per pattern. The pat_unlink_from_head() function was adjusted accordingly.
2020-11-03 08:50:29 -05:00
elt->tree_head = NULL;
elt->list_head = NULL;
MEDIUM: pattern: add delete functions This commit adds a delete function for patterns. It looks up all instances of the pattern to delete and deletes them all. The fetch keyword declarations have been extended to point to the appropriate delete function.
2014-01-15 05:38:49 -05:00
}
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
void pattern_init_expr(struct pattern_expr *expr)
{
LIST_INIT(&expr->patterns);
MINOR: pattern: index duplicates The indexation functions now accept duplicates. This way it is possible to always have some consistency between lists and trees. The "add" command will always add regardless of any previous existence. The new entry will not be used because both trees and list retrieve keys in insertion order. Thus the "add" operation will always succeed (as long as there is enough memory).
2014-01-29 18:27:15 -05:00
expr->pattern_tree = EB_ROOT;
expr->pattern_tree_2 = EB_ROOT;
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
}
void pattern_init_head(struct pattern_head *head)
{
LIST_INIT(&head->head);
}
/* The following functions are relative to the management of the reference
* lists. These lists are used to store the original pattern and associated
* value as string form.
*
* This is used with modifiable ACL and MAPS
MINOR: pattern: Each pattern is identified by unique id. The pattern reference are stored with two identifiers: the unique_id and the reference. The reference identify a file. Each file with the same name point to the same reference. We can register many times one file. If the file is modified, all his dependencies are also modified. The reference can be used with map or acl. The unique_id identify inline acl. The unique id is unique for each acl. You cannot force the same id in the configuration file, because this repport an error. The format of the acl and map listing through the "socket" has changed for displaying these new ids.
2014-03-11 09:29:22 -04:00
*
* The pattern reference are stored with two identifiers: the unique_id and
* the reference.
*
* The reference identify a file. Each file with the same name point to the
* same reference. We can register many times one file. If the file is modified,
* all his dependencies are also modified. The reference can be used with map or
* acl.
*
* The unique_id identify inline acl. The unique id is unique for each acl.
* You cannot force the same id in the configuration file, because this repoort
* an error.
*
* A particular case appears if the filename is a number. In this case, the
* unique_id is set with the number represented by the filename and the
* reference is also set. This method prevent double unique_id.
*
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
*/
CLEANUP: pattern: fix spelling/grammatical/copy-paste in comments The code is horrible to work with because most functions are documented with misleading comments resulting from many spelling and grammatical mistakes, and plenty of remains of copy-paste mentioning arguments that do not exist and return values that are never set. Too many hours wasted writing non-working code because of assumptions resulting from this, let's fix this once for all now!
2020-10-30 11:03:50 -04:00
/* This function looks up a reference by name. If the reference is found, a
* pointer to the struct pat_ref is returned, otherwise NULL is returned.
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
*/
struct pat_ref *pat_ref_lookup(const char *reference)
{
struct pat_ref *ref;
BUG/MINOR: pattern: Fix pattern lookup for map with opt@ prefix When we look for a map file reference, the file@ prefix is removed because if may be omitted. The same is true with opt@ prefix. However this case was not properly performed in pat_ref_lookup(). Let's do so. This patch must be backported as far as 3.0.
2025-09-25 09:21:04 -04:00
/* Skip file@ or opt@ prefix, it is the default case. Can be mixed with ref omitting the prefix */
MEDIUM: pattern: Add support for virtual and optional files for patterns Before this patch, it was not possible to use a list of patterns, map or a list of acls, without an existing file. However, it could be handy to just use an ID, with no file on the disk. It is pretty useful for everyone managing dynamically these lists. It could also be handy to try to load a list from a file if it exists without failing if not. This way, it could be possible to make a cold start without any file (instead of empty file), dynamically add and del patterns, dump the list to the file periodically to reuse it on reload (via an external process). In this patch, we uses some prefixes to be able to use virtual or optional files. The default case remains unchanged. regular files are used. A filename, with no prefix, is used as reference, and it must exist on the disk. With the prefix "file@", the same is performed. Internally this prefix is skipped. Thus the same file, with ou without "file@" prefix, references the same list of patterns. To use a virtual map, "virt@" prefix must be used. No file is read, even if the following name looks like a file. It is just an ID. The prefix is part of ID and must always be used. To use a optional file, ie a file that may or may not exist on a disk at startup, "opt@" prefix must be used. If the file exists, its content is loaded. But HAProxy doesn't complain if not. The prefix is not part of ID. For a given file, optional files and regular files reference the same list of patterns. This patch should fix the issue #2202.
2023-12-01 06:04:35 -05:00
if (strlen(reference) > 5 && strncmp(reference, "file@", 5) == 0)
reference += 5;
BUG/MINOR: pattern: Fix pattern lookup for map with opt@ prefix When we look for a map file reference, the file@ prefix is removed because if may be omitted. The same is true with opt@ prefix. However this case was not properly performed in pat_ref_lookup(). Let's do so. This patch must be backported as far as 3.0.
2025-09-25 09:21:04 -04:00
else if (strlen(reference) > 4 && strncmp(reference, "opt@", 4) == 0)
reference += 4;
MEDIUM: pattern: Add support for virtual and optional files for patterns Before this patch, it was not possible to use a list of patterns, map or a list of acls, without an existing file. However, it could be handy to just use an ID, with no file on the disk. It is pretty useful for everyone managing dynamically these lists. It could also be handy to try to load a list from a file if it exists without failing if not. This way, it could be possible to make a cold start without any file (instead of empty file), dynamically add and del patterns, dump the list to the file periodically to reuse it on reload (via an external process). In this patch, we uses some prefixes to be able to use virtual or optional files. The default case remains unchanged. regular files are used. A filename, with no prefix, is used as reference, and it must exist on the disk. With the prefix "file@", the same is performed. Internally this prefix is skipped. Thus the same file, with ou without "file@" prefix, references the same list of patterns. To use a virtual map, "virt@" prefix must be used. No file is read, even if the following name looks like a file. It is just an ID. The prefix is part of ID and must always be used. To use a optional file, ie a file that may or may not exist on a disk at startup, "opt@" prefix must be used. If the file exists, its content is loaded. But HAProxy doesn't complain if not. The prefix is not part of ID. For a given file, optional files and regular files reference the same list of patterns. This patch should fix the issue #2202.
2023-12-01 06:04:35 -05:00
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
list_for_each_entry(ref, &pattern_reference, list)
MINOR: pattern: Each pattern is identified by unique id. The pattern reference are stored with two identifiers: the unique_id and the reference. The reference identify a file. Each file with the same name point to the same reference. We can register many times one file. If the file is modified, all his dependencies are also modified. The reference can be used with map or acl. The unique_id identify inline acl. The unique id is unique for each acl. You cannot force the same id in the configuration file, because this repport an error. The format of the acl and map listing through the "socket" has changed for displaying these new ids.
2014-03-11 09:29:22 -04:00
if (ref->reference && strcmp(reference, ref->reference) == 0)
return ref;
return NULL;
}
CLEANUP: pattern: fix spelling/grammatical/copy-paste in comments The code is horrible to work with because most functions are documented with misleading comments resulting from many spelling and grammatical mistakes, and plenty of remains of copy-paste mentioning arguments that do not exist and return values that are never set. Too many hours wasted writing non-working code because of assumptions resulting from this, let's fix this once for all now!
2020-10-30 11:03:50 -04:00
/* This function looks up a reference's unique id. If the reference is found, a
* pointer to the struct pat_ref is returned, otherwise NULL is returned.
MINOR: pattern: Each pattern is identified by unique id. The pattern reference are stored with two identifiers: the unique_id and the reference. The reference identify a file. Each file with the same name point to the same reference. We can register many times one file. If the file is modified, all his dependencies are also modified. The reference can be used with map or acl. The unique_id identify inline acl. The unique id is unique for each acl. You cannot force the same id in the configuration file, because this repport an error. The format of the acl and map listing through the "socket" has changed for displaying these new ids.
2014-03-11 09:29:22 -04:00
*/
struct pat_ref *pat_ref_lookupid(int unique_id)
{
struct pat_ref *ref;
list_for_each_entry(ref, &pattern_reference, list)
if (ref->unique_id == unique_id)
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
return ref;
return NULL;
}
MINOR: pattern: introduce pat_ref_delete_by_ptr() to delete a valid reference Till now the only way to remove a known reference was via pat_ref_delete_by_id() which scans the whole list to find a matching pointer. Let's add pat_ref_delete_by_ptr() which takes a valid pointer. It can be called by the function above after the pointer is found, and can also be used to roll back a failed insertion much more efficiently.
2020-11-02 11:30:17 -05:00
/* This function removes from the pattern reference <ref> all the patterns
* attached to the reference element <elt>, and the element itself. The
* reference must be locked.
*/
void pat_ref_delete_by_ptr(struct pat_ref *ref, struct pat_ref_elt *elt)
{
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
struct pat_ref_gen *gen;
MINOR: pattern: introduce pat_ref_delete_by_ptr() to delete a valid reference Till now the only way to remove a known reference was via pat_ref_delete_by_id() which scans the whole list to find a matching pointer. Let's add pat_ref_delete_by_ptr() which takes a valid pointer. It can be called by the function above after the pointer is found, and can also be used to roll back a failed insertion much more efficiently.
2020-11-02 11:30:17 -05:00
struct pattern_expr *expr;
struct bref *bref, *back;
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
gen = pat_ref_gen_get(ref, elt->gen_id);
BUG_ON(!gen);
MINOR: pattern: introduce pat_ref_delete_by_ptr() to delete a valid reference Till now the only way to remove a known reference was via pat_ref_delete_by_id() which scans the whole list to find a matching pointer. Let's add pat_ref_delete_by_ptr() which takes a valid pointer. It can be called by the function above after the pointer is found, and can also be used to roll back a failed insertion much more efficiently.
2020-11-02 11:30:17 -05:00
/*
* we have to unlink all watchers from this reference pattern. We must
* not relink them if this elt was the last one in the list.
*/
list_for_each_entry_safe(bref, back, &elt->back_refs, users) {
CLEANUP: lists/tree-wide: rename some list operations to avoid some confusion The current "ADD" vs "ADDQ" is confusing because when thinking in terms of appending at the end of a list, "ADD" naturally comes to mind, but here it does the opposite, it inserts. Several times already it's been incorrectly used where ADDQ was expected, the latest of which was a fortunate accident explained in 6fa922562 ("CLEANUP: stream: explain why we queue the stream at the head of the server list"). Let's use more explicit (but slightly longer) names now: LIST_ADD -> LIST_INSERT LIST_ADDQ -> LIST_APPEND LIST_ADDED -> LIST_INLIST LIST_DEL -> LIST_DELETE The same is true for MT_LISTs, including their "TRY" variant. LIST_DEL_INIT keeps its short name to encourage to use it instead of the lazier LIST_DELETE which is often less safe. The change is large (~674 non-comment entries) but is mechanical enough to remain safe. No permutation was performed, so any out-of-tree code can easily map older names to new ones. The list doc was updated.
2021-04-21 01:32:39 -04:00
LIST_DELETE(&bref->users);
MINOR: pattern: introduce pat_ref_delete_by_ptr() to delete a valid reference Till now the only way to remove a known reference was via pat_ref_delete_by_id() which scans the whole list to find a matching pointer. Let's add pat_ref_delete_by_ptr() which takes a valid pointer. It can be called by the function above after the pointer is found, and can also be used to roll back a failed insertion much more efficiently.
2020-11-02 11:30:17 -05:00
LIST_INIT(&bref->users);
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
if (elt->list.n != &gen->head)
CLEANUP: lists/tree-wide: rename some list operations to avoid some confusion The current "ADD" vs "ADDQ" is confusing because when thinking in terms of appending at the end of a list, "ADD" naturally comes to mind, but here it does the opposite, it inserts. Several times already it's been incorrectly used where ADDQ was expected, the latest of which was a fortunate accident explained in 6fa922562 ("CLEANUP: stream: explain why we queue the stream at the head of the server list"). Let's use more explicit (but slightly longer) names now: LIST_ADD -> LIST_INSERT LIST_ADDQ -> LIST_APPEND LIST_ADDED -> LIST_INLIST LIST_DEL -> LIST_DELETE The same is true for MT_LISTs, including their "TRY" variant. LIST_DEL_INIT keeps its short name to encourage to use it instead of the lazier LIST_DELETE which is often less safe. The change is large (~674 non-comment entries) but is mechanical enough to remain safe. No permutation was performed, so any out-of-tree code can easily map older names to new ones. The list doc was updated.
2021-04-21 01:32:39 -04:00
LIST_APPEND(&LIST_ELEM(elt->list.n, typeof(elt), list)->back_refs, &bref->users);
MINOR: pattern: introduce pat_ref_delete_by_ptr() to delete a valid reference Till now the only way to remove a known reference was via pat_ref_delete_by_id() which scans the whole list to find a matching pointer. Let's add pat_ref_delete_by_ptr() which takes a valid pointer. It can be called by the function above after the pointer is found, and can also be used to roll back a failed insertion much more efficiently.
2020-11-02 11:30:17 -05:00
bref->ref = elt->list.n;
}
/* delete all entries from all expressions for this pattern */
list_for_each_entry(expr, &ref->pat, list)
HA_RWLOCK_WRLOCK(PATEXP_LOCK, &expr->lock);
pat_delete_gen(ref, elt);
list_for_each_entry(expr, &ref->pat, list)
HA_RWLOCK_WRUNLOCK(PATEXP_LOCK, &expr->lock);
CLEANUP: lists/tree-wide: rename some list operations to avoid some confusion The current "ADD" vs "ADDQ" is confusing because when thinking in terms of appending at the end of a list, "ADD" naturally comes to mind, but here it does the opposite, it inserts. Several times already it's been incorrectly used where ADDQ was expected, the latest of which was a fortunate accident explained in 6fa922562 ("CLEANUP: stream: explain why we queue the stream at the head of the server list"). Let's use more explicit (but slightly longer) names now: LIST_ADD -> LIST_INSERT LIST_ADDQ -> LIST_APPEND LIST_ADDED -> LIST_INLIST LIST_DEL -> LIST_DELETE The same is true for MT_LISTs, including their "TRY" variant. LIST_DEL_INIT keeps its short name to encourage to use it instead of the lazier LIST_DELETE which is often less safe. The change is large (~674 non-comment entries) but is mechanical enough to remain safe. No permutation was performed, so any out-of-tree code can easily map older names to new ones. The list doc was updated.
2021-04-21 01:32:39 -04:00
LIST_DELETE(&elt->list);
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
cebs_item_delete(&gen->elt_root, node, pattern, elt);
MINOR: pattern: introduce pat_ref_delete_by_ptr() to delete a valid reference Till now the only way to remove a known reference was via pat_ref_delete_by_id() which scans the whole list to find a matching pointer. Let's add pat_ref_delete_by_ptr() which takes a valid pointer. It can be called by the function above after the pointer is found, and can also be used to roll back a failed insertion much more efficiently.
2020-11-02 11:30:17 -05:00
free(elt->sample);
free(elt);
MINOR: pattern: add a counter of added/freed patterns Patterns are allocated when loading maps/acls from a file or dynamically via the CLI, and are released only from the CLI (e.g. "clear map xxx"). These ones do not use pools and are much harder to monitor, e.g. in case a script adds many and forgets to clear them, etc. Let's add a new pair of metrics "PatternsAdded" and "PatternsFreed" that will report the number of added and freed patterns respectively. This can allow to simply graph both. The difference between the two normally represents the number of allocated patterns. If Added grows without Freed following, it can indicate a faulty script that doesn't perform the needed cleanup. The metrics are also made available to Prometheus as patterns_added_total and patterns_freed_total respectively.
2025-07-04 18:07:25 -04:00
HA_ATOMIC_INC(&patterns_freed);
MINOR: pattern: introduce pat_ref_delete_by_ptr() to delete a valid reference Till now the only way to remove a known reference was via pat_ref_delete_by_id() which scans the whole list to find a matching pointer. Let's add pat_ref_delete_by_ptr() which takes a valid pointer. It can be called by the function above after the pointer is found, and can also be used to roll back a failed insertion much more efficiently.
2020-11-02 11:30:17 -05:00
}
BUG/MEDIUM: map/acl: pat_ref_{set,delete}_by_id regressions Some regressions were introduced by 5fea59754b ("MEDIUM: map/acl: Accelerate several functions using pat_ref_elt struct ->head list") pat_ref_delete_by_id() fails to properly unlink and free the removed reference because it bypasses the pat_ref_delete_by_ptr() made for that purpose. This function is normally used everywhere the target reference is set for removal, such as the pat_ref_delete() function that matches pattern against a string. The call was probably skipped by accident during the rewrite of the function. With the above commit also comes another undesirable change: both pat_ref_delete_by_id() and pat_ref_set_by_id() directly use the <refelt> argument as a valid pointer (they do dereference it). This is wrong, because <refelt> is unsafe and should be handled as an ID, not a pointer (hence the function name). Indeed, the calling function may directly pass user input from the CLI as <refelt> argument, so we must first ensure that it points to a valid element before using it, else it is probably invalid and we shouldn't touch it. What this patch essentially does, is that it reverts pat_ref_set_by_id() and pat_ref_delete_by_id() to pre 5fea59754b behavior. This seems like it was the only optimization from the patch that doesn't apply. Hopefully, after reviewing the changes with Fred, it seems that the 2 functions are only being involved in commands for manipulating maps or acls on the cli, so the "missed" opportunity to improve their performance shouldn't matter much. Nonetheless, if we wanted to speed up the reference lookup by ID, we could consider adding an eb64 tree for that specific purpose that contains all pattern references IDs (ie: pointers) so that eb lookup functions may be used instead of linear list search. The issue was raised by Marko Juraga as he failed to perform an an acl removal by reference on the CLI on 2.9 which was known to work properly on other versions. It should be backported on 2.9. Co-Authored-by: Frédéric Lécaille <flecaille@haproxy.com>
2023-12-08 05:46:15 -05:00
/* This function removes the pattern matching the pointer <refelt> from
CLEANUP: pattern: fix spelling/grammatical/copy-paste in comments The code is horrible to work with because most functions are documented with misleading comments resulting from many spelling and grammatical mistakes, and plenty of remains of copy-paste mentioning arguments that do not exist and return values that are never set. Too many hours wasted writing non-working code because of assumptions resulting from this, let's fix this once for all now!
2020-10-30 11:03:50 -04:00
* the reference and from each expr member of this reference. This function
* returns 1 if the entry was found and deleted, otherwise zero.
BUG/MEDIUM: map/acl: pat_ref_{set,delete}_by_id regressions Some regressions were introduced by 5fea59754b ("MEDIUM: map/acl: Accelerate several functions using pat_ref_elt struct ->head list") pat_ref_delete_by_id() fails to properly unlink and free the removed reference because it bypasses the pat_ref_delete_by_ptr() made for that purpose. This function is normally used everywhere the target reference is set for removal, such as the pat_ref_delete() function that matches pattern against a string. The call was probably skipped by accident during the rewrite of the function. With the above commit also comes another undesirable change: both pat_ref_delete_by_id() and pat_ref_set_by_id() directly use the <refelt> argument as a valid pointer (they do dereference it). This is wrong, because <refelt> is unsafe and should be handled as an ID, not a pointer (hence the function name). Indeed, the calling function may directly pass user input from the CLI as <refelt> argument, so we must first ensure that it points to a valid element before using it, else it is probably invalid and we shouldn't touch it. What this patch essentially does, is that it reverts pat_ref_set_by_id() and pat_ref_delete_by_id() to pre 5fea59754b behavior. This seems like it was the only optimization from the patch that doesn't apply. Hopefully, after reviewing the changes with Fred, it seems that the 2 functions are only being involved in commands for manipulating maps or acls on the cli, so the "missed" opportunity to improve their performance shouldn't matter much. Nonetheless, if we wanted to speed up the reference lookup by ID, we could consider adding an eb64 tree for that specific purpose that contains all pattern references IDs (ie: pointers) so that eb lookup functions may be used instead of linear list search. The issue was raised by Marko Juraga as he failed to perform an an acl removal by reference on the CLI on 2.9 which was known to work properly on other versions. It should be backported on 2.9. Co-Authored-by: Frédéric Lécaille <flecaille@haproxy.com>
2023-12-08 05:46:15 -05:00
*
* <refelt> is user input: it is provided as an ID and should never be
* dereferenced without making sure that it is valid.
MEDIUM: pattern: delete() function uses the pat_ref_elt to find the element to be removed All the pattern delete function can use her reference to the original "struct pat_ref_elt" to find the element to be remove. The functions pat_del_list_str() and pat_del_meth() were deleted because after applying this modification, they have the same code than pat_del_list_ptr().
2014-01-28 10:43:36 -05:00
*/
int pat_ref_delete_by_id(struct pat_ref *ref, struct pat_ref_elt *refelt)
{
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
struct pat_ref_gen *gen;
BUG/MEDIUM: map/acl: pat_ref_{set,delete}_by_id regressions Some regressions were introduced by 5fea59754b ("MEDIUM: map/acl: Accelerate several functions using pat_ref_elt struct ->head list") pat_ref_delete_by_id() fails to properly unlink and free the removed reference because it bypasses the pat_ref_delete_by_ptr() made for that purpose. This function is normally used everywhere the target reference is set for removal, such as the pat_ref_delete() function that matches pattern against a string. The call was probably skipped by accident during the rewrite of the function. With the above commit also comes another undesirable change: both pat_ref_delete_by_id() and pat_ref_set_by_id() directly use the <refelt> argument as a valid pointer (they do dereference it). This is wrong, because <refelt> is unsafe and should be handled as an ID, not a pointer (hence the function name). Indeed, the calling function may directly pass user input from the CLI as <refelt> argument, so we must first ensure that it points to a valid element before using it, else it is probably invalid and we shouldn't touch it. What this patch essentially does, is that it reverts pat_ref_set_by_id() and pat_ref_delete_by_id() to pre 5fea59754b behavior. This seems like it was the only optimization from the patch that doesn't apply. Hopefully, after reviewing the changes with Fred, it seems that the 2 functions are only being involved in commands for manipulating maps or acls on the cli, so the "missed" opportunity to improve their performance shouldn't matter much. Nonetheless, if we wanted to speed up the reference lookup by ID, we could consider adding an eb64 tree for that specific purpose that contains all pattern references IDs (ie: pointers) so that eb lookup functions may be used instead of linear list search. The issue was raised by Marko Juraga as he failed to perform an an acl removal by reference on the CLI on 2.9 which was known to work properly on other versions. It should be backported on 2.9. Co-Authored-by: Frédéric Lécaille <flecaille@haproxy.com>
2023-12-08 05:46:15 -05:00
struct pat_ref_elt *elt, *safe;
MEDIUM: map/acl: Accelerate several functions using pat_ref_elt struct ->head list Replace as much as possible list_for_each*() around ->head list, member of pat_ref_elt struct by use of its ->ebpt_root member which is an ebtree.
2023-08-22 12:32:13 -04:00
BUG/MEDIUM: map/acl: pat_ref_{set,delete}_by_id regressions Some regressions were introduced by 5fea59754b ("MEDIUM: map/acl: Accelerate several functions using pat_ref_elt struct ->head list") pat_ref_delete_by_id() fails to properly unlink and free the removed reference because it bypasses the pat_ref_delete_by_ptr() made for that purpose. This function is normally used everywhere the target reference is set for removal, such as the pat_ref_delete() function that matches pattern against a string. The call was probably skipped by accident during the rewrite of the function. With the above commit also comes another undesirable change: both pat_ref_delete_by_id() and pat_ref_set_by_id() directly use the <refelt> argument as a valid pointer (they do dereference it). This is wrong, because <refelt> is unsafe and should be handled as an ID, not a pointer (hence the function name). Indeed, the calling function may directly pass user input from the CLI as <refelt> argument, so we must first ensure that it points to a valid element before using it, else it is probably invalid and we shouldn't touch it. What this patch essentially does, is that it reverts pat_ref_set_by_id() and pat_ref_delete_by_id() to pre 5fea59754b behavior. This seems like it was the only optimization from the patch that doesn't apply. Hopefully, after reviewing the changes with Fred, it seems that the 2 functions are only being involved in commands for manipulating maps or acls on the cli, so the "missed" opportunity to improve their performance shouldn't matter much. Nonetheless, if we wanted to speed up the reference lookup by ID, we could consider adding an eb64 tree for that specific purpose that contains all pattern references IDs (ie: pointers) so that eb lookup functions may be used instead of linear list search. The issue was raised by Marko Juraga as he failed to perform an an acl removal by reference on the CLI on 2.9 which was known to work properly on other versions. It should be backported on 2.9. Co-Authored-by: Frédéric Lécaille <flecaille@haproxy.com>
2023-12-08 05:46:15 -05:00
/* delete pattern from reference */
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
pat_ref_gen_foreach(gen, ref) {
list_for_each_entry_safe(elt, safe, &gen->head, list) {
if (elt == refelt) {
event_hdl_publish(&ref->e_subs, EVENT_HDL_SUB_PAT_REF_DEL, NULL);
pat_ref_delete_by_ptr(ref, elt);
return 1;
}
BUG/MEDIUM: map/acl: pat_ref_{set,delete}_by_id regressions Some regressions were introduced by 5fea59754b ("MEDIUM: map/acl: Accelerate several functions using pat_ref_elt struct ->head list") pat_ref_delete_by_id() fails to properly unlink and free the removed reference because it bypasses the pat_ref_delete_by_ptr() made for that purpose. This function is normally used everywhere the target reference is set for removal, such as the pat_ref_delete() function that matches pattern against a string. The call was probably skipped by accident during the rewrite of the function. With the above commit also comes another undesirable change: both pat_ref_delete_by_id() and pat_ref_set_by_id() directly use the <refelt> argument as a valid pointer (they do dereference it). This is wrong, because <refelt> is unsafe and should be handled as an ID, not a pointer (hence the function name). Indeed, the calling function may directly pass user input from the CLI as <refelt> argument, so we must first ensure that it points to a valid element before using it, else it is probably invalid and we shouldn't touch it. What this patch essentially does, is that it reverts pat_ref_set_by_id() and pat_ref_delete_by_id() to pre 5fea59754b behavior. This seems like it was the only optimization from the patch that doesn't apply. Hopefully, after reviewing the changes with Fred, it seems that the 2 functions are only being involved in commands for manipulating maps or acls on the cli, so the "missed" opportunity to improve their performance shouldn't matter much. Nonetheless, if we wanted to speed up the reference lookup by ID, we could consider adding an eb64 tree for that specific purpose that contains all pattern references IDs (ie: pointers) so that eb lookup functions may be used instead of linear list search. The issue was raised by Marko Juraga as he failed to perform an an acl removal by reference on the CLI on 2.9 which was known to work properly on other versions. It should be backported on 2.9. Co-Authored-by: Frédéric Lécaille <flecaille@haproxy.com>
2023-12-08 05:46:15 -05:00
}
}
return 0;
MEDIUM: pattern: delete() function uses the pat_ref_elt to find the element to be removed All the pattern delete function can use her reference to the original "struct pat_ref_elt" to find the element to be remove. The functions pat_del_list_str() and pat_del_meth() were deleted because after applying this modification, they have the same code than pat_del_list_ptr().
2014-01-28 10:43:36 -05:00
}
MINOR: patterns: preliminary changes for reorganization Safe and non-functional changes that only add currently unused structures, field, functions and macros, in preparation of larger changes that alter the way pattern reference elements are stored. This includes code to create and lookup generation objects, and macros to iterate over the generations of a pattern reference.
2025-12-17 23:31:29 -05:00
/* Create a new generation object.
*
* Returns NULL in case of memory allocation failure.
*/
struct pat_ref_gen *pat_ref_gen_new(struct pat_ref *ref, unsigned int gen_id)
{
struct pat_ref_gen *gen, *old;
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
gen = malloc(sizeof(struct pat_ref_gen));
MINOR: patterns: preliminary changes for reorganization Safe and non-functional changes that only add currently unused structures, field, functions and macros, in preparation of larger changes that alter the way pattern reference elements are stored. This includes code to create and lookup generation objects, and macros to iterate over the generations of a pattern reference.
2025-12-17 23:31:29 -05:00
if (!gen)
return NULL;
LIST_INIT(&gen->head);
ceb_init_root(&gen->elt_root);
gen->gen_id = gen_id;
old = cebu32_item_insert(&ref->gen_root, gen_node, gen_id, gen);
BUG_ON(old != gen, "Generation ID already exists");
return gen;
}
/* Find the generation <gen_id> in the pattern reference <ref>.
*
* Returns NULL if the generation cannot be found.
*/
struct pat_ref_gen *pat_ref_gen_get(struct pat_ref *ref, unsigned int gen_id)
{
OPTIM: patterns: cache the current generation This makes a significant difference when loading large files and during commit and clear operations, thanks to improved cache locality. In the measurements below, master refers to the code before any of the changes to the patterns code, not the code before this one commit. Timing the replacement of 10M entries from the CLI with this command which also reports timestamps at start, end of upload and end of clear: $ (echo "prompt i"; echo "show activity"; echo "prepare acl #0"; awk '{print "add acl @1 #0",$0}' < bad-ip.map; echo "show activity"; echo "commit acl @1 #0"; echo "clear acl @0 #0";echo "show activity") | socat -t 10 - /tmp/sock1 | grep ^uptim master, on a 3.7 GHz EPYC, 3 samples: uptime_now: 6.087030 uptime_now: 25.981777 => 21.9 sec insertion time uptime_now: 29.286368 => 3.3 sec commit+clear uptime_now: 5.748087 uptime_now: 25.740675 => 20.0s insertion time uptime_now: 29.039023 => 3.3 s commit+clear uptime_now: 7.065362 uptime_now: 26.769596 => 19.7s insertion time uptime_now: 30.065044 => 3.3s commit+clear And after this commit: uptime_now: 6.119215 uptime_now: 25.023019 => 18.9 sec insertion time uptime_now: 27.155503 => 2.1 sec commit+clear uptime_now: 5.675931 uptime_now: 24.551035 => 18.9s insertion uptime_now: 26.652352 => 2.1s commit+clear uptime_now: 6.722256 uptime_now: 25.593952 => 18.9s insertion uptime_now: 27.724153 => 2.1s commit+clear Now timing the startup time with a 10M entries file (on another machine) on master, 20 samples: Standard Deviation, s: 0.061652677408033 Mean: 4.217 And after this commit: Standard Deviation, s: 0.081821371548669 Mean: 3.78
2025-12-22 09:12:40 -05:00
struct pat_ref_gen *gen;
/* We optimistically try to use the cached generation if it's the current one. */
if (likely(gen_id == ref->curr_gen && gen_id == ref->cached_gen.id && ref->cached_gen.data))
return ref->cached_gen.data;
gen = cebu32_item_lookup(&ref->gen_root, gen_node, gen_id, gen_id, struct pat_ref_gen);
if (unlikely(!gen))
return NULL;
if (gen_id == ref->curr_gen) {
ref->cached_gen.id = gen_id;
ref->cached_gen.data = gen;
}
return gen;
MINOR: patterns: preliminary changes for reorganization Safe and non-functional changes that only add currently unused structures, field, functions and macros, in preparation of larger changes that alter the way pattern reference elements are stored. This includes code to create and lookup generation objects, and macros to iterate over the generations of a pattern reference.
2025-12-17 23:31:29 -05:00
}
MINOR: pattern: add pat_ref_gen_delete() function pat_ref_gen_delete(ref, gen_id, key) tries to delete all samples belonging to <gen_id> and matching <key> under <ref> The goal is to be able to target a single subset from <ref>
2024-11-20 12:18:54 -05:00
/* This function removes all elements belonging to <gen_id> and matching <key>
* from the reference <ref>.
* This function returns 1 if the deletion is done and returns 0 if
* the entry is not found.
*/
int pat_ref_gen_delete(struct pat_ref *ref, unsigned int gen_id, const char *key)
{
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
struct pat_ref_gen *gen;
struct pat_ref_elt *elt;
MINOR: pattern: add pat_ref_gen_delete() function pat_ref_gen_delete(ref, gen_id, key) tries to delete all samples belonging to <gen_id> and matching <key> under <ref> The goal is to be able to target a single subset from <ref>
2024-11-20 12:18:54 -05:00
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
gen = pat_ref_gen_get(ref, gen_id);
if (!gen)
return 0;
MINOR: pattern: add pat_ref_gen_delete() function pat_ref_gen_delete(ref, gen_id, key) tries to delete all samples belonging to <gen_id> and matching <key> under <ref> The goal is to be able to target a single subset from <ref>
2024-11-20 12:18:54 -05:00
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
/* delete pattern from reference */
elt = cebs_item_lookup(&gen->elt_root, node, pattern, key, struct pat_ref_elt);
if (!elt)
return 0;
MINOR: pattern: publish event_hdl events on pat_ref updates Now that PAT_REF events were defined in previous commit, let's actually publish them from pattern API where relevant. Unlike server events, pattern reference events are only published in the pat_ref subscriber's list on purpose, because in some setups patref updates (updates performed on a map for instance from action or cli) are very frequent, and we don't want to impact pattern API performance just for that. Moreover, as the main use case is to be able to subscribe to maps updates from Lua, allowing a per-pattern reference registration is already enough. No additional data is provided for such events (also for performance reason) Care was taken not to publish events when the update doesn't affect the live subset (the one targeted by curr_gen).
2024-10-18 12:40:41 -04:00
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
pat_ref_delete_by_ptr(ref, elt);
event_hdl_publish(&ref->e_subs, EVENT_HDL_SUB_PAT_REF_DEL, NULL);
return 1;
MINOR: pattern: add pat_ref_gen_delete() function pat_ref_gen_delete(ref, gen_id, key) tries to delete all samples belonging to <gen_id> and matching <key> under <ref> The goal is to be able to target a single subset from <ref>
2024-11-20 12:18:54 -05:00
}
MINOR: pattern: introduce pat_ref_delete_by_ptr() to delete a valid reference Till now the only way to remove a known reference was via pat_ref_delete_by_id() which scans the whole list to find a matching pointer. Let's add pat_ref_delete_by_ptr() which takes a valid pointer. It can be called by the function above after the pointer is found, and can also be used to roll back a failed insertion much more efficiently.
2020-11-02 11:30:17 -05:00
/* This function removes all patterns matching <key> from the reference
CLEANUP: Fix typos in the pattern subsystem Fixes typos in the code comments of the pattern subsystem.
2018-11-15 13:22:31 -05:00
* and from each expr member of the reference. This function returns 1
MINOR: pattern: introduce pat_ref_delete_by_ptr() to delete a valid reference Till now the only way to remove a known reference was via pat_ref_delete_by_id() which scans the whole list to find a matching pointer. Let's add pat_ref_delete_by_ptr() which takes a valid pointer. It can be called by the function above after the pointer is found, and can also be used to roll back a failed insertion much more efficiently.
2020-11-02 11:30:17 -05:00
* if the deletion is done and returns 0 is the entry is not found.
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
*/
int pat_ref_delete(struct pat_ref *ref, const char *key)
{
MEDIUM: pattern: always consider gen_id for pat_ref lookup operations Historically, pat_ref lookup operations were performed on the whole pat_ref elements list. As such, set, find and delete operations on a given key would cause any matching element in pat_ref to be considered. When prepare/commit operations were added, gen_id was impelemnted in order to be able to work on a subset from pat_ref without impacting the current (live) version from pat_ref, until a new subset is committed to replace the current one. While the logic was good, there remained a design flaw from the historical implementation: indeed, legacy functions such as pat_ref_set(), pat_ref_delete() and pat_ref_find_elt() kept performing the lookups on the whole set of elements instead of considering only elements from the current subset. Because of this, mixing new prepare/commit operations with legacy operations could yield unexpected results. For instance, before this commit: echo "add map #0 key oldvalue" | socat /tmp/ha.sock - echo "prepare map #0" | socat /tmp/ha.sock - New version created: 1 echo "add map @1 #0 key newvalue" | socat /tmp/ha.sock - echo "del map #0 key" | socat /tmp/ha.sock - echo "commit map @1 #0" | socat /tmp/ha.sock - -> the result would be that "key" entry doesn't exist anymore after the commit, while we would expect the new value to be there instead. Thanks to the previous commits, we may finally fix this issue: for set, find_elt and delete operations, the current generation id is considered. With the above example, it means that the "del map #0 key" would only target elements from the current subset, thus elements in "version 1" of the map would be immune to the delete (as we would expect it to work).
2024-11-20 13:03:00 -05:00
return pat_ref_gen_delete(ref, ref->curr_gen, key);
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
}
MINOR: pattern: add pat_ref_gen_find_elt() function pat_ref_gen_find_elt(ref, gen_id, key) tries to find <elt> element belonging to <gen_id> and matching <key> in <ref> reference. The goal is to be able to target a single subset from <ref>
2024-11-20 12:07:52 -05:00
/*
* find and return an element <elt> belonging to <gen_id> and matching <key> in a
* reference <ref> return NULL if not found
*/
struct pat_ref_elt *pat_ref_gen_find_elt(struct pat_ref *ref, unsigned int gen_id, const char *key)
{
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
struct pat_ref_gen *gen;
MINOR: pattern: add pat_ref_gen_find_elt() function pat_ref_gen_find_elt(ref, gen_id, key) tries to find <elt> element belonging to <gen_id> and matching <key> in <ref> reference. The goal is to be able to target a single subset from <ref>
2024-11-20 12:07:52 -05:00
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
gen = pat_ref_gen_get(ref, gen_id);
if (!gen)
return NULL;
return cebs_item_lookup(&gen->elt_root, node, pattern, key, struct pat_ref_elt);
MINOR: pattern: add pat_ref_gen_find_elt() function pat_ref_gen_find_elt(ref, gen_id, key) tries to find <elt> element belonging to <gen_id> and matching <key> in <ref> reference. The goal is to be able to target a single subset from <ref>
2024-11-20 12:07:52 -05:00
}
MINOR: pattern: find element in a reference This function can be used to look for an entry in either an ACL or a MAP.
2014-04-25 10:57:03 -04:00
/*
* find and return an element <elt> matching <key> in a reference <ref>
* return NULL if not found
*/
struct pat_ref_elt *pat_ref_find_elt(struct pat_ref *ref, const char *key)
{
MEDIUM: pattern: always consider gen_id for pat_ref lookup operations Historically, pat_ref lookup operations were performed on the whole pat_ref elements list. As such, set, find and delete operations on a given key would cause any matching element in pat_ref to be considered. When prepare/commit operations were added, gen_id was impelemnted in order to be able to work on a subset from pat_ref without impacting the current (live) version from pat_ref, until a new subset is committed to replace the current one. While the logic was good, there remained a design flaw from the historical implementation: indeed, legacy functions such as pat_ref_set(), pat_ref_delete() and pat_ref_find_elt() kept performing the lookups on the whole set of elements instead of considering only elements from the current subset. Because of this, mixing new prepare/commit operations with legacy operations could yield unexpected results. For instance, before this commit: echo "add map #0 key oldvalue" | socat /tmp/ha.sock - echo "prepare map #0" | socat /tmp/ha.sock - New version created: 1 echo "add map @1 #0 key newvalue" | socat /tmp/ha.sock - echo "del map #0 key" | socat /tmp/ha.sock - echo "commit map @1 #0" | socat /tmp/ha.sock - -> the result would be that "key" entry doesn't exist anymore after the commit, while we would expect the new value to be there instead. Thanks to the previous commits, we may finally fix this issue: for set, find_elt and delete operations, the current generation id is considered. With the above example, it means that the "del map #0 key" would only target elements from the current subset, thus elements in "version 1" of the map would be immune to the delete (as we would expect it to work).
2024-11-20 13:03:00 -05:00
return pat_ref_gen_find_elt(ref, ref->curr_gen, key);
MINOR: pattern: find element in a reference This function can be used to look for an entry in either an ACL or a MAP.
2014-04-25 10:57:03 -04:00
}
CLEANUP: pattern: fix spelling/grammatical/copy-paste in comments The code is horrible to work with because most functions are documented with misleading comments resulting from many spelling and grammatical mistakes, and plenty of remains of copy-paste mentioning arguments that do not exist and return values that are never set. Too many hours wasted writing non-working code because of assumptions resulting from this, let's fix this once for all now!
2020-10-30 11:03:50 -04:00
/* This function modifies the sample of pat_ref_elt <elt> in all expressions
* found under <ref> to become <value>. It is assumed that the caller has
* already verified that <elt> belongs to <ref>.
*/
MEDIUM: pattern_find_smp: functions find_smp uses the pat_ref_elt to find the element to be removed The find_smp search the smp using the value of the pat_ref_elt pointer. The pat_find_smp_* are no longer used. The function pattern_find_smp() known all pattern indexation, and can be found
2014-01-29 10:24:55 -05:00
static inline int pat_ref_set_elt(struct pat_ref *ref, struct pat_ref_elt *elt,
MEDIUM: dumpstats: Display error message during add of values. This patch adds new display type. This display returns allocated string, when the string is flush into buffers, it is freed. This permit to return the content of "memprintf(err, ...)" messages. The pat_ref_add functions has changed to return error.
2014-01-29 13:08:49 -05:00
const char *value, char **err)
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
{
struct pattern_expr *expr;
MINOR: samples: rename some struct member from "smp" to "data" This members contains data and not sample.
2015-08-19 02:35:43 -04:00
struct sample_data **data;
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
char *sample;
MINOR: samples: rename a struct from sample_storage to sample_data This a first step of sample reorganization.
2015-08-13 18:02:11 -04:00
struct sample_data test;
MEDIUM: map/acl: Improve pat_ref_set_elt() efficiency (for "set-map", "add-acl"action perfs) Store a pointer to the expression (struct pattern_expr) into the data structure used to chain/store the map element references (struct pat_ref_elt) , e.g. the struct pattern_tree when stored into an ebtree or struct pattern_list when chained to a list. Modify pat_ref_set_elt() to stop inspecting all the expressions attached to a map and to look for the <elt> element passed as parameter to retrieve the sample data to be parsed. Indeed, thanks to the pointer added above to each pattern tree nodes or list elements, they all can be inspected directly from the <elt> passed as parameter and its ->tree_head and ->list_head member: the pattern tree nodes are stored into elt->tree_head, and the pattern list elements are chained to elt->list_head list. This inspection was also the job of pattern_find_smp() which is no more useful. This patch removes the code of this function.
2023-08-23 09:58:26 -04:00
struct pattern_tree *tree;
struct pattern_list *pat;
void **node;
MINOR: pattern: The function pat_ref_set() have now atomic behavior Before this patch, this function try to add values in best effort. If the parsing iof the value fail, the operation continue until the end. Now, this function stop on the first error and left the pattern in coherant state.
2014-01-29 13:35:06 -05:00
/* Try all needed converters. */
list_for_each_entry(expr, &ref->pat, list) {
if (!expr->pat_head->parse_smp)
continue;
if (!expr->pat_head->parse_smp(value, &test)) {
memprintf(err, "unable to parse '%s'", value);
return 0;
}
}
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
MEDIUM: pattern_find_smp: functions find_smp uses the pat_ref_elt to find the element to be removed The find_smp search the smp using the value of the pat_ref_elt pointer. The pat_find_smp_* are no longer used. The function pattern_find_smp() known all pattern indexation, and can be found
2014-01-29 10:24:55 -05:00
/* Modify pattern from reference. */
sample = strdup(value);
MEDIUM: dumpstats: Display error message during add of values. This patch adds new display type. This display returns allocated string, when the string is flush into buffers, it is freed. This permit to return the content of "memprintf(err, ...)" messages. The pat_ref_add functions has changed to return error.
2014-01-29 13:08:49 -05:00
if (!sample) {
memprintf(err, "out of memory error");
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
return 0;
MEDIUM: dumpstats: Display error message during add of values. This patch adds new display type. This display returns allocated string, when the string is flush into buffers, it is freed. This permit to return the content of "memprintf(err, ...)" messages. The pat_ref_add functions has changed to return error.
2014-01-29 13:08:49 -05:00
}
CLEANUP: pattern: fix spelling/grammatical/copy-paste in comments The code is horrible to work with because most functions are documented with misleading comments resulting from many spelling and grammatical mistakes, and plenty of remains of copy-paste mentioning arguments that do not exist and return values that are never set. Too many hours wasted writing non-working code because of assumptions resulting from this, let's fix this once for all now!
2020-10-30 11:03:50 -04:00
/* Load sample in each reference. All the conversions are tested
* below, normally these calls don't fail.
MINOR: pattern: The function pat_ref_set() have now atomic behavior Before this patch, this function try to add values in best effort. If the parsing iof the value fail, the operation continue until the end. Now, this function stop on the first error and left the pattern in coherant state.
2014-01-29 13:35:06 -05:00
*/
MEDIUM: map/acl: Improve pat_ref_set_elt() efficiency (for "set-map", "add-acl"action perfs) Store a pointer to the expression (struct pattern_expr) into the data structure used to chain/store the map element references (struct pat_ref_elt) , e.g. the struct pattern_tree when stored into an ebtree or struct pattern_list when chained to a list. Modify pat_ref_set_elt() to stop inspecting all the expressions attached to a map and to look for the <elt> element passed as parameter to retrieve the sample data to be parsed. Indeed, thanks to the pointer added above to each pattern tree nodes or list elements, they all can be inspected directly from the <elt> passed as parameter and its ->tree_head and ->list_head member: the pattern tree nodes are stored into elt->tree_head, and the pattern list elements are chained to elt->list_head list. This inspection was also the job of pattern_find_smp() which is no more useful. This patch removes the code of this function.
2023-08-23 09:58:26 -04:00
for (node = elt->tree_head; node;) {
tree = container_of(node, struct pattern_tree, from_ref);
node = *node;
BUG_ON(tree->ref != elt);
expr = tree->expr;
MEDIUM: pattern_find_smp: functions find_smp uses the pat_ref_elt to find the element to be removed The find_smp search the smp using the value of the pat_ref_elt pointer. The pat_find_smp_* are no longer used. The function pattern_find_smp() known all pattern indexation, and can be found
2014-01-29 10:24:55 -05:00
if (!expr->pat_head->parse_smp)
continue;
MEDIUM: map/acl: Improve pat_ref_set_elt() efficiency (for "set-map", "add-acl"action perfs) Store a pointer to the expression (struct pattern_expr) into the data structure used to chain/store the map element references (struct pat_ref_elt) , e.g. the struct pattern_tree when stored into an ebtree or struct pattern_list when chained to a list. Modify pat_ref_set_elt() to stop inspecting all the expressions attached to a map and to look for the <elt> element passed as parameter to retrieve the sample data to be parsed. Indeed, thanks to the pointer added above to each pattern tree nodes or list elements, they all can be inspected directly from the <elt> passed as parameter and its ->tree_head and ->list_head member: the pattern tree nodes are stored into elt->tree_head, and the pattern list elements are chained to elt->list_head list. This inspection was also the job of pattern_find_smp() which is no more useful. This patch removes the code of this function.
2023-08-23 09:58:26 -04:00
data = &tree->data;
if (data && *data) {
HA_RWLOCK_WRLOCK(PATEXP_LOCK, &expr->lock);
if (!expr->pat_head->parse_smp(sample, *data))
*data = NULL;
HA_RWLOCK_WRUNLOCK(PATEXP_LOCK, &expr->lock);
}
}
for (node = elt->list_head; node;) {
pat = container_of(node, struct pattern_list, from_ref);
node = *node;
BUG_ON(pat->pat.ref != elt);
expr = pat->expr;
if (!expr->pat_head->parse_smp)
continue;
data = &pat->pat.data;
if (data && *data) {
HA_RWLOCK_WRLOCK(PATEXP_LOCK, &expr->lock);
if (!expr->pat_head->parse_smp(sample, *data))
*data = NULL;
HA_RWLOCK_WRUNLOCK(PATEXP_LOCK, &expr->lock);
}
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
}
MAJOR: threads/map: Make acls/maps thread safe locks have been added in pat_ref and pattern_expr structures to protect all accesses to an instance of on of them. Moreover, a global lock has been added to protect the LRU cache used for pattern matching. Patterns are now duplicated after a successfull matching, to avoid modification by other threads when the result is used. Finally, the function reloading a pattern list has been modified to be thread-safe.
2017-07-03 05:34:05 -04:00
/* free old sample only when all exprs are updated */
free(elt->sample);
elt->sample = sample;
MINOR: pattern: The function pat_ref_set() have now atomic behavior Before this patch, this function try to add values in best effort. If the parsing iof the value fail, the operation continue until the end. Now, this function stop on the first error and left the pattern in coherant state.
2014-01-29 13:35:06 -05:00
return 1;
MEDIUM: pattern_find_smp: functions find_smp uses the pat_ref_elt to find the element to be removed The find_smp search the smp using the value of the pat_ref_elt pointer. The pat_find_smp_* are no longer used. The function pattern_find_smp() known all pattern indexation, and can be found
2014-01-29 10:24:55 -05:00
}
CLEANUP: pattern: fix spelling/grammatical/copy-paste in comments The code is horrible to work with because most functions are documented with misleading comments resulting from many spelling and grammatical mistakes, and plenty of remains of copy-paste mentioning arguments that do not exist and return values that are never set. Too many hours wasted writing non-working code because of assumptions resulting from this, let's fix this once for all now!
2020-10-30 11:03:50 -04:00
/* This function modifies the sample of pat_ref_elt <refelt> in all expressions
* found under <ref> to become <value>, after checking that <refelt> really
* belongs to <ref>.
BUG/MEDIUM: map/acl: pat_ref_{set,delete}_by_id regressions Some regressions were introduced by 5fea59754b ("MEDIUM: map/acl: Accelerate several functions using pat_ref_elt struct ->head list") pat_ref_delete_by_id() fails to properly unlink and free the removed reference because it bypasses the pat_ref_delete_by_ptr() made for that purpose. This function is normally used everywhere the target reference is set for removal, such as the pat_ref_delete() function that matches pattern against a string. The call was probably skipped by accident during the rewrite of the function. With the above commit also comes another undesirable change: both pat_ref_delete_by_id() and pat_ref_set_by_id() directly use the <refelt> argument as a valid pointer (they do dereference it). This is wrong, because <refelt> is unsafe and should be handled as an ID, not a pointer (hence the function name). Indeed, the calling function may directly pass user input from the CLI as <refelt> argument, so we must first ensure that it points to a valid element before using it, else it is probably invalid and we shouldn't touch it. What this patch essentially does, is that it reverts pat_ref_set_by_id() and pat_ref_delete_by_id() to pre 5fea59754b behavior. This seems like it was the only optimization from the patch that doesn't apply. Hopefully, after reviewing the changes with Fred, it seems that the 2 functions are only being involved in commands for manipulating maps or acls on the cli, so the "missed" opportunity to improve their performance shouldn't matter much. Nonetheless, if we wanted to speed up the reference lookup by ID, we could consider adding an eb64 tree for that specific purpose that contains all pattern references IDs (ie: pointers) so that eb lookup functions may be used instead of linear list search. The issue was raised by Marko Juraga as he failed to perform an an acl removal by reference on the CLI on 2.9 which was known to work properly on other versions. It should be backported on 2.9. Co-Authored-by: Frédéric Lécaille <flecaille@haproxy.com>
2023-12-08 05:46:15 -05:00
*
* <refelt> is user input: it is provided as an ID and should never be
* dereferenced without making sure that it is valid.
CLEANUP: pattern: fix spelling/grammatical/copy-paste in comments The code is horrible to work with because most functions are documented with misleading comments resulting from many spelling and grammatical mistakes, and plenty of remains of copy-paste mentioning arguments that do not exist and return values that are never set. Too many hours wasted writing non-working code because of assumptions resulting from this, let's fix this once for all now!
2020-10-30 11:03:50 -04:00
*/
MEDIUM: dumpstats: Display error message during add of values. This patch adds new display type. This display returns allocated string, when the string is flush into buffers, it is freed. This permit to return the content of "memprintf(err, ...)" messages. The pat_ref_add functions has changed to return error.
2014-01-29 13:08:49 -05:00
int pat_ref_set_by_id(struct pat_ref *ref, struct pat_ref_elt *refelt, const char *value, char **err)
MEDIUM: pattern_find_smp: functions find_smp uses the pat_ref_elt to find the element to be removed The find_smp search the smp using the value of the pat_ref_elt pointer. The pat_find_smp_* are no longer used. The function pattern_find_smp() known all pattern indexation, and can be found
2014-01-29 10:24:55 -05:00
{
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
struct pat_ref_gen *gen;
BUG/MEDIUM: map/acl: pat_ref_{set,delete}_by_id regressions Some regressions were introduced by 5fea59754b ("MEDIUM: map/acl: Accelerate several functions using pat_ref_elt struct ->head list") pat_ref_delete_by_id() fails to properly unlink and free the removed reference because it bypasses the pat_ref_delete_by_ptr() made for that purpose. This function is normally used everywhere the target reference is set for removal, such as the pat_ref_delete() function that matches pattern against a string. The call was probably skipped by accident during the rewrite of the function. With the above commit also comes another undesirable change: both pat_ref_delete_by_id() and pat_ref_set_by_id() directly use the <refelt> argument as a valid pointer (they do dereference it). This is wrong, because <refelt> is unsafe and should be handled as an ID, not a pointer (hence the function name). Indeed, the calling function may directly pass user input from the CLI as <refelt> argument, so we must first ensure that it points to a valid element before using it, else it is probably invalid and we shouldn't touch it. What this patch essentially does, is that it reverts pat_ref_set_by_id() and pat_ref_delete_by_id() to pre 5fea59754b behavior. This seems like it was the only optimization from the patch that doesn't apply. Hopefully, after reviewing the changes with Fred, it seems that the 2 functions are only being involved in commands for manipulating maps or acls on the cli, so the "missed" opportunity to improve their performance shouldn't matter much. Nonetheless, if we wanted to speed up the reference lookup by ID, we could consider adding an eb64 tree for that specific purpose that contains all pattern references IDs (ie: pointers) so that eb lookup functions may be used instead of linear list search. The issue was raised by Marko Juraga as he failed to perform an an acl removal by reference on the CLI on 2.9 which was known to work properly on other versions. It should be backported on 2.9. Co-Authored-by: Frédéric Lécaille <flecaille@haproxy.com>
2023-12-08 05:46:15 -05:00
struct pat_ref_elt *elt;
/* Look for pattern in the reference. */
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
pat_ref_gen_foreach(gen, ref) {
list_for_each_entry(elt, &gen->head, list) {
if (elt == refelt) {
if (!pat_ref_set_elt(ref, elt, value, err))
return 0;
return 1;
}
BUG/MEDIUM: map/acl: pat_ref_{set,delete}_by_id regressions Some regressions were introduced by 5fea59754b ("MEDIUM: map/acl: Accelerate several functions using pat_ref_elt struct ->head list") pat_ref_delete_by_id() fails to properly unlink and free the removed reference because it bypasses the pat_ref_delete_by_ptr() made for that purpose. This function is normally used everywhere the target reference is set for removal, such as the pat_ref_delete() function that matches pattern against a string. The call was probably skipped by accident during the rewrite of the function. With the above commit also comes another undesirable change: both pat_ref_delete_by_id() and pat_ref_set_by_id() directly use the <refelt> argument as a valid pointer (they do dereference it). This is wrong, because <refelt> is unsafe and should be handled as an ID, not a pointer (hence the function name). Indeed, the calling function may directly pass user input from the CLI as <refelt> argument, so we must first ensure that it points to a valid element before using it, else it is probably invalid and we shouldn't touch it. What this patch essentially does, is that it reverts pat_ref_set_by_id() and pat_ref_delete_by_id() to pre 5fea59754b behavior. This seems like it was the only optimization from the patch that doesn't apply. Hopefully, after reviewing the changes with Fred, it seems that the 2 functions are only being involved in commands for manipulating maps or acls on the cli, so the "missed" opportunity to improve their performance shouldn't matter much. Nonetheless, if we wanted to speed up the reference lookup by ID, we could consider adding an eb64 tree for that specific purpose that contains all pattern references IDs (ie: pointers) so that eb lookup functions may be used instead of linear list search. The issue was raised by Marko Juraga as he failed to perform an an acl removal by reference on the CLI on 2.9 which was known to work properly on other versions. It should be backported on 2.9. Co-Authored-by: Frédéric Lécaille <flecaille@haproxy.com>
2023-12-08 05:46:15 -05:00
}
MEDIUM: pattern_find_smp: functions find_smp uses the pat_ref_elt to find the element to be removed The find_smp search the smp using the value of the pat_ref_elt pointer. The pat_find_smp_* are no longer used. The function pattern_find_smp() known all pattern indexation, and can be found
2014-01-29 10:24:55 -05:00
}
MEDIUM: dumpstats: Display error message during add of values. This patch adds new display type. This display returns allocated string, when the string is flush into buffers, it is freed. This permit to return the content of "memprintf(err, ...)" messages. The pat_ref_add functions has changed to return error.
2014-01-29 13:08:49 -05:00
memprintf(err, "key or pattern not found");
MEDIUM: pattern_find_smp: functions find_smp uses the pat_ref_elt to find the element to be removed The find_smp search the smp using the value of the pat_ref_elt pointer. The pat_find_smp_* are no longer used. The function pattern_find_smp() known all pattern indexation, and can be found
2014-01-29 10:24:55 -05:00
return 0;
}
MEDIUM: migrate the patterns reference to cebs_tree cebs_tree are 24 bytes smaller than ebst_tree (16B vs 40B), and pattern references are only used during map/acl updates, so their storage is pure loss between updates (which most of the time never happen). By switching their indexing to compact trees, we can save 16 to 24 bytes per entry depending on alightment (here it's 24 per struct but 16 practical as malloc's alignment keeps 8 unused). Tested on core i7-8650U running at 3.0 GHz, with a file containing 17.7M IP addresses (16.7M different): $ time ./haproxy -c -f acl-ip.cfg Save 280 MB RAM for 17.7M IP addresses, and slightly speeds up the startup (5.8%, from 19.2s to 18.2s), a part of which possible being attributed to having to write less memory. Note that this is on small strings. On larger ones such as user-agents, ebtree doesn't reread the whole key and might be more efficient. Before: RAM (VSZ/RSS): 4443912 3912444 real 0m19.211s user 0m18.138s sys 0m1.068s Overhead Command Shared Object Symbol 44.79% haproxy haproxy [.] ebst_insert 25.07% haproxy haproxy [.] ebmb_insert_prefix 3.44% haproxy libc-2.33.so [.] __libc_calloc 2.71% haproxy libc-2.33.so [.] _int_malloc 2.33% haproxy haproxy [.] free_pattern_tree 1.78% haproxy libc-2.33.so [.] inet_pton4 1.62% haproxy libc-2.33.so [.] _IO_fgets 1.58% haproxy libc-2.33.so [.] _int_free 1.56% haproxy haproxy [.] pat_ref_push 1.35% haproxy libc-2.33.so [.] malloc_consolidate 1.16% haproxy libc-2.33.so [.] __strlen_avx2 0.79% haproxy haproxy [.] pat_idx_tree_ip 0.76% haproxy haproxy [.] pat_ref_read_from_file 0.60% haproxy libc-2.33.so [.] __strrchr_avx2 0.55% haproxy libc-2.33.so [.] unlink_chunk.constprop.0 0.54% haproxy libc-2.33.so [.] __memchr_avx2 0.46% haproxy haproxy [.] pat_ref_append After: RAM (VSZ/RSS): 4166108 3634768 real 0m18.114s user 0m17.113s sys 0m0.996s Overhead Command Shared Object Symbol 38.99% haproxy haproxy [.] cebs_insert 27.09% haproxy haproxy [.] ebmb_insert_prefix 3.63% haproxy libc-2.33.so [.] __libc_calloc 3.18% haproxy libc-2.33.so [.] _int_malloc 2.69% haproxy haproxy [.] free_pattern_tree 1.99% haproxy libc-2.33.so [.] inet_pton4 1.74% haproxy libc-2.33.so [.] _IO_fgets 1.73% haproxy libc-2.33.so [.] _int_free 1.57% haproxy haproxy [.] pat_ref_push 1.48% haproxy libc-2.33.so [.] malloc_consolidate 1.22% haproxy libc-2.33.so [.] __strlen_avx2 1.05% haproxy libc-2.33.so [.] __strcmp_avx2 0.80% haproxy haproxy [.] pat_idx_tree_ip 0.74% haproxy libc-2.33.so [.] __memchr_avx2 0.69% haproxy libc-2.33.so [.] __strrchr_avx2 0.69% haproxy libc-2.33.so [.] _IO_getline_info 0.62% haproxy haproxy [.] pat_ref_read_from_file 0.56% haproxy libc-2.33.so [.] unlink_chunk.constprop.0 0.56% haproxy libc-2.33.so [.] cfree@GLIBC_2.2.5 0.46% haproxy haproxy [.] pat_ref_append If the addresses are totally disordered (via "shuf" on the input file), we see both implementations reach exactly 68.0s (slower due to much higher cache miss ratio). On large strings such as user agents (1 million here), it's now slightly slower (+9%): Before: real 0m2.475s user 0m2.316s sys 0m0.155s After: real 0m2.696s user 0m2.544s sys 0m0.147s But such patterns are much less common than short ones, and the memory savings do still count. Note that while it could be tempting to get rid of the list that chains all these pat_ref_elt together and only enumerate them by walking along the tree to save 16 extra bytes per entry, that's not possible due to the problem that insertion ordering is critical (think overlapping regex such as /index.* and /index.html). Currently it's not possible to proceed differently because patterns are first pre-loaded into the pat_ref via pat_ref_read_from_file_smp() and later indexed by pattern_read_from_file(), which has to only redo the second part anyway for maps/acls declared multiple times.
2025-01-12 13:38:28 -05:00
static int pat_ref_set_from_elt(struct pat_ref *ref, struct pat_ref_elt *elt, const char *value, char **err)
MEDIUM: pattern_find_smp: functions find_smp uses the pat_ref_elt to find the element to be removed The find_smp search the smp using the value of the pat_ref_elt pointer. The pat_find_smp_* are no longer used. The function pattern_find_smp() known all pattern indexation, and can be found
2014-01-29 10:24:55 -05:00
{
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
struct pat_ref_gen *gen;
BUG/MEDIUM: pattern: fix possible infinite loops on deletion (try 2) Commit e36b3b60b3 ("MEDIUM: migrate the patterns reference to cebs_tree") changed the construction of the loops used to look up matching nodes, and since we don't need two elements anymore, the "continue" statement now loops on the same element when deleting. Let's fix this to make sure it passes through the next one. While this bug is 3.3 only, it turns out that 3.2 is also affected by the incorrect loop construct in pat_ref_set_from_node(), where it's possible to run an infinite loop since commit 010c34b8c7 ("MEDIUM: pattern: consider gen_id in pat_ref_set_from_node()") due to the "continue" statement being placed before the ebmb_next_dup() call. As such the relevant part of this fix (pat_ref_set_from_elt) will need to be backported to 3.2.
2025-09-16 05:49:01 -04:00
struct pat_ref_elt *elt2;
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
int found = 0, publish = 0;
MEDIUM: map/acl: Improve pat_ref_set() efficiency (for "set-map", "add-acl" action perfs) Organize reference to pattern element of map (struct pat_ref_elt) into an ebtree: - add an eb_root member to the map (pat_ref struct) and an ebpt_node to its element (pat_ref_elt struct), - modify the code to insert these nodes into their ebtrees each time they are allocated. This is done in pat_ref_append(). Note that ->head member (struct list) of map (struct pat_ref) is not removed could have been removed. This is not the case because still necessary to dump the map contents from the CLI in the order the map elememnts have been inserted. This patch also modifies http_action_set_map() which is the callback at least used by "set-map" action. The pat_ref_elt element returned by pat_ref_find_elt() is no more ignored, but reused if not NULL by pat_ref_set() as first element to lookup from. This latter is also modified to use the ebtree attached to the map in place of the ->head list attached to each map element (pat_ref_elt struct). Also modify pat_ref_find_elt() to makes it use ->eb_root map ebtree added to the map by this patch in place of inspecting all the elements with a strcmp() call.
2023-08-22 10:52:47 -04:00
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
if (elt) {
if (elt->gen_id == ref->curr_gen)
publish = 1;
gen = pat_ref_gen_get(ref, elt->gen_id);
BUG_ON(!gen);
BUG/MINOR: pattern: pat_ref_set: fix UAF reported by coverity memprintf() performs realloc and updates then the pointer to an output buffer, where it has written the data. So free() is called on the previous buffer address, if it was provided. pat_ref_set_elt() uses memprintf() to write its error message as well as pat_ref_set(). So, when we re-enter into the while loop the second time and pat_ref_set_elt() has returned, the *err ptr (previous value of *merr) is already freed by memprintf() from pat_ref_set_el(). 'if (!found)' condition is false at this point, because we've found a node at the first loop. So, the second memprintf(), in order to write error messages, does again free(*err). This should be backported in all stable versions.
2024-08-12 09:32:00 -04:00
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
for (; elt; elt = elt2) {
char *tmp_err = NULL;
MEDIUM: migrate the patterns reference to cebs_tree cebs_tree are 24 bytes smaller than ebst_tree (16B vs 40B), and pattern references are only used during map/acl updates, so their storage is pure loss between updates (which most of the time never happen). By switching their indexing to compact trees, we can save 16 to 24 bytes per entry depending on alightment (here it's 24 per struct but 16 practical as malloc's alignment keeps 8 unused). Tested on core i7-8650U running at 3.0 GHz, with a file containing 17.7M IP addresses (16.7M different): $ time ./haproxy -c -f acl-ip.cfg Save 280 MB RAM for 17.7M IP addresses, and slightly speeds up the startup (5.8%, from 19.2s to 18.2s), a part of which possible being attributed to having to write less memory. Note that this is on small strings. On larger ones such as user-agents, ebtree doesn't reread the whole key and might be more efficient. Before: RAM (VSZ/RSS): 4443912 3912444 real 0m19.211s user 0m18.138s sys 0m1.068s Overhead Command Shared Object Symbol 44.79% haproxy haproxy [.] ebst_insert 25.07% haproxy haproxy [.] ebmb_insert_prefix 3.44% haproxy libc-2.33.so [.] __libc_calloc 2.71% haproxy libc-2.33.so [.] _int_malloc 2.33% haproxy haproxy [.] free_pattern_tree 1.78% haproxy libc-2.33.so [.] inet_pton4 1.62% haproxy libc-2.33.so [.] _IO_fgets 1.58% haproxy libc-2.33.so [.] _int_free 1.56% haproxy haproxy [.] pat_ref_push 1.35% haproxy libc-2.33.so [.] malloc_consolidate 1.16% haproxy libc-2.33.so [.] __strlen_avx2 0.79% haproxy haproxy [.] pat_idx_tree_ip 0.76% haproxy haproxy [.] pat_ref_read_from_file 0.60% haproxy libc-2.33.so [.] __strrchr_avx2 0.55% haproxy libc-2.33.so [.] unlink_chunk.constprop.0 0.54% haproxy libc-2.33.so [.] __memchr_avx2 0.46% haproxy haproxy [.] pat_ref_append After: RAM (VSZ/RSS): 4166108 3634768 real 0m18.114s user 0m17.113s sys 0m0.996s Overhead Command Shared Object Symbol 38.99% haproxy haproxy [.] cebs_insert 27.09% haproxy haproxy [.] ebmb_insert_prefix 3.63% haproxy libc-2.33.so [.] __libc_calloc 3.18% haproxy libc-2.33.so [.] _int_malloc 2.69% haproxy haproxy [.] free_pattern_tree 1.99% haproxy libc-2.33.so [.] inet_pton4 1.74% haproxy libc-2.33.so [.] _IO_fgets 1.73% haproxy libc-2.33.so [.] _int_free 1.57% haproxy haproxy [.] pat_ref_push 1.48% haproxy libc-2.33.so [.] malloc_consolidate 1.22% haproxy libc-2.33.so [.] __strlen_avx2 1.05% haproxy libc-2.33.so [.] __strcmp_avx2 0.80% haproxy haproxy [.] pat_idx_tree_ip 0.74% haproxy libc-2.33.so [.] __memchr_avx2 0.69% haproxy libc-2.33.so [.] __strrchr_avx2 0.69% haproxy libc-2.33.so [.] _IO_getline_info 0.62% haproxy haproxy [.] pat_ref_read_from_file 0.56% haproxy libc-2.33.so [.] unlink_chunk.constprop.0 0.56% haproxy libc-2.33.so [.] cfree@GLIBC_2.2.5 0.46% haproxy haproxy [.] pat_ref_append If the addresses are totally disordered (via "shuf" on the input file), we see both implementations reach exactly 68.0s (slower due to much higher cache miss ratio). On large strings such as user agents (1 million here), it's now slightly slower (+9%): Before: real 0m2.475s user 0m2.316s sys 0m0.155s After: real 0m2.696s user 0m2.544s sys 0m0.147s But such patterns are much less common than short ones, and the memory savings do still count. Note that while it could be tempting to get rid of the list that chains all these pat_ref_elt together and only enumerate them by walking along the tree to save 16 extra bytes per entry, that's not possible due to the problem that insertion ordering is critical (think overlapping regex such as /index.* and /index.html). Currently it's not possible to proceed differently because patterns are first pre-loaded into the pat_ref via pat_ref_read_from_file_smp() and later indexed by pattern_read_from_file(), which has to only redo the second part anyway for maps/acls declared multiple times.
2025-01-12 13:38:28 -05:00
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
elt2 = cebs_item_next_dup(&gen->elt_root, node, pattern, elt);
if (!pat_ref_set_elt(ref, elt, value, &tmp_err)) {
if (err)
*err = tmp_err;
else
ha_free(&tmp_err);
return 0;
}
found = 1;
MEDIUM: pattern_find_smp: functions find_smp uses the pat_ref_elt to find the element to be removed The find_smp search the smp using the value of the pat_ref_elt pointer. The pat_find_smp_* are no longer used. The function pattern_find_smp() known all pattern indexation, and can be found
2014-01-29 10:24:55 -05:00
}
}
MEDIUM: dumpstats: Display error message during add of values. This patch adds new display type. This display returns allocated string, when the string is flush into buffers, it is freed. This permit to return the content of "memprintf(err, ...)" messages. The pat_ref_add functions has changed to return error.
2014-01-29 13:08:49 -05:00
if (!found) {
memprintf(err, "entry not found");
return 0;
}
MINOR: pattern: publish event_hdl events on pat_ref updates Now that PAT_REF events were defined in previous commit, let's actually publish them from pattern API where relevant. Unlike server events, pattern reference events are only published in the pat_ref subscriber's list on purpose, because in some setups patref updates (updates performed on a map for instance from action or cli) are very frequent, and we don't want to impact pattern API performance just for that. Moreover, as the main use case is to be able to subscribe to maps updates from Lua, allowing a per-pattern reference registration is already enough. No additional data is provided for such events (also for performance reason) Care was taken not to publish events when the update doesn't affect the live subset (the one targeted by curr_gen).
2024-10-18 12:40:41 -04:00
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
if (publish)
MINOR: pattern: publish event_hdl events on pat_ref updates Now that PAT_REF events were defined in previous commit, let's actually publish them from pattern API where relevant. Unlike server events, pattern reference events are only published in the pat_ref subscriber's list on purpose, because in some setups patref updates (updates performed on a map for instance from action or cli) are very frequent, and we don't want to impact pattern API performance just for that. Moreover, as the main use case is to be able to subscribe to maps updates from Lua, allowing a per-pattern reference registration is already enough. No additional data is provided for such events (also for performance reason) Care was taken not to publish events when the update doesn't affect the live subset (the one targeted by curr_gen).
2024-10-18 12:40:41 -04:00
event_hdl_publish(&ref->e_subs, EVENT_HDL_SUB_PAT_REF_SET, NULL);
MEDIUM: dumpstats: Display error message during add of values. This patch adds new display type. This display returns allocated string, when the string is flush into buffers, it is freed. This permit to return the content of "memprintf(err, ...)" messages. The pat_ref_add functions has changed to return error.
2014-01-29 13:08:49 -05:00
return 1;
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
}
MINOR: pattern: split pat_ref_set() split pat_ref_set() function in 2 distinct functions. Indeed, since 0844bed7d3 ("MEDIUM: map/acl: Improve pat_ref_set() efficiency (for "set-map", "add-acl" action perfs)"), pat_ref_set() prototype was updated to include an extra <elt> argument. But the logic behind is not explicit because the function will not only try to set <elt>, but also its duplicate (unlike pat_ref_set_elt() which only tries to update <elt>). Thus, to make it clearer and better distinguish between the key-based lookup version and the elt-based one, restotre pat_ref_set() previous prototype and add a dedicated pat_ref_set_elt_duplicate() that takes <elt> as argument and tries to update <elt> and all duplicates.
2024-11-20 10:22:22 -05:00
/* modifies to <value> the sample for <elt> and all its duplicates */
int pat_ref_set_elt_duplicate(struct pat_ref *ref, struct pat_ref_elt *elt, const char *value,
char **err)
{
MEDIUM: migrate the patterns reference to cebs_tree cebs_tree are 24 bytes smaller than ebst_tree (16B vs 40B), and pattern references are only used during map/acl updates, so their storage is pure loss between updates (which most of the time never happen). By switching their indexing to compact trees, we can save 16 to 24 bytes per entry depending on alightment (here it's 24 per struct but 16 practical as malloc's alignment keeps 8 unused). Tested on core i7-8650U running at 3.0 GHz, with a file containing 17.7M IP addresses (16.7M different): $ time ./haproxy -c -f acl-ip.cfg Save 280 MB RAM for 17.7M IP addresses, and slightly speeds up the startup (5.8%, from 19.2s to 18.2s), a part of which possible being attributed to having to write less memory. Note that this is on small strings. On larger ones such as user-agents, ebtree doesn't reread the whole key and might be more efficient. Before: RAM (VSZ/RSS): 4443912 3912444 real 0m19.211s user 0m18.138s sys 0m1.068s Overhead Command Shared Object Symbol 44.79% haproxy haproxy [.] ebst_insert 25.07% haproxy haproxy [.] ebmb_insert_prefix 3.44% haproxy libc-2.33.so [.] __libc_calloc 2.71% haproxy libc-2.33.so [.] _int_malloc 2.33% haproxy haproxy [.] free_pattern_tree 1.78% haproxy libc-2.33.so [.] inet_pton4 1.62% haproxy libc-2.33.so [.] _IO_fgets 1.58% haproxy libc-2.33.so [.] _int_free 1.56% haproxy haproxy [.] pat_ref_push 1.35% haproxy libc-2.33.so [.] malloc_consolidate 1.16% haproxy libc-2.33.so [.] __strlen_avx2 0.79% haproxy haproxy [.] pat_idx_tree_ip 0.76% haproxy haproxy [.] pat_ref_read_from_file 0.60% haproxy libc-2.33.so [.] __strrchr_avx2 0.55% haproxy libc-2.33.so [.] unlink_chunk.constprop.0 0.54% haproxy libc-2.33.so [.] __memchr_avx2 0.46% haproxy haproxy [.] pat_ref_append After: RAM (VSZ/RSS): 4166108 3634768 real 0m18.114s user 0m17.113s sys 0m0.996s Overhead Command Shared Object Symbol 38.99% haproxy haproxy [.] cebs_insert 27.09% haproxy haproxy [.] ebmb_insert_prefix 3.63% haproxy libc-2.33.so [.] __libc_calloc 3.18% haproxy libc-2.33.so [.] _int_malloc 2.69% haproxy haproxy [.] free_pattern_tree 1.99% haproxy libc-2.33.so [.] inet_pton4 1.74% haproxy libc-2.33.so [.] _IO_fgets 1.73% haproxy libc-2.33.so [.] _int_free 1.57% haproxy haproxy [.] pat_ref_push 1.48% haproxy libc-2.33.so [.] malloc_consolidate 1.22% haproxy libc-2.33.so [.] __strlen_avx2 1.05% haproxy libc-2.33.so [.] __strcmp_avx2 0.80% haproxy haproxy [.] pat_idx_tree_ip 0.74% haproxy libc-2.33.so [.] __memchr_avx2 0.69% haproxy libc-2.33.so [.] __strrchr_avx2 0.69% haproxy libc-2.33.so [.] _IO_getline_info 0.62% haproxy haproxy [.] pat_ref_read_from_file 0.56% haproxy libc-2.33.so [.] unlink_chunk.constprop.0 0.56% haproxy libc-2.33.so [.] cfree@GLIBC_2.2.5 0.46% haproxy haproxy [.] pat_ref_append If the addresses are totally disordered (via "shuf" on the input file), we see both implementations reach exactly 68.0s (slower due to much higher cache miss ratio). On large strings such as user agents (1 million here), it's now slightly slower (+9%): Before: real 0m2.475s user 0m2.316s sys 0m0.155s After: real 0m2.696s user 0m2.544s sys 0m0.147s But such patterns are much less common than short ones, and the memory savings do still count. Note that while it could be tempting to get rid of the list that chains all these pat_ref_elt together and only enumerate them by walking along the tree to save 16 extra bytes per entry, that's not possible due to the problem that insertion ordering is critical (think overlapping regex such as /index.* and /index.html). Currently it's not possible to proceed differently because patterns are first pre-loaded into the pat_ref via pat_ref_read_from_file_smp() and later indexed by pattern_read_from_file(), which has to only redo the second part anyway for maps/acls declared multiple times.
2025-01-12 13:38:28 -05:00
return pat_ref_set_from_elt(ref, elt, value, err);
MINOR: pattern: split pat_ref_set() split pat_ref_set() function in 2 distinct functions. Indeed, since 0844bed7d3 ("MEDIUM: map/acl: Improve pat_ref_set() efficiency (for "set-map", "add-acl" action perfs)"), pat_ref_set() prototype was updated to include an extra <elt> argument. But the logic behind is not explicit because the function will not only try to set <elt>, but also its duplicate (unlike pat_ref_set_elt() which only tries to update <elt>). Thus, to make it clearer and better distinguish between the key-based lookup version and the elt-based one, restotre pat_ref_set() previous prototype and add a dedicated pat_ref_set_elt_duplicate() that takes <elt> as argument and tries to update <elt> and all duplicates.
2024-11-20 10:22:22 -05:00
}
MINOR: pattern: add pat_ref_gen_set() function pat_ref_gen_set(ref, gen_id, value, err) modifies to <value> the sample of all patterns matching <key> and belonging to <gen_id> (generation id) under <ref> The goal is to be able to target a single subset from <ref>
2024-11-20 11:30:39 -05:00
/* This function modifies to <value> the sample of all patterns matching <key>
* and belonging to <gen_id> under <ref>.
*/
int pat_ref_gen_set(struct pat_ref *ref, unsigned int gen_id,
const char *key, const char *value, char **err)
{
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
struct pat_ref_gen *gen;
MINOR: pattern: add pat_ref_gen_set() function pat_ref_gen_set(ref, gen_id, value, err) modifies to <value> the sample of all patterns matching <key> and belonging to <gen_id> (generation id) under <ref> The goal is to be able to target a single subset from <ref>
2024-11-20 11:30:39 -05:00
struct pat_ref_elt *elt;
/* Look for pattern in the reference. */
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
gen = pat_ref_gen_get(ref, gen_id);
if (gen)
elt = cebs_item_lookup(&gen->elt_root, node, pattern, key, struct pat_ref_elt);
else
elt = NULL;
MEDIUM: migrate the patterns reference to cebs_tree cebs_tree are 24 bytes smaller than ebst_tree (16B vs 40B), and pattern references are only used during map/acl updates, so their storage is pure loss between updates (which most of the time never happen). By switching their indexing to compact trees, we can save 16 to 24 bytes per entry depending on alightment (here it's 24 per struct but 16 practical as malloc's alignment keeps 8 unused). Tested on core i7-8650U running at 3.0 GHz, with a file containing 17.7M IP addresses (16.7M different): $ time ./haproxy -c -f acl-ip.cfg Save 280 MB RAM for 17.7M IP addresses, and slightly speeds up the startup (5.8%, from 19.2s to 18.2s), a part of which possible being attributed to having to write less memory. Note that this is on small strings. On larger ones such as user-agents, ebtree doesn't reread the whole key and might be more efficient. Before: RAM (VSZ/RSS): 4443912 3912444 real 0m19.211s user 0m18.138s sys 0m1.068s Overhead Command Shared Object Symbol 44.79% haproxy haproxy [.] ebst_insert 25.07% haproxy haproxy [.] ebmb_insert_prefix 3.44% haproxy libc-2.33.so [.] __libc_calloc 2.71% haproxy libc-2.33.so [.] _int_malloc 2.33% haproxy haproxy [.] free_pattern_tree 1.78% haproxy libc-2.33.so [.] inet_pton4 1.62% haproxy libc-2.33.so [.] _IO_fgets 1.58% haproxy libc-2.33.so [.] _int_free 1.56% haproxy haproxy [.] pat_ref_push 1.35% haproxy libc-2.33.so [.] malloc_consolidate 1.16% haproxy libc-2.33.so [.] __strlen_avx2 0.79% haproxy haproxy [.] pat_idx_tree_ip 0.76% haproxy haproxy [.] pat_ref_read_from_file 0.60% haproxy libc-2.33.so [.] __strrchr_avx2 0.55% haproxy libc-2.33.so [.] unlink_chunk.constprop.0 0.54% haproxy libc-2.33.so [.] __memchr_avx2 0.46% haproxy haproxy [.] pat_ref_append After: RAM (VSZ/RSS): 4166108 3634768 real 0m18.114s user 0m17.113s sys 0m0.996s Overhead Command Shared Object Symbol 38.99% haproxy haproxy [.] cebs_insert 27.09% haproxy haproxy [.] ebmb_insert_prefix 3.63% haproxy libc-2.33.so [.] __libc_calloc 3.18% haproxy libc-2.33.so [.] _int_malloc 2.69% haproxy haproxy [.] free_pattern_tree 1.99% haproxy libc-2.33.so [.] inet_pton4 1.74% haproxy libc-2.33.so [.] _IO_fgets 1.73% haproxy libc-2.33.so [.] _int_free 1.57% haproxy haproxy [.] pat_ref_push 1.48% haproxy libc-2.33.so [.] malloc_consolidate 1.22% haproxy libc-2.33.so [.] __strlen_avx2 1.05% haproxy libc-2.33.so [.] __strcmp_avx2 0.80% haproxy haproxy [.] pat_idx_tree_ip 0.74% haproxy libc-2.33.so [.] __memchr_avx2 0.69% haproxy libc-2.33.so [.] __strrchr_avx2 0.69% haproxy libc-2.33.so [.] _IO_getline_info 0.62% haproxy haproxy [.] pat_ref_read_from_file 0.56% haproxy libc-2.33.so [.] unlink_chunk.constprop.0 0.56% haproxy libc-2.33.so [.] cfree@GLIBC_2.2.5 0.46% haproxy haproxy [.] pat_ref_append If the addresses are totally disordered (via "shuf" on the input file), we see both implementations reach exactly 68.0s (slower due to much higher cache miss ratio). On large strings such as user agents (1 million here), it's now slightly slower (+9%): Before: real 0m2.475s user 0m2.316s sys 0m0.155s After: real 0m2.696s user 0m2.544s sys 0m0.147s But such patterns are much less common than short ones, and the memory savings do still count. Note that while it could be tempting to get rid of the list that chains all these pat_ref_elt together and only enumerate them by walking along the tree to save 16 extra bytes per entry, that's not possible due to the problem that insertion ordering is critical (think overlapping regex such as /index.* and /index.html). Currently it's not possible to proceed differently because patterns are first pre-loaded into the pat_ref via pat_ref_read_from_file_smp() and later indexed by pattern_read_from_file(), which has to only redo the second part anyway for maps/acls declared multiple times.
2025-01-12 13:38:28 -05:00
return pat_ref_set_from_elt(ref, elt, value, err);
MINOR: pattern: add pat_ref_gen_set() function pat_ref_gen_set(ref, gen_id, value, err) modifies to <value> the sample of all patterns matching <key> and belonging to <gen_id> (generation id) under <ref> The goal is to be able to target a single subset from <ref>
2024-11-20 11:30:39 -05:00
}
MINOR: pattern: split pat_ref_set() split pat_ref_set() function in 2 distinct functions. Indeed, since 0844bed7d3 ("MEDIUM: map/acl: Improve pat_ref_set() efficiency (for "set-map", "add-acl" action perfs)"), pat_ref_set() prototype was updated to include an extra <elt> argument. But the logic behind is not explicit because the function will not only try to set <elt>, but also its duplicate (unlike pat_ref_set_elt() which only tries to update <elt>). Thus, to make it clearer and better distinguish between the key-based lookup version and the elt-based one, restotre pat_ref_set() previous prototype and add a dedicated pat_ref_set_elt_duplicate() that takes <elt> as argument and tries to update <elt> and all duplicates.
2024-11-20 10:22:22 -05:00
/* This function modifies to <value> the sample of all patterns matching <key>
* under <ref>.
*/
int pat_ref_set(struct pat_ref *ref, const char *key, const char *value, char **err)
{
MEDIUM: pattern: always consider gen_id for pat_ref lookup operations Historically, pat_ref lookup operations were performed on the whole pat_ref elements list. As such, set, find and delete operations on a given key would cause any matching element in pat_ref to be considered. When prepare/commit operations were added, gen_id was impelemnted in order to be able to work on a subset from pat_ref without impacting the current (live) version from pat_ref, until a new subset is committed to replace the current one. While the logic was good, there remained a design flaw from the historical implementation: indeed, legacy functions such as pat_ref_set(), pat_ref_delete() and pat_ref_find_elt() kept performing the lookups on the whole set of elements instead of considering only elements from the current subset. Because of this, mixing new prepare/commit operations with legacy operations could yield unexpected results. For instance, before this commit: echo "add map #0 key oldvalue" | socat /tmp/ha.sock - echo "prepare map #0" | socat /tmp/ha.sock - New version created: 1 echo "add map @1 #0 key newvalue" | socat /tmp/ha.sock - echo "del map #0 key" | socat /tmp/ha.sock - echo "commit map @1 #0" | socat /tmp/ha.sock - -> the result would be that "key" entry doesn't exist anymore after the commit, while we would expect the new value to be there instead. Thanks to the previous commits, we may finally fix this issue: for set, find_elt and delete operations, the current generation id is considered. With the above example, it means that the "del map #0 key" would only target elements from the current subset, thus elements in "version 1" of the map would be immune to the delete (as we would expect it to work).
2024-11-20 13:03:00 -05:00
return pat_ref_gen_set(ref, ref->curr_gen, key, value, err);
MINOR: pattern: split pat_ref_set() split pat_ref_set() function in 2 distinct functions. Indeed, since 0844bed7d3 ("MEDIUM: map/acl: Improve pat_ref_set() efficiency (for "set-map", "add-acl" action perfs)"), pat_ref_set() prototype was updated to include an extra <elt> argument. But the logic behind is not explicit because the function will not only try to set <elt>, but also its duplicate (unlike pat_ref_set_elt() which only tries to update <elt>). Thus, to make it clearer and better distinguish between the key-based lookup version and the elt-based one, restotre pat_ref_set() previous prototype and add a dedicated pat_ref_set_elt_duplicate() that takes <elt> as argument and tries to update <elt> and all duplicates.
2024-11-20 10:22:22 -05:00
}
MINOR: pattern: add _pat_ref_new() helper func pat_ref_newid() and pat_ref_new() are two functions to create and initialize a pat_ref struct based on input parameters. Both function perform the same generic allocation and initialization for pat_ref struct, thus there is quite a lot of code redundancy. This is error-prone if the pat_ref init sequence has to be updated at some point. To reduce maintenance costs, let's add a _pat_ref_new() helper func that takes care of the generic allocation and base initialization for pat_ref struct.
2024-11-07 04:01:40 -05:00
/* helper function to create and initialize a generic pat_ref struct
*
* Returns the new struct on success and NULL on failure (memory allocation
* error)
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
*/
MINOR: pattern: add _pat_ref_new() helper func pat_ref_newid() and pat_ref_new() are two functions to create and initialize a pat_ref struct based on input parameters. Both function perform the same generic allocation and initialization for pat_ref struct, thus there is quite a lot of code redundancy. This is error-prone if the pat_ref init sequence has to be updated at some point. To reduce maintenance costs, let's add a _pat_ref_new() helper func that takes care of the generic allocation and base initialization for pat_ref struct.
2024-11-07 04:01:40 -05:00
static struct pat_ref *_pat_ref_new(const char *display, unsigned int flags)
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
{
struct pat_ref *ref;
OPTIM: pattern: use malloc() to initialize new pat_ref struct As mentioned in the previous commit, in _pat_ref_new(), it was not strictly needed to explicitly assign all struct members to 0 since the struct was allocated with calloc() which does the zeroing for us. However, it was verified that we already initialize all fields explictly, thus there is no reason to keep using calloc() instead of malloc(). In fact using malloc() is less expensive, so let's use that instead now.
2024-11-07 04:05:30 -05:00
ref = malloc(sizeof(*ref));
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
if (!ref)
return NULL;
OPTIM: pattern: use malloc() to initialize new pat_ref struct As mentioned in the previous commit, in _pat_ref_new(), it was not strictly needed to explicitly assign all struct members to 0 since the struct was allocated with calloc() which does the zeroing for us. However, it was verified that we already initialize all fields explictly, thus there is no reason to keep using calloc() instead of malloc(). In fact using malloc() is less expensive, so let's use that instead now.
2024-11-07 04:05:30 -05:00
/* don't forget to explicitly initialize all pat_ref struct members */
MINOR: pattern: add _pat_ref_new() helper func pat_ref_newid() and pat_ref_new() are two functions to create and initialize a pat_ref struct based on input parameters. Both function perform the same generic allocation and initialization for pat_ref struct, thus there is quite a lot of code redundancy. This is error-prone if the pat_ref init sequence has to be updated at some point. To reduce maintenance costs, let's add a _pat_ref_new() helper func that takes care of the generic allocation and base initialization for pat_ref struct.
2024-11-07 04:01:40 -05:00
MINOR: pattern: store configuration reference for each acl or map pattern. This patch permit to add reference for each pattern reference. This is useful to identify the acl listed.
2014-02-10 21:31:34 -05:00
if (display) {
ref->display = strdup(display);
if (!ref->display) {
free(ref);
return NULL;
}
}
MINOR: pattern: add _pat_ref_new() helper func pat_ref_newid() and pat_ref_new() are two functions to create and initialize a pat_ref struct based on input parameters. Both function perform the same generic allocation and initialization for pat_ref struct, thus there is quite a lot of code redundancy. This is error-prone if the pat_ref init sequence has to be updated at some point. To reduce maintenance costs, let's add a _pat_ref_new() helper func that takes care of the generic allocation and base initialization for pat_ref struct.
2024-11-07 04:01:40 -05:00
ref->reference = NULL;
ref->flags = flags;
ref->curr_gen = 0;
ref->next_gen = 0;
ref->unique_id = -1;
ref->revision = 0;
ref->entry_cnt = 0;
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
ceb_init_root(&ref->gen_root);
OPTIM: patterns: cache the current generation This makes a significant difference when loading large files and during commit and clear operations, thanks to improved cache locality. In the measurements below, master refers to the code before any of the changes to the patterns code, not the code before this one commit. Timing the replacement of 10M entries from the CLI with this command which also reports timestamps at start, end of upload and end of clear: $ (echo "prompt i"; echo "show activity"; echo "prepare acl #0"; awk '{print "add acl @1 #0",$0}' < bad-ip.map; echo "show activity"; echo "commit acl @1 #0"; echo "clear acl @0 #0";echo "show activity") | socat -t 10 - /tmp/sock1 | grep ^uptim master, on a 3.7 GHz EPYC, 3 samples: uptime_now: 6.087030 uptime_now: 25.981777 => 21.9 sec insertion time uptime_now: 29.286368 => 3.3 sec commit+clear uptime_now: 5.748087 uptime_now: 25.740675 => 20.0s insertion time uptime_now: 29.039023 => 3.3 s commit+clear uptime_now: 7.065362 uptime_now: 26.769596 => 19.7s insertion time uptime_now: 30.065044 => 3.3s commit+clear And after this commit: uptime_now: 6.119215 uptime_now: 25.023019 => 18.9 sec insertion time uptime_now: 27.155503 => 2.1 sec commit+clear uptime_now: 5.675931 uptime_now: 24.551035 => 18.9s insertion uptime_now: 26.652352 => 2.1s commit+clear uptime_now: 6.722256 uptime_now: 25.593952 => 18.9s insertion uptime_now: 27.724153 => 2.1s commit+clear Now timing the startup time with a 10M entries file (on another machine) on master, 20 samples: Standard Deviation, s: 0.061652677408033 Mean: 4.217 And after this commit: Standard Deviation, s: 0.081821371548669 Mean: 3.78
2025-12-22 09:12:40 -05:00
ref->cached_gen.id = ref->curr_gen;
ref->cached_gen.data = NULL;
MINOR: pattern: add _pat_ref_new() helper func pat_ref_newid() and pat_ref_new() are two functions to create and initialize a pat_ref struct based on input parameters. Both function perform the same generic allocation and initialization for pat_ref struct, thus there is quite a lot of code redundancy. This is error-prone if the pat_ref init sequence has to be updated at some point. To reduce maintenance costs, let's add a _pat_ref_new() helper func that takes care of the generic allocation and base initialization for pat_ref struct.
2024-11-07 04:01:40 -05:00
LIST_INIT(&ref->pat);
HA_RWLOCK_INIT(&ref->lock);
MINOR: pattern: publish event_hdl events on pat_ref updates Now that PAT_REF events were defined in previous commit, let's actually publish them from pattern API where relevant. Unlike server events, pattern reference events are only published in the pat_ref subscriber's list on purpose, because in some setups patref updates (updates performed on a map for instance from action or cli) are very frequent, and we don't want to impact pattern API performance just for that. Moreover, as the main use case is to be able to subscribe to maps updates from Lua, allowing a per-pattern reference registration is already enough. No additional data is provided for such events (also for performance reason) Care was taken not to publish events when the update doesn't affect the live subset (the one targeted by curr_gen).
2024-10-18 12:40:41 -04:00
event_hdl_sub_list_init(&ref->e_subs);
MINOR: pattern: add _pat_ref_new() helper func pat_ref_newid() and pat_ref_new() are two functions to create and initialize a pat_ref struct based on input parameters. Both function perform the same generic allocation and initialization for pat_ref struct, thus there is quite a lot of code redundancy. This is error-prone if the pat_ref init sequence has to be updated at some point. To reduce maintenance costs, let's add a _pat_ref_new() helper func that takes care of the generic allocation and base initialization for pat_ref struct.
2024-11-07 04:01:40 -05:00
return ref;
}
MINOR: pattern: add pat_ref_free() helper func For now, pat_ref struct are never freed, except during init in case of error. The freeing is done directly in the init functions because we don't have an helper for that. No having an helper func to properly free pat_ref struct doesn't encourage us to free unused pat_ref structs, plus it is error-prone if new dynamic members are added to pat_ref struct in the future. To fix that, let's add a pat_ref_free() helper func and use it where relevant (which means only under pat_ref init function for now..)
2024-11-07 04:11:17 -05:00
/* helper func to properly de-initialize and free pat_ref struct */
static void pat_ref_free(struct pat_ref *ref)
{
ha_free(&ref->reference);
ha_free(&ref->display);
MINOR: pattern: publish event_hdl events on pat_ref updates Now that PAT_REF events were defined in previous commit, let's actually publish them from pattern API where relevant. Unlike server events, pattern reference events are only published in the pat_ref subscriber's list on purpose, because in some setups patref updates (updates performed on a map for instance from action or cli) are very frequent, and we don't want to impact pattern API performance just for that. Moreover, as the main use case is to be able to subscribe to maps updates from Lua, allowing a per-pattern reference registration is already enough. No additional data is provided for such events (also for performance reason) Care was taken not to publish events when the update doesn't affect the live subset (the one targeted by curr_gen).
2024-10-18 12:40:41 -04:00
event_hdl_sub_list_destroy(&ref->e_subs);
MINOR: pattern: add pat_ref_free() helper func For now, pat_ref struct are never freed, except during init in case of error. The freeing is done directly in the init functions because we don't have an helper for that. No having an helper func to properly free pat_ref struct doesn't encourage us to free unused pat_ref structs, plus it is error-prone if new dynamic members are added to pat_ref struct in the future. To fix that, let's add a pat_ref_free() helper func and use it where relevant (which means only under pat_ref init function for now..)
2024-11-07 04:11:17 -05:00
free(ref);
}
MINOR: pattern: add _pat_ref_new() helper func pat_ref_newid() and pat_ref_new() are two functions to create and initialize a pat_ref struct based on input parameters. Both function perform the same generic allocation and initialization for pat_ref struct, thus there is quite a lot of code redundancy. This is error-prone if the pat_ref init sequence has to be updated at some point. To reduce maintenance costs, let's add a _pat_ref_new() helper func that takes care of the generic allocation and base initialization for pat_ref struct.
2024-11-07 04:01:40 -05:00
/* This function creates a new reference. <ref> is the reference name.
* <flags> are PAT_REF_*. /!\ The reference is not checked, and must
* be unique. The user must check the reference with "pat_ref_lookup()"
* before calling this function. If the function fails, it returns NULL,
* otherwise it returns the new struct pat_ref.
*/
struct pat_ref *pat_ref_new(const char *reference, const char *display, unsigned int flags)
{
struct pat_ref *ref;
ref = _pat_ref_new(display, flags);
if (!ref)
return NULL;
MEDIUM: pattern: Add support for virtual and optional files for patterns Before this patch, it was not possible to use a list of patterns, map or a list of acls, without an existing file. However, it could be handy to just use an ID, with no file on the disk. It is pretty useful for everyone managing dynamically these lists. It could also be handy to try to load a list from a file if it exists without failing if not. This way, it could be possible to make a cold start without any file (instead of empty file), dynamically add and del patterns, dump the list to the file periodically to reuse it on reload (via an external process). In this patch, we uses some prefixes to be able to use virtual or optional files. The default case remains unchanged. regular files are used. A filename, with no prefix, is used as reference, and it must exist on the disk. With the prefix "file@", the same is performed. Internally this prefix is skipped. Thus the same file, with ou without "file@" prefix, references the same list of patterns. To use a virtual map, "virt@" prefix must be used. No file is read, even if the following name looks like a file. It is just an ID. The prefix is part of ID and must always be used. To use a optional file, ie a file that may or may not exist on a disk at startup, "opt@" prefix must be used. If the file exists, its content is loaded. But HAProxy doesn't complain if not. The prefix is not part of ID. For a given file, optional files and regular files reference the same list of patterns. This patch should fix the issue #2202.
2023-12-01 06:04:35 -05:00
if (strlen(reference) > 5 && strncmp(reference, "virt@", 5) == 0)
MINOR: pattern: add _pat_ref_new() helper func pat_ref_newid() and pat_ref_new() are two functions to create and initialize a pat_ref struct based on input parameters. Both function perform the same generic allocation and initialization for pat_ref struct, thus there is quite a lot of code redundancy. This is error-prone if the pat_ref init sequence has to be updated at some point. To reduce maintenance costs, let's add a _pat_ref_new() helper func that takes care of the generic allocation and base initialization for pat_ref struct.
2024-11-07 04:01:40 -05:00
ref->flags |= PAT_REF_ID;
MEDIUM: pattern: Add support for virtual and optional files for patterns Before this patch, it was not possible to use a list of patterns, map or a list of acls, without an existing file. However, it could be handy to just use an ID, with no file on the disk. It is pretty useful for everyone managing dynamically these lists. It could also be handy to try to load a list from a file if it exists without failing if not. This way, it could be possible to make a cold start without any file (instead of empty file), dynamically add and del patterns, dump the list to the file periodically to reuse it on reload (via an external process). In this patch, we uses some prefixes to be able to use virtual or optional files. The default case remains unchanged. regular files are used. A filename, with no prefix, is used as reference, and it must exist on the disk. With the prefix "file@", the same is performed. Internally this prefix is skipped. Thus the same file, with ou without "file@" prefix, references the same list of patterns. To use a virtual map, "virt@" prefix must be used. No file is read, even if the following name looks like a file. It is just an ID. The prefix is part of ID and must always be used. To use a optional file, ie a file that may or may not exist on a disk at startup, "opt@" prefix must be used. If the file exists, its content is loaded. But HAProxy doesn't complain if not. The prefix is not part of ID. For a given file, optional files and regular files reference the same list of patterns. This patch should fix the issue #2202.
2023-12-01 06:04:35 -05:00
else if (strlen(reference) > 4 && strncmp(reference, "opt@", 4) == 0) {
MINOR: pattern: add _pat_ref_new() helper func pat_ref_newid() and pat_ref_new() are two functions to create and initialize a pat_ref struct based on input parameters. Both function perform the same generic allocation and initialization for pat_ref struct, thus there is quite a lot of code redundancy. This is error-prone if the pat_ref init sequence has to be updated at some point. To reduce maintenance costs, let's add a _pat_ref_new() helper func that takes care of the generic allocation and base initialization for pat_ref struct.
2024-11-07 04:01:40 -05:00
ref->flags |= (PAT_REF_ID|PAT_REF_FILE); // Will be decided later
MEDIUM: pattern: Add support for virtual and optional files for patterns Before this patch, it was not possible to use a list of patterns, map or a list of acls, without an existing file. However, it could be handy to just use an ID, with no file on the disk. It is pretty useful for everyone managing dynamically these lists. It could also be handy to try to load a list from a file if it exists without failing if not. This way, it could be possible to make a cold start without any file (instead of empty file), dynamically add and del patterns, dump the list to the file periodically to reuse it on reload (via an external process). In this patch, we uses some prefixes to be able to use virtual or optional files. The default case remains unchanged. regular files are used. A filename, with no prefix, is used as reference, and it must exist on the disk. With the prefix "file@", the same is performed. Internally this prefix is skipped. Thus the same file, with ou without "file@" prefix, references the same list of patterns. To use a virtual map, "virt@" prefix must be used. No file is read, even if the following name looks like a file. It is just an ID. The prefix is part of ID and must always be used. To use a optional file, ie a file that may or may not exist on a disk at startup, "opt@" prefix must be used. If the file exists, its content is loaded. But HAProxy doesn't complain if not. The prefix is not part of ID. For a given file, optional files and regular files reference the same list of patterns. This patch should fix the issue #2202.
2023-12-01 06:04:35 -05:00
reference += 4;
}
else {
/* A file by default */
MINOR: pattern: add _pat_ref_new() helper func pat_ref_newid() and pat_ref_new() are two functions to create and initialize a pat_ref struct based on input parameters. Both function perform the same generic allocation and initialization for pat_ref struct, thus there is quite a lot of code redundancy. This is error-prone if the pat_ref init sequence has to be updated at some point. To reduce maintenance costs, let's add a _pat_ref_new() helper func that takes care of the generic allocation and base initialization for pat_ref struct.
2024-11-07 04:01:40 -05:00
ref->flags |= PAT_REF_FILE;
MEDIUM: pattern: Add support for virtual and optional files for patterns Before this patch, it was not possible to use a list of patterns, map or a list of acls, without an existing file. However, it could be handy to just use an ID, with no file on the disk. It is pretty useful for everyone managing dynamically these lists. It could also be handy to try to load a list from a file if it exists without failing if not. This way, it could be possible to make a cold start without any file (instead of empty file), dynamically add and del patterns, dump the list to the file periodically to reuse it on reload (via an external process). In this patch, we uses some prefixes to be able to use virtual or optional files. The default case remains unchanged. regular files are used. A filename, with no prefix, is used as reference, and it must exist on the disk. With the prefix "file@", the same is performed. Internally this prefix is skipped. Thus the same file, with ou without "file@" prefix, references the same list of patterns. To use a virtual map, "virt@" prefix must be used. No file is read, even if the following name looks like a file. It is just an ID. The prefix is part of ID and must always be used. To use a optional file, ie a file that may or may not exist on a disk at startup, "opt@" prefix must be used. If the file exists, its content is loaded. But HAProxy doesn't complain if not. The prefix is not part of ID. For a given file, optional files and regular files reference the same list of patterns. This patch should fix the issue #2202.
2023-12-01 06:04:35 -05:00
/* Skip file@ prefix to be mixed with ref omitting the prefix */
if (strlen(reference) > 5 && strncmp(reference, "file@", 5) == 0)
reference += 5;
}
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
ref->reference = strdup(reference);
if (!ref->reference) {
MINOR: pattern: add pat_ref_free() helper func For now, pat_ref struct are never freed, except during init in case of error. The freeing is done directly in the init functions because we don't have an helper for that. No having an helper func to properly free pat_ref struct doesn't encourage us to free unused pat_ref structs, plus it is error-prone if new dynamic members are added to pat_ref struct in the future. To fix that, let's add a pat_ref_free() helper func and use it where relevant (which means only under pat_ref init function for now..)
2024-11-07 04:11:17 -05:00
pat_ref_free(ref);
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
return NULL;
}
CLEANUP: lists/tree-wide: rename some list operations to avoid some confusion The current "ADD" vs "ADDQ" is confusing because when thinking in terms of appending at the end of a list, "ADD" naturally comes to mind, but here it does the opposite, it inserts. Several times already it's been incorrectly used where ADDQ was expected, the latest of which was a fortunate accident explained in 6fa922562 ("CLEANUP: stream: explain why we queue the stream at the head of the server list"). Let's use more explicit (but slightly longer) names now: LIST_ADD -> LIST_INSERT LIST_ADDQ -> LIST_APPEND LIST_ADDED -> LIST_INLIST LIST_DEL -> LIST_DELETE The same is true for MT_LISTs, including their "TRY" variant. LIST_DEL_INIT keeps its short name to encourage to use it instead of the lazier LIST_DELETE which is often less safe. The change is large (~674 non-comment entries) but is mechanical enough to remain safe. No permutation was performed, so any out-of-tree code can easily map older names to new ones. The list doc was updated.
2021-04-21 01:32:39 -04:00
LIST_APPEND(&pattern_reference, &ref->list);
MINOR: pattern: Each pattern is identified by unique id. The pattern reference are stored with two identifiers: the unique_id and the reference. The reference identify a file. Each file with the same name point to the same reference. We can register many times one file. If the file is modified, all his dependencies are also modified. The reference can be used with map or acl. The unique_id identify inline acl. The unique id is unique for each acl. You cannot force the same id in the configuration file, because this repport an error. The format of the acl and map listing through the "socket" has changed for displaying these new ids.
2014-03-11 09:29:22 -04:00
return ref;
}
CLEANUP: pattern: fix spelling/grammatical/copy-paste in comments The code is horrible to work with because most functions are documented with misleading comments resulting from many spelling and grammatical mistakes, and plenty of remains of copy-paste mentioning arguments that do not exist and return values that are never set. Too many hours wasted writing non-working code because of assumptions resulting from this, let's fix this once for all now!
2020-10-30 11:03:50 -04:00
/* This function creates a new reference. <unique_id> is the unique id. If
MINOR: pattern: Each pattern is identified by unique id. The pattern reference are stored with two identifiers: the unique_id and the reference. The reference identify a file. Each file with the same name point to the same reference. We can register many times one file. If the file is modified, all his dependencies are also modified. The reference can be used with map or acl. The unique_id identify inline acl. The unique id is unique for each acl. You cannot force the same id in the configuration file, because this repport an error. The format of the acl and map listing through the "socket" has changed for displaying these new ids.
2014-03-11 09:29:22 -04:00
* the value of <unique_id> is -1, the unique id is calculated later.
* <flags> are PAT_REF_*. /!\ The reference is not checked, and must
* be unique. The user must check the reference with "pat_ref_lookup()"
* or pat_ref_lookupid before calling this function. If the function
CLEANUP: pattern: fix spelling/grammatical/copy-paste in comments The code is horrible to work with because most functions are documented with misleading comments resulting from many spelling and grammatical mistakes, and plenty of remains of copy-paste mentioning arguments that do not exist and return values that are never set. Too many hours wasted writing non-working code because of assumptions resulting from this, let's fix this once for all now!
2020-10-30 11:03:50 -04:00
* fails, it returns NULL, otherwise it returns the new struct pat_ref.
MINOR: pattern: Each pattern is identified by unique id. The pattern reference are stored with two identifiers: the unique_id and the reference. The reference identify a file. Each file with the same name point to the same reference. We can register many times one file. If the file is modified, all his dependencies are also modified. The reference can be used with map or acl. The unique_id identify inline acl. The unique id is unique for each acl. You cannot force the same id in the configuration file, because this repport an error. The format of the acl and map listing through the "socket" has changed for displaying these new ids.
2014-03-11 09:29:22 -04:00
*/
MINOR: pattern: store configuration reference for each acl or map pattern. This patch permit to add reference for each pattern reference. This is useful to identify the acl listed.
2014-02-10 21:31:34 -05:00
struct pat_ref *pat_ref_newid(int unique_id, const char *display, unsigned int flags)
MINOR: pattern: Each pattern is identified by unique id. The pattern reference are stored with two identifiers: the unique_id and the reference. The reference identify a file. Each file with the same name point to the same reference. We can register many times one file. If the file is modified, all his dependencies are also modified. The reference can be used with map or acl. The unique_id identify inline acl. The unique id is unique for each acl. You cannot force the same id in the configuration file, because this repport an error. The format of the acl and map listing through the "socket" has changed for displaying these new ids.
2014-03-11 09:29:22 -04:00
{
struct pat_ref *ref;
MINOR: pattern: add _pat_ref_new() helper func pat_ref_newid() and pat_ref_new() are two functions to create and initialize a pat_ref struct based on input parameters. Both function perform the same generic allocation and initialization for pat_ref struct, thus there is quite a lot of code redundancy. This is error-prone if the pat_ref init sequence has to be updated at some point. To reduce maintenance costs, let's add a _pat_ref_new() helper func that takes care of the generic allocation and base initialization for pat_ref struct.
2024-11-07 04:01:40 -05:00
ref = _pat_ref_new(display, flags);
MINOR: pattern: Each pattern is identified by unique id. The pattern reference are stored with two identifiers: the unique_id and the reference. The reference identify a file. Each file with the same name point to the same reference. We can register many times one file. If the file is modified, all his dependencies are also modified. The reference can be used with map or acl. The unique_id identify inline acl. The unique id is unique for each acl. You cannot force the same id in the configuration file, because this repport an error. The format of the acl and map listing through the "socket" has changed for displaying these new ids.
2014-03-11 09:29:22 -04:00
if (!ref)
return NULL;
ref->unique_id = unique_id;
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
MINOR: pattern: add _pat_ref_new() helper func pat_ref_newid() and pat_ref_new() are two functions to create and initialize a pat_ref struct based on input parameters. Both function perform the same generic allocation and initialization for pat_ref struct, thus there is quite a lot of code redundancy. This is error-prone if the pat_ref init sequence has to be updated at some point. To reduce maintenance costs, let's add a _pat_ref_new() helper func that takes care of the generic allocation and base initialization for pat_ref struct.
2024-11-07 04:01:40 -05:00
LIST_APPEND(&pattern_reference, &ref->list);
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
return ref;
}
MINOR: pattern: make pat_ref_append() return the newly added element It's more convenient to return the element than to return just 0 or 1, as the next thing we'll want to do is to act on this element! In addition it was using variable arguments instead of consts, causing some reuse constraints which were also addressed. This doesn't change its use as a boolean, hence why call places were not modified.
2020-10-28 05:52:46 -04:00
/* This function adds entry to <ref>. It can fail on memory error. It returns
* the newly added element on success, or NULL on failure. The PATREF_LOCK on
MINOR: pattern: store a generation number in the reference patterns Right now it's not possible to perform a safe reload because we don't know what patterns were recently added or were already present. This patch adds a generation counter to the reference patterns so that it is possible to know what generation of the reference they were loaded with. A reference now has two generations, the current one, used for all additions, and the next one, allocated to those wishing to update the contents. The generation wraps at 2^32 so comparisons must be made relative to the current position. The idea will be that upon full reload, the caller will first get a new generation ID, will insert all new patterns using it, will then switch the current ID to the new one, and will delete all entries older than the current ID. This has the benefit of supporting chunked updates that remain consistent and that won't block the whole process for ages like pat_ref_reload() currently does.
2020-10-28 06:43:49 -04:00
* <ref> must be held. It sets the newly created pattern's generation number
* to the same value as the reference's.
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
*/
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
struct pat_ref_elt *pat_ref_append(struct pat_ref *ref, unsigned int gen_id,
const char *pattern, const char *sample, int line)
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
{
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
struct pat_ref_gen *gen;
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
struct pat_ref_elt *elt;
OPTIM: pattern: save memory and time using ebst instead of ebis In the pat_ref_elt struct, the pattern string is stored outside of the node element, using a pointer to an strdup(). Not only this needlessly wastes at least 16-24 bytes per entry (8 for the pointer, 8-16 for the allocator), it also makes the tree descent less efficient since both the node and the string have to be visited for each layer (hence at least two cache lines). Let's use an ebmb storage and place the pattern right at the end of the pat_ref_elt, making it a variable-sized element instead. The set-map test below jumps from 173 to 182 kreq/s/core, and the memory usage drops from 356 MB to 324 MB: http-request set-map(/dev/null) %[rand(1000000)] 1 This is even more visible with large maps: after loading 16M IP addresses into a map, the process uses this amount of memory: - 3.15 GB with haproxy-2.8 - 4.21 GB with haproxy-2.9-dev11 - 3.68 GB with this patch So that's a net saving of 32 bytes per entry here, which cuts in half the extra cost of the tree, and loading a large map takes about 20% less time.
2023-11-26 05:56:08 -05:00
int len = strlen(pattern);
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
OPTIM: pattern: save memory and time using ebst instead of ebis In the pat_ref_elt struct, the pattern string is stored outside of the node element, using a pointer to an strdup(). Not only this needlessly wastes at least 16-24 bytes per entry (8 for the pointer, 8-16 for the allocator), it also makes the tree descent less efficient since both the node and the string have to be visited for each layer (hence at least two cache lines). Let's use an ebmb storage and place the pattern right at the end of the pat_ref_elt, making it a variable-sized element instead. The set-map test below jumps from 173 to 182 kreq/s/core, and the memory usage drops from 356 MB to 324 MB: http-request set-map(/dev/null) %[rand(1000000)] 1 This is even more visible with large maps: after loading 16M IP addresses into a map, the process uses this amount of memory: - 3.15 GB with haproxy-2.8 - 4.21 GB with haproxy-2.9-dev11 - 3.68 GB with this patch So that's a net saving of 32 bytes per entry here, which cuts in half the extra cost of the tree, and loading a large map takes about 20% less time.
2023-11-26 05:56:08 -05:00
elt = calloc(1, sizeof(*elt) + len + 1);
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
if (!elt)
MINOR: pattern: make pat_ref_append() return the newly added element It's more convenient to return the element than to return just 0 or 1, as the next thing we'll want to do is to act on this element! In addition it was using variable arguments instead of consts, causing some reuse constraints which were also addressed. This doesn't change its use as a boolean, hence why call places were not modified.
2020-10-28 05:52:46 -04:00
goto fail;
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
gen = pat_ref_gen_get(ref, gen_id);
if (!gen) {
gen = pat_ref_gen_new(ref, gen_id);
if (!gen)
goto fail;
}
BUG/MAJOR: set the correct generation ID in pat_ref_append(). This fixes crashes when creating more than one new revision of a map or acl file and purging the previous version.
2025-12-30 13:58:56 -05:00
elt->gen_id = gen_id;
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
elt->line = line;
OPTIM: pattern: save memory and time using ebst instead of ebis In the pat_ref_elt struct, the pattern string is stored outside of the node element, using a pointer to an strdup(). Not only this needlessly wastes at least 16-24 bytes per entry (8 for the pointer, 8-16 for the allocator), it also makes the tree descent less efficient since both the node and the string have to be visited for each layer (hence at least two cache lines). Let's use an ebmb storage and place the pattern right at the end of the pat_ref_elt, making it a variable-sized element instead. The set-map test below jumps from 173 to 182 kreq/s/core, and the memory usage drops from 356 MB to 324 MB: http-request set-map(/dev/null) %[rand(1000000)] 1 This is even more visible with large maps: after loading 16M IP addresses into a map, the process uses this amount of memory: - 3.15 GB with haproxy-2.8 - 4.21 GB with haproxy-2.9-dev11 - 3.68 GB with this patch So that's a net saving of 32 bytes per entry here, which cuts in half the extra cost of the tree, and loading a large map takes about 20% less time.
2023-11-26 05:56:08 -05:00
memcpy((char*)elt->pattern, pattern, len + 1);
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
if (sample) {
elt->sample = strdup(sample);
MINOR: pattern: make pat_ref_append() return the newly added element It's more convenient to return the element than to return just 0 or 1, as the next thing we'll want to do is to act on this element! In addition it was using variable arguments instead of consts, causing some reuse constraints which were also addressed. This doesn't change its use as a boolean, hence why call places were not modified.
2020-10-28 05:52:46 -04:00
if (!elt->sample)
goto fail;
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
}
BUG/MAJOR: map: fix segfault during 'show map/acl' on cli. The reference of the current map/acl element to dump could be destroyed if map is updated from an 'http-request del-map' configuration rule or throught a 'del map/acl' on CLI. We use a 'back_refs' chaining element to fix this. As it is done to dump sessions. This patch needs also fix: 'BUG/MAJOR: cli: fix custom io_release was crushed by NULL.' To clean the back_ref and avoid a crash on a further del/clear map operation. Those fixes should be backported on mainline branches 1.7 and 1.6. This patch wont directly apply on 1.6.
2017-06-29 09:40:33 -04:00
LIST_INIT(&elt->back_refs);
MEDIUM: pattern: turn the pattern chaining to single-linked list It does not require heavy deletion from the expr anymore, so we can now turn this to a single-linked list since most of the time we want to delete all instances of a given pattern from the head. By doing so we save 32 bytes of memory per pattern. The pat_unlink_from_head() function was adjusted accordingly.
2020-11-03 08:50:29 -05:00
elt->list_head = NULL;
elt->tree_head = NULL;
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
LIST_APPEND(&gen->head, &elt->list);
cebs_item_insert(&gen->elt_root, node, pattern, elt);
MINOR: pattern: add a counter of added/freed patterns Patterns are allocated when loading maps/acls from a file or dynamically via the CLI, and are released only from the CLI (e.g. "clear map xxx"). These ones do not use pools and are much harder to monitor, e.g. in case a script adds many and forgets to clear them, etc. Let's add a new pair of metrics "PatternsAdded" and "PatternsFreed" that will report the number of added and freed patterns respectively. This can allow to simply graph both. The difference between the two normally represents the number of allocated patterns. If Added grows without Freed following, it can indicate a faulty script that doesn't perform the needed cleanup. The metrics are also made available to Prometheus as patterns_added_total and patterns_freed_total respectively.
2025-07-04 18:07:25 -04:00
HA_ATOMIC_INC(&patterns_added);
MINOR: pattern: make pat_ref_append() return the newly added element It's more convenient to return the element than to return just 0 or 1, as the next thing we'll want to do is to act on this element! In addition it was using variable arguments instead of consts, causing some reuse constraints which were also addressed. This doesn't change its use as a boolean, hence why call places were not modified.
2020-10-28 05:52:46 -04:00
return elt;
fail:
free(elt);
return NULL;
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
}
CLEANUP: pattern: fix spelling/grammatical/copy-paste in comments The code is horrible to work with because most functions are documented with misleading comments resulting from many spelling and grammatical mistakes, and plenty of remains of copy-paste mentioning arguments that do not exist and return values that are never set. Too many hours wasted writing non-working code because of assumptions resulting from this, let's fix this once for all now!
2020-10-30 11:03:50 -04:00
/* This function creates sample found in <elt>, parses the pattern also
* found in <elt> and inserts it in <expr>. The function copies <patflags>
* into <expr>. If the function fails, it returns 0 and <err> is filled.
CLEANUP: assorted typo fixes in the code and comments This is 10th iteration of typo fixes
2020-06-21 12:42:57 -04:00
* In success case, the function returns 1.
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
*/
int pat_ref_push(struct pat_ref_elt *elt, struct pattern_expr *expr,
int patflags, char **err)
{
MINOR: samples: rename some struct member from "smp" to "data" This members contains data and not sample.
2015-08-19 02:35:43 -04:00
struct sample_data *data;
MINOR: pattern: Merge function pattern_add() with pat_ref_push(). The function Pattern_add() is only used by pat_ref_push(). This patch remove the function pattern_add() and merge his code in the function pat_ref_push().
2014-01-28 09:34:35 -05:00
struct pattern pattern;
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
/* Create sample */
if (elt->sample && expr->pat_head->parse_smp) {
/* New sample. */
MINOR: samples: rename some struct member from "smp" to "data" This members contains data and not sample.
2015-08-19 02:35:43 -04:00
data = malloc(sizeof(*data));
if (!data)
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
return 0;
/* Parse value. */
MINOR: samples: rename some struct member from "smp" to "data" This members contains data and not sample.
2015-08-19 02:35:43 -04:00
if (!expr->pat_head->parse_smp(elt->sample, data)) {
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
memprintf(err, "unable to parse '%s'", elt->sample);
MINOR: samples: rename some struct member from "smp" to "data" This members contains data and not sample.
2015-08-19 02:35:43 -04:00
free(data);
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
return 0;
}
}
else
MINOR: samples: rename some struct member from "smp" to "data" This members contains data and not sample.
2015-08-19 02:35:43 -04:00
data = NULL;
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
MINOR: pattern: Merge function pattern_add() with pat_ref_push(). The function Pattern_add() is only used by pat_ref_push(). This patch remove the function pattern_add() and merge his code in the function pat_ref_push().
2014-01-28 09:34:35 -05:00
/* initialise pattern */
memset(&pattern, 0, sizeof(pattern));
MINOR: samples: rename some struct member from "smp" to "data" This members contains data and not sample.
2015-08-19 02:35:43 -04:00
pattern.data = data;
MINOR: pattern: Each pattern expression element store the reference struct. Now, each pattern entry known the original "struct pat_ref_elt" from that was built. This patch permit to delete each pattern entry without confusion. After this patch, each reference can use his pointer to be targeted.
2014-01-28 09:54:36 -05:00
pattern.ref = elt;
MINOR: pattern: Merge function pattern_add() with pat_ref_push(). The function Pattern_add() is only used by pat_ref_push(). This patch remove the function pattern_add() and merge his code in the function pat_ref_push().
2014-01-28 09:34:35 -05:00
/* parse pattern */
BUG/MEDIUM: patterns: last fix was still not enough Last fix did address the issue for inlined patterns, but it was not enough because the flags are lost as well when updating patterns dynamically over the CLI. Also if the same file was used once with -i and another time without -i, their references would have been merged and both would have used the same matching method. It's appear that the patterns have two types of flags. The first ones are relative to the pattern matching, and the second are relative to the pattern storage. The pattern matching flags are the same for all the patterns of one expression. Now they are stored in the expression. The storage flags are information returned by the pattern mathing function. This information is relative to each entry and is stored in the "struct pattern". Now, the expression matching flags are forwarded to the parse and index functions. These flags are stored during the configuration parsing, and they are used during the parse and index actions. This issue was introduced in dev23 with the major pattern rework, and is a continuation of commit a631fc8 ("BUG/MAJOR: patterns: -i and -n are ignored for inlined patterns"). No backport is needed.
2014-04-28 05:18:57 -04:00
if (!expr->pat_head->parse(elt->pattern, &pattern, expr->mflags, err)) {
MINOR: samples: rename some struct member from "smp" to "data" This members contains data and not sample.
2015-08-19 02:35:43 -04:00
free(data);
MINOR: pattern: Merge function pattern_add() with pat_ref_push(). The function Pattern_add() is only used by pat_ref_push(). This patch remove the function pattern_add() and merge his code in the function pat_ref_push().
2014-01-28 09:34:35 -05:00
return 0;
}
BUILD: threads: Rename SPIN/RWLOCK macros using HA_ prefix This remove any name conflicts, especially on Solaris.
2017-11-07 04:42:54 -05:00
HA_RWLOCK_WRLOCK(PATEXP_LOCK, &expr->lock);
MINOR: pattern: Merge function pattern_add() with pat_ref_push(). The function Pattern_add() is only used by pat_ref_push(). This patch remove the function pattern_add() and merge his code in the function pat_ref_push().
2014-01-28 09:34:35 -05:00
/* index pattern */
if (!expr->pat_head->index(expr, &pattern, err)) {
BUILD: threads: Rename SPIN/RWLOCK macros using HA_ prefix This remove any name conflicts, especially on Solaris.
2017-11-07 04:42:54 -05:00
HA_RWLOCK_WRUNLOCK(PATEXP_LOCK, &expr->lock);
MINOR: samples: rename some struct member from "smp" to "data" This members contains data and not sample.
2015-08-19 02:35:43 -04:00
free(data);
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
return 0;
}
BUILD: threads: Rename SPIN/RWLOCK macros using HA_ prefix This remove any name conflicts, especially on Solaris.
2017-11-07 04:42:54 -05:00
HA_RWLOCK_WRUNLOCK(PATEXP_LOCK, &expr->lock);
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
return 1;
}
MINOR: pattern: add pat_ref_commit() to commit a previously inserted element This function will be used after a successful pat_ref_append() to propagate the pattern to all use places (including parsing and indexing). On failure, it will entirely roll back all insertions and free the pattern itself. It also preserves the generation number so that it is convenient for use in association with pat_ref_append(). pat_ref_add() was modified to rely on it instead of open-coding the insertion and roll-back.
2020-10-28 13:45:45 -04:00
/* This function tries to commit entry <elt> into <ref>. The new entry must
* have already been inserted using pat_ref_append(), and its generation number
* may have been adjusted as it will not be changed. <err> must point to a NULL
* pointer. The PATREF lock on <ref> must be held. All the pattern_expr for
* this reference will be updated (parsing, indexing). On success, non-zero is
* returned. On failure, all the operation is rolled back (the element is
* deleted from all expressions and is freed), zero is returned and the error
* pointer <err> may have been updated (and the caller must free it). Failure
* causes include memory allocation, parsing error or indexing error.
*/
CLEANUP: pattern: rename pat_ref_commit() to pat_ref_commit_elt() It's about the third time I get confused by these functions, half of which manipulate the reference as a whole and those manipulating only an entry. For me "pat_ref_commit" means committing the pattern reference, not just an element, so let's rename it. A number of other ones should really be renamed before 2.4 gets released :-/
2021-01-15 08:11:59 -05:00
int pat_ref_commit_elt(struct pat_ref *ref, struct pat_ref_elt *elt, char **err)
MINOR: pattern: add pat_ref_commit() to commit a previously inserted element This function will be used after a successful pat_ref_append() to propagate the pattern to all use places (including parsing and indexing). On failure, it will entirely roll back all insertions and free the pattern itself. It also preserves the generation number so that it is convenient for use in association with pat_ref_append(). pat_ref_add() was modified to rely on it instead of open-coding the insertion and roll-back.
2020-10-28 13:45:45 -04:00
{
struct pattern_expr *expr;
list_for_each_entry(expr, &ref->pat, list) {
if (!pat_ref_push(elt, expr, 0, err)) {
pat_ref_delete_by_ptr(ref, elt);
return 0;
}
}
return 1;
}
MINOR: pattern: implement pat_ref_load() to load a pattern at a given generation pat_ref_load() basically combines pat_ref_append() and pat_ref_commit(). It's very similar to pat_ref_add() except that it also allows to set the generation ID and the line number. pat_ref_add() was modified to directly rely on it to avoid code duplication. Note that a previous declaration of pat_ref_load() was removed as it was just a leftover of an earlier incarnation of something possibly similar, so no existing functionality was changed here.
2020-10-29 04:21:43 -04:00
/* Loads <pattern>:<sample> into <ref> for generation <gen>. <sample> may be
* NULL if none exists (e.g. ACL). If not needed, the generation number should
* be set to ref->curr_gen. The error pointer must initially point to NULL. The
* new entry will be propagated to all use places, involving allocation, parsing
* and indexing. On error (parsing, allocation), the operation will be rolled
* back, an error may be reported, and NULL will be reported. On success, the
* freshly allocated element will be returned. The PATREF lock on <ref> must be
* held during the operation.
*/
struct pat_ref_elt *pat_ref_load(struct pat_ref *ref, unsigned int gen,
const char *pattern, const char *sample,
int line, char **err)
{
struct pat_ref_elt *elt;
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
elt = pat_ref_append(ref, gen, pattern, sample, line);
MINOR: pattern: implement pat_ref_load() to load a pattern at a given generation pat_ref_load() basically combines pat_ref_append() and pat_ref_commit(). It's very similar to pat_ref_add() except that it also allows to set the generation ID and the line number. pat_ref_add() was modified to directly rely on it to avoid code duplication. Note that a previous declaration of pat_ref_load() was removed as it was just a leftover of an earlier incarnation of something possibly similar, so no existing functionality was changed here.
2020-10-29 04:21:43 -04:00
if (elt) {
CLEANUP: pattern: rename pat_ref_commit() to pat_ref_commit_elt() It's about the third time I get confused by these functions, half of which manipulate the reference as a whole and those manipulating only an entry. For me "pat_ref_commit" means committing the pattern reference, not just an element, so let's rename it. A number of other ones should really be renamed before 2.4 gets released :-/
2021-01-15 08:11:59 -05:00
if (!pat_ref_commit_elt(ref, elt, err))
MINOR: pattern: implement pat_ref_load() to load a pattern at a given generation pat_ref_load() basically combines pat_ref_append() and pat_ref_commit(). It's very similar to pat_ref_add() except that it also allows to set the generation ID and the line number. pat_ref_add() was modified to directly rely on it to avoid code duplication. Note that a previous declaration of pat_ref_load() was removed as it was just a leftover of an earlier incarnation of something possibly similar, so no existing functionality was changed here.
2020-10-29 04:21:43 -04:00
elt = NULL;
} else
memprintf(err, "out of memory error");
MINOR: pattern: publish event_hdl events on pat_ref updates Now that PAT_REF events were defined in previous commit, let's actually publish them from pattern API where relevant. Unlike server events, pattern reference events are only published in the pat_ref subscriber's list on purpose, because in some setups patref updates (updates performed on a map for instance from action or cli) are very frequent, and we don't want to impact pattern API performance just for that. Moreover, as the main use case is to be able to subscribe to maps updates from Lua, allowing a per-pattern reference registration is already enough. No additional data is provided for such events (also for performance reason) Care was taken not to publish events when the update doesn't affect the live subset (the one targeted by curr_gen).
2024-10-18 12:40:41 -04:00
/* ignore if update requires committing to be seen */
if (elt && gen == ref->curr_gen)
event_hdl_publish(&ref->e_subs, EVENT_HDL_SUB_PAT_REF_ADD, NULL);
MINOR: pattern: implement pat_ref_load() to load a pattern at a given generation pat_ref_load() basically combines pat_ref_append() and pat_ref_commit(). It's very similar to pat_ref_add() except that it also allows to set the generation ID and the line number. pat_ref_add() was modified to directly rely on it to avoid code duplication. Note that a previous declaration of pat_ref_load() was removed as it was just a leftover of an earlier incarnation of something possibly similar, so no existing functionality was changed here.
2020-10-29 04:21:43 -04:00
return elt;
}
MINOR: pattern: make pat_ref_add() rely on pat_ref_append() Let's remove unneeded code duplication, both are exactly the same.
2020-10-28 05:58:05 -04:00
/* This function adds entry to <ref>. It can fail on memory error. The new
MINOR: pattern: index duplicates The indexation functions now accept duplicates. This way it is possible to always have some consistency between lists and trees. The "add" command will always add regardless of any previous existence. The new entry will not be used because both trees and list retrieve keys in insertion order. Thus the "add" operation will always succeed (as long as there is enough memory).
2014-01-29 18:27:15 -05:00
* entry is added at all the pattern_expr registered in this reference. The
MINOR: pattern: make pat_ref_add() rely on pat_ref_append() Let's remove unneeded code duplication, both are exactly the same.
2020-10-28 05:58:05 -04:00
* function stops on the first error encountered. It returns 0 and <err> is
MINOR: pattern: index duplicates The indexation functions now accept duplicates. This way it is possible to always have some consistency between lists and trees. The "add" command will always add regardless of any previous existence. The new entry will not be used because both trees and list retrieve keys in insertion order. Thus the "add" operation will always succeed (as long as there is enough memory).
2014-01-29 18:27:15 -05:00
* filled. If an error is encountered, the complete add operation is cancelled.
* If the insertion is a success the function returns 1.
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
*/
int pat_ref_add(struct pat_ref *ref,
const char *pattern, const char *sample,
BUG/MEDIUM: patterns: last fix was still not enough Last fix did address the issue for inlined patterns, but it was not enough because the flags are lost as well when updating patterns dynamically over the CLI. Also if the same file was used once with -i and another time without -i, their references would have been merged and both would have used the same matching method. It's appear that the patterns have two types of flags. The first ones are relative to the pattern matching, and the second are relative to the pattern storage. The pattern matching flags are the same for all the patterns of one expression. Now they are stored in the expression. The storage flags are information returned by the pattern mathing function. This information is relative to each entry and is stored in the "struct pattern". Now, the expression matching flags are forwarded to the parse and index functions. These flags are stored during the configuration parsing, and they are used during the parse and index actions. This issue was introduced in dev23 with the major pattern rework, and is a continuation of commit a631fc8 ("BUG/MAJOR: patterns: -i and -n are ignored for inlined patterns"). No backport is needed.
2014-04-28 05:18:57 -04:00
char **err)
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
{
MINOR: pattern: implement pat_ref_load() to load a pattern at a given generation pat_ref_load() basically combines pat_ref_append() and pat_ref_commit(). It's very similar to pat_ref_add() except that it also allows to set the generation ID and the line number. pat_ref_add() was modified to directly rely on it to avoid code duplication. Note that a previous declaration of pat_ref_load() was removed as it was just a leftover of an earlier incarnation of something possibly similar, so no existing functionality was changed here.
2020-10-29 04:21:43 -04:00
return !!pat_ref_load(ref, ref->curr_gen, pattern, sample, -1, err);
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
}
MINOR: pattern: support purging arbitrary ranges of generations Instead of being able to purge only values older than a specific value, let's support arbitrary ranges and make pat_ref_purge_older() just be one special case of this one.
2021-04-30 07:19:37 -04:00
/* This function purges all elements from <ref> whose generation is included in
* the range of <from> to <to> (inclusive), taking wrapping into consideration.
* It will not purge more than <budget> entries at once, in order to remain
* responsive. If budget is negative, no limit is applied.
MINOR: pattern: add pat_ref_purge_older() to purge old entries This function will be usable to purge at most a specified number of old entries from a reference. Entries are declared old if their generation number is in the past compared to the one passed in argument. This will ease removal of early entries when new ones have been appended. We also call malloc_trim() when available, at the end of the series, because this is one place where there is a lot of memory to save. Reloads of 1M IP addresses used in an ACL made the process grow up to 1.7 GB RSS after 10 reloads and roughly stabilize there without this call, versus only 260 MB when the call is present. Sadly there is no direct equivalent for jemalloc, which stabilizes around 800MB-1GB.
2020-10-28 13:23:49 -04:00
* The caller must already hold the PATREF_LOCK on <ref>. The function will
* take the PATEXP_LOCK on all expressions of the pattern as needed. It returns
* non-zero on completion, or zero if it had to stop before the end after
* <budget> was depleted.
*/
MINOR: pattern: support purging arbitrary ranges of generations Instead of being able to purge only values older than a specific value, let's support arbitrary ranges and make pat_ref_purge_older() just be one special case of this one.
2021-04-30 07:19:37 -04:00
int pat_ref_purge_range(struct pat_ref *ref, uint from, uint to, int budget)
MINOR: pattern: add pat_ref_purge_older() to purge old entries This function will be usable to purge at most a specified number of old entries from a reference. Entries are declared old if their generation number is in the past compared to the one passed in argument. This will ease removal of early entries when new ones have been appended. We also call malloc_trim() when available, at the end of the series, because this is one place where there is a lot of memory to save. Reloads of 1M IP addresses used in an ACL made the process grow up to 1.7 GB RSS after 10 reloads and roughly stabilize there without this call, versus only 260 MB when the call is present. Sadly there is no direct equivalent for jemalloc, which stabilizes around 800MB-1GB.
2020-10-28 13:23:49 -04:00
{
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
struct pat_ref_gen *gen, *gen2;
MINOR: pattern: add pat_ref_purge_older() to purge old entries This function will be usable to purge at most a specified number of old entries from a reference. Entries are declared old if their generation number is in the past compared to the one passed in argument. This will ease removal of early entries when new ones have been appended. We also call malloc_trim() when available, at the end of the series, because this is one place where there is a lot of memory to save. Reloads of 1M IP addresses used in an ACL made the process grow up to 1.7 GB RSS after 10 reloads and roughly stabilize there without this call, versus only 260 MB when the call is present. Sadly there is no direct equivalent for jemalloc, which stabilizes around 800MB-1GB.
2020-10-28 13:23:49 -04:00
struct pat_ref_elt *elt, *elt_bck;
struct bref *bref, *bref_bck;
struct pattern_expr *expr;
int done;
list_for_each_entry(expr, &ref->pat, list)
HA_RWLOCK_WRLOCK(PATEXP_LOCK, &expr->lock);
/* all expr are locked, we can safely remove all pat_ref */
/* assume completion for e.g. empty lists */
done = 1;
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
pat_ref_gen_foreach_safe(gen, gen2, ref) {
if (gen->gen_id - from > to - from) {
if (from <= to) {
break;
}
MINOR: pattern: add pat_ref_purge_older() to purge old entries This function will be usable to purge at most a specified number of old entries from a reference. Entries are declared old if their generation number is in the past compared to the one passed in argument. This will ease removal of early entries when new ones have been appended. We also call malloc_trim() when available, at the end of the series, because this is one place where there is a lot of memory to save. Reloads of 1M IP addresses used in an ACL made the process grow up to 1.7 GB RSS after 10 reloads and roughly stabilize there without this call, versus only 260 MB when the call is present. Sadly there is no direct equivalent for jemalloc, which stabilizes around 800MB-1GB.
2020-10-28 13:23:49 -04:00
continue;
}
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
list_for_each_entry_safe(elt, elt_bck, &gen->head, list) {
if (budget >= 0 && !budget--) {
done = 0;
break;
}
BUG_ON(elt->gen_id != gen->gen_id);
/*
* we have to unlink all watchers from this reference pattern. We must
* not relink them if this elt was the last one in the list.
*/
list_for_each_entry_safe(bref, bref_bck, &elt->back_refs, users) {
LIST_DELETE(&bref->users);
LIST_INIT(&bref->users);
if (elt->list.n != &gen->head)
LIST_APPEND(&LIST_ELEM(elt->list.n, typeof(elt), list)->back_refs, &bref->users);
bref->ref = elt->list.n;
}
/* delete the storage for all representations of this pattern. */
pat_delete_gen(ref, elt);
LIST_DELETE(&elt->list);
cebs_item_delete(&gen->elt_root, node, pattern, elt);
free(elt->sample);
free(elt);
HA_ATOMIC_INC(&patterns_freed);
MINOR: pattern: add pat_ref_purge_older() to purge old entries This function will be usable to purge at most a specified number of old entries from a reference. Entries are declared old if their generation number is in the past compared to the one passed in argument. This will ease removal of early entries when new ones have been appended. We also call malloc_trim() when available, at the end of the series, because this is one place where there is a lot of memory to save. Reloads of 1M IP addresses used in an ACL made the process grow up to 1.7 GB RSS after 10 reloads and roughly stabilize there without this call, versus only 260 MB when the call is present. Sadly there is no direct equivalent for jemalloc, which stabilizes around 800MB-1GB.
2020-10-28 13:23:49 -04:00
}
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
if (!done)
break;
MINOR: pattern: add pat_ref_purge_older() to purge old entries This function will be usable to purge at most a specified number of old entries from a reference. Entries are declared old if their generation number is in the past compared to the one passed in argument. This will ease removal of early entries when new ones have been appended. We also call malloc_trim() when available, at the end of the series, because this is one place where there is a lot of memory to save. Reloads of 1M IP addresses used in an ACL made the process grow up to 1.7 GB RSS after 10 reloads and roughly stabilize there without this call, versus only 260 MB when the call is present. Sadly there is no direct equivalent for jemalloc, which stabilizes around 800MB-1GB.
2020-10-28 13:23:49 -04:00
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
BUG_ON(!LIST_ISEMPTY(&gen->head));
BUG_ON(!ceb_isempty(&gen->elt_root));
cebu32_item_delete(&ref->gen_root, gen_node, gen_id, gen);
OPTIM: patterns: cache the current generation This makes a significant difference when loading large files and during commit and clear operations, thanks to improved cache locality. In the measurements below, master refers to the code before any of the changes to the patterns code, not the code before this one commit. Timing the replacement of 10M entries from the CLI with this command which also reports timestamps at start, end of upload and end of clear: $ (echo "prompt i"; echo "show activity"; echo "prepare acl #0"; awk '{print "add acl @1 #0",$0}' < bad-ip.map; echo "show activity"; echo "commit acl @1 #0"; echo "clear acl @0 #0";echo "show activity") | socat -t 10 - /tmp/sock1 | grep ^uptim master, on a 3.7 GHz EPYC, 3 samples: uptime_now: 6.087030 uptime_now: 25.981777 => 21.9 sec insertion time uptime_now: 29.286368 => 3.3 sec commit+clear uptime_now: 5.748087 uptime_now: 25.740675 => 20.0s insertion time uptime_now: 29.039023 => 3.3 s commit+clear uptime_now: 7.065362 uptime_now: 26.769596 => 19.7s insertion time uptime_now: 30.065044 => 3.3s commit+clear And after this commit: uptime_now: 6.119215 uptime_now: 25.023019 => 18.9 sec insertion time uptime_now: 27.155503 => 2.1 sec commit+clear uptime_now: 5.675931 uptime_now: 24.551035 => 18.9s insertion uptime_now: 26.652352 => 2.1s commit+clear uptime_now: 6.722256 uptime_now: 25.593952 => 18.9s insertion uptime_now: 27.724153 => 2.1s commit+clear Now timing the startup time with a 10M entries file (on another machine) on master, 20 samples: Standard Deviation, s: 0.061652677408033 Mean: 4.217 And after this commit: Standard Deviation, s: 0.081821371548669 Mean: 3.78
2025-12-22 09:12:40 -05:00
if (gen->gen_id == ref->cached_gen.id)
ref->cached_gen.data = NULL;
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
free(gen);
MINOR: pattern: add pat_ref_purge_older() to purge old entries This function will be usable to purge at most a specified number of old entries from a reference. Entries are declared old if their generation number is in the past compared to the one passed in argument. This will ease removal of early entries when new ones have been appended. We also call malloc_trim() when available, at the end of the series, because this is one place where there is a lot of memory to save. Reloads of 1M IP addresses used in an ACL made the process grow up to 1.7 GB RSS after 10 reloads and roughly stabilize there without this call, versus only 260 MB when the call is present. Sadly there is no direct equivalent for jemalloc, which stabilizes around 800MB-1GB.
2020-10-28 13:23:49 -04:00
}
list_for_each_entry(expr, &ref->pat, list)
HA_RWLOCK_WRUNLOCK(PATEXP_LOCK, &expr->lock);
MINOR: pattern: publish event_hdl events on pat_ref updates Now that PAT_REF events were defined in previous commit, let's actually publish them from pattern API where relevant. Unlike server events, pattern reference events are only published in the pat_ref subscriber's list on purpose, because in some setups patref updates (updates performed on a map for instance from action or cli) are very frequent, and we don't want to impact pattern API performance just for that. Moreover, as the main use case is to be able to subscribe to maps updates from Lua, allowing a per-pattern reference registration is already enough. No additional data is provided for such events (also for performance reason) Care was taken not to publish events when the update doesn't affect the live subset (the one targeted by curr_gen).
2024-10-18 12:40:41 -04:00
/* only publish when we're done and if curr_gen was impacted by the
* purge
*/
if (done && ref->curr_gen - from <= to - from)
event_hdl_publish(&ref->e_subs, EVENT_HDL_SUB_PAT_REF_CLEAR, NULL);
MINOR: pattern: add pat_ref_purge_older() to purge old entries This function will be usable to purge at most a specified number of old entries from a reference. Entries are declared old if their generation number is in the past compared to the one passed in argument. This will ease removal of early entries when new ones have been appended. We also call malloc_trim() when available, at the end of the series, because this is one place where there is a lot of memory to save. Reloads of 1M IP addresses used in an ACL made the process grow up to 1.7 GB RSS after 10 reloads and roughly stabilize there without this call, versus only 260 MB when the call is present. Sadly there is no direct equivalent for jemalloc, which stabilizes around 800MB-1GB.
2020-10-28 13:23:49 -04:00
return done;
}
MEDIUM: pattern: make pat_ref_prune() rely on pat_ref_purge_older() When purging all of a reference, it's much more efficient to scan the reference patterns from the reference head and delete all derivative patterns than to scan the expressions. The only thing is that we need to proceed both for the current and next generations, in case there is a huge gap between the two. With this, purging 20M IP addresses in small batches of 100 takes roughly 3 seconds.
2020-11-03 04:37:31 -05:00
/* This function prunes all entries of <ref> and all their associated
* pattern_expr. It may return before the end of the list is reached,
* returning 0, to yield, indicating to the caller that it must call it again.
* until it returns non-zero. All patterns are purged, both current ones and
* future or incomplete ones. This is used by "clear map" or "clear acl".
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
*/
MEDIUM: map: make the "clear map" operation yield As reported in issue #419, a "clear map" operation on a very large map can take a lot of time and freeze the entire process for several seconds. This patch makes sure that pat_ref_prune() can regularly yield after clearing some entries so that the rest of the process continues to work. The first part, the removal of the patterns, can take quite some time by itself in one run but it's still relatively fast. It may block for up to 100ms for 16M IP addresses in a tree typically. This change needed to declare an I/O handler for the clear operation so that we can get back to it after yielding. The second part can be much slower because it deconstructs the elements and its users, but it iterates progressively so we can yield less often here. The patch was tested with traffic in parallel sollicitating the map being released and showed no problem. Some traffic will definitely notice an incomplete map but the filling is already not atomic anyway thus this is not different. It may be backported to stable versions once sufficiently tested for side effects, at least as far as 2.0 in order to avoid the watchdog triggering when the process is frozen there. For a better behaviour, all these prune_* functions should support yielding so that the callers have a chance to continue also yield in turn.
2019-12-20 12:22:02 -05:00
int pat_ref_prune(struct pat_ref *ref)
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
{
MINOR: pattern: support purging arbitrary ranges of generations Instead of being able to purge only values older than a specific value, let's support arbitrary ranges and make pat_ref_purge_older() just be one special case of this one.
2021-04-30 07:19:37 -04:00
return pat_ref_purge_range(ref, 0, ~0, 100);
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
}
CLEANUP: pattern: fix spelling/grammatical/copy-paste in comments The code is horrible to work with because most functions are documented with misleading comments resulting from many spelling and grammatical mistakes, and plenty of remains of copy-paste mentioning arguments that do not exist and return values that are never set. Too many hours wasted writing non-working code because of assumptions resulting from this, let's fix this once for all now!
2020-10-30 11:03:50 -04:00
/* This function looks up any existing reference <ref> in pattern_head <head>, and
* returns the associated pattern_expr pointer if found, otherwise NULL.
*/
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
struct pattern_expr *pattern_lookup_expr(struct pattern_head *head, struct pat_ref *ref)
{
MEDIUM: pattern: merge same pattern Sometimes the same pattern file is used with the same index, parse and parse_smp functions. If this two condition are true, these two pattern are identical and the same struct can be used.
2014-01-20 08:29:33 -05:00
struct pattern_expr_list *expr;
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
MEDIUM: pattern: merge same pattern Sometimes the same pattern file is used with the same index, parse and parse_smp functions. If this two condition are true, these two pattern are identical and the same struct can be used.
2014-01-20 08:29:33 -05:00
list_for_each_entry(expr, &head->head, list)
if (expr->expr->ref == ref)
return expr->expr;
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
return NULL;
}
CLEANUP: Fix typos in the pattern subsystem Fixes typos in the code comments of the pattern subsystem.
2018-11-15 13:22:31 -05:00
/* This function creates new pattern_expr associated to the reference <ref>.
* <ref> can be NULL. If an error occurs, the function returns NULL and
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
* <err> is filled. Otherwise, the function returns new pattern_expr linked
* with <head> and <ref>.
BUG/MEDIUM: pattern: don't load more than once a pattern list. A memory optimization can use the same pattern expression for many equal pattern list (same parse method, index method and index_smp method). The pattern expression is returned by "pattern_new_expr", but this function dont indicate if the returned pattern is already in use. So, the caller function reload the list of patterns in addition with the existing patterns. This behavior is not a problem with tree indexed pattern, but it grows the lists indexed patterns. This fix add a "reuse" flag in return of the function "pattern_new_expr". If the flag is set, I suppose that the patterns are already loaded. This fix must be backported into 1.5.
2014-11-24 05:14:42 -05:00
*
CLEANUP: Fix typos in the pattern subsystem Fixes typos in the code comments of the pattern subsystem.
2018-11-15 13:22:31 -05:00
* The returned value can be an already filled pattern list, in this case the
BUG/MEDIUM: pattern: don't load more than once a pattern list. A memory optimization can use the same pattern expression for many equal pattern list (same parse method, index method and index_smp method). The pattern expression is returned by "pattern_new_expr", but this function dont indicate if the returned pattern is already in use. So, the caller function reload the list of patterns in addition with the existing patterns. This behavior is not a problem with tree indexed pattern, but it grows the lists indexed patterns. This fix add a "reuse" flag in return of the function "pattern_new_expr". If the flag is set, I suppose that the patterns are already loaded. This fix must be backported into 1.5.
2014-11-24 05:14:42 -05:00
* flag <reuse> is set.
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
*/
BUG/MEDIUM: pattern: don't load more than once a pattern list. A memory optimization can use the same pattern expression for many equal pattern list (same parse method, index method and index_smp method). The pattern expression is returned by "pattern_new_expr", but this function dont indicate if the returned pattern is already in use. So, the caller function reload the list of patterns in addition with the existing patterns. This behavior is not a problem with tree indexed pattern, but it grows the lists indexed patterns. This fix add a "reuse" flag in return of the function "pattern_new_expr". If the flag is set, I suppose that the patterns are already loaded. This fix must be backported into 1.5.
2014-11-24 05:14:42 -05:00
struct pattern_expr *pattern_new_expr(struct pattern_head *head, struct pat_ref *ref,
BUG/MEDIUM: map/acl: fix unwanted flags inheritance. The bug: Maps/ACLs using the same file/id can mistakenly inherit their flags from the last declared one. i.e. $ cat haproxy.conf listen mylistener mode http bind 0.0.0.0:8080 acl myacl1 url -i -f mine.acl acl myacl2 url -f mine.acl acl myacl3 url -i -f mine.acl redirect location / if myacl2 $ cat mine.acl foobar Shows an unexpected redirect for request 'GET /FOObAR HTTP/1.0\n\n'. This fix should be backported on mainline branches v1.6 and v1.7.
2017-07-03 11:54:23 -04:00
int patflags, char **err, int *reuse)
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
{
struct pattern_expr *expr;
MEDIUM: pattern: merge same pattern Sometimes the same pattern file is used with the same index, parse and parse_smp functions. If this two condition are true, these two pattern are identical and the same struct can be used.
2014-01-20 08:29:33 -05:00
struct pattern_expr_list *list;
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
BUG/MEDIUM: pattern: don't load more than once a pattern list. A memory optimization can use the same pattern expression for many equal pattern list (same parse method, index method and index_smp method). The pattern expression is returned by "pattern_new_expr", but this function dont indicate if the returned pattern is already in use. So, the caller function reload the list of patterns in addition with the existing patterns. This behavior is not a problem with tree indexed pattern, but it grows the lists indexed patterns. This fix add a "reuse" flag in return of the function "pattern_new_expr". If the flag is set, I suppose that the patterns are already loaded. This fix must be backported into 1.5.
2014-11-24 05:14:42 -05:00
if (reuse)
*reuse = 0;
MEDIUM: pattern: merge same pattern Sometimes the same pattern file is used with the same index, parse and parse_smp functions. If this two condition are true, these two pattern are identical and the same struct can be used.
2014-01-20 08:29:33 -05:00
/* Memory and initialization of the chain element. */
CLEANUP: pattern: use calloc() rather than malloc for structures It's particularly difficult to make sure that the various pattern structures are properly initialized given that they can be allocated at multiple places and systematically via malloc() instead of calloc(), thus not even leaving the possibility of default values. Let's adjust a few of them.
2020-10-30 10:35:11 -04:00
list = calloc(1, sizeof(*list));
MEDIUM: pattern: merge same pattern Sometimes the same pattern file is used with the same index, parse and parse_smp functions. If this two condition are true, these two pattern are identical and the same struct can be used.
2014-01-20 08:29:33 -05:00
if (!list) {
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
memprintf(err, "out of memory");
return NULL;
}
MEDIUM: pattern: merge same pattern Sometimes the same pattern file is used with the same index, parse and parse_smp functions. If this two condition are true, these two pattern are identical and the same struct can be used.
2014-01-20 08:29:33 -05:00
/* Look for existing similar expr. No that only the index, parse and
* parse_smp function must be identical for having similar pattern.
CLEANUP: Fix typos in the pattern subsystem Fixes typos in the code comments of the pattern subsystem.
2018-11-15 13:22:31 -05:00
* The other function depends of these first.
MEDIUM: pattern: merge same pattern Sometimes the same pattern file is used with the same index, parse and parse_smp functions. If this two condition are true, these two pattern are identical and the same struct can be used.
2014-01-20 08:29:33 -05:00
*/
if (ref) {
list_for_each_entry(expr, &ref->pat, list)
if (expr->pat_head->index == head->index &&
expr->pat_head->parse == head->parse &&
BUG/MEDIUM: map/acl: fix unwanted flags inheritance. The bug: Maps/ACLs using the same file/id can mistakenly inherit their flags from the last declared one. i.e. $ cat haproxy.conf listen mylistener mode http bind 0.0.0.0:8080 acl myacl1 url -i -f mine.acl acl myacl2 url -f mine.acl acl myacl3 url -i -f mine.acl redirect location / if myacl2 $ cat mine.acl foobar Shows an unexpected redirect for request 'GET /FOObAR HTTP/1.0\n\n'. This fix should be backported on mainline branches v1.6 and v1.7.
2017-07-03 11:54:23 -04:00
expr->pat_head->parse_smp == head->parse_smp &&
expr->mflags == patflags)
MEDIUM: pattern: merge same pattern Sometimes the same pattern file is used with the same index, parse and parse_smp functions. If this two condition are true, these two pattern are identical and the same struct can be used.
2014-01-20 08:29:33 -05:00
break;
if (&expr->list == &ref->pat)
expr = NULL;
}
else
expr = NULL;
/* If no similar expr was found, we create new expr. */
if (!expr) {
/* Get a lot of memory for the expr struct. */
CLEANUP: pattern: use calloc() rather than malloc for structures It's particularly difficult to make sure that the various pattern structures are properly initialized given that they can be allocated at multiple places and systematically via malloc() instead of calloc(), thus not even leaving the possibility of default values. Let's adjust a few of them.
2020-10-30 10:35:11 -04:00
expr = calloc(1, sizeof(*expr));
MEDIUM: pattern: merge same pattern Sometimes the same pattern file is used with the same index, parse and parse_smp functions. If this two condition are true, these two pattern are identical and the same struct can be used.
2014-01-20 08:29:33 -05:00
if (!expr) {
BUG/MINOR: pattern: Avoid memory leak on out-of-memory condition pattern_new_expr() failed to free the allocated list element when an out-of-memory error occurs during initialization of the element. As this only happens when loading the configuration file or evaluating commands via the CLI, it is unlikely for this leak to be relevant unless the user makes automated, heavy use of the CLI. Found in HAProxy 1.5.14.
2016-03-03 14:20:23 -05:00
free(list);
MEDIUM: pattern: merge same pattern Sometimes the same pattern file is used with the same index, parse and parse_smp functions. If this two condition are true, these two pattern are identical and the same struct can be used.
2014-01-20 08:29:33 -05:00
memprintf(err, "out of memory");
return NULL;
}
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
MEDIUM: pattern: merge same pattern Sometimes the same pattern file is used with the same index, parse and parse_smp functions. If this two condition are true, these two pattern are identical and the same struct can be used.
2014-01-20 08:29:33 -05:00
/* Initialize this new expr. */
pattern_init_expr(expr);
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
BUG/MEDIUM: map/acl: fix unwanted flags inheritance. The bug: Maps/ACLs using the same file/id can mistakenly inherit their flags from the last declared one. i.e. $ cat haproxy.conf listen mylistener mode http bind 0.0.0.0:8080 acl myacl1 url -i -f mine.acl acl myacl2 url -f mine.acl acl myacl3 url -i -f mine.acl redirect location / if myacl2 $ cat mine.acl foobar Shows an unexpected redirect for request 'GET /FOObAR HTTP/1.0\n\n'. This fix should be backported on mainline branches v1.6 and v1.7.
2017-07-03 11:54:23 -04:00
/* Copy the pattern matching and indexing flags. */
expr->mflags = patflags;
MEDIUM: pattern: merge same pattern Sometimes the same pattern file is used with the same index, parse and parse_smp functions. If this two condition are true, these two pattern are identical and the same struct can be used.
2014-01-20 08:29:33 -05:00
/* This new pattern expression reference one of his heads. */
expr->pat_head = head;
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
CLEANUP: lists/tree-wide: rename some list operations to avoid some confusion The current "ADD" vs "ADDQ" is confusing because when thinking in terms of appending at the end of a list, "ADD" naturally comes to mind, but here it does the opposite, it inserts. Several times already it's been incorrectly used where ADDQ was expected, the latest of which was a fortunate accident explained in 6fa922562 ("CLEANUP: stream: explain why we queue the stream at the head of the server list"). Let's use more explicit (but slightly longer) names now: LIST_ADD -> LIST_INSERT LIST_ADDQ -> LIST_APPEND LIST_ADDED -> LIST_INLIST LIST_DEL -> LIST_DELETE The same is true for MT_LISTs, including their "TRY" variant. LIST_DEL_INIT keeps its short name to encourage to use it instead of the lazier LIST_DELETE which is often less safe. The change is large (~674 non-comment entries) but is mechanical enough to remain safe. No permutation was performed, so any out-of-tree code can easily map older names to new ones. The list doc was updated.
2021-04-21 01:32:39 -04:00
/* Link with ref, or to self to facilitate LIST_DELETE() */
MEDIUM: pattern: merge same pattern Sometimes the same pattern file is used with the same index, parse and parse_smp functions. If this two condition are true, these two pattern are identical and the same struct can be used.
2014-01-20 08:29:33 -05:00
if (ref)
CLEANUP: lists/tree-wide: rename some list operations to avoid some confusion The current "ADD" vs "ADDQ" is confusing because when thinking in terms of appending at the end of a list, "ADD" naturally comes to mind, but here it does the opposite, it inserts. Several times already it's been incorrectly used where ADDQ was expected, the latest of which was a fortunate accident explained in 6fa922562 ("CLEANUP: stream: explain why we queue the stream at the head of the server list"). Let's use more explicit (but slightly longer) names now: LIST_ADD -> LIST_INSERT LIST_ADDQ -> LIST_APPEND LIST_ADDED -> LIST_INLIST LIST_DEL -> LIST_DELETE The same is true for MT_LISTs, including their "TRY" variant. LIST_DEL_INIT keeps its short name to encourage to use it instead of the lazier LIST_DELETE which is often less safe. The change is large (~674 non-comment entries) but is mechanical enough to remain safe. No permutation was performed, so any out-of-tree code can easily map older names to new ones. The list doc was updated.
2021-04-21 01:32:39 -04:00
LIST_APPEND(&ref->pat, &expr->list);
MEDIUM: pattern: merge same pattern Sometimes the same pattern file is used with the same index, parse and parse_smp functions. If this two condition are true, these two pattern are identical and the same struct can be used.
2014-01-20 08:29:33 -05:00
else
LIST_INIT(&expr->list);
expr->ref = ref;
BUILD: threads: Rename SPIN/RWLOCK macros using HA_ prefix This remove any name conflicts, especially on Solaris.
2017-11-07 04:42:54 -05:00
HA_RWLOCK_INIT(&expr->lock);
MEDIUM: pattern: merge same pattern Sometimes the same pattern file is used with the same index, parse and parse_smp functions. If this two condition are true, these two pattern are identical and the same struct can be used.
2014-01-20 08:29:33 -05:00
}
else {
BUG/MEDIUM: pattern: don't load more than once a pattern list. A memory optimization can use the same pattern expression for many equal pattern list (same parse method, index method and index_smp method). The pattern expression is returned by "pattern_new_expr", but this function dont indicate if the returned pattern is already in use. So, the caller function reload the list of patterns in addition with the existing patterns. This behavior is not a problem with tree indexed pattern, but it grows the lists indexed patterns. This fix add a "reuse" flag in return of the function "pattern_new_expr". If the flag is set, I suppose that the patterns are already loaded. This fix must be backported into 1.5.
2014-11-24 05:14:42 -05:00
if (reuse)
*reuse = 1;
MEDIUM: pattern: merge same pattern Sometimes the same pattern file is used with the same index, parse and parse_smp functions. If this two condition are true, these two pattern are identical and the same struct can be used.
2014-01-20 08:29:33 -05:00
}
BUG/MEDIUM: pattern: prevent UAF on reused pattern expr Since c5959fd ("MEDIUM: pattern: merge same pattern"), UAF (leading to crash) can be experienced if the same pattern file (and match method) is used in two default sections and the first one is not referenced later in the config. In this case, the first default section will be cleaned up. However, due to an unhandled case in the above optimization, the original expr which the second default section relies on is mistakenly freed. This issue was discovered while trying to reproduce GH #2708. The issue was particularly tricky to reproduce given the config and sequence required to make the UAF happen. Hopefully, Github user @asmnek not only provided useful informations, but since he was able to consistently trigger the crash in his environment he was able to nail down the crash to the use of pattern file involved with 2 named default sections. Big thanks to him. To fix the issue, let's push the logic from c5959fd a bit further. Instead of relying on "do_free" variable to know if the expression should be freed or not (which proved to be insufficient in our case), let's switch to a simple refcounting logic. This way, no matter who owns the expression, the last one attempting to free it will be responsible for freeing it. Refcount is implemented using a 32bit value which fills a previous 4 bytes structure gap: int mflags; /* 80 4 */ /* XXX 4 bytes hole, try to pack */ long unsigned int lock; /* 88 8 */ (output from pahole) Even though it was not reproduced in 2.6 or below by @asmnek (the bug was revealed thanks to another bugfix), this issue theorically affects all stable versions (up to c5959fd), thus it should be backported to all stable versions.
2024-09-09 08:59:19 -04:00
HA_ATOMIC_INC(&expr->refcount);
MEDIUM: pattern: merge same pattern Sometimes the same pattern file is used with the same index, parse and parse_smp functions. If this two condition are true, these two pattern are identical and the same struct can be used.
2014-01-20 08:29:33 -05:00
/* The new list element reference the pattern_expr. */
list->expr = expr;
/* Link the list element with the pattern_head. */
CLEANUP: lists/tree-wide: rename some list operations to avoid some confusion The current "ADD" vs "ADDQ" is confusing because when thinking in terms of appending at the end of a list, "ADD" naturally comes to mind, but here it does the opposite, it inserts. Several times already it's been incorrectly used where ADDQ was expected, the latest of which was a fortunate accident explained in 6fa922562 ("CLEANUP: stream: explain why we queue the stream at the head of the server list"). Let's use more explicit (but slightly longer) names now: LIST_ADD -> LIST_INSERT LIST_ADDQ -> LIST_APPEND LIST_ADDED -> LIST_INLIST LIST_DEL -> LIST_DELETE The same is true for MT_LISTs, including their "TRY" variant. LIST_DEL_INIT keeps its short name to encourage to use it instead of the lazier LIST_DELETE which is often less safe. The change is large (~674 non-comment entries) but is mechanical enough to remain safe. No permutation was performed, so any out-of-tree code can easily map older names to new ones. The list doc was updated.
2021-04-21 01:32:39 -04:00
LIST_APPEND(&head->head, &list->list);
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
return expr;
}
MINOR: pattern/map/acl: Centralization of the file parsers The acl and map function do the same work with the file parsing. This patch merge these code in only one. Note that the function map_read_entries_from_file() in the file "map.c" is moved to the the function pat_ref_read_from_file_smp() in the file "pattern.c". The code of this function is not modified, only the the name and the arguments order has changed.
2014-01-29 07:29:45 -05:00
/* Reads patterns from a file. If <err_msg> is non-NULL, an error message will
* be returned there on errors and the caller will have to free it.
*
* The file contains one key + value per line. Lines which start with '#' are
* ignored, just like empty lines. Leading tabs/spaces are stripped. The key is
* then the first "word" (series of non-space/tabs characters), and the value is
* what follows this series of space/tab till the end of the line excluding
* trailing spaces/tabs.
*
* Example :
*
* # this is a comment and is ignored
* 62.212.114.60 1wt.eu \n
* <-><-----------><---><----><---->
* | | | | `--- trailing spaces ignored
* | | | `-------- value
* | | `--------------- middle spaces ignored
* | `------------------------ key
* `-------------------------------- leading spaces ignored
*
CLEANUP: assorted typo fixes in the code and comments This is 10th iteration of typo fixes
2020-06-21 12:42:57 -04:00
* Return non-zero in case of success, otherwise 0.
MINOR: pattern/map/acl: Centralization of the file parsers The acl and map function do the same work with the file parsing. This patch merge these code in only one. Note that the function map_read_entries_from_file() in the file "map.c" is moved to the the function pat_ref_read_from_file_smp() in the file "pattern.c". The code of this function is not modified, only the the name and the arguments order has changed.
2014-01-29 07:29:45 -05:00
*/
MINOR: pattern: Use reference name as filename to read patterns from a file It is only a small API refactoring. The filename is no longer used when pat_ref_read_from_file_smp() or pat_ref_read_from_file() functions are called. The filename was already used to create the reference on the list of patterns. Thus, we now directly use info from this reference.
2023-12-01 06:04:13 -05:00
int pat_ref_read_from_file_smp(struct pat_ref *ref, char **err)
MINOR: pattern/map/acl: Centralization of the file parsers The acl and map function do the same work with the file parsing. This patch merge these code in only one. Note that the function map_read_entries_from_file() in the file "map.c" is moved to the the function pat_ref_read_from_file_smp() in the file "pattern.c". The code of this function is not modified, only the the name and the arguments order has changed.
2014-01-29 07:29:45 -05:00
{
FILE *file;
char *c;
int ret = 0;
int line = 0;
char *key_beg;
char *key_end;
char *value_beg;
char *value_end;
MINOR: pattern: Use reference name as filename to read patterns from a file It is only a small API refactoring. The filename is no longer used when pat_ref_read_from_file_smp() or pat_ref_read_from_file() functions are called. The filename was already used to create the reference on the list of patterns. Thus, we now directly use info from this reference.
2023-12-01 06:04:13 -05:00
file = fopen(ref->reference, "r");
MINOR: pattern/map/acl: Centralization of the file parsers The acl and map function do the same work with the file parsing. This patch merge these code in only one. Note that the function map_read_entries_from_file() in the file "map.c" is moved to the the function pat_ref_read_from_file_smp() in the file "pattern.c". The code of this function is not modified, only the the name and the arguments order has changed.
2014-01-29 07:29:45 -05:00
if (!file) {
MEDIUM: pattern: Add support for virtual and optional files for patterns Before this patch, it was not possible to use a list of patterns, map or a list of acls, without an existing file. However, it could be handy to just use an ID, with no file on the disk. It is pretty useful for everyone managing dynamically these lists. It could also be handy to try to load a list from a file if it exists without failing if not. This way, it could be possible to make a cold start without any file (instead of empty file), dynamically add and del patterns, dump the list to the file periodically to reuse it on reload (via an external process). In this patch, we uses some prefixes to be able to use virtual or optional files. The default case remains unchanged. regular files are used. A filename, with no prefix, is used as reference, and it must exist on the disk. With the prefix "file@", the same is performed. Internally this prefix is skipped. Thus the same file, with ou without "file@" prefix, references the same list of patterns. To use a virtual map, "virt@" prefix must be used. No file is read, even if the following name looks like a file. It is just an ID. The prefix is part of ID and must always be used. To use a optional file, ie a file that may or may not exist on a disk at startup, "opt@" prefix must be used. If the file exists, its content is loaded. But HAProxy doesn't complain if not. The prefix is not part of ID. For a given file, optional files and regular files reference the same list of patterns. This patch should fix the issue #2202.
2023-12-01 06:04:35 -05:00
if (ref->flags & PAT_REF_ID) {
/* file not found for an optional file, switch it to a virtual list of patterns */
ref->flags &= ~PAT_REF_FILE;
return 1;
}
MINOR: pattern: Use reference name as filename to read patterns from a file It is only a small API refactoring. The filename is no longer used when pat_ref_read_from_file_smp() or pat_ref_read_from_file() functions are called. The filename was already used to create the reference on the list of patterns. Thus, we now directly use info from this reference.
2023-12-01 06:04:13 -05:00
memprintf(err, "failed to open pattern file <%s>", ref->reference);
MINOR: pattern/map/acl: Centralization of the file parsers The acl and map function do the same work with the file parsing. This patch merge these code in only one. Note that the function map_read_entries_from_file() in the file "map.c" is moved to the the function pat_ref_read_from_file_smp() in the file "pattern.c". The code of this function is not modified, only the the name and the arguments order has changed.
2014-01-29 07:29:45 -05:00
return 0;
}
MEDIUM: pattern: Add support for virtual and optional files for patterns Before this patch, it was not possible to use a list of patterns, map or a list of acls, without an existing file. However, it could be handy to just use an ID, with no file on the disk. It is pretty useful for everyone managing dynamically these lists. It could also be handy to try to load a list from a file if it exists without failing if not. This way, it could be possible to make a cold start without any file (instead of empty file), dynamically add and del patterns, dump the list to the file periodically to reuse it on reload (via an external process). In this patch, we uses some prefixes to be able to use virtual or optional files. The default case remains unchanged. regular files are used. A filename, with no prefix, is used as reference, and it must exist on the disk. With the prefix "file@", the same is performed. Internally this prefix is skipped. Thus the same file, with ou without "file@" prefix, references the same list of patterns. To use a virtual map, "virt@" prefix must be used. No file is read, even if the following name looks like a file. It is just an ID. The prefix is part of ID and must always be used. To use a optional file, ie a file that may or may not exist on a disk at startup, "opt@" prefix must be used. If the file exists, its content is loaded. But HAProxy doesn't complain if not. The prefix is not part of ID. For a given file, optional files and regular files reference the same list of patterns. This patch should fix the issue #2202.
2023-12-01 06:04:35 -05:00
ref->flags |= PAT_REF_FILE;
MINOR: pattern/map/acl: Centralization of the file parsers The acl and map function do the same work with the file parsing. This patch merge these code in only one. Note that the function map_read_entries_from_file() in the file "map.c" is moved to the the function pat_ref_read_from_file_smp() in the file "pattern.c". The code of this function is not modified, only the the name and the arguments order has changed.
2014-01-29 07:29:45 -05:00
/* now parse all patterns. The file may contain only one pattern
* followed by one value per line. The start spaces, separator spaces
* and and spaces are stripped. Each can contain comment started by '#'
*/
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
while (fgets(trash.area, trash.size, file) != NULL) {
MINOR: pattern/map/acl: Centralization of the file parsers The acl and map function do the same work with the file parsing. This patch merge these code in only one. Note that the function map_read_entries_from_file() in the file "map.c" is moved to the the function pat_ref_read_from_file_smp() in the file "pattern.c". The code of this function is not modified, only the the name and the arguments order has changed.
2014-01-29 07:29:45 -05:00
line++;
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
c = trash.area;
MINOR: pattern/map/acl: Centralization of the file parsers The acl and map function do the same work with the file parsing. This patch merge these code in only one. Note that the function map_read_entries_from_file() in the file "map.c" is moved to the the function pat_ref_read_from_file_smp() in the file "pattern.c". The code of this function is not modified, only the the name and the arguments order has changed.
2014-01-29 07:29:45 -05:00
/* ignore lines beginning with a dash */
if (*c == '#')
continue;
/* strip leading spaces and tabs */
while (*c == ' ' || *c == '\t')
c++;
/* empty lines are ignored too */
if (*c == '\0' || *c == '\r' || *c == '\n')
continue;
/* look for the end of the key */
key_beg = c;
while (*c && *c != ' ' && *c != '\t' && *c != '\n' && *c != '\r')
c++;
key_end = c;
/* strip middle spaces and tabs */
while (*c == ' ' || *c == '\t')
c++;
/* look for the end of the value, it is the end of the line */
value_beg = c;
while (*c && *c != '\n' && *c != '\r')
c++;
value_end = c;
/* trim possibly trailing spaces and tabs */
while (value_end > value_beg && (value_end[-1] == ' ' || value_end[-1] == '\t'))
value_end--;
/* set final \0 and check entries */
*key_end = '\0';
*value_end = '\0';
/* insert values */
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
if (!pat_ref_append(ref, ref->curr_gen, key_beg, value_beg, line)) {
MINOR: pattern/map/acl: Centralization of the file parsers The acl and map function do the same work with the file parsing. This patch merge these code in only one. Note that the function map_read_entries_from_file() in the file "map.c" is moved to the the function pat_ref_read_from_file_smp() in the file "pattern.c". The code of this function is not modified, only the the name and the arguments order has changed.
2014-01-29 07:29:45 -05:00
memprintf(err, "out of memory");
goto out_close;
}
}
BUG/MINOR: pattern: handle errors from fgets when trying to load patterns We need to do some error handling after we call fgets to make sure everything went fine. If we don't users can be fooled into thinking they can load pattens from directory because cfgparse doesn't flinch. This applies to acl patterns map files. This should be backported to all supported versions.
2020-01-17 10:09:33 -05:00
if (ferror(file)) {
memprintf(err, "error encountered while reading <%s> : %s",
MINOR: pattern: Use reference name as filename to read patterns from a file It is only a small API refactoring. The filename is no longer used when pat_ref_read_from_file_smp() or pat_ref_read_from_file() functions are called. The filename was already used to create the reference on the list of patterns. Thus, we now directly use info from this reference.
2023-12-01 06:04:13 -05:00
ref->reference, strerror(errno));
BUG/MINOR: pattern: handle errors from fgets when trying to load patterns We need to do some error handling after we call fgets to make sure everything went fine. If we don't users can be fooled into thinking they can load pattens from directory because cfgparse doesn't flinch. This applies to acl patterns map files. This should be backported to all supported versions.
2020-01-17 10:09:33 -05:00
goto out_close;
}
CLEANUP: assorted typo fixes in the code and comments This is 10th iteration of typo fixes
2020-06-21 12:42:57 -04:00
/* success */
MINOR: pattern/map/acl: Centralization of the file parsers The acl and map function do the same work with the file parsing. This patch merge these code in only one. Note that the function map_read_entries_from_file() in the file "map.c" is moved to the the function pat_ref_read_from_file_smp() in the file "pattern.c". The code of this function is not modified, only the the name and the arguments order has changed.
2014-01-29 07:29:45 -05:00
ret = 1;
out_close:
fclose(file);
return ret;
}
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
/* Reads patterns from a file. If <err_msg> is non-NULL, an error message will
* be returned there on errors and the caller will have to free it.
*/
MINOR: pattern: Use reference name as filename to read patterns from a file It is only a small API refactoring. The filename is no longer used when pat_ref_read_from_file_smp() or pat_ref_read_from_file() functions are called. The filename was already used to create the reference on the list of patterns. Thus, we now directly use info from this reference.
2023-12-01 06:04:13 -05:00
int pat_ref_read_from_file(struct pat_ref *ref, char **err)
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
{
FILE *file;
char *c;
char *arg;
int ret = 0;
int line = 0;
MINOR: pattern: Use reference name as filename to read patterns from a file It is only a small API refactoring. The filename is no longer used when pat_ref_read_from_file_smp() or pat_ref_read_from_file() functions are called. The filename was already used to create the reference on the list of patterns. Thus, we now directly use info from this reference.
2023-12-01 06:04:13 -05:00
file = fopen(ref->reference, "r");
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
if (!file) {
MEDIUM: pattern: Add support for virtual and optional files for patterns Before this patch, it was not possible to use a list of patterns, map or a list of acls, without an existing file. However, it could be handy to just use an ID, with no file on the disk. It is pretty useful for everyone managing dynamically these lists. It could also be handy to try to load a list from a file if it exists without failing if not. This way, it could be possible to make a cold start without any file (instead of empty file), dynamically add and del patterns, dump the list to the file periodically to reuse it on reload (via an external process). In this patch, we uses some prefixes to be able to use virtual or optional files. The default case remains unchanged. regular files are used. A filename, with no prefix, is used as reference, and it must exist on the disk. With the prefix "file@", the same is performed. Internally this prefix is skipped. Thus the same file, with ou without "file@" prefix, references the same list of patterns. To use a virtual map, "virt@" prefix must be used. No file is read, even if the following name looks like a file. It is just an ID. The prefix is part of ID and must always be used. To use a optional file, ie a file that may or may not exist on a disk at startup, "opt@" prefix must be used. If the file exists, its content is loaded. But HAProxy doesn't complain if not. The prefix is not part of ID. For a given file, optional files and regular files reference the same list of patterns. This patch should fix the issue #2202.
2023-12-01 06:04:35 -05:00
if (ref->flags & PAT_REF_ID) {
/* file not found for an optional file, switch it to a virtual list of patterns */
ref->flags &= ~PAT_REF_FILE;
return 1;
}
MINOR: pattern: Use reference name as filename to read patterns from a file It is only a small API refactoring. The filename is no longer used when pat_ref_read_from_file_smp() or pat_ref_read_from_file() functions are called. The filename was already used to create the reference on the list of patterns. Thus, we now directly use info from this reference.
2023-12-01 06:04:13 -05:00
memprintf(err, "failed to open pattern file <%s>", ref->reference);
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
return 0;
}
/* now parse all patterns. The file may contain only one pattern per
* line. If the line contains spaces, they will be part of the pattern.
* The pattern stops at the first CR, LF or EOF encountered.
*/
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
while (fgets(trash.area, trash.size, file) != NULL) {
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
line++;
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
c = trash.area;
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
/* ignore lines beginning with a dash */
if (*c == '#')
continue;
/* strip leading spaces and tabs */
while (*c == ' ' || *c == '\t')
c++;
arg = c;
while (*c && *c != '\n' && *c != '\r')
c++;
*c = 0;
/* empty lines are ignored too */
if (c == arg)
continue;
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
if (!pat_ref_append(ref, ref->curr_gen, arg, NULL, line)) {
MINOR: pattern: Use reference name as filename to read patterns from a file It is only a small API refactoring. The filename is no longer used when pat_ref_read_from_file_smp() or pat_ref_read_from_file() functions are called. The filename was already used to create the reference on the list of patterns. Thus, we now directly use info from this reference.
2023-12-01 06:04:13 -05:00
memprintf(err, "out of memory when loading patterns from file <%s>", ref->reference);
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
goto out_close;
}
}
BUG/MINOR: pattern: handle errors from fgets when trying to load patterns We need to do some error handling after we call fgets to make sure everything went fine. If we don't users can be fooled into thinking they can load pattens from directory because cfgparse doesn't flinch. This applies to acl patterns map files. This should be backported to all supported versions.
2020-01-17 10:09:33 -05:00
if (ferror(file)) {
memprintf(err, "error encountered while reading <%s> : %s",
MINOR: pattern: Use reference name as filename to read patterns from a file It is only a small API refactoring. The filename is no longer used when pat_ref_read_from_file_smp() or pat_ref_read_from_file() functions are called. The filename was already used to create the reference on the list of patterns. Thus, we now directly use info from this reference.
2023-12-01 06:04:13 -05:00
ref->reference, strerror(errno));
BUG/MINOR: pattern: handle errors from fgets when trying to load patterns We need to do some error handling after we call fgets to make sure everything went fine. If we don't users can be fooled into thinking they can load pattens from directory because cfgparse doesn't flinch. This applies to acl patterns map files. This should be backported to all supported versions.
2020-01-17 10:09:33 -05:00
goto out_close;
}
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
ret = 1; /* success */
out_close:
fclose(file);
return ret;
}
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
int pattern_read_from_file(struct pattern_head *head, unsigned int refflags,
MINOR: pattern/map/acl: Centralization of the file parsers The acl and map function do the same work with the file parsing. This patch merge these code in only one. Note that the function map_read_entries_from_file() in the file "map.c" is moved to the the function pat_ref_read_from_file_smp() in the file "pattern.c". The code of this function is not modified, only the the name and the arguments order has changed.
2014-01-29 07:29:45 -05:00
const char *filename, int patflags, int load_smp,
MINOR: dumpstat/conf: display all the configuration lines that using pattern reference
2014-02-11 08:36:45 -05:00
char **err, const char *file, int line)
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
{
struct pat_ref *ref;
struct pattern_expr *expr;
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
struct pat_ref_gen *gen;
MINOR: pattern/map/acl: Centralization of the file parsers The acl and map function do the same work with the file parsing. This patch merge these code in only one. Note that the function map_read_entries_from_file() in the file "map.c" is moved to the the function pat_ref_read_from_file_smp() in the file "pattern.c". The code of this function is not modified, only the the name and the arguments order has changed.
2014-01-29 07:29:45 -05:00
struct pat_ref_elt *elt;
BUG/MEDIUM: patterns: previous fix was incomplete Dmitry Sivachenko <trtrmitya@gmail.com> reported that commit 315ec42 ("BUG/MEDIUM: pattern: don't load more than once a pattern list.") relies on an uninitialised variable in the stack. While it used to work fine during the tests, if the uninitialized variable is non-null, some patterns may be aggregated if loaded multiple times, resulting in slower processing, which was the original issue it tried to address. The fix needs to be backported to 1.5.
2014-11-26 07:17:03 -05:00
int reuse = 0;
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
MINOR: pattern: Each pattern is identified by unique id. The pattern reference are stored with two identifiers: the unique_id and the reference. The reference identify a file. Each file with the same name point to the same reference. We can register many times one file. If the file is modified, all his dependencies are also modified. The reference can be used with map or acl. The unique_id identify inline acl. The unique id is unique for each acl. You cannot force the same id in the configuration file, because this repport an error. The format of the acl and map listing through the "socket" has changed for displaying these new ids.
2014-03-11 09:29:22 -04:00
/* Lookup for the existing reference. */
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
ref = pat_ref_lookup(filename);
MINOR: pattern: Each pattern is identified by unique id. The pattern reference are stored with two identifiers: the unique_id and the reference. The reference identify a file. Each file with the same name point to the same reference. We can register many times one file. If the file is modified, all his dependencies are also modified. The reference can be used with map or acl. The unique_id identify inline acl. The unique id is unique for each acl. You cannot force the same id in the configuration file, because this repport an error. The format of the acl and map listing through the "socket" has changed for displaying these new ids.
2014-03-11 09:29:22 -04:00
/* If the reference doesn't exists, create it and load associated file. */
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
if (!ref) {
MINOR: dumpstat/conf: display all the configuration lines that using pattern reference
2014-02-11 08:36:45 -05:00
chunk_printf(&trash,
"pattern loaded from file '%s' used by %s at file '%s' line %d",
filename, refflags & PAT_REF_MAP ? "map" : "acl", file, line);
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
ref = pat_ref_new(filename, trash.area, refflags);
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
if (!ref) {
memprintf(err, "out of memory");
return 0;
}
MEDIUM: pattern: Add support for virtual and optional files for patterns Before this patch, it was not possible to use a list of patterns, map or a list of acls, without an existing file. However, it could be handy to just use an ID, with no file on the disk. It is pretty useful for everyone managing dynamically these lists. It could also be handy to try to load a list from a file if it exists without failing if not. This way, it could be possible to make a cold start without any file (instead of empty file), dynamically add and del patterns, dump the list to the file periodically to reuse it on reload (via an external process). In this patch, we uses some prefixes to be able to use virtual or optional files. The default case remains unchanged. regular files are used. A filename, with no prefix, is used as reference, and it must exist on the disk. With the prefix "file@", the same is performed. Internally this prefix is skipped. Thus the same file, with ou without "file@" prefix, references the same list of patterns. To use a virtual map, "virt@" prefix must be used. No file is read, even if the following name looks like a file. It is just an ID. The prefix is part of ID and must always be used. To use a optional file, ie a file that may or may not exist on a disk at startup, "opt@" prefix must be used. If the file exists, its content is loaded. But HAProxy doesn't complain if not. The prefix is not part of ID. For a given file, optional files and regular files reference the same list of patterns. This patch should fix the issue #2202.
2023-12-01 06:04:35 -05:00
if (ref->flags & PAT_REF_FILE) {
if (load_smp) {
ref->flags |= PAT_REF_SMP;
if (!pat_ref_read_from_file_smp(ref, err))
return 0;
}
else {
if (!pat_ref_read_from_file(ref, err))
return 0;
}
MINOR: pattern/map/acl: Centralization of the file parsers The acl and map function do the same work with the file parsing. This patch merge these code in only one. Note that the function map_read_entries_from_file() in the file "map.c" is moved to the the function pat_ref_read_from_file_smp() in the file "pattern.c". The code of this function is not modified, only the the name and the arguments order has changed.
2014-01-29 07:29:45 -05:00
}
BUG/MINOR: pattern: Properly flag virtual maps as using samples When a map file is load, internally, the pattern reference is flagged as based on a sample. However it is not performed for virtual maps. This flag is only used during startup to check the map compatibility when it used at different places. At runtime this does not change anything. But errors can be triggered during configuration parsing. For instance, the following valid config will trigger an error: http-request set-map(virt@test) foo bar if !{ str(foo),map(virt@test) -m found } http-request set-var(txn.foo) str(foo),map(virt@test) The fix is quite obvious. PAT_REF_SMP flag must be set for virtual map as any other map. A workaround is to use optional map (opt@...) by checking the map id cannot reference an existing file. This patch must be backported as far as 3.0.
2025-09-25 04:03:41 -04:00
else if ((ref->flags & PAT_REF_ID) && load_smp)
ref->flags |= PAT_REF_SMP;
MINOR: pattern/map/acl: Centralization of the file parsers The acl and map function do the same work with the file parsing. This patch merge these code in only one. Note that the function map_read_entries_from_file() in the file "map.c" is moved to the the function pat_ref_read_from_file_smp() in the file "pattern.c". The code of this function is not modified, only the the name and the arguments order has changed.
2014-01-29 07:29:45 -05:00
}
else {
MINOR: pattern: Check if the file reference is not used with acl and map The format of the acl file are not the same than the format of the map files. In some case, the same file can be used, but this is ambiguous for the user because the patterns are not the expected.
2014-01-29 06:32:58 -05:00
/* The reference already exists, check the map compatibility. */
/* If the load require samples and the flag PAT_REF_SMP is not set,
* the reference doesn't contain sample, and cannot be used.
*/
if (load_smp) {
if (!(ref->flags & PAT_REF_SMP)) {
memprintf(err, "The file \"%s\" is already used as one column file "
"and cannot be used by as two column file.",
filename);
return 0;
}
}
else {
/* The load doesn't require samples. If the flag PAT_REF_SMP is
* set, the reference contains a sample, and cannot be used.
*/
if (ref->flags & PAT_REF_SMP) {
memprintf(err, "The file \"%s\" is already used as two column file "
"and cannot be used by as one column file.",
filename);
return 0;
}
}
MINOR: dumpstat/conf: display all the configuration lines that using pattern reference
2014-02-11 08:36:45 -05:00
/* Extends display */
chunk_printf(&trash, "%s", ref->display);
chunk_appendf(&trash, ", by %s at file '%s' line %d",
refflags & PAT_REF_MAP ? "map" : "acl", file, line);
free(ref->display);
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
ref->display = strdup(trash.area);
MINOR: dumpstat/conf: display all the configuration lines that using pattern reference
2014-02-11 08:36:45 -05:00
if (!ref->display) {
memprintf(err, "out of memory");
return 0;
}
MINOR: pattern: Check if the file reference is not used with acl and map The format of the acl file are not the same than the format of the map files. In some case, the same file can be used, but this is ambiguous for the user because the patterns are not the expected.
2014-01-29 06:32:58 -05:00
/* Merge flags. */
MINOR: pattern/map/acl: Centralization of the file parsers The acl and map function do the same work with the file parsing. This patch merge these code in only one. Note that the function map_read_entries_from_file() in the file "map.c" is moved to the the function pat_ref_read_from_file_smp() in the file "pattern.c". The code of this function is not modified, only the the name and the arguments order has changed.
2014-01-29 07:29:45 -05:00
ref->flags |= refflags;
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
}
/* Now, we can loading patterns from the reference. */
/* Lookup for existing reference in the head. If the reference
* doesn't exists, create it.
*/
expr = pattern_lookup_expr(head, ref);
BUG/MEDIUM: patterns: last fix was still not enough Last fix did address the issue for inlined patterns, but it was not enough because the flags are lost as well when updating patterns dynamically over the CLI. Also if the same file was used once with -i and another time without -i, their references would have been merged and both would have used the same matching method. It's appear that the patterns have two types of flags. The first ones are relative to the pattern matching, and the second are relative to the pattern storage. The pattern matching flags are the same for all the patterns of one expression. Now they are stored in the expression. The storage flags are information returned by the pattern mathing function. This information is relative to each entry and is stored in the "struct pattern". Now, the expression matching flags are forwarded to the parse and index functions. These flags are stored during the configuration parsing, and they are used during the parse and index actions. This issue was introduced in dev23 with the major pattern rework, and is a continuation of commit a631fc8 ("BUG/MAJOR: patterns: -i and -n are ignored for inlined patterns"). No backport is needed.
2014-04-28 05:18:57 -04:00
if (!expr || (expr->mflags != patflags)) {
BUG/MEDIUM: map/acl: fix unwanted flags inheritance. The bug: Maps/ACLs using the same file/id can mistakenly inherit their flags from the last declared one. i.e. $ cat haproxy.conf listen mylistener mode http bind 0.0.0.0:8080 acl myacl1 url -i -f mine.acl acl myacl2 url -f mine.acl acl myacl3 url -i -f mine.acl redirect location / if myacl2 $ cat mine.acl foobar Shows an unexpected redirect for request 'GET /FOObAR HTTP/1.0\n\n'. This fix should be backported on mainline branches v1.6 and v1.7.
2017-07-03 11:54:23 -04:00
expr = pattern_new_expr(head, ref, patflags, err, &reuse);
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
if (!expr)
return 0;
}
BUG/MEDIUM: pattern: don't load more than once a pattern list. A memory optimization can use the same pattern expression for many equal pattern list (same parse method, index method and index_smp method). The pattern expression is returned by "pattern_new_expr", but this function dont indicate if the returned pattern is already in use. So, the caller function reload the list of patterns in addition with the existing patterns. This behavior is not a problem with tree indexed pattern, but it grows the lists indexed patterns. This fix add a "reuse" flag in return of the function "pattern_new_expr". If the flag is set, I suppose that the patterns are already loaded. This fix must be backported into 1.5.
2014-11-24 05:14:42 -05:00
/* The returned expression may be not empty, because the function
* "pattern_new_expr" lookup for similar pattern list and can
* reuse a already filled pattern list. In this case, we can not
* reload the patterns.
*/
if (reuse)
return 1;
BUG/MINOR: map: list-based matching potential ordering regression An unexpected side-effect was introduced by 5fea597 ("MEDIUM: map/acl: Accelerate several functions using pat_ref_elt struct ->head list") The above commit tried to use eb tree API to manipulate elements as much as possible in the hope to accelerate some functions. Prior to 5fea597, pattern_read_from_file() used to iterate over all elements from the map file in the same order they were seen in the file (using list_for_each_entry) to push them in the pattern expression. Now, since eb api is used to iterate over elements, the ordering is lost very early. This is known to cause behavior changes with existing setups (same conf and map file) when compared with previous versions for some list-based matching methods as described in GH #2400. For instance, the map_dom() converter may return a different matching key from the one that was returned by older haproxy versions. For IP or STR matching, matching is based on tree lookups for better efficiency, so in this case the ordering is lost at the name of performance. The order in which they are loaded doesn't matter because tree ordering is based on the content, it is not positional. But with some other types, matching is based on list lookups (e.g.: dom), and the order in which elements are pushed into the list can affect the matching element that will be returned (in case of multiple matches, since only the first matching element in the list will be returned). Despite the documentation not officially stating that the file ordering should be preserved for list-based matching methods, it's probably best to be conservative here and stick to historical behavior. Moreover, there was no performance benefit from using the eb tree api to iterate over elements in pattern_read_from_file() since all elements are visited anyway. This should be backported to 2.9.
2024-01-03 05:54:03 -05:00
/* Load reference content in the pattern expression.
* We need to load elements in the same order they were seen in the
MINOR: map: mapfile ordering also matters for tree-based match types Willy made me realize that tree-based matching may also suffer from out-of-order mapfile loading, as opposed to what's being said in b546bb6d ("BUG/MINOR: map: list-based matching potential ordering regression") and the associated REGTEST. Indeed, in case of duplicated keys, we want to be sure that only the key that was first seen in the file will be returned (as long as it is not removed). The above fix is still valid, and the list-based match regtest will also prevent regressions for tree-based match since mapfile loading logic is currently match-type agnostic. But let's clarify that by making both the code comment and the regtest more precise.
2024-01-11 04:31:04 -05:00
* file. Indeed, some list-based matching types may rely on it as the
* list is positional, and for tree-based matching, even if the tree is
* content-based in case of duplicated keys we only want the first key
* in the file to be considered.
BUG/MINOR: map: list-based matching potential ordering regression An unexpected side-effect was introduced by 5fea597 ("MEDIUM: map/acl: Accelerate several functions using pat_ref_elt struct ->head list") The above commit tried to use eb tree API to manipulate elements as much as possible in the hope to accelerate some functions. Prior to 5fea597, pattern_read_from_file() used to iterate over all elements from the map file in the same order they were seen in the file (using list_for_each_entry) to push them in the pattern expression. Now, since eb api is used to iterate over elements, the ordering is lost very early. This is known to cause behavior changes with existing setups (same conf and map file) when compared with previous versions for some list-based matching methods as described in GH #2400. For instance, the map_dom() converter may return a different matching key from the one that was returned by older haproxy versions. For IP or STR matching, matching is based on tree lookups for better efficiency, so in this case the ordering is lost at the name of performance. The order in which they are loaded doesn't matter because tree ordering is based on the content, it is not positional. But with some other types, matching is based on list lookups (e.g.: dom), and the order in which elements are pushed into the list can affect the matching element that will be returned (in case of multiple matches, since only the first matching element in the list will be returned). Despite the documentation not officially stating that the file ordering should be preserved for list-based matching methods, it's probably best to be conservative here and stick to historical behavior. Moreover, there was no performance benefit from using the eb tree api to iterate over elements in pattern_read_from_file() since all elements are visited anyway. This should be backported to 2.9.
2024-01-03 05:54:03 -05:00
*/
MEDIUM: patterns: reorganize pattern reference elements Instead of a global list (and tree) of pattern reference elements, we now have an intermediate pat_ref_gen structure and store the elements in those. This simplifies the logic of some operations such as commit and clear, and improves performance in some cases - numbers to be provided in a subsequent commit after one important optimization is added. A lot of the changes are due to adding an extra level of indirection, changing many cases where we iterate over all elements to an outer loop iterating over the generation and an inner one iterating over the elements of the current generation. It is therefore easier to read this patch using 'git diff -w'.
2025-12-17 23:37:44 -05:00
pat_ref_gen_foreach(gen, ref) {
list_for_each_entry(elt, &gen->head, list) {
if (!pat_ref_push(elt, expr, patflags, err)) {
if (elt->line > 0)
memprintf(err, "%s at line %d of file '%s'",
*err, elt->line, filename);
return 0;
}
MINOR: pattern/map/acl: Centralization of the file parsers The acl and map function do the same work with the file parsing. This patch merge these code in only one. Note that the function map_read_entries_from_file() in the file "map.c" is moved to the the function pat_ref_read_from_file_smp() in the file "pattern.c". The code of this function is not modified, only the the name and the arguments order has changed.
2014-01-29 07:29:45 -05:00
}
}
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
return 1;
}
MEDIUM: pattern: The function pattern_exec_match() returns "struct pattern" if the patten match. Before this commit, the pattern_exec_match() function returns the associate sample, the associate struct pattern or the associate struct pattern_tree. This is complex to use, because we can check the type of information returned. Now the function return always a "struct pattern". If <fill> is not set, only the value of the pointer can be used as boolean (NULL or other). If <fill> is set, you can use the <smp> pointer and the pattern information. If information must be duplicated, it is stored in trash buffer. Otherwise, the pattern can point on existing strings.
2014-01-17 09:25:13 -05:00
/* This function executes a pattern match on a sample. It applies pattern <expr>
CLEANUP: Fix spelling errors in comments This is from the output of codespell. It's done at once over a bunch of files and only affects comments, so there is nothing user-visible. No backport needed.
2021-01-07 23:35:52 -05:00
* to sample <smp>. The function returns NULL if the sample don't match. It returns
MEDIUM: pattern: The function pattern_exec_match() returns "struct pattern" if the patten match. Before this commit, the pattern_exec_match() function returns the associate sample, the associate struct pattern or the associate struct pattern_tree. This is complex to use, because we can check the type of information returned. Now the function return always a "struct pattern". If <fill> is not set, only the value of the pointer can be used as boolean (NULL or other). If <fill> is set, you can use the <smp> pointer and the pattern information. If information must be duplicated, it is stored in trash buffer. Otherwise, the pattern can point on existing strings.
2014-01-17 09:25:13 -05:00
* non-null if the sample match. If <fill> is true and the sample match, the
* function returns the matched pattern. In many cases, this pattern can be a
* static buffer.
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
*/
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
struct pattern *pattern_exec_match(struct pattern_head *head, struct sample *smp, int fill)
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
{
MEDIUM: pattern: merge same pattern Sometimes the same pattern file is used with the same index, parse and parse_smp functions. If this two condition are true, these two pattern are identical and the same struct can be used.
2014-01-20 08:29:33 -05:00
struct pattern_expr_list *list;
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
struct pattern *pat;
if (!head->match) {
MEDIUM: pattern: The function pattern_exec_match() returns "struct pattern" if the patten match. Before this commit, the pattern_exec_match() function returns the associate sample, the associate struct pattern or the associate struct pattern_tree. This is complex to use, because we can check the type of information returned. Now the function return always a "struct pattern". If <fill> is not set, only the value of the pointer can be used as boolean (NULL or other). If <fill> is set, you can use the <smp> pointer and the pattern information. If information must be duplicated, it is stored in trash buffer. Otherwise, the pattern can point on existing strings.
2014-01-17 09:25:13 -05:00
if (fill) {
MINOR: samples: rename some struct member from "smp" to "data" This members contains data and not sample.
2015-08-19 02:35:43 -04:00
static_pattern.data = NULL;
MINOR: pattern: Each pattern expression element store the reference struct. Now, each pattern entry known the original "struct pat_ref_elt" from that was built. This patch permit to delete each pattern entry without confusion. After this patch, each reference can use his pointer to be targeted.
2014-01-28 09:54:36 -05:00
static_pattern.ref = NULL;
BUG/MEDIUM: patterns: last fix was still not enough Last fix did address the issue for inlined patterns, but it was not enough because the flags are lost as well when updating patterns dynamically over the CLI. Also if the same file was used once with -i and another time without -i, their references would have been merged and both would have used the same matching method. It's appear that the patterns have two types of flags. The first ones are relative to the pattern matching, and the second are relative to the pattern storage. The pattern matching flags are the same for all the patterns of one expression. Now they are stored in the expression. The storage flags are information returned by the pattern mathing function. This information is relative to each entry and is stored in the "struct pattern". Now, the expression matching flags are forwarded to the parse and index functions. These flags are stored during the configuration parsing, and they are used during the parse and index actions. This issue was introduced in dev23 with the major pattern rework, and is a continuation of commit a631fc8 ("BUG/MAJOR: patterns: -i and -n are ignored for inlined patterns"). No backport is needed.
2014-04-28 05:18:57 -04:00
static_pattern.sflags = 0;
MAJOR: sample: converts uint and sint in 64 bits signed integer This patch removes the 32 bits unsigned integer and the 32 bit signed integer. It replaces these types by a unique type 64 bit signed. This makes easy the usage of integer and clarify signed and unsigned use. With the previous version, signed and unsigned are used ones in place of others, and sometimes the converter loose the sign. For example, divisions are processed with "unsigned", if one entry is negative, the result is wrong. Note that the integer pattern matching and dotted version pattern matching are already working with signed 64 bits integer values. There is one user-visible change : the "uint()" and "sint()" sample fetch functions which used to return a constant integer have been replaced with a new more natural, unified "int()" function. These functions were only introduced in the latest 1.6-dev2 so there's no impact on regular deployments.
2015-07-06 17:43:03 -04:00
static_pattern.type = SMP_T_SINT;
MEDIUM: pattern: The match function browse itself the list or the tree. The match function known the format of the pattern. The pattern can be stored in a list or in a tree. The pattern matching function use itself the good entry point and indexation type. Each pattern matching function return the struct pattern that match. If the flag "fill" is set, the struct pattern is filled, otherwise the content of this struct must not be used. With this feature, the general pattern matching function cannot have exceptions for building the "struct pattern".
2013-12-16 08:22:13 -05:00
static_pattern.val.i = 1;
MEDIUM: pattern: The function pattern_exec_match() returns "struct pattern" if the patten match. Before this commit, the pattern_exec_match() function returns the associate sample, the associate struct pattern or the associate struct pattern_tree. This is complex to use, because we can check the type of information returned. Now the function return always a "struct pattern". If <fill> is not set, only the value of the pointer can be used as boolean (NULL or other). If <fill> is set, you can use the <smp> pointer and the pattern information. If information must be duplicated, it is stored in trash buffer. Otherwise, the pattern can point on existing strings.
2014-01-17 09:25:13 -05:00
}
return &static_pattern;
}
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
MEDIUM: pattern: The expected type is stored in the pattern head, and conversion is executed once. This patch extract the expect_type variable from the "struct pattern" to "struct pattern_head". This variable is set during the declaration of ACL and MAP. With this change, the function "pat_parse_len()" become useless and can be replaced by "pat_parse_int()". Implicit ACLs by default rely on the fetch's output type, so let's simply do the same for all other ones. It has been verified that they all match.
2014-01-27 08:19:53 -05:00
/* convert input to string */
if (!sample_convert(smp, head->expect_type))
return NULL;
MEDIUM: pattern: merge same pattern Sometimes the same pattern file is used with the same index, parse and parse_smp functions. If this two condition are true, these two pattern are identical and the same struct can be used.
2014-01-20 08:29:33 -05:00
list_for_each_entry(list, &head->head, list) {
BUILD: threads: Rename SPIN/RWLOCK macros using HA_ prefix This remove any name conflicts, especially on Solaris.
2017-11-07 04:42:54 -05:00
HA_RWLOCK_RDLOCK(PATEXP_LOCK, &list->expr->lock);
MEDIUM: pattern: merge same pattern Sometimes the same pattern file is used with the same index, parse and parse_smp functions. If this two condition are true, these two pattern are identical and the same struct can be used.
2014-01-20 08:29:33 -05:00
pat = head->match(smp, list->expr, fill);
MAJOR: threads/map: Make acls/maps thread safe locks have been added in pat_ref and pattern_expr structures to protect all accesses to an instance of on of them. Moreover, a global lock has been added to protect the LRU cache used for pattern matching. Patterns are now duplicated after a successfull matching, to avoid modification by other threads when the result is used. Finally, the function reloading a pattern list has been modified to be thread-safe.
2017-07-03 05:34:05 -04:00
if (pat) {
/* We duplicate the pattern cause it could be modified
by another thread */
if (pat != &static_pattern) {
memcpy(&static_pattern, pat, sizeof(struct pattern));
pat = &static_pattern;
}
/* We also duplicate the sample data for
same reason */
if (pat->data && (pat->data != &static_sample_data)) {
BUG/MINOR: pattern: Rely on the sample type to copy it in pattern_exec_match To be thread safe, the function pattern_exec_match copy data (the pattern and the inner sample) in thread-local variables. But when the sample is duplicated, we must check its type and not the pattern one. This is specific to threads, no backport is needed.
2017-11-09 10:14:16 -05:00
switch(pat->data->type) {
MAJOR: threads/map: Make acls/maps thread safe locks have been added in pat_ref and pattern_expr structures to protect all accesses to an instance of on of them. Moreover, a global lock has been added to protect the LRU cache used for pattern matching. Patterns are now duplicated after a successfull matching, to avoid modification by other threads when the result is used. Finally, the function reloading a pattern list has been modified to be thread-safe.
2017-07-03 05:34:05 -04:00
case SMP_T_STR:
static_sample_data.type = SMP_T_STR;
static_sample_data.u.str = *get_trash_chunk();
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
static_sample_data.u.str.data = pat->data->u.str.data;
if (static_sample_data.u.str.data >= static_sample_data.u.str.size)
static_sample_data.u.str.data = static_sample_data.u.str.size - 1;
memcpy(static_sample_data.u.str.area,
BUG/MEDIUM: pattern: fix thread safety of pattern matching Commit b5997f740 ("MAJOR: threads/map: Make acls/maps thread safe") introduced a subtle bug in the pattern matching code. In order to cope with the possibility that another thread might be modifying the pattern's sample_data while it's being used, we return a thread-local static sample_data which is a copy of the one found in the matched pattern. The copy is performed depending on the sample_data's type. But the switch statement misses some breaks and doesn't set the new sample_data pointer at the right place, resulting in the original sample_data being restored at the end before returning. The net effect overall is that the correct sample_data is returned (hence functionally speaking the matching works fine) but it's not thread-safe so any del_map() or set_map() action could modify the pattern on one thread while it's being used on another one. It doesn't seem likely to cause a crash but could result in corrupted data appearing where the value is consumed (e.g. when appended in a header or when logged) or an ACL occasionally not matching after a map lookup. This fix should be backported as far as 1.8. Thanks to Tim for reporting it and to Emeric for the analysis.
2020-06-11 10:37:35 -04:00
pat->data->u.str.area, static_sample_data.u.str.data);
MEDIUM: chunks: make the chunk struct's fields match the buffer struct Chunks are only a subset of a buffer (a non-wrapping version with no head offset). Despite this we still carry a lot of duplicated code between buffers and chunks. Replacing chunks with buffers would significantly reduce the maintenance efforts. This first patch renames the chunk's fields to match the name and types used by struct buffers, with the goal of isolating the code changes from the declaration changes. Most of the changes were made with spatch using this coccinelle script : @rule_d1@ typedef chunk; struct chunk chunk; @@ - chunk.str + chunk.area @rule_d2@ typedef chunk; struct chunk chunk; @@ - chunk.len + chunk.data @rule_i1@ typedef chunk; struct chunk *chunk; @@ - chunk->str + chunk->area @rule_i2@ typedef chunk; struct chunk *chunk; @@ - chunk->len + chunk->data Some minor updates to 3 http functions had to be performed to take size_t ints instead of ints in order to match the unsigned length here.
2018-07-13 04:54:26 -04:00
static_sample_data.u.str.area[static_sample_data.u.str.data] = 0;
BUG/MEDIUM: pattern: fix thread safety of pattern matching Commit b5997f740 ("MAJOR: threads/map: Make acls/maps thread safe") introduced a subtle bug in the pattern matching code. In order to cope with the possibility that another thread might be modifying the pattern's sample_data while it's being used, we return a thread-local static sample_data which is a copy of the one found in the matched pattern. The copy is performed depending on the sample_data's type. But the switch statement misses some breaks and doesn't set the new sample_data pointer at the right place, resulting in the original sample_data being restored at the end before returning. The net effect overall is that the correct sample_data is returned (hence functionally speaking the matching works fine) but it's not thread-safe so any del_map() or set_map() action could modify the pattern on one thread while it's being used on another one. It doesn't seem likely to cause a crash but could result in corrupted data appearing where the value is consumed (e.g. when appended in a header or when logged) or an ACL occasionally not matching after a map lookup. This fix should be backported as far as 1.8. Thanks to Tim for reporting it and to Emeric for the analysis.
2020-06-11 10:37:35 -04:00
pat->data = &static_sample_data;
break;
MAJOR: threads/map: Make acls/maps thread safe locks have been added in pat_ref and pattern_expr structures to protect all accesses to an instance of on of them. Moreover, a global lock has been added to protect the LRU cache used for pattern matching. Patterns are now duplicated after a successfull matching, to avoid modification by other threads when the result is used. Finally, the function reloading a pattern list has been modified to be thread-safe.
2017-07-03 05:34:05 -04:00
case SMP_T_IPV4:
case SMP_T_IPV6:
case SMP_T_SINT:
memcpy(&static_sample_data, pat->data, sizeof(struct sample_data));
BUG/MEDIUM: pattern: fix thread safety of pattern matching Commit b5997f740 ("MAJOR: threads/map: Make acls/maps thread safe") introduced a subtle bug in the pattern matching code. In order to cope with the possibility that another thread might be modifying the pattern's sample_data while it's being used, we return a thread-local static sample_data which is a copy of the one found in the matched pattern. The copy is performed depending on the sample_data's type. But the switch statement misses some breaks and doesn't set the new sample_data pointer at the right place, resulting in the original sample_data being restored at the end before returning. The net effect overall is that the correct sample_data is returned (hence functionally speaking the matching works fine) but it's not thread-safe so any del_map() or set_map() action could modify the pattern on one thread while it's being used on another one. It doesn't seem likely to cause a crash but could result in corrupted data appearing where the value is consumed (e.g. when appended in a header or when logged) or an ACL occasionally not matching after a map lookup. This fix should be backported as far as 1.8. Thanks to Tim for reporting it and to Emeric for the analysis.
2020-06-11 10:37:35 -04:00
pat->data = &static_sample_data;
break;
MAJOR: threads/map: Make acls/maps thread safe locks have been added in pat_ref and pattern_expr structures to protect all accesses to an instance of on of them. Moreover, a global lock has been added to protect the LRU cache used for pattern matching. Patterns are now duplicated after a successfull matching, to avoid modification by other threads when the result is used. Finally, the function reloading a pattern list has been modified to be thread-safe.
2017-07-03 05:34:05 -04:00
default:
BUG/MEDIUM: pattern: fix thread safety of pattern matching Commit b5997f740 ("MAJOR: threads/map: Make acls/maps thread safe") introduced a subtle bug in the pattern matching code. In order to cope with the possibility that another thread might be modifying the pattern's sample_data while it's being used, we return a thread-local static sample_data which is a copy of the one found in the matched pattern. The copy is performed depending on the sample_data's type. But the switch statement misses some breaks and doesn't set the new sample_data pointer at the right place, resulting in the original sample_data being restored at the end before returning. The net effect overall is that the correct sample_data is returned (hence functionally speaking the matching works fine) but it's not thread-safe so any del_map() or set_map() action could modify the pattern on one thread while it's being used on another one. It doesn't seem likely to cause a crash but could result in corrupted data appearing where the value is consumed (e.g. when appended in a header or when logged) or an ACL occasionally not matching after a map lookup. This fix should be backported as far as 1.8. Thanks to Tim for reporting it and to Emeric for the analysis.
2020-06-11 10:37:35 -04:00
/* unimplemented pattern type */
MAJOR: threads/map: Make acls/maps thread safe locks have been added in pat_ref and pattern_expr structures to protect all accesses to an instance of on of them. Moreover, a global lock has been added to protect the LRU cache used for pattern matching. Patterns are now duplicated after a successfull matching, to avoid modification by other threads when the result is used. Finally, the function reloading a pattern list has been modified to be thread-safe.
2017-07-03 05:34:05 -04:00
pat->data = NULL;
BUG/MEDIUM: pattern: fix thread safety of pattern matching Commit b5997f740 ("MAJOR: threads/map: Make acls/maps thread safe") introduced a subtle bug in the pattern matching code. In order to cope with the possibility that another thread might be modifying the pattern's sample_data while it's being used, we return a thread-local static sample_data which is a copy of the one found in the matched pattern. The copy is performed depending on the sample_data's type. But the switch statement misses some breaks and doesn't set the new sample_data pointer at the right place, resulting in the original sample_data being restored at the end before returning. The net effect overall is that the correct sample_data is returned (hence functionally speaking the matching works fine) but it's not thread-safe so any del_map() or set_map() action could modify the pattern on one thread while it's being used on another one. It doesn't seem likely to cause a crash but could result in corrupted data appearing where the value is consumed (e.g. when appended in a header or when logged) or an ACL occasionally not matching after a map lookup. This fix should be backported as far as 1.8. Thanks to Tim for reporting it and to Emeric for the analysis.
2020-06-11 10:37:35 -04:00
break;
MAJOR: threads/map: Make acls/maps thread safe locks have been added in pat_ref and pattern_expr structures to protect all accesses to an instance of on of them. Moreover, a global lock has been added to protect the LRU cache used for pattern matching. Patterns are now duplicated after a successfull matching, to avoid modification by other threads when the result is used. Finally, the function reloading a pattern list has been modified to be thread-safe.
2017-07-03 05:34:05 -04:00
}
}
BUILD: threads: Rename SPIN/RWLOCK macros using HA_ prefix This remove any name conflicts, especially on Solaris.
2017-11-07 04:42:54 -05:00
HA_RWLOCK_RDUNLOCK(PATEXP_LOCK, &list->expr->lock);
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
return pat;
MAJOR: threads/map: Make acls/maps thread safe locks have been added in pat_ref and pattern_expr structures to protect all accesses to an instance of on of them. Moreover, a global lock has been added to protect the LRU cache used for pattern matching. Patterns are now duplicated after a successfull matching, to avoid modification by other threads when the result is used. Finally, the function reloading a pattern list has been modified to be thread-safe.
2017-07-03 05:34:05 -04:00
}
BUILD: threads: Rename SPIN/RWLOCK macros using HA_ prefix This remove any name conflicts, especially on Solaris.
2017-11-07 04:42:54 -05:00
HA_RWLOCK_RDUNLOCK(PATEXP_LOCK, &list->expr->lock);
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
}
return NULL;
REORG: acl/pattern: extract pattern matching from the acl file and create pattern.c This patch just moves code without any change. The ACL are just the association between sample and pattern. The pattern contains the match method and the parse method. These two things are different. This patch cleans the code by splitting it.
2013-11-28 05:05:19 -05:00
}
CLEANUP: pattern: fix spelling/grammatical/copy-paste in comments The code is horrible to work with because most functions are documented with misleading comments resulting from many spelling and grammatical mistakes, and plenty of remains of copy-paste mentioning arguments that do not exist and return values that are never set. Too many hours wasted writing non-working code because of assumptions resulting from this, let's fix this once for all now!
2020-10-30 11:03:50 -04:00
/* This function prunes the pattern expressions starting at pattern_head <head>. */
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
void pattern_prune(struct pattern_head *head)
MEDIUM: pattern: add prune function This path add specific pointer to each expression to point on prune function. Now, each pattern expression embed his own prune function.
2014-01-14 10:24:51 -05:00
{
MEDIUM: pattern: merge same pattern Sometimes the same pattern file is used with the same index, parse and parse_smp functions. If this two condition are true, these two pattern are identical and the same struct can be used.
2014-01-20 08:29:33 -05:00
struct pattern_expr_list *list, *safe;
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
MEDIUM: pattern: merge same pattern Sometimes the same pattern file is used with the same index, parse and parse_smp functions. If this two condition are true, these two pattern are identical and the same struct can be used.
2014-01-20 08:29:33 -05:00
list_for_each_entry_safe(list, safe, &head->head, list) {
CLEANUP: lists/tree-wide: rename some list operations to avoid some confusion The current "ADD" vs "ADDQ" is confusing because when thinking in terms of appending at the end of a list, "ADD" naturally comes to mind, but here it does the opposite, it inserts. Several times already it's been incorrectly used where ADDQ was expected, the latest of which was a fortunate accident explained in 6fa922562 ("CLEANUP: stream: explain why we queue the stream at the head of the server list"). Let's use more explicit (but slightly longer) names now: LIST_ADD -> LIST_INSERT LIST_ADDQ -> LIST_APPEND LIST_ADDED -> LIST_INLIST LIST_DEL -> LIST_DELETE The same is true for MT_LISTs, including their "TRY" variant. LIST_DEL_INIT keeps its short name to encourage to use it instead of the lazier LIST_DELETE which is often less safe. The change is large (~674 non-comment entries) but is mechanical enough to remain safe. No permutation was performed, so any out-of-tree code can easily map older names to new ones. The list doc was updated.
2021-04-21 01:32:39 -04:00
LIST_DELETE(&list->list);
BUG/MEDIUM: pattern: prevent UAF on reused pattern expr Since c5959fd ("MEDIUM: pattern: merge same pattern"), UAF (leading to crash) can be experienced if the same pattern file (and match method) is used in two default sections and the first one is not referenced later in the config. In this case, the first default section will be cleaned up. However, due to an unhandled case in the above optimization, the original expr which the second default section relies on is mistakenly freed. This issue was discovered while trying to reproduce GH #2708. The issue was particularly tricky to reproduce given the config and sequence required to make the UAF happen. Hopefully, Github user @asmnek not only provided useful informations, but since he was able to consistently trigger the crash in his environment he was able to nail down the crash to the use of pattern file involved with 2 named default sections. Big thanks to him. To fix the issue, let's push the logic from c5959fd a bit further. Instead of relying on "do_free" variable to know if the expression should be freed or not (which proved to be insufficient in our case), let's switch to a simple refcounting logic. This way, no matter who owns the expression, the last one attempting to free it will be responsible for freeing it. Refcount is implemented using a 32bit value which fills a previous 4 bytes structure gap: int mflags; /* 80 4 */ /* XXX 4 bytes hole, try to pack */ long unsigned int lock; /* 88 8 */ (output from pahole) Even though it was not reproduced in 2.6 or below by @asmnek (the bug was revealed thanks to another bugfix), this issue theorically affects all stable versions (up to c5959fd), thus it should be backported to all stable versions.
2024-09-09 08:59:19 -04:00
if (HA_ATOMIC_SUB_FETCH(&list->expr->refcount, 1) == 0) {
CLEANUP: lists/tree-wide: rename some list operations to avoid some confusion The current "ADD" vs "ADDQ" is confusing because when thinking in terms of appending at the end of a list, "ADD" naturally comes to mind, but here it does the opposite, it inserts. Several times already it's been incorrectly used where ADDQ was expected, the latest of which was a fortunate accident explained in 6fa922562 ("CLEANUP: stream: explain why we queue the stream at the head of the server list"). Let's use more explicit (but slightly longer) names now: LIST_ADD -> LIST_INSERT LIST_ADDQ -> LIST_APPEND LIST_ADDED -> LIST_INLIST LIST_DEL -> LIST_DELETE The same is true for MT_LISTs, including their "TRY" variant. LIST_DEL_INIT keeps its short name to encourage to use it instead of the lazier LIST_DELETE which is often less safe. The change is large (~674 non-comment entries) but is mechanical enough to remain safe. No permutation was performed, so any out-of-tree code can easily map older names to new ones. The list doc was updated.
2021-04-21 01:32:39 -04:00
LIST_DELETE(&list->expr->list);
BUILD: threads: Rename SPIN/RWLOCK macros using HA_ prefix This remove any name conflicts, especially on Solaris.
2017-11-07 04:42:54 -05:00
HA_RWLOCK_WRLOCK(PATEXP_LOCK, &list->expr->lock);
MEDIUM: pattern: merge same pattern Sometimes the same pattern file is used with the same index, parse and parse_smp functions. If this two condition are true, these two pattern are identical and the same struct can be used.
2014-01-20 08:29:33 -05:00
head->prune(list->expr);
BUILD: threads: Rename SPIN/RWLOCK macros using HA_ prefix This remove any name conflicts, especially on Solaris.
2017-11-07 04:42:54 -05:00
HA_RWLOCK_WRUNLOCK(PATEXP_LOCK, &list->expr->lock);
MEDIUM: pattern: merge same pattern Sometimes the same pattern file is used with the same index, parse and parse_smp functions. If this two condition are true, these two pattern are identical and the same struct can be used.
2014-01-20 08:29:33 -05:00
free(list->expr);
}
free(list);
MAJOR: pattern/map: Extends the map edition system in the patterns This patch add the following socket command line options: show acl [<id>] clear acl <id> get acl <id> <pattern> del acl <id> <pattern> add acl <id> <pattern> The system used for maps is backported in the pattern functions.
2014-02-11 05:31:40 -05:00
}
MEDIUM: pattern: add prune function This path add specific pointer to each expression to point on prune function. Now, each pattern expression embed his own prune function.
2014-01-14 10:24:51 -05:00
}
CLEANUP: pattern: fix spelling/grammatical/copy-paste in comments The code is horrible to work with because most functions are documented with misleading comments resulting from many spelling and grammatical mistakes, and plenty of remains of copy-paste mentioning arguments that do not exist and return values that are never set. Too many hours wasted writing non-working code because of assumptions resulting from this, let's fix this once for all now!
2020-10-30 11:03:50 -04:00
/* This function compares two pat_ref** on their unique_id, and returns -1/0/1
* depending on their order (suitable for sorting).
*/
OPTIM: startup: fast unique_id allocation for acl. pattern_finalize_config() uses an inefficient algorithm which is a problem with very large configuration files. This affects startup, and therefore reload time. When haproxy is deployed as a router in a Kubernetes cluster the generated configuration file may be large and reloads are frequently occuring, which makes this a significant issue. The old algorithm is O(n^2) * allocate missing uids - O(n^2) * sort linked list - O(n^2) The new algorithm is O(n log n): * find the user allocated uids - O(n) * store them for efficient lookup - O(n log n) * allocate missing uids - n times O(log n) * sort all uids - O(n log n) * convert back to linked list - O(n) Performance examples, startup time in seconds: pat_refs old new 1000 0.02 0.01 10000 2.1 0.04 20000 12.3 0.07 30000 27.9 0.10 40000 52.5 0.14 50000 77.5 0.17 Please backport to 1.8, 2.0 and 2.1.
2020-02-27 10:45:50 -05:00
static int cmp_pat_ref(const void *_a, const void *_b)
{
struct pat_ref * const *a = _a;
struct pat_ref * const *b = _b;
if ((*a)->unique_id < (*b)->unique_id)
return -1;
else if ((*a)->unique_id > (*b)->unique_id)
return 1;
return 0;
}
CLEANUP: pattern: fix spelling/grammatical/copy-paste in comments The code is horrible to work with because most functions are documented with misleading comments resulting from many spelling and grammatical mistakes, and plenty of remains of copy-paste mentioning arguments that do not exist and return values that are never set. Too many hours wasted writing non-working code because of assumptions resulting from this, let's fix this once for all now!
2020-10-30 11:03:50 -04:00
/* This function finalizes the configuration parsing. It sets all the
* automatic ids.
MINOR: pattern: Each pattern is identified by unique id. The pattern reference are stored with two identifiers: the unique_id and the reference. The reference identify a file. Each file with the same name point to the same reference. We can register many times one file. If the file is modified, all his dependencies are also modified. The reference can be used with map or acl. The unique_id identify inline acl. The unique id is unique for each acl. You cannot force the same id in the configuration file, because this repport an error. The format of the acl and map listing through the "socket" has changed for displaying these new ids.
2014-03-11 09:29:22 -04:00
*/
OPTIM: startup: fast unique_id allocation for acl. pattern_finalize_config() uses an inefficient algorithm which is a problem with very large configuration files. This affects startup, and therefore reload time. When haproxy is deployed as a router in a Kubernetes cluster the generated configuration file may be large and reloads are frequently occuring, which makes this a significant issue. The old algorithm is O(n^2) * allocate missing uids - O(n^2) * sort linked list - O(n^2) The new algorithm is O(n log n): * find the user allocated uids - O(n) * store them for efficient lookup - O(n log n) * allocate missing uids - n times O(log n) * sort all uids - O(n log n) * convert back to linked list - O(n) Performance examples, startup time in seconds: pat_refs old new 1000 0.02 0.01 10000 2.1 0.04 20000 12.3 0.07 30000 27.9 0.10 40000 52.5 0.14 50000 77.5 0.17 Please backport to 1.8, 2.0 and 2.1.
2020-02-27 10:45:50 -05:00
int pattern_finalize_config(void)
MINOR: pattern: Each pattern is identified by unique id. The pattern reference are stored with two identifiers: the unique_id and the reference. The reference identify a file. Each file with the same name point to the same reference. We can register many times one file. If the file is modified, all his dependencies are also modified. The reference can be used with map or acl. The unique_id identify inline acl. The unique id is unique for each acl. You cannot force the same id in the configuration file, because this repport an error. The format of the acl and map listing through the "socket" has changed for displaying these new ids.
2014-03-11 09:29:22 -04:00
{
BUG/MINOR: pattern: Do not pass len = 0 to calloc() The behavior of calloc() when being passed `0` as `nelem` is implementation defined. It may return a NULL pointer. Avoid this issue by checking before allocating. While doing so adjust the local integer variables that are used to refer to memory offsets to `size_t`. This issue was introced in commit f91ac19299fe216a793ba6550dca06b688b31549. This patch should be backported together with that commit.
2020-03-17 16:08:24 -04:00
size_t len = 0;
size_t unassigned_pos = 0;
OPTIM: startup: fast unique_id allocation for acl. pattern_finalize_config() uses an inefficient algorithm which is a problem with very large configuration files. This affects startup, and therefore reload time. When haproxy is deployed as a router in a Kubernetes cluster the generated configuration file may be large and reloads are frequently occuring, which makes this a significant issue. The old algorithm is O(n^2) * allocate missing uids - O(n^2) * sort linked list - O(n^2) The new algorithm is O(n log n): * find the user allocated uids - O(n) * store them for efficient lookup - O(n log n) * allocate missing uids - n times O(log n) * sort all uids - O(n log n) * convert back to linked list - O(n) Performance examples, startup time in seconds: pat_refs old new 1000 0.02 0.01 10000 2.1 0.04 20000 12.3 0.07 30000 27.9 0.10 40000 52.5 0.14 50000 77.5 0.17 Please backport to 1.8, 2.0 and 2.1.
2020-02-27 10:45:50 -05:00
int next_unique_id = 0;
BUG/MINOR: pattern: Do not pass len = 0 to calloc() The behavior of calloc() when being passed `0` as `nelem` is implementation defined. It may return a NULL pointer. Avoid this issue by checking before allocating. While doing so adjust the local integer variables that are used to refer to memory offsets to `size_t`. This issue was introced in commit f91ac19299fe216a793ba6550dca06b688b31549. This patch should be backported together with that commit.
2020-03-17 16:08:24 -04:00
size_t i, j;
OPTIM: startup: fast unique_id allocation for acl. pattern_finalize_config() uses an inefficient algorithm which is a problem with very large configuration files. This affects startup, and therefore reload time. When haproxy is deployed as a router in a Kubernetes cluster the generated configuration file may be large and reloads are frequently occuring, which makes this a significant issue. The old algorithm is O(n^2) * allocate missing uids - O(n^2) * sort linked list - O(n^2) The new algorithm is O(n log n): * find the user allocated uids - O(n) * store them for efficient lookup - O(n log n) * allocate missing uids - n times O(log n) * sort all uids - O(n log n) * convert back to linked list - O(n) Performance examples, startup time in seconds: pat_refs old new 1000 0.02 0.01 10000 2.1 0.04 20000 12.3 0.07 30000 27.9 0.10 40000 52.5 0.14 50000 77.5 0.17 Please backport to 1.8, 2.0 and 2.1.
2020-02-27 10:45:50 -05:00
struct pat_ref *ref, **arr;
MINOR: pattern: Each pattern is identified by unique id. The pattern reference are stored with two identifiers: the unique_id and the reference. The reference identify a file. Each file with the same name point to the same reference. We can register many times one file. If the file is modified, all his dependencies are also modified. The reference can be used with map or acl. The unique_id identify inline acl. The unique id is unique for each acl. You cannot force the same id in the configuration file, because this repport an error. The format of the acl and map listing through the "socket" has changed for displaying these new ids.
2014-03-11 09:29:22 -04:00
struct list pr = LIST_HEAD_INIT(pr);
BUG/MEDIUM: random: implement a thread-safe and process-safe PRNG This is the replacement of failed attempt to add thread safety and per-process sequences of random numbers initally tried with commit 1c306aa84d ("BUG/MEDIUM: random: implement per-thread and per-process random sequences"). This new version takes a completely different approach and doesn't try to work around the horrible OS-specific and non-portable random API anymore. Instead it implements "xoroshiro128**", a reputedly high quality random number generator, which is one of the many variants of xorshift, which passes all quality tests and which is described here: http://prng.di.unimi.it/ While not cryptographically secure, it is fast and features a 2^128-1 period. It supports fast jumps allowing to cut the period into smaller non-overlapping sequences, which we use here to support up to 2^32 processes each having their own, non-overlapping sequence of 2^96 numbers (~7*10^28). This is enough to provide 1 billion randoms per second and per process for 2200 billion years. The implementation was made thread-safe either by using a double 64-bit CAS on platforms supporting it (x86_64, aarch64) or by using a local lock for the time needed to perform the shift operations. This ensures that all threads pick numbers from the same pool so that it is not needed to assign per-thread ranges. For processes we use the fast jump method to advance the sequence by 2^96 for each process. Before this patch, the following config: global nbproc 8 frontend f bind :4445 mode http log stdout format raw daemon log-format "%[uuid] %pid" redirect location / Would produce this output: a4d0ad64-2645-4b74-b894-48acce0669af 12987 a4d0ad64-2645-4b74-b894-48acce0669af 12992 a4d0ad64-2645-4b74-b894-48acce0669af 12986 a4d0ad64-2645-4b74-b894-48acce0669af 12988 a4d0ad64-2645-4b74-b894-48acce0669af 12991 a4d0ad64-2645-4b74-b894-48acce0669af 12989 a4d0ad64-2645-4b74-b894-48acce0669af 12990 82d5f6cd-f6c1-4f85-a89c-36ae85d26fb9 12987 82d5f6cd-f6c1-4f85-a89c-36ae85d26fb9 12992 82d5f6cd-f6c1-4f85-a89c-36ae85d26fb9 12986 (...) And now produces: f94b29b3-da74-4e03-a0c5-a532c635bad9 13011 47470c02-4862-4c33-80e7-a952899570e5 13014 86332123-539a-47bf-853f-8c8ea8b2a2b5 13013 8f9efa99-3143-47b2-83cf-d618c8dea711 13012 3cc0f5c7-d790-496b-8d39-bec77647af5b 13015 3ec64915-8f95-4374-9e66-e777dc8791e0 13009 0f9bf894-dcde-408c-b094-6e0bb3255452 13011 49c7bfde-3ffb-40e9-9a8d-8084d650ed8f 13014 e23f6f2e-35c5-4433-a294-b790ab902653 13012 There are multiple benefits to using this method. First, it doesn't depend anymore on a non-portable API. Second it's thread safe. Third it is fast and more proven than any hack we could attempt to try to work around the deficiencies of the various implementations around. This commit depends on previous patches "MINOR: tools: add 64-bit rotate operators" and "BUG/MEDIUM: random: initialize the random pool a bit better", all of which will need to be backported at least as far as version 2.0. It doesn't require to backport the build fixes for circular include files dependecy anymore.
2020-03-07 18:42:37 -05:00
pat_lru_seed = ha_random();
MAJOR: pattern: add LRU-based cache on pattern matching The principle of this cache is to have a global cache for all pattern matching operations which rely on lists (reg, sub, dir, dom, ...). The input data, the expression and a random seed are used as a hashing key. The cached entries contains a pointer to the expression and a revision number for that expression so that we don't accidently used obsolete data after a pattern update or a very unlikely hash collision. Regarding the risk of collisions, 10k entries at 10k req/s mean 1% risk of a collision after 60 years, that's already much less than the memory's reliability in most machines and more durable than most admin's life expectancy. A collision will result in a valid result to be returned for a different entry from the same list. If this is not acceptable, the cache can be disabled using tune.pattern.cache-size. A test on a file containing 10k small regex showed that the regex matching was limited to 6k/s instead of 70k with regular strings. When enabling the LRU cache, the performance was back to 70k/s.
2015-04-29 10:24:50 -04:00
OPTIM: startup: fast unique_id allocation for acl. pattern_finalize_config() uses an inefficient algorithm which is a problem with very large configuration files. This affects startup, and therefore reload time. When haproxy is deployed as a router in a Kubernetes cluster the generated configuration file may be large and reloads are frequently occuring, which makes this a significant issue. The old algorithm is O(n^2) * allocate missing uids - O(n^2) * sort linked list - O(n^2) The new algorithm is O(n log n): * find the user allocated uids - O(n) * store them for efficient lookup - O(n log n) * allocate missing uids - n times O(log n) * sort all uids - O(n log n) * convert back to linked list - O(n) Performance examples, startup time in seconds: pat_refs old new 1000 0.02 0.01 10000 2.1 0.04 20000 12.3 0.07 30000 27.9 0.10 40000 52.5 0.14 50000 77.5 0.17 Please backport to 1.8, 2.0 and 2.1.
2020-02-27 10:45:50 -05:00
/* Count pat_refs with user defined unique_id and totalt count */
MINOR: pattern: Each pattern is identified by unique id. The pattern reference are stored with two identifiers: the unique_id and the reference. The reference identify a file. Each file with the same name point to the same reference. We can register many times one file. If the file is modified, all his dependencies are also modified. The reference can be used with map or acl. The unique_id identify inline acl. The unique id is unique for each acl. You cannot force the same id in the configuration file, because this repport an error. The format of the acl and map listing through the "socket" has changed for displaying these new ids.
2014-03-11 09:29:22 -04:00
list_for_each_entry(ref, &pattern_reference, list) {
OPTIM: startup: fast unique_id allocation for acl. pattern_finalize_config() uses an inefficient algorithm which is a problem with very large configuration files. This affects startup, and therefore reload time. When haproxy is deployed as a router in a Kubernetes cluster the generated configuration file may be large and reloads are frequently occuring, which makes this a significant issue. The old algorithm is O(n^2) * allocate missing uids - O(n^2) * sort linked list - O(n^2) The new algorithm is O(n log n): * find the user allocated uids - O(n) * store them for efficient lookup - O(n log n) * allocate missing uids - n times O(log n) * sort all uids - O(n log n) * convert back to linked list - O(n) Performance examples, startup time in seconds: pat_refs old new 1000 0.02 0.01 10000 2.1 0.04 20000 12.3 0.07 30000 27.9 0.10 40000 52.5 0.14 50000 77.5 0.17 Please backport to 1.8, 2.0 and 2.1.
2020-02-27 10:45:50 -05:00
len++;
if (ref->unique_id != -1)
unassigned_pos++;
}
MINOR: pattern: Each pattern is identified by unique id. The pattern reference are stored with two identifiers: the unique_id and the reference. The reference identify a file. Each file with the same name point to the same reference. We can register many times one file. If the file is modified, all his dependencies are also modified. The reference can be used with map or acl. The unique_id identify inline acl. The unique id is unique for each acl. You cannot force the same id in the configuration file, because this repport an error. The format of the acl and map listing through the "socket" has changed for displaying these new ids.
2014-03-11 09:29:22 -04:00
BUG/MINOR: pattern: Do not pass len = 0 to calloc() The behavior of calloc() when being passed `0` as `nelem` is implementation defined. It may return a NULL pointer. Avoid this issue by checking before allocating. While doing so adjust the local integer variables that are used to refer to memory offsets to `size_t`. This issue was introced in commit f91ac19299fe216a793ba6550dca06b688b31549. This patch should be backported together with that commit.
2020-03-17 16:08:24 -04:00
if (len == 0) {
return 0;
}
OPTIM: startup: fast unique_id allocation for acl. pattern_finalize_config() uses an inefficient algorithm which is a problem with very large configuration files. This affects startup, and therefore reload time. When haproxy is deployed as a router in a Kubernetes cluster the generated configuration file may be large and reloads are frequently occuring, which makes this a significant issue. The old algorithm is O(n^2) * allocate missing uids - O(n^2) * sort linked list - O(n^2) The new algorithm is O(n log n): * find the user allocated uids - O(n) * store them for efficient lookup - O(n log n) * allocate missing uids - n times O(log n) * sort all uids - O(n log n) * convert back to linked list - O(n) Performance examples, startup time in seconds: pat_refs old new 1000 0.02 0.01 10000 2.1 0.04 20000 12.3 0.07 30000 27.9 0.10 40000 52.5 0.14 50000 77.5 0.17 Please backport to 1.8, 2.0 and 2.1.
2020-02-27 10:45:50 -05:00
arr = calloc(len, sizeof(*arr));
if (arr == NULL) {
ha_alert("Out of memory error.\n");
return ERR_ALERT | ERR_FATAL;
MINOR: pattern: Each pattern is identified by unique id. The pattern reference are stored with two identifiers: the unique_id and the reference. The reference identify a file. Each file with the same name point to the same reference. We can register many times one file. If the file is modified, all his dependencies are also modified. The reference can be used with map or acl. The unique_id identify inline acl. The unique id is unique for each acl. You cannot force the same id in the configuration file, because this repport an error. The format of the acl and map listing through the "socket" has changed for displaying these new ids.
2014-03-11 09:29:22 -04:00
}
OPTIM: startup: fast unique_id allocation for acl. pattern_finalize_config() uses an inefficient algorithm which is a problem with very large configuration files. This affects startup, and therefore reload time. When haproxy is deployed as a router in a Kubernetes cluster the generated configuration file may be large and reloads are frequently occuring, which makes this a significant issue. The old algorithm is O(n^2) * allocate missing uids - O(n^2) * sort linked list - O(n^2) The new algorithm is O(n log n): * find the user allocated uids - O(n) * store them for efficient lookup - O(n log n) * allocate missing uids - n times O(log n) * sort all uids - O(n log n) * convert back to linked list - O(n) Performance examples, startup time in seconds: pat_refs old new 1000 0.02 0.01 10000 2.1 0.04 20000 12.3 0.07 30000 27.9 0.10 40000 52.5 0.14 50000 77.5 0.17 Please backport to 1.8, 2.0 and 2.1.
2020-02-27 10:45:50 -05:00
i = 0;
j = unassigned_pos;
list_for_each_entry(ref, &pattern_reference, list) {
if (ref->unique_id != -1)
arr[i++] = ref;
else
arr[j++] = ref;
}
/* Sort first segment of array with user-defined unique ids for
* fast lookup when generating unique ids
*/
qsort(arr, unassigned_pos, sizeof(*arr), cmp_pat_ref);
/* Assign unique ids to the rest of the elements */
for (i = unassigned_pos; i < len; i++) {
do {
arr[i]->unique_id = next_unique_id++;
} while (bsearch(&arr[i], arr, unassigned_pos, sizeof(*arr), cmp_pat_ref));
MINOR: pattern: Each pattern is identified by unique id. The pattern reference are stored with two identifiers: the unique_id and the reference. The reference identify a file. Each file with the same name point to the same reference. We can register many times one file. If the file is modified, all his dependencies are also modified. The reference can be used with map or acl. The unique_id identify inline acl. The unique id is unique for each acl. You cannot force the same id in the configuration file, because this repport an error. The format of the acl and map listing through the "socket" has changed for displaying these new ids.
2014-03-11 09:29:22 -04:00
}
OPTIM: startup: fast unique_id allocation for acl. pattern_finalize_config() uses an inefficient algorithm which is a problem with very large configuration files. This affects startup, and therefore reload time. When haproxy is deployed as a router in a Kubernetes cluster the generated configuration file may be large and reloads are frequently occuring, which makes this a significant issue. The old algorithm is O(n^2) * allocate missing uids - O(n^2) * sort linked list - O(n^2) The new algorithm is O(n log n): * find the user allocated uids - O(n) * store them for efficient lookup - O(n log n) * allocate missing uids - n times O(log n) * sort all uids - O(n log n) * convert back to linked list - O(n) Performance examples, startup time in seconds: pat_refs old new 1000 0.02 0.01 10000 2.1 0.04 20000 12.3 0.07 30000 27.9 0.10 40000 52.5 0.14 50000 77.5 0.17 Please backport to 1.8, 2.0 and 2.1.
2020-02-27 10:45:50 -05:00
/* Sort complete array */
qsort(arr, len, sizeof(*arr), cmp_pat_ref);
/* Convert back to linked list */
for (i = 0; i < len; i++)
CLEANUP: lists/tree-wide: rename some list operations to avoid some confusion The current "ADD" vs "ADDQ" is confusing because when thinking in terms of appending at the end of a list, "ADD" naturally comes to mind, but here it does the opposite, it inserts. Several times already it's been incorrectly used where ADDQ was expected, the latest of which was a fortunate accident explained in 6fa922562 ("CLEANUP: stream: explain why we queue the stream at the head of the server list"). Let's use more explicit (but slightly longer) names now: LIST_ADD -> LIST_INSERT LIST_ADDQ -> LIST_APPEND LIST_ADDED -> LIST_INLIST LIST_DEL -> LIST_DELETE The same is true for MT_LISTs, including their "TRY" variant. LIST_DEL_INIT keeps its short name to encourage to use it instead of the lazier LIST_DELETE which is often less safe. The change is large (~674 non-comment entries) but is mechanical enough to remain safe. No permutation was performed, so any out-of-tree code can easily map older names to new ones. The list doc was updated.
2021-04-21 01:32:39 -04:00
LIST_APPEND(&pr, &arr[i]->list);
OPTIM: startup: fast unique_id allocation for acl. pattern_finalize_config() uses an inefficient algorithm which is a problem with very large configuration files. This affects startup, and therefore reload time. When haproxy is deployed as a router in a Kubernetes cluster the generated configuration file may be large and reloads are frequently occuring, which makes this a significant issue. The old algorithm is O(n^2) * allocate missing uids - O(n^2) * sort linked list - O(n^2) The new algorithm is O(n log n): * find the user allocated uids - O(n) * store them for efficient lookup - O(n log n) * allocate missing uids - n times O(log n) * sort all uids - O(n log n) * convert back to linked list - O(n) Performance examples, startup time in seconds: pat_refs old new 1000 0.02 0.01 10000 2.1 0.04 20000 12.3 0.07 30000 27.9 0.10 40000 52.5 0.14 50000 77.5 0.17 Please backport to 1.8, 2.0 and 2.1.
2020-02-27 10:45:50 -05:00
MINOR: pattern: Each pattern is identified by unique id. The pattern reference are stored with two identifiers: the unique_id and the reference. The reference identify a file. Each file with the same name point to the same reference. We can register many times one file. If the file is modified, all his dependencies are also modified. The reference can be used with map or acl. The unique_id identify inline acl. The unique id is unique for each acl. You cannot force the same id in the configuration file, because this repport an error. The format of the acl and map listing through the "socket" has changed for displaying these new ids.
2014-03-11 09:29:22 -04:00
/* swap root */
CLEANUP: lists/tree-wide: rename some list operations to avoid some confusion The current "ADD" vs "ADDQ" is confusing because when thinking in terms of appending at the end of a list, "ADD" naturally comes to mind, but here it does the opposite, it inserts. Several times already it's been incorrectly used where ADDQ was expected, the latest of which was a fortunate accident explained in 6fa922562 ("CLEANUP: stream: explain why we queue the stream at the head of the server list"). Let's use more explicit (but slightly longer) names now: LIST_ADD -> LIST_INSERT LIST_ADDQ -> LIST_APPEND LIST_ADDED -> LIST_INLIST LIST_DEL -> LIST_DELETE The same is true for MT_LISTs, including their "TRY" variant. LIST_DEL_INIT keeps its short name to encourage to use it instead of the lazier LIST_DELETE which is often less safe. The change is large (~674 non-comment entries) but is mechanical enough to remain safe. No permutation was performed, so any out-of-tree code can easily map older names to new ones. The list doc was updated.
2021-04-21 01:32:39 -04:00
LIST_INSERT(&pr, &pattern_reference);
LIST_DELETE(&pr);
OPTIM: startup: fast unique_id allocation for acl. pattern_finalize_config() uses an inefficient algorithm which is a problem with very large configuration files. This affects startup, and therefore reload time. When haproxy is deployed as a router in a Kubernetes cluster the generated configuration file may be large and reloads are frequently occuring, which makes this a significant issue. The old algorithm is O(n^2) * allocate missing uids - O(n^2) * sort linked list - O(n^2) The new algorithm is O(n log n): * find the user allocated uids - O(n) * store them for efficient lookup - O(n log n) * allocate missing uids - n times O(log n) * sort all uids - O(n log n) * convert back to linked list - O(n) Performance examples, startup time in seconds: pat_refs old new 1000 0.02 0.01 10000 2.1 0.04 20000 12.3 0.07 30000 27.9 0.10 40000 52.5 0.14 50000 77.5 0.17 Please backport to 1.8, 2.0 and 2.1.
2020-02-27 10:45:50 -05:00
free(arr);
return 0;
MINOR: pattern: Each pattern is identified by unique id. The pattern reference are stored with two identifiers: the unique_id and the reference. The reference identify a file. Each file with the same name point to the same reference. We can register many times one file. If the file is modified, all his dependencies are also modified. The reference can be used with map or acl. The unique_id identify inline acl. The unique id is unique for each acl. You cannot force the same id in the configuration file, because this repport an error. The format of the acl and map listing through the "socket" has changed for displaying these new ids.
2014-03-11 09:29:22 -04:00
}
BUG/MEDIUM: pattern: make the pattern LRU cache thread-local and lockless As reported in issue #335, a lot of contention happens on the PATLRU lock when performing expensive regex lookups. This is absurd since the purpose of the LRU cache was to have a fast cache for expressions, thus the cache must not be shared between threads and must remain lockless. This commit makes the LRU cache thread-local and gets rid of the PATLRU lock. A test with 7 threads on 4 cores climbed from 67kH/s to 369kH/s, or a scalability factor of 5.5. Given the huge performance difference and the regression caused to users migrating from processes to threads, this should be backported at least to 2.0. Thanks to Brian Diekelman for his detailed report about this regression.
2019-10-23 00:59:31 -04:00
static int pattern_per_thread_lru_alloc()
{
if (!global.tune.pattern_cache)
return 1;
pat_lru_tree = lru64_new(global.tune.pattern_cache);
return !!pat_lru_tree;
}
static void pattern_per_thread_lru_free()
{
lru64_destroy(pat_lru_tree);
}
REGISTER_PER_THREAD_ALLOC(pattern_per_thread_lru_alloc);
REGISTER_PER_THREAD_FREE(pattern_per_thread_lru_free);
Reference in a new issue Copy permalink