Commit graph

415 commits

Author SHA1 Message Date
Julian Brost
61c6c7f110
Merge pull request #10619 from Icinga/efficient-config-and-state-update-queue
IcingaDB: better config and state update queueing
2026-04-17 11:19:50 +02:00
Yonas Habteab
bfb0e7db12 Inline SendNextUpdate & remove superfluous m_RconWorker checks 2026-04-16 08:53:58 +02:00
Yonas Habteab
99328ec417 Log pending items stats regularly & include them as perfdata in IcingaDB check 2026-04-15 17:33:43 +02:00
Julian Brost
267675e80b RedisConnection: simplify GetOldestPendingQueryTs function 2026-04-02 16:37:57 +02:00
Yonas Habteab
7d7159c033 Reduce min queue item age from 1000ms to 300ms 2026-04-02 16:37:57 +02:00
Yonas Habteab
390ee8c02f Inline DequeueAndProcessOne & don't process items out of order
Now, the individual `ProcessQueueItem` functions decide whether to
acquire an `olock` or not instead of probing this from within the
worker loop. This is way easier than having to deal with the potential
out of order processing of items in the queue in both ways, i.e., we
don't want to send delete events for objects while their created events
haven't been processed yet and vice versa.
2026-04-02 16:37:57 +02:00
Julian Brost
855f6c7c0c IcingaDB: use key extractor for worker queue
This commit restructures the queue items so that each one now has a method
`GetQueueLookupKey()` that is used to derive which elements of the queue are
considered to be equal. For this, there is a key extractor for the
`multi_index_container` that takes the `variant` from the queue item, calls
that method on it, and puts the result in a second variant type. The types in
that variant type are automatically deduced from the return types of the
individual methods.
2026-04-02 16:37:57 +02:00
Yonas Habteab
2048450159 IcingaDB: put all queue related stuff into icingadb:task_queue namespace 2026-04-02 16:37:57 +02:00
Julian Brost
8375934d19 Simplify IcingaDB::PendingItemsThreadProc() event loop 2026-04-02 16:37:57 +02:00
Yonas Habteab
b633d6b0d0 IcingaDB: remove unused UpdateObjectAttrs method 2026-04-02 16:37:57 +02:00
Yonas Habteab
89d8c326e6 Fix missing olock for dependency child registration
Previously, the checkable was locked while processing all the dependency
registration stuff, so the worker thread should also do the same to
avoid any potential race conditions.
2026-04-02 16:37:57 +02:00
Julian Brost
9d5883df78 IcingaDB: use polymorphism for queue entries 2026-04-02 16:37:57 +02:00
Yonas Habteab
e88366ddae IcingaDB: subscribe to OnNextCheckChanged signal
We can't drop the `OnNextCheckUpdated` signal entirely yet, as IDO still
relies on it.
2026-04-02 16:37:57 +02:00
Yonas Habteab
bbb7d0249e RedisConnection: enhance WriteQueueItem & related usages 2026-04-02 16:37:57 +02:00
Yonas Habteab
cbb4147055 RedisConnection: simplify query prioritization logic
As opposed to the previous version which used a complex data structure
to correctly manage the query priorities, this version uses two separate
queues for the high and normal priority writes. All high priority writes
are processed in FIFO order but over take all queries from the normal
priority queue. The later queue only be processed when the high priority
queue is empty.
2026-04-02 16:37:57 +02:00
Yonas Habteab
adbefa5540 Revert "IcingaDB: suppress state sync until config sync finished"
This reverts commit f6f7d9b635 and all
other its new users.
2026-04-02 14:06:28 +02:00
Yonas Habteab
d364ad981e IcingaDB: enqueue config runtime updates to the worker queue 2026-04-02 14:06:28 +02:00
Yonas Habteab
374cc6e282 Cache Icinga DB env_id in Application class as well
So that other components can use it without having to import any Icinga
DB related header files, but only the base library.
2026-04-01 12:15:58 +02:00
Julian Brost
4f13651cb0
Merge pull request #10727 from Icinga/icingadb-missing-exception-messages
Redis exceptions: add proper what() messages
2026-03-31 10:27:33 +02:00
Julian Brost
207764584a Redis exceptions: remove inline specifiers
Remove them as they are redundant, as requested in the PR review.
2026-03-27 17:02:05 +01:00
Julian Brost
221382486e Redis exceptions: use BOOST_THROW_EXCEPTION
Use it for consistency, as requested in the PR review.
2026-03-27 17:01:51 +01:00
Julian Brost
862f012381 Redis exceptions: add proper what() messages
RedisDisconnected::what() and RedisProtocolError::what() always returned an
empty string. Similarly, BadRedisType::what() and BadRedisInt::what() only
return the value that couldn't be parsed without any information about the
exception type. If only what() is used when printing the exception, as it's
typical, this results in unhelpful log messages like the following when simply
stopping the Redis server:

    [2026-02-23 14:33:33 +0100] critical/IcingaDB: Error during receiving the response to a query which has been fired and forgotten: Connection reset by peer [system:104 at /usr/include/boost/asio/detail/reactive_socket_recv_op.hpp:134 in function 'do_complete']
    [2026-02-23 14:33:33 +0100] critical/IcingaDB: Error during receiving the response to a query which has been fired and forgotten:
    [2026-02-23 14:33:33 +0100] critical/IcingaDB: Error during receiving the response to a query which has been fired and forgotten:
    [2026-02-23 14:33:33 +0100] critical/IcingaDB: Cannot connect to redis-1:6379: Connection refused [system:111 at /usr/include/boost/asio/detail/reactive_socket_connect_op.hpp:98 in function 'do_complete']

This commit changes these messages so that something like "Redis disconnected",
"Redis protocol error: bad int: foo", or "Redis protocol error: bad type: ?" is
returned. In doing so, it also removes a member of type std::vector<char> in
BadRedisInt as this is unsafe to use in exceptions (it violates the requirement
that copy constructor and assignment must be nothrow, see
https://en.cppreference.com/w/cpp/error/exception.html#Standard_exception_requirements).

With this commit, the log messages are now a bit more helpful:

    [2026-02-23 15:08:23 +0100] critical/IcingaDB: Error during receiving the response to a query which has been fired and forgotten: Connection reset by peer [system:104 at /usr/include/boost/asio/detail/reactive_socket_recv_op.hpp:134 in function 'do_complete']
    [2026-02-23 15:08:23 +0100] critical/IcingaDB: Error during receiving the response to a query which has been fired and forgotten: Redis disconnected
    [2026-02-23 15:08:23 +0100] critical/IcingaDB: Error during receiving the response to a query which has been fired and forgotten: Redis disconnected
    [2026-02-23 15:08:23 +0100] critical/IcingaDB: Cannot connect to redis-1:6379: Connection refused [system:111 at /usr/include/boost/asio/detail/reactive_socket_connect_op.hpp:98 in function 'do_complete']
2026-03-27 17:01:49 +01:00
Yonas Habteab
26d23bd569 IcingaDB: enqueue all dependency related updates to the worker queue 2026-03-23 11:26:48 +01:00
Yonas Habteab
e6dc594313 IcingaDB: don't send nextupdate:{host,service} as part of the initial sync
Don't send nextupdate as part of the intial sync. Instead, enque them to
the background worker to be sent after the initial dump is done.
2026-03-23 11:26:48 +01:00
Yonas Habteab
eab507f892 IcingaDB: track object changesets efficiently 2026-03-23 11:26:48 +01:00
Yonas Habteab
105c066649 IcingaDB: use a separate Redis conn for history & heartbeat streams 2026-03-23 11:26:47 +01:00
Yonas Habteab
fa514e6336 IcingaDB: use a statically initialized list of syncable types for Redis keys 2026-03-16 08:39:12 +01:00
Yonas Habteab
cb872ee5c8 IcingaDB: consistently use const char* literals everywhere
Co-Authored-By: Julian Brost <julian.brost@icinga.com>
2026-03-13 16:14:12 +01:00
Yonas Habteab
f593330735 IcingaDB: change const T to const T& to avoid stupid String copying 2026-03-13 16:14:12 +01:00
Alexander Aleksandrovič Klimov
55d35b19dd
Merge pull request #10391 from Icinga/redis-query-variant
Some checks failed
Container Image / Container Image (push) Has been cancelled
Linux / alpine:bash (push) Has been cancelled
Linux / amazonlinux:2 (push) Has been cancelled
Linux / amazonlinux:2023 (push) Has been cancelled
Linux / debian:11 (linux/386) (push) Has been cancelled
Linux / debian:11 (push) Has been cancelled
Linux / debian:12 (linux/386) (push) Has been cancelled
Linux / debian:12 (push) Has been cancelled
Linux / debian:13 (push) Has been cancelled
Linux / fedora:41 (push) Has been cancelled
Linux / fedora:42 (push) Has been cancelled
Linux / fedora:43 (push) Has been cancelled
Linux / opensuse/leap:15.6 (push) Has been cancelled
Linux / opensuse/leap:16.0 (push) Has been cancelled
Linux / registry.suse.com/bci/bci-base:16.0 (push) Has been cancelled
Linux / registry.suse.com/suse/sle15:15.6 (push) Has been cancelled
Linux / registry.suse.com/suse/sle15:15.7 (push) Has been cancelled
Linux / rockylinux/rockylinux:10 (push) Has been cancelled
Linux / rockylinux:8 (push) Has been cancelled
Linux / rockylinux:9 (push) Has been cancelled
Linux / ubuntu:22.04 (push) Has been cancelled
Linux / ubuntu:24.04 (push) Has been cancelled
Linux / ubuntu:25.04 (push) Has been cancelled
Linux / ubuntu:25.10 (push) Has been cancelled
Windows / Windows (push) Has been cancelled
Change Redis query type from `String` to an optional `std::string_view`
2026-03-04 10:38:12 +01:00
Alexander A. Klimov
1702eebd41 Change RedisConnection::Query::value_type to a string/view variant
Especially our history messages contain lots of hardcoded C string literals
`"like this one"`. At runtime, they get translated to pointers to constant
global memory, `const char*`. String `malloc(3)`s and copies these data every
time. In contrast, the new type just stores the address if any. (Actually,
`const char*` is wrapped by `std::string_view` to not compute its length every
time.)
2026-03-03 11:29:07 +01:00
Yonas Habteab
dd5289bf4e RedisConnection: drop superfluous GetConnected method 2026-03-02 15:36:59 +01:00
Yonas Habteab
e0c239161b Allow to track query stats in child Redis connections
So that when we want the query stats of this specific connection we can
easily get them, since the session leader contains the aggregated stats
of all its children.
2026-03-02 15:36:58 +01:00
Yonas Habteab
4d4650b86a RedisConnection: introduce & use RedisConnInfo helper struct 2026-03-02 15:34:04 +01:00
Alexander A. Klimov
c351f6a88c Use RedisConnection::Quer* type aliases
This expresses what kind of vector it is
and allows to easily change those types in the future.
2026-03-02 12:45:09 +01:00
Alexander A. Klimov
ba1bb0e758 Introduce RedisConnection::QueryArg, an alias of String
to ease changing that type in the future.
2026-03-02 12:45:09 +01:00
Alexander A. Klimov
808a6c77e8 IcingaDB#AddObjectDataToRuntimeUpdates(): take String by value
It was copied anyway.
2026-03-02 12:45:09 +01:00
Johannes Schmidt
e976450bbb Fix undefined behavior introduced by initializing last states with 99 2026-02-12 16:00:36 +01:00
Yonas Habteab
91c7e60df8 Replace all existing copyright headers with SPDX headers
I've used the following command to replace the original copyright header
lines in a C-style comment block:

```
$ find . \( -type d \( -name '\..*' -o -name third-party -o -name scripts -o -name prefix -o -name malloc -o -name server -o -name docker -o -name build -o -name doc \) -prune \) -o -type f -exec perl -pi -e 's{/\*[^*]*\(\s*c\s*\)\s*(\d{4})\s*Icinga\s+GmbH[^*]*\*/}{// SPDX-FileCopyrightText: \1 Icinga GmbH <https://icinga.com>\n// SPDX-License-Identifier: GPL-2.0-or-later}gi' {} +
```

For files that use shell-style comments (#) like CMakeLists.txt, I've
used this command:

```
$ find . \( -type d \( -name '\..*' -o -name third-party -o -name scripts -o -name prefix -o -name malloc -o -name server -o -name docker -o -name build -o -name doc \) -prune \) -o -type f -exec perl -pi -e 's{#.*\(\s*c\s*\)\s(\d{4})\sIcinga\s+GmbH.*}{# SPDX-FileCopyrightText: \1 Icinga GmbH <https://icinga.com>\n# SPDX-License-Identifier: GPL-2.0-or-later}gi' {} +
```

And for SQL files:

```
$ find . \( -type d \( -name '\..*' -o -name third-party -o -name scripts -o -name prefix -o -name malloc -o -name server -o -name docker -o -name build -o -name doc \) -prune \) -o -type f \( -name '*.sql' \) -exec perl -pi -e 's{--.*\(c\)\s(\d{4})\sIcinga\sGmbH.*}{-- SPDX-FileCopyrightText: \1 Icinga GmbH <https://icinga.com>\n-- SPDX-License-Identifier: GPL-2.0-or-later}gi' {} +
$ find . \( -type d \( -name '\..*' -o -name third-party -o -name scripts -o -name prefix -o -name malloc -o -name server -o -name docker -o -name build -o -name doc \) -prune \) -o -type f \( -name '*.sql' \) -exec perl -pi -e 's{-- Copyright \(c\)\s(\d{4})\sIcinga\s+Development\sTeam.*}{-- SPDX-FileCopyrightText: \1 Icinga GmbH <https://icinga.com>\n-- SPDX-License-Identifier: GPL-2.0-or-later}gi' {} +
```
2026-02-04 14:00:05 +01:00
Alexander A. Klimov
4ba46f9eb2 Silence compiler warnings about unused parameters
Every of these parameters exists for a reason.
The best we can do is to convince the compiler.
2026-01-23 13:31:01 +01:00
Johannes Schmidt
11e83b6274 Fix -Wunused-lambda-capture warning 2025-11-19 12:00:58 +01:00
Johannes Schmidt
93e69ed0e8 Fix -Wsign-compare warnings 2025-11-19 12:00:58 +01:00
Johannes Schmidt
f42510f981
Merge pull request #9411 from Icinga/compiler-warnings
Some checks failed
Container Image / Container Image (push) Has been cancelled
Linux / alpine:bash (push) Has been cancelled
Linux / amazonlinux:2 (push) Has been cancelled
Linux / amazonlinux:2023 (push) Has been cancelled
Linux / debian:11 (linux/386) (push) Has been cancelled
Linux / debian:11 (push) Has been cancelled
Linux / debian:12 (linux/386) (push) Has been cancelled
Linux / debian:12 (push) Has been cancelled
Linux / debian:13 (push) Has been cancelled
Linux / fedora:41 (push) Has been cancelled
Linux / fedora:42 (push) Has been cancelled
Linux / opensuse/leap:15.6 (push) Has been cancelled
Linux / opensuse/leap:16.0 (push) Has been cancelled
Linux / registry.suse.com/suse/sle15:15.6 (push) Has been cancelled
Linux / registry.suse.com/suse/sle15:15.7 (push) Has been cancelled
Linux / rockylinux/rockylinux:10 (push) Has been cancelled
Linux / rockylinux:8 (push) Has been cancelled
Linux / rockylinux:9 (push) Has been cancelled
Linux / ubuntu:22.04 (push) Has been cancelled
Linux / ubuntu:24.04 (push) Has been cancelled
Linux / ubuntu:25.04 (push) Has been cancelled
Windows / Windows (push) Has been cancelled
Fix compiler warnings
2025-10-21 15:23:31 +02:00
Alexander A. Klimov
d877e818db Fix compiler warnings 2025-10-17 17:08:31 +02:00
Alexander A. Klimov
37b5c39e20 Fix compiler warnings by re-ordering member init in constructors 2025-10-17 15:33:57 +02:00
Alexander A. Klimov
3d69a31043 Fix compiler warnings by removing unused variables 2025-10-17 09:56:46 +02:00
Julian Brost
b27310fb6c
Merge pull request #10467 from Icinga/icingadb-calceventid-no-double-timestamptomilliseconds
IcingaDB::CalcEventID: No milliseconds as eventTime
2025-06-10 17:09:16 +02:00
Alvar Penning
9cdbbf4ede
IcingaDB::CalcEventID: No milliseconds as eventTime
CalcEventID's internal logic uses the TimestampToMilliseconds function
to convert the given eventTime to milliseconds. Within this function,
the timestamp is capped to prevent an overflow.

On three occasions, the input timestamp given to CalcEventID had already
been converted using TimestampToMilliseconds. The second
TimestampToMilliseconds function then checked the value and always
returned the capped maximum value. Consequently, CalcEventID returned
the same hash value for different timestamps.

This affected SendFlappingChange, SendAcknowledgementSet, and
SendAcknowledgementCleared. For example, when two acknowledgments were
created for the same service, the calculated event_id representing the
history table row would be identical.

Fixes #10465
2025-06-06 16:50:01 +02:00
Yonas Habteab
9e65a8b63b Fix compiler warnings of missing NotificationTypeAll case 2025-06-06 13:31:44 +02:00
Yonas Habteab
fd1927115a IcingaDB: Make is_acknowledged a bool & add is_sticky_acknowledgement field 2025-06-06 13:31:44 +02:00