upstream/icingadb
1
0
Fork 0
mirror of https://github.com/Icinga/icingadb.git synced 2026-05-28 04:35:54 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions 1
1972 commits 29 branches 11 tags 14 MiB
Commit graph

242 commits

Author SHA1 Message Date
Alexander A. Klimov
2c23b145b7 cmd/ido2icingadb: resume migration progress bar where interrupted 2022-10-11 12:45:36 +02:00
Alexander A. Klimov
bc5ad21a6d cmd/ido2icingadb: simplify computeProgress() a lot 2022-10-11 12:45:36 +02:00
Alexander A. Klimov
6e7e217b3c cmd/ido2icingadb: sliceIdoHistory(): take named args 2022-10-11 12:45:36 +02:00
Alexander A. Klimov
068052f75b cmd/ido2icingadb: migrate(): don't REPLACE, but upsert (one by one) 2022-10-11 12:45:36 +02:00
Alexander A. Klimov
77fa207eb5 cmd/ido2icingadb: simplify unnecessarily complex datatype 2022-10-11 12:45:36 +02:00
Alexander A. Klimov
4fe3134bdd cmd/ido2icingadb: document misc.go 2022-10-11 12:45:36 +02:00
Alexander A. Klimov
662744ee48 cmd/ido2icingadb: migrate(): UPDATE after REPLACE 
... not to loose data.
 2022-10-11 12:45:36 +02:00
Alexander A. Klimov
7e0d3d59a5 cmd/ido2icingadb: document cache.go 2022-10-11 12:45:33 +02:00
Alexander A. Klimov
3ca0b493af cmd/ido2icingadb: document main.go 2022-10-11 12:44:24 +02:00
Alexander A. Klimov
f63892ea01 Revert "Outsource cmd/icingadb.Exit* to cmd/internal" 
This reverts commit fb02ea67ad17af35e72434e97b3e611434405392.
 2022-10-11 12:44:24 +02:00
Alexander A. Klimov
f4a2741f00 cmd/ido2icingadb: remove redundant function 2022-10-11 12:44:24 +02:00
Alexander A. Klimov
7d2ea6c8b5 cmd/ido2icingadb: don't re-invent sqlx.In() 2022-10-11 12:44:24 +02:00
Alexander A. Klimov
31e7c38404 cmd/ido2icingadb: types: add keys and delete zero values 2022-10-11 12:44:24 +02:00
Alexander A. Klimov
21115334c4 cmd/ido2icingadb: document sliceIdoHistory() 2022-10-11 12:44:24 +02:00
Alexander A. Klimov
87a6f8ede7 cmd/ido2icingadb: shorten SugaredLogger#With().Info() to #Infow() 2022-10-11 12:44:24 +02:00
Alexander A. Klimov
d9e2c7daf4 cmd/ido2icingadb: reason not actually used INNER JOINs 2022-10-11 12:44:24 +02:00
Alexander A. Klimov
ae57d4fe6f cmd/ido2icingadb: reduce IDE warnings 2022-10-11 12:44:24 +02:00
Alexander A. Klimov
517fdcb475 cmd/ido2icingadb: correct log levels 2022-10-11 12:44:24 +02:00
Alexander A. Klimov
5396afe32a cmd/ido2icingadb: actually migrate 2022-10-11 12:44:24 +02:00
Alexander A. Klimov
6b583242eb cmd/ido2icingadb: chunk all large queries 2022-10-11 12:44:24 +02:00
Alexander A. Klimov
6ae22d5fec cmd/ido2icingadb: fill cache 2022-10-11 12:44:20 +02:00
Alexander A. Klimov
0704750f85 cmd/ido2icingadb: generalize cache schemata 2022-10-11 10:17:47 +02:00
Alexander A. Klimov
63ab8d938e cmd/ido2icingadb: support SELECT xh.foo_bar->FooBar int 2022-10-11 10:17:47 +02:00
Alexander A. Klimov
423c948d30 cmd/ido2icingadb: introduce barIncrementer 2022-10-11 10:17:47 +02:00
Alexander A. Klimov
d9d8fb7576 cmd/ido2icingadb: make bulk a const 2022-10-11 10:17:47 +02:00
Alexander A. Klimov
4de76da239 cmd/ido2icingadb: introduce historyType#setupBar() 2022-10-11 10:17:47 +02:00
Alexander A. Klimov
d9381ec2fa cmd/ido2icingadb: separate cache schema 2022-10-11 10:17:47 +02:00
Alexander A. Klimov
b005b4c08b cmd/ido2icingadb: make log a global var 2022-10-11 10:17:47 +02:00
Alexander A. Klimov
1d111b7dd3 cmd/ido2icingadb: prepare cache 2022-10-11 10:17:47 +02:00
Alexander A. Klimov
1141058b9c cmd/ido2icingadb: split main() 2022-10-11 10:17:47 +02:00
Alexander A. Klimov
495e0afe32 cmd/ido2icingadb: deduplicate parallelism 2022-10-11 10:17:47 +02:00
Alexander A. Klimov
2f5d5ea931 cmd/ido2icingadb: compute previous progress 2022-10-11 10:17:47 +02:00
Alexander A. Klimov
4eb2ba1ea4 cmd/ido2icingadb: count total IDO events 2022-10-11 10:17:47 +02:00
Alexander A. Klimov
890b66ddb9 cmd/ido2icingadb: connect to databases 2022-10-11 10:17:47 +02:00
Alexander A. Klimov
dce3889645 cmd/ido2icingadb: parse config file 2022-10-11 10:17:47 +02:00
Alexander A. Klimov
700f88a2ff cmd/ido2icingadb: parse CLI flags 2022-10-11 10:17:47 +02:00
Alexander A. Klimov
3c1a83e222 Outsource cmd/icingadb.Exit* to cmd/internal 2022-10-11 10:17:47 +02:00
Eric Lippmann
cd96f0de6f Block XREADs for a maxium of one second 
I just had the observation that blocking XREADs without timeouts (BLOCK
0) on multiple consecutive Redis restarts and I/O timeouts exceeds Redis
internal retries and eventually leads to fatal errors. @julianbrost
looked at this for clarification, here is his finding:

go-redis only considers a command successful when it returned something,
so a successfully started blocking XREAD consumes a retry attempt each
time the underlying Redis connection is terminated. If this happens
often before any element appears in the stream, this error is
propagated. (This also means that even with this PR, when restarting
Redis often enough so that a query never reaches the BLOCK 1sec, this
would still happen.)

https://github.com/Icinga/icingadb/pull/504#issuecomment-1164589244
 2022-06-28 16:09:29 +02:00
Eric Lippmann
5f29caecbe
Merge pull request #464 from Icinga/OwnHeartbeat 
Write own status into Redis
 2022-06-28 15:21:03 +02:00
Alexander A. Klimov
e1ff704aff Write own heartbeat into icingadb:telemetry:heartbeat 
including version, current DB error and HA status quo.
 2022-06-23 18:31:45 +02:00
Alexander A. Klimov
d85d070d1f Clear icinga:runtime* and read from 0-0 later 
instead of preserving the (never read) data and reading beyond its end later.
This indicates the correct number of pending runtime updates
(for monitoring by Icinga 2) from the beginning.
 2022-06-23 18:31:45 +02:00
Yonas Habteab
c06a9ed1ca Add host:port log info when trying to connecto to database/redis 2022-06-21 15:42:57 +02:00
Yonas Habteab
ad575ad552 Make Database/Redis schema mismatch error more user friendly 2022-06-21 15:42:57 +02:00
Alexander A. Klimov
fac9f5e4e5 Write ops/s by op and s to icingadb:telemetry:stats 2022-06-15 09:51:59 +02:00
Yonas Habteab
79003429ab Bump expected redis schema version to 5 2022-06-07 12:57:57 +02:00
Julian Brost
19170ecbcf SLA reporting: history retention for SLA tables 2022-05-13 12:04:47 +02:00
Ravi Kumar Kempapura Srinivasa
2c09dd9e1c Start history retention after config and state sync 
Co-authored-by: Eric Lippmann <eric.lippmann@icinga.com>
 2022-03-22 15:55:04 +01:00
Alexander A. Klimov
d898681d3c Wrap "icingadb-*sql" in constants 
refs #136
 2022-03-10 17:07:06 +01:00
Alexander A. Klimov
c3b8f80abf Decouple *SQL schema versions 
refs #136
 2022-03-10 17:04:22 +01:00
Alexander A. Klimov
3213d18249 Schema: make notification_history#text longtext 2022-01-28 11:51:58 +01:00
Eric Lippmann
81692395b1 Use own connections for heartbeat and HA 
Use dedicated connections for heartbeat and HA to ensure that
heartbeats are always processed and the instance table is updated.
Otherwise, the connections can be too busy due to the
synchronization of configuration, status, history, etc.,
which can lead to handover / takeover loops because the heartbeat
is not read while HA gets stuck when updating the instance table.
 2021-11-12 17:46:54 +01:00
Alexander A. Klimov
fecf332b8e RuntimeUpdates#Sync(): force FIFO for config updates 
so they don't interfer.

Co-authored-by: Eric Lippmann <eric.lippmann@icinga.com>
 2021-11-12 12:35:12 +01:00
Eric Lippmann
9a47f50828 Raise expectedRedisSchemaVersion 2021-11-09 15:11:09 +01:00
Eric Lippmann
ccda48234e Use custom logger for accessing the interval for periodic logging 2021-11-05 17:57:22 +01:00
Eric Lippmann
6232773943 Use debug instead of info for some log messages 
These log messages are not relevant for the info level.
 2021-11-05 17:52:48 +01:00
Eric Lippmann
313e3cc735 Log time taken for the config and initial state sync 2021-11-05 17:52:11 +01:00
Eric Lippmann
986e685ee0 Allow to configure interval for periodic logging 2021-11-05 17:18:05 +01:00
Eric Lippmann
dc7511cd25 Don't log if context is canceled 2021-11-05 17:16:57 +01:00
Julian Brost
74fb238923 Increase Redis schema version 
PR #391 and icinga2#9036 changed where the environment is read from in an
incompatible way, therefore the schema version bump.
 2021-11-03 15:47:38 +01:00
Julian Brost
3342191b5e Exit on environment ID changes 
There's a small risk that when the environment ID changes, Icinga DB could
update write into the wrong environment in the database. Therefore,
https://github.com/Icinga/icinga2/pull/9036 introduced a new default
environment ID based on the CA public key so that there should be no cases
where it's required to change the actual environment ID. So if this happens
nonetheless, just bail out.
 2021-11-03 15:47:38 +01:00
Eric Lippmann
a081927672 Only sync entities that belong to the current environment 
Previously, we selected each entity from the database.
Now we only select entities that belong to the current environment.
 2021-11-03 15:47:38 +01:00
Eric Lippmann
084e0409bc Restart HA after environment change 
If the environment changes during runtime, we have to restart HA
in order to stop a possibly running config sync and start a new
one.
 2021-11-03 14:51:19 +01:00
Eric Lippmann
e74b09daa1 Use synctx instead of ctx 
Otherwise, code that is executed with ctx will not be cancelled if
the synchronization is cancelled, e.g. in the case of heartbeat
handovers.
 2021-11-03 14:51:13 +01:00
Julian Brost
cd9e683faa
Merge pull request #393 from Icinga/feature/history-ha 
Sync history independent of HA
 2021-11-03 14:17:14 +01:00
Julian Brost
f290a0c9b7
Merge pull request #392 from Icinga/feature/history-deterministic-ids 
Make {NotificationHistory,StateHistory,History*}#Id UUID -> SHA1
 2021-11-03 13:18:11 +01:00
Alexander A. Klimov
47d5e80a05 Raise expectedRedisSchemaVersion 1 -> 2 2021-11-03 12:15:25 +01:00
Alexander A. Klimov
660d8c7d5a Sync history independent of HA 2021-10-29 20:32:04 +02:00
Eric Lippmann
e433aa7ec3 Move custom var sync to a new method 2021-10-26 09:31:41 +02:00
Eric Lippmann
5b18a7dd93 Respect context in custom var sync 2021-10-26 09:31:41 +02:00
Eric Lippmann
b48792cf36 Handle flat custom vars explicitly in runtime updates 
This also requires explicit handling of custom variables as we need
to multiplex the original values to handle flat custom variables.
 2021-10-26 09:31:41 +02:00
Eric Lippmann
4d65c62f77 Handle contracts.Initer in common.NewSyncSubject() 
contracts.EntitiyFactoryFunc.WithInit() checked for
contracts.Initer every time.
Now it is only done once in common.NewSyncSubject().
 2021-10-26 09:31:36 +02:00
Eric Lippmann
cacc710ff6 Use ExpandCustomvars() and synctx instead of ctx 2021-10-26 09:27:00 +02:00
Alexander Aleksandrovič Klimov
d9c626ced8
Merge pull request #383 from Icinga/feature/icingadb-schema-version 
Only do anything while icinga:schema version meets our expectations
 2021-10-25 15:49:58 +02:00
Alexander A. Klimov
b5e024e68d Sync state runtime updates ASAP 
I.e. don't wait for the complete initial sync first.
 2021-10-14 10:15:28 +02:00
Eric Lippmann
c9914a1668 Notify systemd that Icinga DB finished starting up 2021-10-13 09:46:12 +02:00
Eric Lippmann
537a4cf37f Allow to configure the logging output 2021-10-13 09:46:12 +02:00
Eric Lippmann
a9afcea25c Allow systemd-journald (and console) as log outputs 2021-10-13 09:46:12 +02:00
Eric Lippmann
9e49b62c4d Use the app name as the default logger name 2021-10-13 09:46:12 +02:00
Eric Lippmann
bdeed69337 Move logging from internal to pkg 2021-10-13 09:20:55 +02:00
Alexander A. Klimov
17dc99b078 Only do anything while icinga:schema version meets our expectations 
... not to work on unknown data structure.
 2021-10-06 12:11:16 +02:00
Ravi Kumar Kempapura Srinivasa
f8e9290618 Use child loggers 2021-09-28 17:30:11 +02:00
Ravi Kumar Kempapura Srinivasa
0aa646c218 Use child loggers for Redis and database 2021-09-28 17:30:11 +02:00
Ravi Kumar Kempapura Srinivasa
d3cdc8023b Use logger from logging pkg 
This now also uses info as the default log level and enables the
configuration of the default log level as well as log levels of
the child loggers that will be implemented in a later commit.
 2021-09-28 17:30:11 +02:00
Alexander A. Klimov
3e452d9561 On shutdown: give up HA handover after 3s, not 5m 
... not to hang for 5m if DB is down.

refs #350
 2021-09-22 15:42:58 +02:00
Alexander A. Klimov
ea7668d99a HA#Close(): allow custom context 
refs #350
 2021-09-22 14:12:27 +02:00
Eric Lippmann
ffd66c5333 Add newline before return 2021-08-09 10:30:53 +02:00
Alexander A. Klimov
f06bad5f23 Don't append(x,y) if x is not to be modified 
... as append(x,y) would modify x.

refs #324
 2021-08-03 18:59:40 +02:00
Eric Lippmann
a936d26c75 Get the last IDs of the runtime update streams before starting anything else 
Otherwise updates may be lost.
 2021-08-03 14:06:55 +02:00
Eric Lippmann
e35a1609fc Stream state updates from icinga:runtime:state 
Icinga now sends runtime updates in two separate channels,
icinga:runtime for config updates and icinga:runtime:state for
state updates. With this change, Icinga DB reads from these two
streams. This is a preparation so that state updates can be
streamed directly after a (re)start of Icinga or Icinga DB without
waiting for the config sync, as it is currently done.
 2021-08-03 14:06:55 +02:00
Alexander A. Klimov
6a17911e6b Assert the database schema of the expected version being present 2021-07-06 19:00:20 +02:00
Eric Lippmann
752db18a91 Print stack traces 2021-06-21 13:20:44 +02:00
Eric Lippmann
dac7cceff7 Use Info() instead of Infof() if no format string 2021-06-21 12:13:24 +02:00
Eric Lippmann
03e899701e Add custom vars to the runtime update sync 
The customvar runtime update sync may change because the customvar
flat runtime update sync is not yet implemented.
 2021-05-31 13:59:13 +02:00
Alexander A. Klimov
64433ec674 Make channel for signal.Notify() buffered 2021-05-28 14:24:36 +02:00
Alexander A. Klimov
621c1b9537 Ensure context cancellation 2021-05-28 14:24:36 +02:00
Noah Hilverling
44c734f72d Improve database and HA logging 2021-05-25 09:49:48 +02:00
Alexander A. Klimov
b31e012cf0 Sync overdue indicators 2021-05-12 18:51:13 +02:00
Eric Lippmann
52e5eb774e Sync customvar flat 2021-05-11 16:40:08 +02:00
Eric Lippmann
442e04bbf1 Use type common.SyncSubject 2021-05-11 16:40:08 +02:00
Eric Lippmann
feda71f06d Merge pull request #49 from lippserd/bugfix/ha-error-handling 
Handle HA errors
 2021-05-07 14:54:33 +02:00
Julian Brost
ae743817da Add comment explaining the purpose of ha.Close() 2021-05-07 14:51:58 +02:00
Julian Brost
e0d66fe0ee Exit with an error code if not exiting due to a signal 
Icinga DB should not exit with code 0 if it does so due to an error.
 2021-05-07 14:51:58 +02:00
Eric Lippmann
a1ceeeb9eb Merge pull request #36 from lippserd/feature/runtime-updates 
Implement runtime updates
 2021-05-07 14:37:56 +02:00
Eric Lippmann
345bcb952f Runtime updates must wait for config sync to complete 2021-05-07 12:04:41 +02:00
Noah Hilverling
91a8228e2d Implement runtime updates 2021-05-07 12:04:41 +02:00
Julian Brost
3649b367e1 Handle HA errors 2021-05-07 11:46:45 +02:00
Alexander A. Klimov
ff6ae994d2 Remove always-false if 2021-05-07 11:06:54 +02:00
Julian Brost
a7e5395f66 Read icinga:dump stream and respect dump signals 2021-05-03 11:19:26 +02:00
Julian Brost
ab24bdd1f7 Main loop: return if ctx has an error 
If ctx.Done(), there's no point in looping again.
 2021-04-28 16:05:56 +02:00
Julian Brost
d138b8e0ac Handle SIGINT/SIGTERM/SIGHUP 
On any of these signals, HA removes the row of the local instance from
icingadb_instance. This allows another instance to take over immediately.
 2021-04-28 12:01:02 +02:00
Eric Lippmann
4fbcd4c90f Move factories away from main and use v1.Factories 2021-04-27 23:36:43 +02:00
Alexander A. Klimov
5e473e8b37 main(): don't panic on (intentional) context cancel 2021-04-26 14:15:10 +02:00
Eric Lippmann
a9c6cdabc9 Remove comment 2021-04-23 11:50:27 +02:00
Eric Lippmann
dfb7ea5dc9 Remove customvar_flat sync draft 2021-04-23 11:34:07 +02:00
Eric Lippmann
0c8a9139b5 Merge pull request #28 from lippserd/bugfix/delta-race 
Delta#start(): avoid a race across maps by using a mutex
 2021-04-19 16:20:42 +02:00
Alexander A. Klimov
fac47fb330 Delta: don't over-lock 2021-04-12 17:41:18 +02:00
Alexander A. Klimov
7ccf627df6 Sync history 2021-04-12 13:28:03 +02:00
Eric Lippmann
63d8bf8e24 Merge pull request #15 from lippserd/feature/timeperiod 
Sync timeperiods as well
 2021-03-30 10:12:33 +02:00
Eric Lippmann
9525ef476e Merge pull request #13 from lippserd/feature/notification 
Sync notifications as well
 2021-03-23 10:09:06 +01:00
Eric Lippmann
b9673e1a94 Merge pull request #11 from lippserd/feature/comment 
Sync comments as well
 2021-03-23 10:08:01 +01:00
Alexander A. Klimov
fd2e068de6 Sync timeperiods as well 2021-03-22 14:07:52 +01:00
Alexander A. Klimov
7ab45082ad Sync notifications as well 2021-03-22 14:06:51 +01:00
Alexander A. Klimov
3ec88ac53d Sync comments as well 2021-03-22 14:04:58 +01:00
Alexander A. Klimov
b90bc86bb4 Sync zones as well 2021-03-22 14:03:46 +01:00
Eric Lippmann
49d50a779e Merge pull request #12 from lippserd/feature/downtime 
Sync downtimes as well
 2021-03-22 13:37:35 +01:00
Eric Lippmann
dd1fad4b13 Merge pull request #10 from lippserd/feature/command 
Sync commands as well
 2021-03-22 13:36:46 +01:00
Eric Lippmann
9f936eba95 Merge pull request #7 from lippserd/feature/groups 
Sync groups as well
 2021-03-22 13:36:12 +01:00
Alexander A. Klimov
13c672ce6e Sync downtimes as well 2021-03-18 13:56:07 +01:00
Alexander A. Klimov
a56d312c81 Sync commands as well 2021-03-18 13:51:33 +01:00
Eric Lippmann
e4539d90fc Merge pull request #8 from lippserd/feature/endpoint 
Sync endpoints as well
 2021-03-18 13:42:20 +01:00
Alexander A. Klimov
71d1b4a9f4 Sync endpoints as well 2021-03-18 12:32:09 +01:00
Alexander A. Klimov
055e63e272 Sync groups as well 2021-03-18 12:31:11 +01:00
Alexander A. Klimov
0dd5e54b14 Sync users as well 2021-03-18 12:19:23 +01:00
Alexander A. Klimov
3a66f0372c Introduce CustomvarMeta 2021-03-18 12:19:20 +01:00
Eric Lippmann
c2fe0dfe7c Merge pull request #16 from lippserd/feature/channels 
Make channels more specific
 2021-03-17 10:03:24 +01:00
Eric Lippmann
f9648b00c5 Merge pull request #14 from lippserd/feature/url 
Sync URLs as well
 2021-03-17 10:02:20 +01:00
Eric Lippmann
c5f1b88c0c Merge pull request #4 from lippserd/feature/initer 
Make NameCiMeta an Initer
 2021-03-17 09:56:10 +01:00
Alexander A. Klimov
4dffbad76e Make channels more specific 2021-03-15 16:34:58 +01:00
Alexander A. Klimov
aa3dfc2afc Sync URLs as well 2021-03-12 16:44:19 +01:00
Alexander A. Klimov
8cd62118b2 Auto-init Initers 2021-03-12 10:55:12 +01:00
Julian Brost
a474c83651 Replace persisted instance ID with a random one and remove old rows from icingadb_instance 2021-03-08 13:01:52 +01:00
Eric Lippmann
6ca9d36c8b Add cmd 2021-03-04 00:50:24 +01:00