upstream/icingadb
1
0
Fork 0
mirror of https://github.com/Icinga/icingadb.git synced 2026-04-15 14:29:48 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions 1
1971 commits 29 branches 11 tags 14 MiB
Commit graph

242 commits

Author SHA1 Message Date
Alexander A. Klimov
6794252ba1 cmd/ido2icingadb: remove unused code 2022-10-11 12:46:48 +02:00
Alexander A. Klimov
6804bbdb54 cmd/ido2icingadb: increase parallelism 2022-10-11 12:46:48 +02:00
Alexander A. Klimov
23130d7be8 cmd/ido2icingadb: remove unnecessary transactions 
for the sake of speed.
 2022-10-11 12:46:48 +02:00
Alexander A. Klimov
690fcfa190 cmd/ido2icingadb: chunkCacheTx(): commit less often 
for the sake of speed.
 2022-10-11 12:46:48 +02:00
Alexander A. Klimov
aa571f0856 cmd/ido2icingadb: remove unnecessary mutex 
for the sake of speed.
 2022-10-11 12:46:48 +02:00
Alexander A. Klimov
8bf8a6fe75 cmd/ido2icingadb: rename to Icinga DB Migration 2022-10-11 12:46:48 +02:00
Alexander A. Klimov
36d07aa7cd cmd/ido2icingadb: add docs 2022-10-11 12:46:48 +02:00
Alexander A. Klimov
1c381cfef8 cmd/ido2icingadb: make cache filling fast again 
by COMMITing less frequently.
 2022-10-11 12:46:48 +02:00
Alexander A. Klimov
f0ec8b0158 cmd/ido2icingadb: allow to migrate multiple IDO databases 
into one Icinga DB database as separate envs.
 2022-10-11 12:46:48 +02:00
Alexander A. Klimov
59c77eb90a cmd/ido2icingadb: reduce bulk size 2022-10-11 12:46:48 +02:00
Alexander A. Klimov
fcae6759d3 cmd/ido2icingadb: remove unused code 2022-10-11 12:46:48 +02:00
Alexander A. Klimov
34ef6bec44 cmd/ido2icingadb: upsert, not update, to make bulk statements 2022-10-11 12:46:47 +02:00
Alexander A. Klimov
58cfbf46b5 cmd/ido2icingadb: allow converters to upsert 2022-10-11 12:46:47 +02:00
Alexander A. Klimov
cc98c34d18 cmd/ido2icingadb: centralise notification type conversion 2022-10-11 12:46:47 +02:00
Alexander A. Klimov
98bb53b2f7 cmd/ido2icingadb: clean up cache 2022-10-11 12:46:47 +02:00
Alexander A. Klimov
1caca0b8b8 cmd/ido2icingadb: fix duplicate comment 2022-10-11 12:46:47 +02:00
Alexander A. Klimov
74fce9b3de cmd/ido2icingadb: fix missing flapping_history#percent_state_change_end 2022-10-11 12:46:47 +02:00
Alexander A. Klimov
d69ccd13d7 cmd/ido2icingadb: don't unnecessarily overwrite already written/migrated history 2022-10-11 12:46:47 +02:00
Alexander A. Klimov
9601b1685f cmd/ido2icingadb: fix missing ack clearings 2022-10-11 12:46:47 +02:00
Alexander A. Klimov
90bc748aec cmd/ido2icingadb: indicate no notification author as "", not "-" 
It looks nicer in Icinga DB Web.
 2022-10-11 12:46:47 +02:00
Alexander A. Klimov
7896bc5024 cmd/ido2icingadb: fix missing flapping end events 2022-10-11 12:46:47 +02:00
Alexander A. Klimov
f944844fb5 cmd/ido2icingadb: build fix 2022-10-11 12:46:47 +02:00
Alexander A. Klimov
fe24c2d3a0 cmd/ido2icingadb: fix missing config defaults 2022-10-11 12:46:47 +02:00
Alexander A. Klimov
4f871ecd3d cmd/ido2icingadb: bulk, not prepare, upserts 
for the sake of performance.
 2022-10-11 12:46:47 +02:00
Alexander A. Klimov
d9eb3689b8 cmd/ido2icingadb: fix downtime SLA end time 2022-10-11 12:46:47 +02:00
Alexander A. Klimov
752a9090ab cmd/ido2icingadb: improve code docs 2022-10-11 12:46:47 +02:00
Alexander A. Klimov
f40a39415c cmd/ido2icingadb: don't unnecessarily pre-fill ido_migration_progress 2022-10-11 12:46:47 +02:00
Alexander A. Klimov
0e9c93cd27 cmd/ido2icingadb: rename onNewUncommittedDml to commitPeriodically 2022-10-11 12:46:47 +02:00
Alexander A. Klimov
ad0b6cad91 cmd/ido2icingadb: remove obsolete code 2022-10-11 12:46:47 +02:00
Alexander A. Klimov
3cc8641276 cmd/ido2icingadb: let SQLite VACUUM automatically 2022-10-11 12:46:47 +02:00
Alexander A. Klimov
1a1f19163f cmd/ido2icingadb: write SLA 2022-10-11 12:46:47 +02:00
Alexander A. Klimov
041d4a0a74 cmd/ido2icingadb: avoid &(*ht)[i] 2022-10-11 12:46:47 +02:00
Alexander A. Klimov
7c6f9ddbbd cmd/ido2icingadb: improve comments 2022-10-11 12:46:47 +02:00
Alexander A. Klimov
5470c297b1 cmd/ido2icingadb: go:embed large queries 
for the sake of syntax highlighting.
 2022-10-11 12:46:47 +02:00
Alexander A. Klimov
67c7fe9257 cmd/ido2icingadb: go:embed schemata 
for the sake of syntax highlighting.
 2022-10-11 12:46:47 +02:00
Alexander A. Klimov
3b634ec499 cmd/ido2icingadb: build fix 2022-10-11 12:46:47 +02:00
Alexander A. Klimov
7dd911d970 cmd/ido2icingadb: better ETA 2022-10-11 12:46:47 +02:00
Alexander A. Klimov
f522b2923a cmd/ido2icingadb: prefer generics over reflection 2022-10-11 12:46:45 +02:00
Alexander A. Klimov
581270ffee cmd/ido2icingadb: support Postgres 2022-10-11 12:46:20 +02:00
Alexander A. Klimov
d9a3e04537 cmd/ido2icingadb: migrate(): run only one transaction at a time 
to avoid inter-transaction lock timeouts.
 2022-10-11 12:46:20 +02:00
Alexander A. Klimov
bcf2256796 cmd/ido2icingadb: correct mismatching FK ID 2022-10-11 12:46:19 +02:00
Alexander A. Klimov
aa585e9d72 cmd/ido2icingadb: don't hash non-hashable types 2022-10-11 12:46:19 +02:00
Alexander A. Klimov
a2dae38b00 cmd/ido2icingadb: build fix 2022-10-11 12:46:19 +02:00
Alexander A. Klimov
597cd63164 cmd/ido2icingadb: reduce historyType#convertRows call complexity 2022-10-11 12:46:19 +02:00
Alexander A. Klimov
2d2cad1677 cmd/ido2icingadb: sliceIdoHistory(): reduce onRows call complexity 2022-10-11 12:46:18 +02:00
Alexander A. Klimov
086b3d3f07 cmd/ido2icingadb: sliceIdoHistory(): re-use memory 2022-10-11 12:45:36 +02:00
Alexander A. Klimov
163c8629e9 cmd/ido2icingadb: env: "Environment" config constant value -> hex ID 2022-10-11 12:45:36 +02:00
Alexander A. Klimov
4e58631f8e cmd/ido2icingadb: make IDs UUID -> SHA1 2022-10-11 12:45:36 +02:00
Alexander A. Klimov
34b3ee1767 cmd/ido2icingadb: merge countIdoHistory() and computeProgress() 2022-10-11 12:45:36 +02:00
Alexander A. Klimov
b78b15e32c cmd/ido2icingadb: ensure migration progress bar finishes 2022-10-11 12:45:36 +02:00
Alexander A. Klimov
2c23b145b7 cmd/ido2icingadb: resume migration progress bar where interrupted 2022-10-11 12:45:36 +02:00
Alexander A. Klimov
bc5ad21a6d cmd/ido2icingadb: simplify computeProgress() a lot 2022-10-11 12:45:36 +02:00
Alexander A. Klimov
6e7e217b3c cmd/ido2icingadb: sliceIdoHistory(): take named args 2022-10-11 12:45:36 +02:00
Alexander A. Klimov
068052f75b cmd/ido2icingadb: migrate(): don't REPLACE, but upsert (one by one) 2022-10-11 12:45:36 +02:00
Alexander A. Klimov
77fa207eb5 cmd/ido2icingadb: simplify unnecessarily complex datatype 2022-10-11 12:45:36 +02:00
Alexander A. Klimov
4fe3134bdd cmd/ido2icingadb: document misc.go 2022-10-11 12:45:36 +02:00
Alexander A. Klimov
662744ee48 cmd/ido2icingadb: migrate(): UPDATE after REPLACE 
... not to loose data.
 2022-10-11 12:45:36 +02:00
Alexander A. Klimov
7e0d3d59a5 cmd/ido2icingadb: document cache.go 2022-10-11 12:45:33 +02:00
Alexander A. Klimov
3ca0b493af cmd/ido2icingadb: document main.go 2022-10-11 12:44:24 +02:00
Alexander A. Klimov
f63892ea01 Revert "Outsource cmd/icingadb.Exit* to cmd/internal" 
This reverts commit fb02ea67ad17af35e72434e97b3e611434405392.
 2022-10-11 12:44:24 +02:00
Alexander A. Klimov
f4a2741f00 cmd/ido2icingadb: remove redundant function 2022-10-11 12:44:24 +02:00
Alexander A. Klimov
7d2ea6c8b5 cmd/ido2icingadb: don't re-invent sqlx.In() 2022-10-11 12:44:24 +02:00
Alexander A. Klimov
31e7c38404 cmd/ido2icingadb: types: add keys and delete zero values 2022-10-11 12:44:24 +02:00
Alexander A. Klimov
21115334c4 cmd/ido2icingadb: document sliceIdoHistory() 2022-10-11 12:44:24 +02:00
Alexander A. Klimov
87a6f8ede7 cmd/ido2icingadb: shorten SugaredLogger#With().Info() to #Infow() 2022-10-11 12:44:24 +02:00
Alexander A. Klimov
d9e2c7daf4 cmd/ido2icingadb: reason not actually used INNER JOINs 2022-10-11 12:44:24 +02:00
Alexander A. Klimov
ae57d4fe6f cmd/ido2icingadb: reduce IDE warnings 2022-10-11 12:44:24 +02:00
Alexander A. Klimov
517fdcb475 cmd/ido2icingadb: correct log levels 2022-10-11 12:44:24 +02:00
Alexander A. Klimov
5396afe32a cmd/ido2icingadb: actually migrate 2022-10-11 12:44:24 +02:00
Alexander A. Klimov
6b583242eb cmd/ido2icingadb: chunk all large queries 2022-10-11 12:44:24 +02:00
Alexander A. Klimov
6ae22d5fec cmd/ido2icingadb: fill cache 2022-10-11 12:44:20 +02:00
Alexander A. Klimov
0704750f85 cmd/ido2icingadb: generalize cache schemata 2022-10-11 10:17:47 +02:00
Alexander A. Klimov
63ab8d938e cmd/ido2icingadb: support SELECT xh.foo_bar->FooBar int 2022-10-11 10:17:47 +02:00
Alexander A. Klimov
423c948d30 cmd/ido2icingadb: introduce barIncrementer 2022-10-11 10:17:47 +02:00
Alexander A. Klimov
d9d8fb7576 cmd/ido2icingadb: make bulk a const 2022-10-11 10:17:47 +02:00
Alexander A. Klimov
4de76da239 cmd/ido2icingadb: introduce historyType#setupBar() 2022-10-11 10:17:47 +02:00
Alexander A. Klimov
d9381ec2fa cmd/ido2icingadb: separate cache schema 2022-10-11 10:17:47 +02:00
Alexander A. Klimov
b005b4c08b cmd/ido2icingadb: make log a global var 2022-10-11 10:17:47 +02:00
Alexander A. Klimov
1d111b7dd3 cmd/ido2icingadb: prepare cache 2022-10-11 10:17:47 +02:00
Alexander A. Klimov
1141058b9c cmd/ido2icingadb: split main() 2022-10-11 10:17:47 +02:00
Alexander A. Klimov
495e0afe32 cmd/ido2icingadb: deduplicate parallelism 2022-10-11 10:17:47 +02:00
Alexander A. Klimov
2f5d5ea931 cmd/ido2icingadb: compute previous progress 2022-10-11 10:17:47 +02:00
Alexander A. Klimov
4eb2ba1ea4 cmd/ido2icingadb: count total IDO events 2022-10-11 10:17:47 +02:00
Alexander A. Klimov
890b66ddb9 cmd/ido2icingadb: connect to databases 2022-10-11 10:17:47 +02:00
Alexander A. Klimov
dce3889645 cmd/ido2icingadb: parse config file 2022-10-11 10:17:47 +02:00
Alexander A. Klimov
700f88a2ff cmd/ido2icingadb: parse CLI flags 2022-10-11 10:17:47 +02:00
Alexander A. Klimov
3c1a83e222 Outsource cmd/icingadb.Exit* to cmd/internal 2022-10-11 10:17:47 +02:00
Eric Lippmann
cd96f0de6f Block XREADs for a maxium of one second 
I just had the observation that blocking XREADs without timeouts (BLOCK
0) on multiple consecutive Redis restarts and I/O timeouts exceeds Redis
internal retries and eventually leads to fatal errors. @julianbrost
looked at this for clarification, here is his finding:

go-redis only considers a command successful when it returned something,
so a successfully started blocking XREAD consumes a retry attempt each
time the underlying Redis connection is terminated. If this happens
often before any element appears in the stream, this error is
propagated. (This also means that even with this PR, when restarting
Redis often enough so that a query never reaches the BLOCK 1sec, this
would still happen.)

https://github.com/Icinga/icingadb/pull/504#issuecomment-1164589244
 2022-06-28 16:09:29 +02:00
Eric Lippmann
5f29caecbe
Merge pull request #464 from Icinga/OwnHeartbeat 
Write own status into Redis
 2022-06-28 15:21:03 +02:00
Alexander A. Klimov
e1ff704aff Write own heartbeat into icingadb:telemetry:heartbeat 
including version, current DB error and HA status quo.
 2022-06-23 18:31:45 +02:00
Alexander A. Klimov
d85d070d1f Clear icinga:runtime* and read from 0-0 later 
instead of preserving the (never read) data and reading beyond its end later.
This indicates the correct number of pending runtime updates
(for monitoring by Icinga 2) from the beginning.
 2022-06-23 18:31:45 +02:00
Yonas Habteab
c06a9ed1ca Add host:port log info when trying to connecto to database/redis 2022-06-21 15:42:57 +02:00
Yonas Habteab
ad575ad552 Make Database/Redis schema mismatch error more user friendly 2022-06-21 15:42:57 +02:00
Alexander A. Klimov
fac9f5e4e5 Write ops/s by op and s to icingadb:telemetry:stats 2022-06-15 09:51:59 +02:00
Yonas Habteab
79003429ab Bump expected redis schema version to 5 2022-06-07 12:57:57 +02:00
Julian Brost
19170ecbcf SLA reporting: history retention for SLA tables 2022-05-13 12:04:47 +02:00
Ravi Kumar Kempapura Srinivasa
2c09dd9e1c Start history retention after config and state sync 
Co-authored-by: Eric Lippmann <eric.lippmann@icinga.com>
 2022-03-22 15:55:04 +01:00
Alexander A. Klimov
d898681d3c Wrap "icingadb-*sql" in constants 
refs #136
 2022-03-10 17:07:06 +01:00
Alexander A. Klimov
c3b8f80abf Decouple *SQL schema versions 
refs #136
 2022-03-10 17:04:22 +01:00
Alexander A. Klimov
3213d18249 Schema: make notification_history#text longtext 2022-01-28 11:51:58 +01:00