upstream/icingadb
1
0
Fork 0
mirror of https://github.com/Icinga/icingadb.git synced 2026-05-28 04:35:54 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions 1
1971 commits 29 branches 11 tags 14 MiB
Commit graph

602 commits

Author SHA1 Message Date
Eric Lippmann
12525c7872 Use pkg periodic for overdue sync logs 2021-11-05 17:18:05 +01:00
Eric Lippmann
b10d038ba8 Use internal.LoggingInterval() 2021-11-05 17:18:05 +01:00
Eric Lippmann
dbb64a0de3 Log how many items to insert 2021-11-05 17:18:05 +01:00
Eric Lippmann
addfabbde1 Speak of items instead of rows 2021-11-05 17:18:05 +01:00
Eric Lippmann
5f1639aca2 Use pkg periodic for Redis logs 2021-11-05 17:18:05 +01:00
Eric Lippmann
a6e02e7f3c Introduce Counter.Total() 2021-11-05 17:18:05 +01:00
Eric Lippmann
c335a3c99c Introduce package periodic 2021-11-05 17:18:05 +01:00
Eric Lippmann
986e685ee0 Allow to configure interval for periodic logging 2021-11-05 17:18:05 +01:00
Eric Lippmann
b067ed2147 Introduce Counter.Reset() 2021-11-05 17:18:05 +01:00
Eric Lippmann
8a03745273 Speak of Icinga heartbeat not Icinga 2 heartbeat 2021-11-05 17:18:03 +01:00
Eric Lippmann
dc7511cd25 Don't log if context is canceled 2021-11-05 17:16:57 +01:00
Julian Brost
54dbe0cfbe
Merge pull request #391 from Icinga/bugfix/multi-environment 
Better handling of multiple environments
 2021-11-05 16:55:21 +01:00
Julian Brost
82cf600c55
Merge pull request #401 from Icinga/flawed-config-keys-and-validation 
Fix flawed config keys and validation
 2021-11-04 15:01:51 +01:00
Julian Brost
3342191b5e Exit on environment ID changes 
There's a small risk that when the environment ID changes, Icinga DB could
update write into the wrong environment in the database. Therefore,
https://github.com/Icinga/icinga2/pull/9036 introduced a new default
environment ID based on the CA public key so that there should be no cases
where it's required to change the actual environment ID. So if this happens
nonetheless, just bail out.
 2021-11-03 15:47:38 +01:00
Julian Brost
9b02b18f46 Use new environment ID 
https://github.com/Icinga/icinga2/pull/9036 introduced a new environment ID for
Icinga DB that's written to the icinga:stats stream as field
"icingadb_environment". This commit updates the code to make use of this ID
instead of the one derived from the Icinga 2 Environment constant.
 2021-11-03 15:47:38 +01:00
Eric Lippmann
a081927672 Only sync entities that belong to the current environment 
Previously, we selected each entity from the database.
Now we only select entities that belong to the current environment.
 2021-11-03 15:47:38 +01:00
Eric Lippmann
6b5aac9154 Config: Validate max_rows_per_transaction 2021-11-03 15:26:20 +01:00
Eric Lippmann
dac4a7246a Config: Validate max_placeholders_per_statement 2021-11-03 15:26:01 +01:00
Eric Lippmann
15eb9b471f Don't use CamelCase for config keys 2021-11-03 15:25:16 +01:00
Eric Lippmann
563aafaf90 Config: Validate xread_count 2021-11-03 15:23:40 +01:00
Eric Lippmann
084e0409bc Restart HA after environment change 
If the environment changes during runtime, we have to restart HA
in order to stop a possibly running config sync and start a new
one.
 2021-11-03 14:51:19 +01:00
Eric Lippmann
26a184d953 Make v1.Environment#Name types.String 
The default environment of Icinga is the empty string.
In Golang, the zero value of string is also the empty string.
But it makes sense to distinguish whether the name is not set
or set to the empty string. That is possible with this change.
 2021-11-03 14:49:56 +01:00
Eric Lippmann
54f9ef5a12 Insert environment 
With this change Icinga DB will insert the environment after each
heartbeat takeover if it does not already exist in the database as
the environment may have changed, although this is likely to happen
very rarely,

Instead of checking whether the environment already exists,
uses an INSERT statement that does nothing if it does.
 2021-11-03 14:49:56 +01:00
Eric Lippmann
4a659fd5c4 Add db.BuildIgnoreStmt() 2021-11-03 14:49:56 +01:00
Julian Brost
f290a0c9b7
Merge pull request #392 from Icinga/feature/history-deterministic-ids 
Make {NotificationHistory,StateHistory,History*}#Id UUID -> SHA1
 2021-11-03 13:18:11 +01:00
Alexander A. Klimov
b39eac660f Replace Apache 2.0 licensed gopkg.in/yaml.v3 with MIT licensed github.com/goccy/go-yaml 
... not to have GPLv2<->Apache 2.0 (app<->deps) license conflicts.
 2021-11-03 12:22:21 +01:00
Alexander A. Klimov
d903d05c82 Make History*#Id UUID -> SHA1 2021-11-03 12:15:25 +01:00
Alexander A. Klimov
52ae34a6f8 Make {NotificationHistory,StateHistory}#Id UUID -> SHA1 2021-11-03 12:15:25 +01:00
Eric Lippmann
e433aa7ec3 Move custom var sync to a new method 2021-10-26 09:31:41 +02:00
Eric Lippmann
65440ee8fe Expect no custom var clear events during runtime 2021-10-26 09:31:41 +02:00
Eric Lippmann
b48792cf36 Handle flat custom vars explicitly in runtime updates 
This also requires explicit handling of custom variables as we need
to multiplex the original values to handle flat custom variables.
 2021-10-26 09:31:41 +02:00
Eric Lippmann
4d65c62f77 Handle contracts.Initer in common.NewSyncSubject() 
contracts.EntitiyFactoryFunc.WithInit() checked for
contracts.Initer every time.
Now it is only done once in common.NewSyncSubject().
 2021-10-26 09:31:36 +02:00
Eric Lippmann
c78326ad1b Use SyncSubject in RuntimeUpdates.Sync() 2021-10-26 09:27:00 +02:00
Eric Lippmann
fe6915447e Add method SyncSubject.Name() 2021-10-26 09:27:00 +02:00
Eric Lippmann
a73a882c6f Introduce function ExpandCustomvars() 2021-10-26 09:27:00 +02:00
Eric Lippmann
d017a05d05 Export DB.getSemaphoreForTable() 2021-10-26 09:27:00 +02:00
Eric Lippmann
16dd4663ad Move method DB.getSemaphoreForTable() 2021-10-26 09:27:00 +02:00
Eric Lippmann
44b45fc429 Use context from errgroup 2021-10-26 09:27:00 +02:00
Alexander A. Klimov
b5e024e68d Sync state runtime updates ASAP 
I.e. don't wait for the complete initial sync first.
 2021-10-14 10:15:28 +02:00
Eric Lippmann
537a4cf37f Allow to configure the logging output 2021-10-13 09:46:12 +02:00
Eric Lippmann
a9afcea25c Allow systemd-journald (and console) as log outputs 2021-10-13 09:46:12 +02:00
Eric Lippmann
b582995e37 Introduce zapcore.Core that sends logs to systemd-journald 2021-10-13 09:46:12 +02:00
Eric Lippmann
9e49b62c4d Use the app name as the default logger name 2021-10-13 09:46:12 +02:00
Eric Lippmann
3061e3d0c5 Introduce function utils.AppName() 2021-10-13 09:46:12 +02:00
Eric Lippmann
66d34b4a9f Add ConvertCamelCase utility function 2021-10-13 09:46:12 +02:00
Eric Lippmann
bdeed69337 Move logging from internal to pkg 2021-10-13 09:20:55 +02:00
Eric Lippmann
d8ba0c374a
Merge pull request #364 from Icinga/feature/history-sync-foreign-keys 
Add foreign key constraints to history tables
 2021-10-07 18:38:33 +02:00
Julian Brost
7c782e3eb8 History sync: use information from notification stream for user_notification_history table 2021-10-05 18:35:02 +02:00
Julian Brost
8b4e4d68a6 History sync: use indefinitely blocking XREAD 
Just like we do it throughout the rest of the code.
 2021-10-05 18:35:02 +02:00
Julian Brost
bfcc324535 History sync: rewrite to use a sequential pipeline 
This is in preparation for adding foreign key constraints to the history
tables. For this, is is required to insert the rows into the different history
tables in a defined order.
 2021-10-05 18:35:02 +02:00
Julian Brost
82530c771d Redis/DB: export options member 
This change allows the history sync to use values configured in these options.
 2021-10-05 18:34:55 +02:00
Julian Brost
c5af0cd287 HA: only set realize timeout when active 
When inactive, this is the only query running so it has to retry for longer to
eventually trigger a fatal error if the database is gone for too long (5
minutes at the moment).
 2021-10-04 16:58:35 +02:00
Julian Brost
239d2ea410 HA: after heartbeat expiry, stop writing to database and hand over 
If it's not possible for Icinga DB to write through the heartbeat within its
validity period it cannot signal to other instances that it still is alive and
has the hand over. There's also no point in retrying for this individual
heartbeat any longer.
 2021-10-04 16:58:35 +02:00
Julian Brost
a34aef4fc5 retry: if stopped due to outer context, return that error 
If there is an outer context that is canceled or exceeds its deadline before
the internal timeout is reached, its error should be passed on as the failure
didn't happen due to retry giving up.
 2021-10-04 16:58:35 +02:00
Julian Brost
217ab03e59 heartbeat: wrap messages with a timestamp 
Track when a heartbeat was received to allow other components to check when it
will expire.
 2021-10-04 16:58:35 +02:00
Julian Brost
8b2cb3acb8 heartbeat: use a single channel for all beat/loss events 
Using Cond does not allow to reliably catch all events as one will only receive
events that occour after starting to listen. For heartbeat loss events it's
import to reliably catch them to not remain in an HA active state incorrectly.

fixes #360
 2021-10-04 16:36:09 +02:00
Julian Brost
a1b78e0f23 Add XMessageBulker 
Generics would be nice but we don't have them yet unfortunatly, so for now, yet
another copy of Bulker (as it already exists for EntityBulker).
 2021-10-04 14:44:50 +02:00
Alexander Aleksandrovič Klimov
d99e0586a5
Merge pull request #236 from Icinga/feature/tls 
Support TLS
 2021-10-04 12:27:21 +02:00
Alexander A. Klimov
82c26b187e Support TLS 2021-09-30 12:25:23 +02:00
Ravi Kumar Kempapura Srinivasa
414057830e Allow to configure logging in the YAML configuration 2021-09-28 17:30:11 +02:00
Ravi Kumar Kempapura Srinivasa
acde6ade69 Introduce Logging config struct 2021-09-28 17:30:11 +02:00
Julian Brost
65074f5755
Merge pull request #370 from Icinga/feature/icingadb-scheduling_source-160 
Include CheckResult#scheduling_source in state and history
 2021-09-27 17:32:42 +02:00
Julian Brost
6e3df7d63b
Merge pull request #373 from Icinga/feature/single-threaded-delta 
Rewrite delta to use only a single goroutine
 2021-09-24 16:53:23 +02:00
Julian Brost
0c9fb2f22f
Merge pull request #369 from Icinga/feature/log-reconnects-351 
Log all different failed and recovered reconnects to backends
 2021-09-24 11:58:34 +02:00
Julian Brost
66d9b0e6e6 Rewrite delta to use only a single goroutine 2021-09-24 11:52:15 +02:00
Julian Brost
1e9a88bee6 Add tests and benchmarks for delta computation 2021-09-24 11:52:13 +02:00
Alexander A. Klimov
b4bfee92d9 Log all recovered reconnects to backends 
... to give the admin the all-clear.

refs #351
 2021-09-23 16:07:41 +02:00
Julian Brost
e0c903bfdc Redis HYield: remove duplicates returned by HSCAN 
fixes #349
 2021-09-23 14:36:51 +02:00
Julian Brost
4457f9f440
Merge pull request #365 from Icinga/data-races 
Fix data races
 2021-09-23 12:32:19 +02:00
Eric Lippmann
454381c820 Use uint64 instead of Counter 
Use uint64 as there is no longer any concurrent access.
 2021-09-23 12:18:08 +02:00
Eric Lippmann
98202e1257 Use buffered channel 
Use a buffered channel so that the next HSCAN call does not have
to wait until the previous result has been processed.
 2021-09-23 09:37:31 +02:00
Eric Lippmann
c1e722f5fa Do not close channel too early 
This fixes a data race where the pairs channel was closed too early
when the context is canceled and therefore the outer errgroup
returns from Redis operations before Wait() is called on the inner
errgroup. Unfinished Go methods in the inner errgroup would then
try to work on a closed channel.
 2021-09-23 09:37:31 +02:00
Eric Lippmann
7351559793 Use pointer receiver for Counter.Val() 
This fixes a data race as Val() was previously operating on a copy
of the counter while Inc() and Add() may haved changed the original
value.
 2021-09-23 09:37:31 +02:00
Eric Lippmann
9ce2cff5c0 Introduce WaiterFunc type 
The WaiterFunc type is an adapter to allow the use of ordinary
functions as Waiter.
 2021-09-23 09:37:31 +02:00
Julian Brost
17321cdfc3 Fix use of wrong log function on heartbeat loss 
Has to use the Warnw function as it passes additional zap attributes.
 2021-09-23 09:27:26 +02:00
Alexander A. Klimov
82d8f830af Include CheckResult#scheduling_source in state and history 
refs #160
 2021-09-22 17:30:13 +02:00
Alexander Aleksandrovič Klimov
585d1e6bb5
Merge pull request #368 from Icinga/bugfix/icinga-db-does-not-exit-when-reconnecting-to-the-database-350 
On shutdown: give up HA handover after 3s, not 5m
 2021-09-22 16:22:51 +02:00
Alexander A. Klimov
f554fa9dfe Log all different failed reconnects to backends 
E.g. the first "connection refused" and the first "hostname mismatch".

refs #351
 2021-09-22 16:15:37 +02:00
Alexander A. Klimov
321db0eecf Introduce Settings#OnSuccess 
refs #351
 2021-09-22 15:35:08 +02:00
Alexander A. Klimov
653df82a1e Introduce Settings#OnError 
refs #351
 2021-09-22 15:34:39 +02:00
Alexander A. Klimov
ea7668d99a HA#Close(): allow custom context 
refs #350
 2021-09-22 14:12:27 +02:00
Alexander A. Klimov
5a146645f2 HA#removeInstance(): allow custom context 
refs #350
 2021-09-22 14:12:27 +02:00
Alexander A. Klimov
8d57ec107a WithBackoff(): aggregate optional settings in one struct 
refs #351
 2021-09-22 13:37:12 +02:00
Alexander Aleksandrovič Klimov
4371d04d5e
Merge pull request #356 from Icinga/bugfix/mustpackany 
Clarify what MustPackAny() does
 2021-09-21 16:16:56 +02:00
Alexander Aleksandrovič Klimov
956590cb99
Merge pull request #301 from Icinga/feature/scheduled_by 
Introduce downtime#scheduled_by
 2021-09-02 10:31:59 +02:00
Julian Brost
be9054628a Ensure extra config options are properly initialized 
YAML is decoded by the structure of the YAML source document, not the Go
destination data structure. Therefore, the old code did not always call
UnmarshalYAML() on all sub-structs. Therefore, defaults were not always set but
zero values were used, resulting in all kind of strange behavior.

This commit changes the code so that it no longer relies on individual
UnmarshalYAML() functions to set the defaults for each sub-struct but instead
just sets all of them when creating the surrounding Config instance. It also
moves the config validation to separate Validate() functions.
 2021-09-01 18:49:38 +02:00
Alexander A. Klimov
3f8703310c Clarify what MustPackAny() does 
It always packs a slice, even if only one item given.
 2021-08-20 11:54:00 +02:00
Eric Lippmann
0bf4f4df0a
Merge pull request #343 from Icinga/config-options 
Separate required and optional configuration for database and Redis
 2021-08-10 12:33:22 +02:00
Alexander Aleksandrovič Klimov
5a91d8acf6
Merge pull request #338 from Icinga/feature/support-icingadb-version-again-335 
Support --version
 2021-08-10 11:17:51 +02:00
Eric Lippmann
fbbb9bfacd Don't allow 0 for timeout redis option 
0 stands for deactivate, which makes no sense here.
 2021-08-10 09:29:27 +02:00
Eric Lippmann
8232000524 Don't allow 0 for max_connections database option 
0 stands for deactivate, which makes no sense here.
 2021-08-10 08:55:24 +02:00
Eric Lippmann
50473bca70 Remove UnmarshalYAML 
Config options are no longer inlined.
 2021-08-09 22:06:55 +02:00
Eric Lippmann
1c386c9c2f Don't inline Database options 
There is now the options key to separate required and optional
configuration. Before, both were mixed.
 2021-08-09 22:06:50 +02:00
Eric Lippmann
8927e942f1 Remove UnmarshalYAML 
Config options are no longer inlined.
 2021-08-09 22:06:50 +02:00
Eric Lippmann
559b27cd8b Don't inline Redis options 
There is now the options key to separate required and optional
configuration. Before, both were mixed.
 2021-08-09 21:48:27 +02:00
Eric Lippmann
f9e12d9df7 Move method 2021-08-09 21:45:08 +02:00
Alexander A. Klimov
825dcbc817 Introduce downtime#scheduled_by 2021-08-09 20:12:10 +02:00
Eric Lippmann
783b3a6bfe
Merge pull request #323 from Icinga/feature/downtime-parent-downtime-id 
Downtime: Add parent_id
 2021-08-09 13:43:16 +02:00
Eric Lippmann
0a521a6e4a Add missing doc in meta 2021-08-09 10:30:53 +02:00
Eric Lippmann
1d5ae198aa Add missing doc in meta 2021-08-09 10:30:53 +02:00
Eric Lippmann
a4b77c6a45 Add missing doc in sync 2021-08-09 10:30:53 +02:00
Eric Lippmann
77ab2753f9 Add missing doc in ha 2021-08-09 10:30:53 +02:00
Eric Lippmann
ac0f26e59b Add missing doc in entitiesbyid 2021-08-09 10:30:53 +02:00
Eric Lippmann
1d361594ee Add missing doc in dump_signals 2021-08-09 10:30:53 +02:00
Eric Lippmann
fc5e2882ff Add missing doc in delta 2021-08-09 10:30:53 +02:00
Eric Lippmann
1fda4ea6ee Rename start() to run() 2021-08-09 10:30:53 +02:00
Eric Lippmann
a13788073d Add missing doc entitiy_bulker 2021-08-09 10:30:53 +02:00
Eric Lippmann
2c3a58e365 Add missing doc in bulker 2021-08-09 10:30:53 +02:00
Eric Lippmann
bf415f2e1c Add missing doc in stats_message 2021-08-09 10:30:53 +02:00
Eric Lippmann
ff88cb73f7 Add missing doc in icinga_status 2021-08-09 10:30:53 +02:00
Eric Lippmann
92bc1b26c7 Add missing doc in redis utils 2021-08-09 10:30:53 +02:00
Eric Lippmann
fee30380d5 Add missing doc in client 2021-08-09 10:30:53 +02:00
Eric Lippmann
7d59a98f90 Add missing doc in utils 2021-08-09 10:30:53 +02:00
Eric Lippmann
d1c20b6946 Add missing doc in db 2021-08-09 10:30:53 +02:00
Eric Lippmann
270f1930aa Remove useless comments 2021-08-09 10:30:53 +02:00
Eric Lippmann
7bda89e79d Return error instead of panicking 2021-08-09 10:29:47 +02:00
Eric Lippmann
ee36691f3f Remove --datadir config flag 
It's currently not used anywhere.
 2021-08-09 10:29:47 +02:00
Eric Lippmann
858dbe7481 Remove config.ValidateFile() 
YAML already complains that the file is a directory:
"can't parse YAML file pkg: yaml: input error: read pkg: is a directory"
 2021-08-09 10:29:47 +02:00
Eric Lippmann
0b1610c69b Use cancelCtx() instead of just cancel() 2021-08-09 10:29:47 +02:00
Eric Lippmann
f3f07a29cc Always use data as paramter name in UnmarshalJSON() 2021-08-09 10:29:47 +02:00
Eric Lippmann
63b8d98237 Always use text as paramter name in UnmarshalText() 2021-08-09 10:29:47 +02:00
Eric Lippmann
42935ae962 Fix comments 2021-08-09 10:29:47 +02:00
Eric Lippmann
83866f3a70 Remove unused variables Yes and No 2021-08-09 10:29:47 +02:00
Eric Lippmann
ec70babc91 Fix typo 2021-08-09 10:29:47 +02:00
Eric Lippmann
d40768ee64 Fix different receiver names 2021-08-09 10:29:47 +02:00
Eric Lippmann
e1d27bd93f Remove unused function PipeError 2021-08-09 10:29:47 +02:00
Eric Lippmann
f7be60623c Use QueryxContext() instead of Query() 2021-08-09 10:29:47 +02:00
Alexander A. Klimov
41e473b244 Remove unused functions 2021-08-09 10:29:47 +02:00
Eric Lippmann
868d46219a
Merge pull request #340 from Icinga/retry-badconn 
Also retry driver.ErrBadConn
 2021-08-09 10:27:03 +02:00
Alexander A. Klimov
46da390499 Validate specified config file only once 
... after processed --version to allow --version w/o config file.

refs #335
 2021-08-06 12:40:15 +02:00
Alexander A. Klimov
31fd1963f2 Support --version 
refs #335
 2021-08-06 12:40:15 +02:00
Eric Lippmann
ebd3f10d69 Log MySQL driver errors at the debug level instead of discarding them 
The MySQL driver logs something like unexpected EOF, busy buffer,
broken pipe, bad connection, invalid connection etc. in the event
of an error. Most of these are retryable and are handled either by
the MySQL driver or our retry logic. Previously these messages
were discarded, but since they can be useful when debugging
connectivity issues, we'll now log them with the debug severity as
otherwise they would add too much noise to the logs where you might
see error messages with no actual effect.
 2021-08-06 10:44:50 +02:00
Eric Lippmann
9b5e016e57 Also retry driver.ErrBadConn 
ErrBadConn is returned by a driver to signal that a driver.Conn is
in a bad state. We have to retry with these errors as well.
 2021-08-06 10:02:14 +02:00
Eric Lippmann
b5b169aea8
Merge pull request #312 from Icinga/bulk-statement-size 
Bulk statement size
 2021-08-05 00:40:25 +02:00
Alexander A. Klimov
ec37415261 AcknowledgementHistory#Author: make missing value NULL, not "" 
refs #305
 2021-08-04 12:02:08 +02:00
Eric Lippmann
bbba443529 Reduce the size of bulk statements and make the size configurable 
Reduce the size of the bulk create, update, and delete statements to
also reduce query execution time so that the database server
can better execute statements in parallel.
 2021-08-04 00:14:31 +02:00
Alexander Aleksandrovič Klimov
261c5aab8d
Merge pull request #322 from Icinga/feature/icingadb-last_comment_id 
Icinga DB: introduce Checkable#last_comment_id
 2021-08-03 16:29:40 +02:00
Eric Lippmann
e35a1609fc Stream state updates from icinga:runtime:state 
Icinga now sends runtime updates in two separate channels,
icinga:runtime for config updates and icinga:runtime:state for
state updates. With this change, Icinga DB reads from these two
streams. This is a preparation so that state updates can be
streamed directly after a (re)start of Icinga or Icinga DB without
waiting for the config sync, as it is currently done.
 2021-08-03 14:06:55 +02:00
Alexander A. Klimov
7346edb836 Icinga DB: introduce Checkable#last_comment_id 2021-07-26 18:10:28 +02:00
Noah Hilverling
9f6c73ca56 Downtime: Add parent_id 2021-07-23 13:54:39 +02:00
Eric Lippmann
ab4caa32a2
Merge pull request #319 from Icinga/heartbeat 
Pointer receivers, Cond usage, pass ctx and Godoc for Heartbeat
 2021-07-21 19:06:13 +02:00
Eric Lippmann
725e70f0b9 Pointer receivers, Cond usage, pass ctx and Godoc for Heartbeat 
Heartbeat now uses pointer receivers for its methods because
some methods actually change the heartbeat values.
The context is no longer stored in the structure,
but passed to the controller loop.
The beat and the lost channels are replaced by Cond and
the last heartbeat is stored independently to not be affected by
a slow HA receiver. If the database connections are occupied by
the config, HA cannot update the instance and does not read from
the beat channel in time.
In addition, heartbeat errors are no longer swallowed,
but handled in HA.
 2021-07-20 10:17:05 +02:00
Eric Lippmann
b7ecfb9df2 Do not store ctx inside Cond and add Godoc 2021-07-15 14:06:21 +02:00
Eric Lippmann
320fcd84bb
Merge pull request #308 from Icinga/feature/normalized_performance_data 
Introduce *_state#normalized_performance_data
 2021-07-13 15:16:38 +02:00
Eric Lippmann
2eb914a48c
Merge pull request #303 from Icinga/bugfix/cfg-segv 
Fix SEGV due to empty config
 2021-07-08 10:50:58 +02:00
Alexander A. Klimov
9bf1512e06 Introduce *_state#normalized_performance_data 2021-07-07 19:02:55 +02:00
Alexander A. Klimov
f47b6e7657 Fix SEGV due to empty config 2021-07-06 18:42:40 +02:00
Eric Lippmann
43624ecebe Use bulk upsert for bulk updates in sync 2021-07-05 08:51:32 +02:00
Eric Lippmann
f5777f1055 Use bulk upsert in runtime updates 2021-07-01 11:20:43 +02:00
Eric Lippmann
73865f819a Activate bulk upsert 
sqlx >= v1.3.4 supports `VALUES(col_name)` to refer to column values.
 2021-07-01 11:15:49 +02:00
Eric Lippmann
5cba0f9e22 Return number of placeholders 2021-07-01 11:13:17 +02:00
Eric Lippmann
7eb55ae81b Support DriverContext 
Our driver now supports DriverContext and uses contexts.
This basically supports that the connect function returns
immediately when a context is cancelled.
 2021-06-22 14:54:33 +02:00
Eric Lippmann
40bbe3117f
Merge pull request #289 from Icinga/error-handling 
Error handling
 2021-06-21 17:58:45 +02:00
Eric Lippmann
40095ad0db Call errors.Wrap*() unconditionally where appropriate 2021-06-21 13:20:44 +02:00
Eric Lippmann
dac942a9a8 Use errors.As() and Is() in favor of Unwrap() 2021-06-21 13:20:44 +02:00
Eric Lippmann
e12425d8dc Wrap errors 2021-06-21 12:13:24 +02:00
Eric Lippmann
6b2ecec65b Replace custom err w/ func returing a wrapped err 
This ensures that there's also a stack trace associated with the error.
 2021-06-21 12:13:24 +02:00
Eric Lippmann
8e216d2f83 Fix imports 2021-06-21 12:13:24 +02:00
Alexander A. Klimov
8cbf24932e Add stack of current goroutine to errors 2021-05-31 16:53:57 +02:00
Alexander A. Klimov
c25c47ad72 Compare errors smartly, i.e. unwrap them first 2021-05-31 16:52:47 +02:00
Eric Lippmann
a693b52118 HA: Respect context when operating channels 
The transmission processes of the handover and takeover channel can
block because in the event of a context cancellation, nobody reads
it anymore.
 2021-05-31 15:31:55 +02:00
Julian Brost
31aed435cc HA: context used to start transaction must not be canceled before the transaction is committed 
This was introduced by 621c1b9537 and leads to
tx.Commit() always returning an "context canceled" error.
 2021-05-31 14:48:15 +02:00
Alexander A. Klimov
b0354e3503 Don't misuse loop as if 2021-05-28 14:24:36 +02:00
Alexander A. Klimov
afe0a90487 s/CondClosed/ErrCondClosed/ 2021-05-28 14:24:36 +02:00
Alexander A. Klimov
1329c68cf3 Use !bytes.Equal(x,y), not bytes.Compare(x,y)!=0 2021-05-28 14:24:36 +02:00
Alexander A. Klimov
a4abdce1f0 Use time.Since(x), not time.Now().Sub(x) 2021-05-28 14:24:36 +02:00
Alexander A. Klimov
35349262ce Use time.NewTicker(), not time.Tick() 2021-05-28 14:24:36 +02:00
Alexander A. Klimov
5a084ba7a9 Simplify code 2021-05-28 14:24:36 +02:00
Alexander A. Klimov
c636de294a Un-capitalize error messages 2021-05-28 14:24:36 +02:00
Alexander A. Klimov
dac83f2773 Drop unused stuff 2021-05-28 14:24:36 +02:00
Alexander A. Klimov
c3ea4d9490 Avoid unreachable code 2021-05-28 14:24:36 +02:00
Alexander A. Klimov
621c1b9537 Ensure context cancellation 2021-05-28 14:24:36 +02:00
Alexander A. Klimov
cabcd458ff Don't "misuse" unsafe.Pointer 2021-05-28 14:24:36 +02:00
Eric Lippmann
9ba2c01ce2 Reduce "Can't update or insert instance. Retrying" log noise 
Only log after the third retry with the info level. Also, before
executing the transaction, sleep dependent on the retry count.
 2021-05-25 16:35:41 +02:00
Eric Lippmann
372f5cae7c Also log environment info 2021-05-25 16:25:04 +02:00
Eric Lippmann
a1ffc53998 Log first Redis connection error while retrying 2021-05-25 14:00:06 +02:00
Eric Lippmann
be3180a54c Log first database connection error while retrying 2021-05-25 14:00:05 +02:00
Noah Hilverling
44c734f72d Improve database and HA logging 2021-05-25 09:49:48 +02:00
Julian Brost
37538b891c db.BulkExec(): Use ExecContext() instead of Query() 
Query() returns a cursor that has to be closed to not block the underlying
database connection. Use ExecContext() instead to both avoid the cursor leak
and also properly pass the context.

This bug lead to the DB connection pool be blocked completely after a certain
number of runtime delete queries.
 2021-05-21 11:24:53 +02:00
Eric Lippmann
c5e875cfce Merge pull request #64 from lippserd/feature/mysql-retry-5m 
Re-use Redis dialer logic in retry.WithBackoff() and the SQL driver
 2021-05-21 10:39:29 +02:00
Alexander A. Klimov
867d5b67dd SQL driver: de-duplicate retry.WithBackoff() logic 2021-05-20 13:13:55 +02:00
Alexander A. Klimov
e4e138aaa4 Redis dialer: de-duplicate retry.WithBackoff() logic 2021-05-20 12:10:20 +02:00
Noah Hilverling
7bbc4e931e Merge pull request #61 from lippserd/bugfix/change-id-fields-to-match-sql-schema 
Change ID fields to match SQL schema
 2021-05-20 10:46:06 +02:00
Eric Lippmann
91052b1f69 Merge pull request #62 from lippserd/feature/wrap-redis-err 
Wrap Redis errors
 2021-05-20 09:05:18 +02:00
Alexander A. Klimov
5bcd5339b4 retry.WithBackoff(): return the most descriptive error 2021-05-19 19:28:22 +02:00
Alexander A. Klimov
f77d394041 retry.WithBackoff(): add optional timeout 2021-05-19 19:12:18 +02:00
Alexander A. Klimov
00fe3fe6f7 retry.WithBackoff(): pass a context to the function to be tried 2021-05-19 19:00:55 +02:00
Julian Brost
7056cf2b92 Don't execute runtime update upset queries concurrently 
These updates must be executed in order, therefore prevent concurrency by using
a separate semaphore.
 2021-05-19 18:32:47 +02:00
Eric Lippmann
ff5bded004 Merge pull request #26 from lippserd/feature/redis-retry 
Survive a Redis restart
 2021-05-19 16:57:24 +02:00
Eric Lippmann
417ba462ed Merge pull request #31 from lippserd/feature/overdue 
Sync overdue indicators
 2021-05-19 16:57:03 +02:00
Alexander A. Klimov
1026d4cabf Wrap Redis errors 2021-05-19 11:57:58 +02:00
Alexander A. Klimov
d08f32397a Introduce icingaredis.WrapCmdErr() 2021-05-19 11:57:58 +02:00
Alexander A. Klimov
05c1361f1a Introduce utils.Ellipsize() 2021-05-19 11:55:17 +02:00
Eric Lippmann
20465d078a Merge pull request #57 from lippserd/bugfix/include-instance-id-in-ha-query 
HA: Add own instance ID to responsibility query
 2021-05-19 09:48:48 +02:00
Noah Hilverling
26086db211 HA: Add own instance ID to responsibility query 
Without this PR Icinga DB thinks someone else is responsible, when it is responsible. This leads to Icinga DB only updating the heartbeat after the timeout. This could also cause random responsibility switching.
 2021-05-19 09:32:12 +02:00
Alexander A. Klimov
3e45567368 config.Redis#NewClient(): work-around go-redis/redis#1737 
... by re-trying once more often than there are connections in the pool.
 2021-05-18 18:47:41 +02:00
Alexander A. Klimov
45b626c914 Redis: retry dial on syscall.ECONNREFUSED 2021-05-18 18:47:41 +02:00
Noah Hilverling
1282df748f Change ID fields to match SQL schema 2021-05-18 09:42:46 +02:00
Noah Hilverling
586e99a548 types.AcknowledgementState: Fix state map 2021-05-17 09:39:29 +02:00
Alexander A. Klimov
b31e012cf0 Sync overdue indicators 2021-05-12 18:51:13 +02:00
Eric Lippmann
7fa8147303 Merge pull request #34 from lippserd/bugfix/cv-flat-nil 
Flat CVs: represent null as "null", not "<nil>"
 2021-05-12 14:16:49 +02:00
Alexander A. Klimov
188ce4cb4b Flat CVs: represent null as "null", not "<nil>" 2021-05-12 12:08:13 +02:00
Julian Brost
f82de0c1a9 Make Redis/MySQL concurrency and batch sizes configurable 2021-05-12 11:54:36 +02:00
Eric Lippmann
58f80bc8d8 Introduce objectpacker.MustPackAny() 2021-05-11 16:40:08 +02:00
Eric Lippmann
ac9aa0365a Remove time taken debug log for named queries 2021-05-11 16:40:08 +02:00
Eric Lippmann
bf27824980 Remove NewCustomvar from Factories 
We now explicitly sync it in main.go.
 2021-05-11 16:40:08 +02:00
Eric Lippmann
e71833defb Flatten: Fix keys 2021-05-11 16:40:08 +02:00
Eric Lippmann
52e5eb774e Sync customvar flat 2021-05-11 16:40:08 +02:00
Eric Lippmann
f4ec939b9f Add func icingadb.v1.FlattenCustomvars() 2021-05-11 16:40:08 +02:00
Eric Lippmann
766740974f Precise log messages 2021-05-11 16:40:08 +02:00
Eric Lippmann
850de4d371 Remove log message 2021-05-11 16:40:08 +02:00
Eric Lippmann
eea1d0e486 Fix race 2021-05-11 16:40:08 +02:00
Eric Lippmann
9c86198302 Remove icingadb.Sync.fromRedis() 2021-05-11 16:40:08 +02:00
Eric Lippmann
5550c47be3 Add NewCustomvarFlat() 2021-05-11 16:40:08 +02:00
Eric Lippmann
7c70a72991 Introduce icingadb.Sync.ApplyDelta() 2021-05-11 16:40:08 +02:00
Eric Lippmann
34363e19ac Expect common.SyncSubject in icingadb.Delta 2021-05-11 16:40:08 +02:00
Eric Lippmann
442e04bbf1 Use type common.SyncSubject 2021-05-11 16:40:08 +02:00
Eric Lippmann
51d2532f18 Introduce func icingaredis.Client.YieldAll() 2021-05-11 16:40:08 +02:00
Eric Lippmann
961a3650e8 Introduce type common.SyncSubject 2021-05-11 16:40:08 +02:00
Eric Lippmann
420193f228 Merge pull request #50 from lippserd/bugfix/obj-pack-bytes 
PackAny(): support types.Binary
 2021-05-11 13:53:54 +02:00
Alexander A. Klimov
40c18e3492 PackAny(): Golangify spec doc 2021-05-10 14:01:52 +02:00
Alexander A. Klimov
a6da8ab90a PackAny(): pack [I]byte as string in map keys as well 2021-05-10 13:15:34 +02:00
Alexander A. Klimov
c8fa629f32 PackAny(): restore unit test case 2021-05-10 12:38:54 +02:00
Alexander A. Klimov
1b4b207465 PackAny(): enhance code docs 2021-05-10 12:35:23 +02:00
Alexander A. Klimov
9df2f9d7e6 PackAny(): panic() explicitly, not due to a programming error 2021-05-10 12:13:24 +02:00
Eric Lippmann
a1ceeeb9eb Merge pull request #36 from lippserd/feature/runtime-updates 
Implement runtime updates
 2021-05-07 14:37:56 +02:00
Alexander A. Klimov
23127e3245 PackAny(): disallow types recursively more strictly 2021-05-07 12:38:18 +02:00
Alexander A. Klimov
a66cfca9c8 PackAny(): support numbers only as float64 2021-05-07 12:16:09 +02:00
Noah Hilverling
91a8228e2d Implement runtime updates 2021-05-07 12:04:41 +02:00
Noah Hilverling
cf195c9b98 Fix UsergroupMember.UserId JSON tag 2021-05-07 09:18:41 +02:00
Alexander A. Klimov
ab43673593 PackAny(): support types.Binary 2021-05-06 18:00:47 +02:00
Alexander A. Klimov
bb850ddb5d PackAny(): pack []byte as string, not array of numbers 2021-05-06 17:53:54 +02:00
Noah Hilverling
ca9936e5b6 icingadb.DB: Rename functions that take a channel to *Streamed() 2021-05-04 14:30:53 +02:00
Noah Hilverling
25558f9827 icingadb.DB: Add Delete() and DeleteStreamed() 2021-05-04 14:30:23 +02:00
Noah Hilverling
4b56dd1de2 com.Bulker: Change type from contracts.Entity to interface{} 2021-05-04 11:46:33 +02:00
Noah Hilverling
507a414ca7 Add com.EntityBulker 2021-05-04 11:46:33 +02:00
Noah Hilverling
b05a00a8d5 icingaredis.Client: Add StreamLastId() 2021-05-04 11:46:33 +02:00
Noah Hilverling
26c889ad66 types.NotificationTypes: Implement UnmarshalText() 2021-05-04 11:46:33 +02:00
Noah Hilverling
9e3a8b4d1d types.NotificationStates: Implement UnmarshalText() 2021-05-04 11:46:33 +02:00
Noah Hilverling
564d3d594e types.Int: Implement UnmarshalText() 2021-05-04 11:46:33 +02:00
Noah Hilverling
96835ace04 Structify: Allow for string ptr (e.g. NameCi) 2021-05-04 11:46:33 +02:00
Noah Hilverling
094e374070 NameCiMeta: Add missing json tag to NameMeta 2021-05-04 11:46:33 +02:00
Alexander A. Klimov
429ea1ca48 Introduce PackAny() 2021-05-04 11:09:24 +02:00
Noah Hilverling
7bc8bddc5a Add service state 2021-05-04 09:55:43 +02:00
Noah Hilverling
4dc38c42e7 Add host state 2021-05-04 09:55:43 +02:00
Noah Hilverling
ebf6698540 Add v1.State 2021-05-04 09:55:43 +02:00
Noah Hilverling
0dac6aec10 Add types.AcknowledgementState 2021-05-04 09:25:06 +02:00
Noah Hilverling
39589dcb7d types.StateType: Implement UnmarshalJson() 2021-05-04 09:25:06 +02:00
Noah Hilverling
a0a1eec35f Change redis key prefix to 'icinga:*' instead of 'icinga:config:*' 2021-05-04 09:22:38 +02:00
Julian Brost
a7e5395f66 Read icinga:dump stream and respect dump signals 2021-05-03 11:19:26 +02:00
Julian Brost
ab24bdd1f7 Main loop: return if ctx has an error 
If ctx.Done(), there's no point in looping again.
 2021-04-28 16:05:56 +02:00
Julian Brost
d138b8e0ac Handle SIGINT/SIGTERM/SIGHUP 
On any of these signals, HA removes the row of the local instance from
icingadb_instance. This allows another instance to take over immediately.
 2021-04-28 12:01:02 +02:00
Eric Lippmann
3a4c96fa2e Introduce v1.Factories 2021-04-27 23:34:35 +02:00
Eric Lippmann
adcf9a6cfd Add EntityFactoryFunc.WithInit() 2021-04-27 23:33:49 +02:00
Alexander A. Klimov
0ec2b6ca60 Return ctx.Err() on ctx.Done() 2021-04-26 14:15:10 +02:00
Eric Lippmann
cab03bd8f6 Merge pull request #19 from lippserd/bugfix/hmyield 
HMYield(): handle missing values
 2021-04-22 10:20:21 +02:00
Alexander A. Klimov
3304f0486c HMYield(): handle missing values 2021-04-21 15:26:31 +02:00
Alexander A. Klimov
b8d2372899 Sync#Sync(): don't run no-ops 2021-04-19 18:23:18 +02:00
Alexander A. Klimov
4dc935534c Fix missing import 2021-04-19 16:48:02 +02:00
Eric Lippmann
0c8a9139b5 Merge pull request #28 from lippserd/bugfix/delta-race 
Delta#start(): avoid a race across maps by using a mutex
 2021-04-19 16:20:42 +02:00
Eric Lippmann
3ea98313c3 Merge pull request #17 from lippserd/feature/history-sync 
Sync history
 2021-04-15 22:23:49 +02:00
Alexander A. Klimov
bf1a77a67c Drop utils.SyncMap*() 2021-04-12 17:41:18 +02:00
Alexander A. Klimov
fac47fb330 Delta: don't over-lock 2021-04-12 17:41:18 +02:00
Alexander A. Klimov
3974a7bd4f Introduce EntitiesById 2021-04-12 17:29:32 +02:00
Alexander A. Klimov
7ccf627df6 Sync history 2021-04-12 13:28:03 +02:00
Alexander A. Klimov
9aa1070db0 Introduce contracts.TableNamer 2021-04-12 13:28:03 +02:00
Alexander A. Klimov
3fbc9fa25f DB#Upsert(): workaround jmoiron/sqlx#694 2021-04-12 13:28:03 +02:00
Alexander A. Klimov
4e87ca6de3 DB#Upsert(): allow to update not all columns 2021-04-12 13:28:03 +02:00
Alexander A. Klimov
ebfabaffc2 Introduce DB#Upsert() 2021-04-12 13:28:03 +02:00
Alexander A. Klimov
500dbac66a Deduplicate DB#Create() and DB#Update() 2021-04-12 13:28:03 +02:00
Alexander A. Klimov
a9a4450068 DB#NamedBulkExec*(): reduce channel requirements 2021-04-12 13:28:03 +02:00
Alexander A. Klimov
df0124de09 DB#NamedBulkExec(): optionally inform about successfully inserted rows 2021-04-12 13:28:03 +02:00
Alexander A. Klimov
b13d2c3cd7 Restrict Bulker to contracts.Entity 2021-04-12 13:28:03 +02:00
Alexander A. Klimov
581cac27aa Make types.UnixMilli an encoding.TextUnmarshaler 2021-04-12 13:28:03 +02:00
Alexander A. Klimov
270dfa6159 Make types.String an encoding.TextUnmarshaler 2021-04-12 13:28:03 +02:00
Alexander A. Klimov
bfd22b1f39 Make types.Bool an encoding.TextUnmarshaler 2021-04-12 13:28:03 +02:00
Alexander A. Klimov
fb4fd4c964 Introduce types.StateType 2021-04-12 13:28:03 +02:00
Alexander A. Klimov
ba69cef6cb Introduce types.NotificationType 2021-04-12 13:28:03 +02:00
Alexander A. Klimov
c86a8df20b Introduce types.Float 2021-04-12 13:28:03 +02:00
Alexander A. Klimov
f2adf2e8c1 Introduce types.UUID 2021-04-12 13:28:03 +02:00
Alexander A. Klimov
ee023b1202 Introduce utils.MakeMapStructifier() 2021-04-12 13:28:03 +02:00
Eric Lippmann
b36d198075 Merge pull request #29 from lippserd/bugfix/address-bin 
Host: sync also address{,6}_bin
 2021-04-12 10:53:49 +02:00
Eric Lippmann
d96b5938b1 Merge pull request #27 from lippserd/feature/json-name_ci 
NameCiMeta#NameCi: don't (unneccessarily) decode from JSON
 2021-04-12 10:52:36 +02:00
Alexander A. Klimov
5f4d04e5a7 Host: sync also address{,6}_bin 2021-04-08 17:33:43 +02:00
Alexander A. Klimov
d36928afd3 Delta#start(): avoid a race across maps by using a mutex 
Imagine an Icinga restart w/o any config changes and a full dump already being done.
One goroutine reads Redis, the other the database.
Both get the same object at the same time and check it in the map of the other goroutine - not present.
So they store it in their own map.
I.e. the same object hasn't been changed, but has to be deleted and inserted.
If the insert comes first, that causes a duplicate key error.
 2021-04-08 16:28:42 +02:00
Alexander A. Klimov
0b660a7e7a NameCiMeta#NameCi: don't (unneccessarily) decode from JSON 2021-04-08 13:50:38 +02:00
Alexander A. Klimov
e092e6de68 shouldRetry(): retry on driver.ErrBadConn as well 2021-04-07 13:56:43 +02:00
Alexander A. Klimov
f5003e6119 Actually use re-trying SQL driver 2021-04-07 13:55:40 +02:00
Eric Lippmann
d1b1693d14 Merge pull request #23 from lippserd/bugfix/comment-entrytype 
Introduce types.CommentType
 2021-04-06 10:10:28 +02:00
Eric Lippmann
cf110d74bf Merge pull request #18 from lippserd/bugfix/notification-bitmask 
Correctly parse notification bitmask types
 2021-04-06 10:07:54 +02:00
Eric Lippmann
8345202fc6 Merge pull request #21 from lippserd/bugfix/json-inline 
EntityWithChecksum: fix missing `json:",inline"`
 2021-03-30 10:57:11 +02:00
Eric Lippmann
32ece5ea43 Merge pull request #20 from lippserd/bugfix/cmd-arg-bool-null 
Make CommandArgument an Initer
 2021-03-30 10:56:28 +02:00
Eric Lippmann
63d8bf8e24 Merge pull request #15 from lippserd/feature/timeperiod 
Sync timeperiods as well
 2021-03-30 10:12:33 +02:00
Alexander A. Klimov
4743693872 Introduce types.CommentType 2021-03-26 14:16:20 +01:00
Alexander A. Klimov
4a90674823 EntityWithChecksum: fix missing json:",inline" 2021-03-23 19:26:49 +01:00
Alexander A. Klimov
96a93847f4 Make CommandArgument an Initer 
... to auto-set the required defaults.
 2021-03-23 17:23:10 +01:00
Alexander A. Klimov
d782d41b79 Introduce types.NotificationStates 2021-03-23 16:27:29 +01:00
Alexander A. Klimov
c2cdb82fc4 Introduce types.NotificationTypes 2021-03-23 16:27:29 +01:00
Eric Lippmann
9525ef476e Merge pull request #13 from lippserd/feature/notification 
Sync notifications as well
 2021-03-23 10:09:06 +01:00
Eric Lippmann
b9673e1a94 Merge pull request #11 from lippserd/feature/comment 
Sync comments as well
 2021-03-23 10:08:01 +01:00