1477 commits

Author	SHA1	Message	Date
bwplotka	49ea7b05ab	feat(storage)[PART4b]: add AppenderV2 to the rest of storage.Storage impl ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2026-01-12 09:19:19 +00:00
Bryan Boreham	da754cd240	Merge remote-tracking branch 'origin/release-3.9' into merge-3.9-into-main	2026-01-07 15:51:44 +00:00
Bryan Boreham	f1719fa1d4	[BUGFIX] Agent: fix crash from invalid type in pool (#17802) ... We have separate pools for Appender and AppenderV2 objects, and must not put another kind of object into them. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2026-01-07 14:01:02 +00:00
Julius Hinze	22463b1e9f	tsdb: add support for OOO exemplars in CircularExemplarStorage (#17469) ... * tsdb: add support for OOO exemplars in CircularExemplarStorage Doubly linked exemplar storage resize. Split exemplar buffer resize into shrink and grow functions. Skip duplicate OOO exemplars, re-initialize emptied index after deleting its last exemplar. Signed-off-by: Julius Hinze <julius.hinze@grafana.com>	2026-01-07 13:25:50 +01:00
Patryk Prus	dcda4840a7	tsdb/index: export sentinel errors for size limit failures (#17773) ... * tsdb/index: export sentinel errors for size limit failures --------- Signed-off-by: Patryk Prus <p@trykpr.us> Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>	2026-01-06 14:07:23 +01:00
Ben Kochie	e14795bbf4	Remove copyright date from headers (#17785) ... Remove copyright dates from various files as part of [PROM-50]. [PROM-50]: https://github.com/prometheus/proposals/blob/main/proposals/0050-remove-copyright-dates.md Signed-off-by: SuperQ <superq@gmail.com>	2026-01-05 13:46:21 +01:00
Jarkko Pöyry	e4b6d443fc	tsdb: fix handle leak on mmap failure on MS Windows (#17310) ... Signed-off-by: Jarkko Pöyry <jarkko.poyry@metaplay.io>	2025-12-21 20:55:02 +01:00
Bartlomiej Plotka	1c0537dc02	Merge pull request #17677 from prometheus/bwplotka/a2-agent ... refactor(tsdb/agent)[PART3]: add AppenderV2 support to agent	2025-12-17 14:27:51 +00:00
bwplotka	96ff5b8f9c	addressed comment ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-12-17 05:47:28 +00:00
bwplotka	bab7614d1b	fix: ensure no race on lset ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-12-17 05:39:07 +00:00
bwplotka	fc019d6628	refactor(scrape): DRY-ed getOrCreate flow ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-12-17 05:29:30 +00:00
Charles Korn	69aa3ac67b	tsdb: correctly log error in headAppenderV2.appendExemplars ... Signed-off-by: Charles Korn <charles.korn@grafana.com>	2025-12-17 10:04:14 +11:00
NamanParlecha	c94101d023	TSDB: Option to configure TSDB Block Reload Interval (#16728) ... Add --storage.tsdb.block-reload-interval flag to configure TSDB block reload interval. --------- Signed-off-by: Naman-B-Parlecha <namanparlecha@gmail.com> Signed-off-by: NamanParlecha <namanparlecha@gmail.com> Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>	2025-12-15 09:31:17 +01:00
bwplotka	ad367b504b	refactor(tsdb/agent)[PART3]: add AppenderV2 support to agent ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-12-13 05:17:39 +00:00
bwplotka	763b935b45	refactor(tsdb/agent): 1:1 copy of db.go and db_test.go for starting point ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-12-12 15:20:42 +00:00
Bryan Boreham	41665a4a55	[CHORE] TSDB: Remove unused LabelValueFor function ... The last use of this was removed 4 years ago. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2025-12-12 13:44:13 +00:00
Bryan Boreham	583bc01cc9	Refactor: TSDB: small improvement to Postings (#17661) ... * [TESTS] TSDB: Check that ListPostings are sorted `newListPostings` is a convenient place to do this; move it into the test file because it's not needed for anything else. Also simplify the existing `SliceIsSorted` check. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * [COMMENTS] TSDB: Document that Postings are ordered. The description of `Seek()` implies they are, but it's better to be explicit. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * [REFACTOR] Unexport ListPostings type It's only used within the one package, and it would be surprising if some downstream code did rely on this type, given all of its members are unexported. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> --------- Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2025-12-11 08:14:35 +00:00
Julien	f73aba34cd	Merge pull request #17427 from roidelapluie/roidelapluie/ffapi ... API: Add a /api/v1/features endpoint	2025-12-10 10:14:03 +01:00
Julien Pivotto	a5671a002f	API: Add a /api/v1/features endpoint ... Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>	2025-12-09 16:13:14 +01:00
Bartlomiej Plotka	be419d80dc	Merge pull request #17629 from prometheus/bwplotka/a2-tsdb ... refactor(appenderV2)[PART1]: add AppenderV2 interface; add TSDB AppenderV2 implementation	2025-12-09 11:41:00 +00:00
bwplotka	e7e45090e4	refactor(appenderV2): port TSDB non-head tests ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-12-09 10:39:45 +00:00
bwplotka	0b70a07572	refactor(appenderV2): add TSDB AppenderV2 implementation ... Signed-off-by: bwplotka <bwplotka@gmail.com> tmp Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-12-09 10:39:43 +00:00
bwplotka	a41e1144dd	refactor(appenderV2): 1:1 copy of db_test.go to db_append_v2_test.go (starting point) ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-12-09 09:53:39 +00:00
bwplotka	efdfb8fed6	refactor(appenderV2): 1:1 copy of head append test files for v2 (starting point) ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-12-09 09:53:39 +00:00
bwplotka	129650df9d	refactor(appenderV2): 1:1 copy of head_append.go -> head_append_v2.go (starting point) ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-12-09 09:53:39 +00:00
dongjiang	3239723098	Update golangci-lint and add modernize check (#17640) ... * add modernize check Signed-off-by: dongjiang1989 <dongjiang1989@126.com> * fix golangci lint Signed-off-by: dongjiang1989 <dongjiang1989@126.com> --------- Signed-off-by: dongjiang1989 <dongjiang1989@126.com>	2025-12-05 09:29:10 +01:00
Bartlomiej Plotka	f6ca7145ca	refactor(tsdb): use one test newTestDB constructor (#17638) ... For tests only, we had various ways of opening DB. Reduced to one instead of: * Open * newTestDB * newTestDBOpts * openTestDB This so https://github.com/prometheus/prometheus/pull/17629 is smaller and bit easier. Also for test maintainability and consistency. Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-12-03 07:55:48 +00:00
Łukasz Mierzwa	8a1086a128	feat: Add flag that blocks lvl 1 compactions until upload is confirmed in an external JSON file (#17435) ... * Delay compactions until Thanos uploads all blocks Using Thanos sidecar with Prometheus requires us to disable TSDB compactions on Prometheus side by setting --storage.tsdb.min-block-duration and --storage.tsdb.max-block-duration to the same value. See https://thanos.io/tip/components/sidecar.md. The main problem this avoids is that Prometheus might compact given block before Thanos uploads it, creating a gap in Thanos metrics. Thanos does not upload compacted blocks because that would upload the same sample multiple times. You can tell Thanos to upload compacted blocks but that is aimed at one time migrations. This patch creates a bridge between Thanos and Prometheus by allowing Prometheus to read the shipper file Thanos creates, where it tracks which blocks were already uploaded, and using that data delays compaction of blocks until they are marked as uploaded by Thanos. Thanks to this both services can coordinate with each other (in a way) and we can stop disabling compaction on Prometheus side when Thanos uploads are enabled. The reason to have this is that disabling compactions have very dramatic performance cost. Since most time series exist for longer than a single block duration (2h by default) large chunks of block index will reference the same series, so 10 * 2h blocks will each have an index that is usually fairly big and is almost the same for all 10 blocks. Compaction de-duplicates the index so merging 10 blocks together would leave us with a single index that is around the same size as each of these 10 2h blocks would have (plus some extra for series that only exists in some blocks, but not all). Every range query that iterates over all 10 blocks would then have to read each index and so we're doing 10x more work then if we had a single compacted block. Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com> * Rename structs and functions to make this more generic Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com> * Address review comments Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com> * Cache UploadMeta for 1 minute Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com> --------- Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com>	2025-12-02 10:39:45 +00:00
Björn Rabenstein	b8d19543b8	Add histogram validation in remote-read and during reducing resolution (#17561) ... ReduceResolution is currently called before validation during ingestion. This will cause a panic if there are not enough buckets in the histogram. If there are too many buckets, the spurious buckets are ignored, and therefore the error in the input histogram is masked. Furthermore, invalid negative offsets might cause problems, too. Therefore, we need to do some minimal validation in reduceResolution. Fortunately, it is easy and shouldn't slow things down. Sadly, it requires to return errors, which triggers a bunch of code changes. Even here is a bright side, we can get rud of a few panics. (Remember: Don't panic!) In different news, we haven't done a full validation of histograms read via remote-read. This is not so much a security concern (as you can throw off Prometheus easily by feeding it bogus data via remote-read) but more that remote-read sources might be makeshift and could accidentally create invalid histograms. We really don't want to panic in that case. So this commit does not only add a check of the spans and buckets as needed for resolution reduction but also a full validation during remote-read. Signed-off-by: beorn7 <beorn@grafana.com>	2025-11-21 00:22:24 +01:00
0xkato	ae00fd45ab	tsdb: guard chunk length overflow in head chunk reader (#17533) ... Signed-off-by: 0xkato <0xkkato@gmail.com>	2025-11-15 21:09:00 +01:00
Bryan Boreham	1240402620	Merge pull request #17439 from bboreham/faster-postings ... tsdb: couple of postings optimizations	2025-11-14 18:36:34 +01:00
Bartlomiej Plotka	f50ff0a40a	feat: rename CreatedTimestamp to StartTimestamp (#17523) ... Partially fixes https://github.com/prometheus/prometheus/issues/17416 by renaming all CT* names to ST* in the whole codebase except RW2 (this is done in separate [PR](https://github.com/prometheus/prometheus/pull/17411)) and PrometheusProto exposition proto. ``` CreatedTimestamp -> StartTimestamp CreatedTimeStamp -> StartTimestamp created_timestamp -> start_timestamp CT -> ST ct -> st ``` Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-11-13 14:17:51 +00:00
Ben Ye	2e609511bb	Register missing metric prometheus_tsdb_sample_ooo_delta (#17477) ... * register missing metric prometheus_tsdb_sample_ooo_delta Signed-off-by: yeya24 <benye@amazon.com> * changelog Signed-off-by: yeya24 <benye@amazon.com> --------- Signed-off-by: yeya24 <benye@amazon.com>	2025-11-11 11:07:08 +01:00
Bryan Boreham	c1e0ab11c6	[PERF] TSDB: Speed up intersectPostings.Next ... Check if the next position is already a match, in which case we don't have to call `Seek`. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2025-11-10 17:00:19 +00:00
Bryan Boreham	0e1e7441e4	[PERF] TSDB: ListPostings: check next item before binary search ... It is fairly common that the next item is the one we want, and cheap to check. We could also start the binary search one position on, but strangely that slows it down. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2025-11-10 17:00:19 +00:00
Bryan Boreham	be8307db58	[TEST] Refactor BenchmarkIntersect to reduce memory allocations ... Extract functions which pre-create all the memory for the benchmark itself. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2025-11-10 17:00:15 +00:00
Bryan Boreham	393ab9e12e	[TEST] TSDB: More realistic BenchmarkIntersect ... 100,000 matchers is not something that could happen while using Prometheus. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2025-11-10 16:58:14 +00:00
Jan Fajerski	49254f45e9	Merge pull request #17351 from bboreham/simplify-precreate ... TSDB: Allocate series ID after seriesLifecycleCallback; simplify code.	2025-11-07 14:39:51 +01:00
Ben Kochie	204249fcb5	Update golangci-lint (#17478) ... * Update golangci-lint to v2.6.0 * Fixup various linting issues. * Fixup deprecations. * Add exception for `labels.MetricName` deprecation. Signed-off-by: SuperQ <superq@gmail.com>	2025-11-05 13:47:34 +01:00
Ben Kochie	48956f60d7	Update modernize (#17471) ... Apply additional Go modernize tool improvements. Signed-off-by: SuperQ <superq@gmail.com>	2025-11-04 05:13:49 +00:00
geogrego	58dbe927d5	docs: minor improvement for docs ... Signed-off-by: geogrego <geogrego@outlook.com>	2025-10-29 14:42:14 +08:00
Fiona Liao	b004db49af	Reduce samples for TestRuntimeRetentionConfigChange (#17422) ... * Reduce samples for TestRuntimeRetentionConfigChange --------- Signed-off-by: Fiona Liao <fiona.liao@grafana.com>	2025-10-28 18:23:32 +01:00
Arve Knudsen	df8a9076b9	tsdb: Reduce TestHeadSeriesChunkRace number of iterations to 100 (#17410) ... Reduce tsdb.TestHeadSeriesChunkRace number of iterations from 1000 to 100, to stop this test from timing out under CI. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2025-10-28 13:57:20 +01:00
Minh Nguyen	ad4b59c504	tsdb: Deprecate retention flags; add tsdb.retention runtime configuration (#17026) ... * Move storage from CL to config file Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * Fix .md Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * run make cli-documentation Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * run make cli-documentation Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * nit_fixed Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * add test and update configuration.md Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix lint Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> --------- Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>	2025-10-27 14:51:33 +00:00
György Krajcsovits	18efd9d629	feat(ui): mark native histograms as stable in ui strings ... Plus some docstrings Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2025-10-24 12:32:15 +02:00
Ayoub Mrini	504587c724	chore(direct_io): fix constructor's name (#17371) ... Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2025-10-23 11:35:16 +02:00
Bryan Boreham	42b52ecc4b	TSDB: Allocate series ID after seriesLifecycleCallback ... This callback is not used by Prometheus, but in downstream projects it is wasteful to allocate an ID only to abandon it. Remove lengthy commment which I feel is distracting from the flow. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2025-10-17 11:06:22 +01:00
Bryan Boreham	2852c9c431	[REFACTOR] TSDB: Simplify series creation ... Refactor the code so that everything proceeds linearly. Also renamed `getOrSet` to `setUnlessAlreadySet` to emphasise that the caller is expecting it not to be set. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2025-10-17 10:46:22 +01:00
beorn7	ad7d1aed99	Phase out native histogram feature flag ... The detailed plan for this is laid out in https://github.com/prometheus/prometheus/issues/16572 . This commit adds a global and local scrape config option `scrape_native_histograms`, which has to be set to true to ingest native histograms. To ease the transition, the feature flag is changed to simply set the default of `scrape_native_histograms` to true. Further implications: - The default scrape protocols now depend on the `scrape_native_histograms` setting. - Everywhere else, histograms are now "on by default". Documentation beyond the one for the feature flag and the scrape config are deliberately left out. See https://github.com/prometheus/prometheus/pull/17232 for that. Signed-off-by: beorn7 <beorn@grafana.com>	2025-10-15 14:50:52 +02:00
Björn Rabenstein	1caac94026	Merge pull request #17302 from prometheus/release-3.7 ... Merge release-3.7 back into main.	2025-10-07 18:40:08 +02:00