VictoriaMetrics already had downsampling, under another name

My home Prometheus kept fifteen days. That was not a choice, it was the disk. At a 15 second scrape interval the TSDB grew about 40 MB a day, 594 MB for the fifteen days I had, and two years of that would be 29 GB on a container with 12 GB free. So fifteen days it was, and every question about last winter’s furnace or last summer’s UPS load got the same answer: gone.

The standard fix is downsampling. Keep raw data for a month or three, then thin it to one sample every five minutes, then one an hour. Prometheus does not do it at all. Thanos and Mimir do, at the cost of running Thanos or Mimir at home. VictoriaMetrics does it, in the enterprise build. The issue asking for it in the open-source build was opened in 2019, collected 107 thumbs and 49 comments, and was closed in 2021 when the feature shipped behind a licence. People are still posting workarounds in it in 2024.

One of those workarounds gave the whole thing away.

The workaround that is the feature#

One suggestion in that thread was: run two storage groups. Send every sample to both. Give the second one a long retention and a big -dedup.minScrapeInterval. Point long-range queries at it.

-dedup.minScrapeInterval exists for a different reason. If two Prometheus instances scrape the same targets for redundancy and both write to VictoriaMetrics, you get every sample twice. The flag says: within each interval of this length, keep only the last sample. Set it to your scrape interval and the duplicates collapse.

Set it to five minutes and it is downsampling. Last sample per five-minute bucket, no aggregation, applied during background merges so the disk actually shrinks. That is exactly what the enterprise -downsampling.period=30d:5m does to data older than 30 days. The only thing dedup cannot express is the “older than 30 days” part. It uses one interval for everything.

So the question was never how to build downsampling. It was how to make one integer depend on the age of the sample.

The open-source tree agrees, quietly. partition.go defines a flag called isDedupScheduled and exports metrics named vm_downsampling_partitions_scheduled and vm_downsampling_partitions_scheduled_size_bytes. They are in every open-source binary. Nothing in the open-source binary ever sets them, so they read zero forever. The seam the enterprise build plugs into was left in place.

Where dedup runs#

Two places, and they matter differently.

The destructive one is in lib/storage/block.go. When VictoriaMetrics merges parts on disk, every block of samples goes through deduplicateSamplesDuringMerge, which asks for the global interval and drops what falls inside it. What the merge removes is gone. This is the path that reclaims disk.

The cosmetic one is in app/vmselect/netstorage/netstorage.go. Query results are deduplicated again on the way out, so a query sees thinned data even if the merge has not run yet. Nothing is deleted here. It just makes the answer consistent with what the merge will eventually produce.

Both call the same function with the same interval:

dedupInterval := GetDedupInterval()
if dedupInterval <= 0 {
    return
}

Replace GetDedupInterval() with a function of a timestamp and the rest follows.

The patch#

A new file, lib/storage/downsampling.go, parses tiers from a flag with the same syntax as the enterprise one, offset:interval,offset:interval, sorts them oldest first, and answers one question:

// dedupIntervalForTimestamp returns the dedup interval for a sample with the given timestamp:
// the interval of the matching downsampling tier, but not smaller than the global dedup interval.
func dedupIntervalForTimestamp(ts, nowMsecs int64) int64 {
    d := globalDedupInterval
    for _, dsp := range downsamplingPeriods {
        if nowMsecs-ts >= dsp.offsetMsecs {
            if dsp.intervalMsecs > d {
                d = dsp.intervalMsecs
            }
            break
        }
    }
    return d
}

The merge path now picks its interval from the newest sample in the block:

dedupInterval := dedupIntervalForTimestamp(srcTimestamps[len(srcTimestamps)-1], time.Now().UnixMilli())

Newest, not oldest, and that choice is the one piece of the patch I would defend in a review. A block that straddles a tier boundary has some samples that should be thinned to five minutes and some that should not. Thinning the whole block at the older tier’s interval would eat raw samples early. Using the newest sample’s age means a block is never thinned harder than its youngest member allows. It gets thinned properly on a later merge, once all of it has crossed the line.

The query path cannot make that compromise, because a single range query can span every tier at once. So it splits the sorted samples at each boundary and dedups each segment with its own interval:

for _, dsp := range downsamplingPeriods {
    if start >= len(srcTimestamps) {
        break
    }
    boundary := nowMsecs - dsp.offsetMsecs
    end := start + sort.Search(len(srcTimestamps)-start, func(i int) bool { return srcTimestamps[start+i] > boundary })
    if end > start {
        interval := max(dsp.intervalMsecs, dedupInterval)
        timestamps, values := DeduplicateSamples(srcTimestamps[start:end], srcValues[start:end], interval)
        dstTimestamps = append(dstTimestamps, timestamps...)
        dstValues = append(dstValues, values...)
    }
    start = end
}

DeduplicateSamples works in place, and every segment’s output lands at or before its input, so appending segment results back into the source slice never overwrites anything unread. Oldest segment first is what makes that true. I did not enjoy convincing myself of it.

Getting the disk back#

Thinning during merges only helps if merges happen. Old partitions are not being written to, so nothing merges them. VictoriaMetrics already has an answer for this too, because dedup has the same problem: if you raise -dedup.minScrapeInterval on an existing database, old data should eventually get re-deduplicated at the new interval. An hourly watcher walks the partitions, and each part’s header records the MinDedupInterval it was last merged with. If a partition’s parts were merged at a smaller interval than the current one, the watcher schedules a final merge.

That watcher is the whole disk-reclaim story for the patch. Two hunks in lib/storage/partition.go:

func (pt *partition) isFinalDedupNeeded() bool {
    dedupInterval := dedupIntervalForTimestamp(pt.tr.MaxTimestamp, time.Now().UnixMilli())
    // ...compares against the smallest MinDedupInterval stamped on the parts

ph.MinDedupInterval = dedupIntervalForTimestamp(ph.MaxTimestamp, currentTimestamp)
ph.MustWriteMetadata(dstPartPath)

A partition is a month of data. Its target interval is decided by its newest sample, so the whole month has to age past a tier boundary before the watcher touches it, and then it re-merges once at the new interval and stamps the parts. Next tier, same thing. No new goroutine, no scheduler, no bookkeeping beyond a field that was already there.

The storage format does not change. A stock binary reads the data afterwards and simply sees fewer samples.

I tested it the blunt way. Tiers 30d:5m,180d:1h, retention two years, then sixty samples a minute apart with timestamps 200 days in the past. After the merge, two remained on disk. An export returned the same two. A forced merge afterwards removed nothing more, which is the part header stamp doing its job. The first attempt exported nothing at all, because the default retention is one month and the storage had thrown the samples away on ingest. That is why the README tells you to set retention before you set tiers.

What it is not#

Last sample per bucket. Counters do not care, rate() only needs the endpoints of the window. Gauges lose whatever happened inside a bucket: the five-minute tier keeps one temperature reading out of twenty, and max_over_time of the old data is the max of the survivors. Enterprise has the same semantics, so this is not a shortcut, but it is the thing to know before you set a tier.

No per-series filters. Enterprise takes {__name__=~"node_.*"}:30d:1h and applies rules to subsets. This is one policy for everything. That bit me immediately. I have two years of five-minute weather history backfilled from Wunderground, and a 730d:1h tier would have thinned the oldest of it on the first night. So the box runs 90d:5m and no hourly tier. At five minutes forever the weather costs about a megabyte a day. Fine.

No cluster support. Single node only. And it is destructive: once a merge has run, backups taken after it hold the thinned data.

Numbers#

The patch went into production on the same container that used to hold fifteen days of Prometheus, now set to ten years. The scrape config is the old prometheus.yml byte for byte. Grafana kept its datasource UID and only the port changed. History came across with vmctl from a Prometheus snapshot, and the duplicate samples from the overlap collapsed under the 15 second dedup interval, which is what the flag was for in the first place.

Every live series still shows 5,760 samples a day at every age I checked, because the tier has not fired yet. A partition is a calendar month, and the whole month has to age past 90 days before the watcher touches it. The first month of raw data ends on July 31, so the first thinning merge is due at the end of October. Until then vm_downsampling_partitions_scheduled reads zero here too, honestly this time. The arithmetic from the measured rates says 90 days of raw is about 2 GB, and the 5 minute tier, with 11,000 active series at the 0.43 bytes a sample this data compresses to, adds about half a gigabyte a year. Ten years lands near 7 GB on a container with 10.7 GB free. I will know in November whether the arithmetic was right.

It applies to every upstream release since July, five of them so far. Three hunks move by three lines. git apply takes it without fuzz. I expected to be rebasing this every month and instead I have done nothing.

The first working version was 445 lines. Most of the difference was comments explaining the code to a reviewer who does not exist. What is left is the code.

It is a patch, not a fork: four files in a repo, apply to a checkout, copy the test in, build. Apache 2.0 like upstream, an independent reimplementation that touches no enterprise code. If you need filters or a cluster, buy the licence. If you need your home metrics to outlive the disk, 213 lines was enough.