feat: allow the specification of a write window for retention policies #25517
Conversation
Add FutureWriteLimit and PastWriteLimit to retention policies. Points which are outside of now() + FutureWriteLimit or now() - PastWriteLimit will be rejected on write with a PartialWriteError. closes #25424
|
@davidby-influx It looks like if an RP has a future limit or a past limit that there is no way to remove it completely, just alter the duration. At the very least, I don't seem to see a test for removing a write limit from an RP. |
|
@gwossum - You are correct - there is not explicit removal. Write limits that are set to zero are ignored/not printed/removed. I will add a test for that. |
coordinator/points_writer.go
Outdated
| if (rp != nil) && | ||
| (((rp.FutureWriteLimit > 0) && p.Time().After(time.Now().Add(rp.FutureWriteLimit))) || | ||
| ((rp.PastWriteLimit > 0) && p.Time().Before(time.Now().Add(-rp.PastWriteLimit)))) { |
There was a problem hiding this comment.
I'm wondering if we shouldn't move this check into PointsWriter.MapShards. Having the check in here could allow us to create shards outside of the write window before we drop the points here. Moving the check into PointsWriter.MapShards would prevent the creation of shards outside the write windows.
| (((rp.FutureWriteLimit > 0) && p.Time().After(time.Now().Add(rp.FutureWriteLimit))) || | ||
| ((rp.PastWriteLimit > 0) && p.Time().Before(time.Now().Add(-rp.PastWriteLimit)))) { |
There was a problem hiding this comment.
Suggestion for future optimization: Create a writeWindow object that caches the value of time.Now().Add(rp.FutureWriteLimit) and time.Now().Add(-rp.PastWriteLimit) to avoid calculating them twice for every point in request.
|
In the release notes, we should mention the addition of InfluxQL reserved tokens for |
Add
FutureWriteLimitandPastWriteLimittoretention policies. Points which are outside of
now() + FutureWriteLimitor
now() - PastWriteLimitwill be rejected with a
PartialWriteError.Closes #25424