feat: implement Sliding Window Bloom Filter #715

nesty92 · 2025-01-06T16:18:30Z

Sliding Window Bloom Filter

How It Works

Core Concept

Contains two Bloom filters: current and next
Elements are added to both filters.
At the half of the window the filters are rotated current is dropped, next becomes current
this is done in Redis to warranty an atomic operation.

Redis implementation.

Key structure:
- '{name}' is the current filter.
- '{name:c}' is the current counter.
- '{name}:n' is the next filter
- '{name}:nc' is the next counter.
- '{name}:lr' is the last rotation lock to avoid multiples rotations in a given period
Scripts:
- Add: Sets bits in the current and next filters.
- Exists: Checks just bits in the current filter.
- Rotate: swaps the next filter with the current one, creating a new next filter.
Rotation Coordination:
- Clients track rotation timing locally and just trigger the rotation that will only be executed if the last rotation lock is not present

nesty92 · 2025-01-06T16:26:42Z

Hi @rueian, I was working on this and maybe is a good plus to the lib. Any feedback is more than welcome

I had to borrow some of the internal utilities for this module, but I'm not sure if that's the best way to do it.

I made some improvements for the indexes that could also be applied to existing filters, it won't improve that much the execution time because the slowest part is waiting for Redis, but it will reduce the GC pressure when the filter is on a "hot path".

rueian · 2025-01-06T16:31:40Z

Hi @nesty92, thank you. I will look into it later. @proost would you like to take a look as well?

rueidisprob/README.md

rueidisprob/slidingbloomfilter_retry.go

proost

Clever Idea!

rueidisprob/slidingbloomfilter.go

proost

LGTM.

I just wandering:

there are partitioned and discrete bloom filter idea for unbounded input. your choice is overlapping and sliding window bloom filter. do you think partitioned and discrete bloom filter is too expensive?

rueidisprob/slidingbloomfilter.go

nesty92 · 2025-01-08T12:22:09Z

Hi @rueian @proost, Following the idea of moving the rotation to redis, i found that some times we get a weird behavior with the expirations.

In the test case the PX is 500 but when running it on multiples test Redis says that the pttl is ~6s, see the video

Cursor_tIzIte8ahI.mp4

1736337047.130384 [0 lua] "SET" "{test}:lr" "1736337047130" "PX" "500" "NX"
1736337047.633866 [0 lua] "SET" "{test}:lr" "1736337047633" "PX" "500" "NX"
1736337048.134485 [0 lua] "SET" "{test}:lr" "1736337048134" "PX" "500" "NX"
1736337048.145867 [0 lua] "SET" "{test}:lr" "1736337048145" "PX" "500" "NX"
1736337054.405496 [0 lua] "SET" "{test}:lr" "1736337054405" "PX" "500" "NX"
1736337059.410190 [0 lua] "SET" "{test}:lr" "1736337059410" "PX" "500" "NX"
1736337057.153281 [0 lua] "SET" "{test}:lr" "1736337057153" "PX" "500" "NX"
1736337057.657655 [0 lua] "SET" "{test}:lr" "1736337057657" "PX" "500" "NX"
1736337058.161746 [0 lua] "SET" "{test}:lr" "1736337058161" "PX" "500" "NX"
1736337058.171059 [0 lua] "SET" "{test}:lr" "1736337058170" "PX" "500" "NX"
1736337064.421463 [0 lua] "SET" "{test}:lr" "1736337064421" "PX" "500" "NX"
1736337069.431727 [0 lua] "SET" "{test}:lr" "1736337069431" "PX" "500" "NX"
1736337069.448439 [0 lua] "SET" "{test}:lr" "1736337069448" "PX" "500" "NX"
1736337074.444494 [0 lua] "SET" "{test}:lr" "1736337074444" "PX" "500" "NX"
1736337072.370417 [0 lua] "SET" "{test}:lr" "1736337072370" "PX" "500" "NX"
1736337072.870308 [0 lua] "SET" "{test}:lr" "1736337072870" "PX" "500" "NX"
1736337072.872326 [0 lua] "SET" "{test}:lr" "1736337072872" "PX" "500" "NX"
1736337079.452635 [0 lua] "SET" "{test}:lr" "1736337079452" "PX" "500" "NX"
1736337079.467039 [0 lua] "SET" "{test}:lr" "1736337079466" "PX" "500" "NX"
1736337084.460596 [0 lua] "SET" "{test}:lr" "1736337084460" "PX" "500" "NX"
1736337082.378464 [0 lua] "SET" "{test}:lr" "1736337082378" "PX" "500" "NX"
1736337082.878354 [0 lua] "SET" "{test}:lr" "1736337082878" "PX" "500" "NX"
1736337082.881565 [0 lua] "SET" "{test}:lr" "1736337082881" "PX" "500" "NX"
1736337089.463478 [0 lua] "SET" "{test}:lr" "1736337089463" "PX" "500" "NX"
1736337089.477189 [0 lua] "SET" "{test}:lr" "1736337089477" "PX" "500" "NX"
1736337094.470253 [0 lua] "SET" "{test}:lr" "1736337094470" "PX" "500" "NX"
1736337092.386336 [0 lua] "SET" "{test}:lr" "1736337092386" "PX" "500" "NX"
1736337092.886976 [0 lua] "SET" "{test}:lr" "1736337092886" "PX" "500" "NX"
1736337092.887048 [0 lua] "SET" "{test}:lr" "1736337092887" "PX" "500" "NX"
1736337099.481321 [0 lua] "SET" "{test}:lr" "1736337099481" "PX" "500" "NX"

Do you have any idea why this happens?

After running it several's times I can't reproduce it now but from time to time I saw ~1s of pttl.

nesty92 · 2025-01-08T14:53:08Z

LGTM.

I just wandering:

there are partitioned and discrete bloom filter idea for unbounded input. your choice is overlapping and sliding window bloom filter. do you think partitioned and discrete bloom filter is too expensive?

@proost The first implementation that I did was a partitioned filter but partitioned filters can be slower if you need to query multiple partitions to get a result. Then i found this idea of overlapping that is simpler and you only have check the current filter

proost

Hmm, that is very weird operation. I read code again, but i don't know how can happen.

rueidisprob/slidingbloomfilter.go

proost

I found that code is changed. is weird pttl phenomenon gone? i guess same situation happen.

rueidisprob/slidingbloomfilter.go

nesty92 · 2025-01-08T15:45:32Z

I found that code is changed. is weird pttl phenomenon gone? i guess same situation happen.

@proost, It continue from time to time, even try with redis-benchmark to discard any possible error from our side and still, I'll prepare a issue on Redis to get more info about it

This feature enhances the existing Bloom Filter capabilities by allowing time-based item tracking, making it suitable for use cases requiring temporary membership checks. Signed-off-by: Ernesto Alejandro Santana Hidalgo <[email protected]>

nesty92 · 2025-01-08T16:22:56Z

issue on Redis to get more info about it

redis/redis#13733

rueian · 2025-01-10T23:38:14Z

Do you have any idea why this happens?

Hi @nesty92, I can't reproduce the issue with valkey 8. These are the commands I used:

watch -n0 ./valkey-cli PTTL a
watch -n0 ./valkey-cli SET a 1 PX 500 NX

Which version of Redis are you using? But, anyway, I think we can merge this already.

nesty92 · 2025-01-11T07:30:23Z

Hi @rueian , I was using 7.4 that is the same in the cluster used for testing. Not sure yet why this behavior but there's nothing to do from our side.

I will try with valkey later

rueian · 2025-01-11T12:14:18Z

rueidisprob/slidingbloomfilter.go

+if acquiredLock then
+	redis.call('RENAME', nextFilterKey, filterKey)
+	redis.call('RENAME', nextCounterKey, counterKey)
+	redis.call('SET', nextFilterKey, "")


Here the script always rotate the filters once if it acquires the lock. But shouldn’t it rotate the filters twice if we access it after idling for a while (> 2*windowHalf)?

You're absolutely correct. If more than half of the window has passed since the last rotation, it means that both the current and next filters are no longer valid since they no longer contain relevant data. In this case, saving the last rotation and reseting the filter if to much time as passed will solve it

nesty92 force-pushed the main branch from c1f6dbd to 6af0e70 Compare January 6, 2025 16:20

rueian reviewed Jan 6, 2025

View reviewed changes

rueidisprob/README.md Outdated Show resolved Hide resolved

rueidisprob/slidingbloomfilter_retry.go Outdated Show resolved Hide resolved

nesty92 force-pushed the main branch 5 times, most recently from def7e26 to 04268ac Compare January 6, 2025 16:55

proost reviewed Jan 7, 2025

View reviewed changes

rueidisprob/slidingbloomfilter.go Outdated Show resolved Hide resolved

nesty92 force-pushed the main branch 2 times, most recently from 0014c39 to a976b89 Compare January 7, 2025 12:41

nesty92 commented Jan 7, 2025

View reviewed changes

rueidisprob/slidingbloomfilter.go Outdated Show resolved Hide resolved

proost approved these changes Jan 7, 2025

View reviewed changes

rueian reviewed Jan 7, 2025

View reviewed changes

rueidisprob/slidingbloomfilter.go Outdated Show resolved Hide resolved

proost reviewed Jan 8, 2025

View reviewed changes

rueidisprob/slidingbloomfilter.go Outdated Show resolved Hide resolved

nesty92 force-pushed the main branch from d265a05 to 169ae16 Compare January 8, 2025 15:34

proost reviewed Jan 8, 2025

View reviewed changes

rueidisprob/slidingbloomfilter.go Outdated Show resolved Hide resolved

feat: implement Sliding Window Bloom Filter

a0707b1

This feature enhances the existing Bloom Filter capabilities by allowing time-based item tracking, making it suitable for use cases requiring temporary membership checks. Signed-off-by: Ernesto Alejandro Santana Hidalgo <[email protected]>

nesty92 force-pushed the main branch from 169ae16 to a0707b1 Compare January 8, 2025 15:45

rueian reviewed Jan 11, 2025

View reviewed changes

rueian merged commit 09ad427 into redis:main Jan 11, 2025
30 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: implement Sliding Window Bloom Filter #715

feat: implement Sliding Window Bloom Filter #715

nesty92 commented Jan 6, 2025

nesty92 commented Jan 6, 2025

rueian commented Jan 6, 2025

proost left a comment

proost left a comment •

edited

Loading

nesty92 commented Jan 8, 2025 •

edited

Loading

nesty92 commented Jan 8, 2025

proost left a comment •

edited

Loading

proost left a comment

nesty92 commented Jan 8, 2025

nesty92 commented Jan 8, 2025

rueian commented Jan 10, 2025

nesty92 commented Jan 11, 2025 •

edited

Loading

rueian Jan 11, 2025

nesty92 Jan 11, 2025 •

edited

Loading

feat: implement Sliding Window Bloom Filter #715

feat: implement Sliding Window Bloom Filter #715

Conversation

nesty92 commented Jan 6, 2025

Sliding Window Bloom Filter

How It Works

Core Concept

Redis implementation.

nesty92 commented Jan 6, 2025

rueian commented Jan 6, 2025

proost left a comment

Choose a reason for hiding this comment

proost left a comment • edited Loading

Choose a reason for hiding this comment

nesty92 commented Jan 8, 2025 • edited Loading

nesty92 commented Jan 8, 2025

proost left a comment • edited Loading

Choose a reason for hiding this comment

proost left a comment

Choose a reason for hiding this comment

nesty92 commented Jan 8, 2025

nesty92 commented Jan 8, 2025

rueian commented Jan 10, 2025

nesty92 commented Jan 11, 2025 • edited Loading

rueian Jan 11, 2025

Choose a reason for hiding this comment

nesty92 Jan 11, 2025 • edited Loading

Choose a reason for hiding this comment

proost left a comment •

edited

Loading

nesty92 commented Jan 8, 2025 •

edited

Loading

proost left a comment •

edited

Loading

nesty92 commented Jan 11, 2025 •

edited

Loading

nesty92 Jan 11, 2025 •

edited

Loading