[RFC] Why don't we rate limit Notion imports according to their spec?

[angeloreale]

Notion integrations rate limit requests to 3 per sec, and punishes for longer.
We should queue/debounce/throttle requests within this limit, so we can import larger data sets, at a smaller timeframe (before import breaks/times-out).

Solution:

• Just memoize requests and check a counter per timestamp. If 2 within last 1000ms, wait for (1000ms- (now() - oldest))

[angeloreale]

I'll be working on those, that's why I'm not writing a full spec. This is to keep track.

[AnastasiaShemyakinskaya]

Notion integrations rate limit requests to 3 per sec, and punishes for longer. We should queue/debounce/throttle requests within this limit, so we can import larger data sets, at a smaller timeframe (before import breaks/times-out).

Solution:

• Just memoize requests and check a counter per timestamp. If 2 within last 1000ms, wait for (1000ms- (now() - oldest))

We do up to 30 requests per second. And in case we got rate-limited we wait 300 sec (returned in Retry-After HTTP header). For big accounts we are working on task to decrease the number of requests to Notion, so import of such accounts would be more faster