Add PollingAlertCheckWorker to copy with 429s blocking Notify callbacks

[KludgeKML]

The problem of blocks of 429s sometimes being returned when notify attempts to make callbacks for delivered emails means that we're at risk of getting false alarms for emails not being sent when they actually have been, but the callback has failed. As a fallback, we add in an active polling step.

• If a single positive callback has been recorded, we act as normal (no alert).
• If no positive callbacks have been records, we start actively polling Notify with the IDs (reference ids in Notify parlance) of emails that we've had no callback for, querying whether they have been received. We expect in most cases this will result in one or two calls to Notify before a positive delivery is found, although the worst case (no emails delivered at all) would result in us asking Notify about every single email.

https://trello.com/c/KR2THVrN/2323-add-active-polling-to-mitigate-false-alarms-from-429-responses

⚠️ This repo is Continuously Deployed: make sure you follow the guidance ⚠️

[1pretz1]

Excellent work on this Keith! Overall everything looks great, just left a few comments.

We should also remove AlertCheckWorker, the spec.

Also need to update the testing documentation, I don't think the current method would suffice anymore as the canary account will receive the email and Notify should respond with the email delivery. I think we could probably achieve the same test by just deleting all the emails vs marking the Notify status as nil.

[1pretz1]

Changes look good, thanks for working through the comments! Left one minor suggestion but non blocking so approving 🍏

[1pretz1]

We could probably clean up the code slightly here:

        email3 = create(:email, content_id:, notify_status: nil, status: "sent")
        request = stub_notify_email_delivery_status_delivered(email3.id)
        perform
        expect(request).not_to have_been_made

[KludgeKML]

Done