@recap.email didn't capture an email from PACER. Help?

• We can begin by identifying the EmailProcessingQueues related to the issue.

  To do this, we can utilize the recap-email API and perform a filter by date_created and court.

  Example:

  https://www.courtlistener.com/api/rest/v3/recap-email/?court=cod&date_created__gte=2024-04-02&date_created__lte=2024-04-03
  

  {
    "id": 63045,
    "court": "cod",
    "date_created": "2024-04-02T19:34:14.747334-07:00",
    "date_modified": "2024-04-02T19:34:14.779103-07:00",
    "message_id": "6h6oelebp8tnelo2rokq70j98qjhaepunoird201",
    "destination_emails": [
        "[email protected]"
    ],
    "filepath": null,
    "status": 4,
    "status_message": "",
    "recap_documents": []
  }
  

  If we know the sender of the recap.email, we can identify it by the destination_emails field.

  We can also obtain the message_id, which is useful for retrieving it from S3 in case we need to analyze the email body.

  From here, we can also check the status_message and the linked recap_documents. If there is no message or RDs, it indicates that the processing failed.

  • The next step is to analyze if there is something wrong with the email body and analyze other related PQ objects.

  With the message_id identified in the previous step, we can use it to locate the file in the recap.email bucket in S3 using the AWS console.

  If we were unable to identify the message_id, we can alternatively look for the file directly in the bucket. We can download the emails using an FTP client for S3 like Cyberduck and download the files around the dates the failure occurred. Then, search within the plain text files for the metadata that can help us identify the file: recap.email sender, docket_number, case_name, etc.

  If the file is not there, it indicates that the problem could be at a different level, possibly because SES didn’t receive the email.

  At this stage, we can also search for the documents using the RECAP search to confirm if the filings are in the docket and if they were created by recap.email or another source, looking at their date of creation.

  • Once we have the file, we can use Juriscraper to confirm its parsing is correct. To do this, the file should be renamed to court_id_test_number, e.g., cand_30.txt:

    • If it’s a district/bankruptcy court email, place it in juriscraper/tests/examples/pacer/nef/s3.
    • If it’s an appellate email, place it in juriscraper/tests/examples/pacer/nda.

  Then, run the tests:

  • python -m unittest -v tests.local.test_PacerNotificationEmailTest.S3PacerNotificationEmailTest
  • python -m unittest -v tests.local.test_PacerNotificationEmailTest.S3PacerNdaNotificationEmailTests

  According to the type of file, the test should generate a new JSON file, for instance, cand_30.json.

  We should verify that the JSON file contains valid content for the entries related to the notification. If it’s blank or the content is not as expected, we should fix the parsing.

  If the tests throw an error related to parsing, the parsing should also be fixed. If this is the case, the error should also be logged in Sentry.

  If we have confirmed that parsing is correct, then we can check some of the other related PQs to the notification.

• We can confirm if the mail was properly downloaded. It’s stored in a PQ, and to find it, we can use the court and the pacer_doc_id from the email file and filter them using the admin, e.g.:

https://www.courtlistener.com/admin/recap/processingqueue/?court=cod&pacer_doc_id=039011213053

• If there are no PQs, that means that the failure occurred before the main document was downloaded.

• If there is a PQ, we can check its status and whether there is a file; if so, that means that the document was properly downloaded but the process failed to create the related RECAPDocuments and assign them.

• In either case, we should look at this stage for the error in Sentry related to process_recap_email.

  We can identify the error in Sentry using the docket/document metadata, court, pacer_doc_id, docket_number, etc.

Once we find the error, we can fix it.

Finally, if the problem is solved or it was related to a transient issue, we can use the admin to select the affected EmailProcessingQueue and reprocess it:

https://www.courtlistener.com/admin/recap/emailprocessingqueue/

• If the reported error occurred in the LAMBDA function, follow these steps:

1. Navigate to the related event on Sentry.
2. Open the stack trace associated with the send_to_court_listener method and inspect the email object.
3. Locate the second item within the headers. Its value should resemble the following format:

from yyy.aaa.domain.gov (yyy.aaa.domain.gov) []) by inbound-smtp.yyyy.domain.com with SMTP id xxxxxxxxxxxxxxxxxxx for …

• Copy the message_id, which appears after the SMTP id xxxxxxxxxxxxxxxxxxx.

• Use the recap-email endpoint to filter results by court and the date the email was received, for example: https://www.courtlistener.com/api/rest/v3/recap-email/?court=cod&date_created__gte=2024-12-05&date_created__lte=2024-12-06

• In the results, search for the message_id retrieved in the previous step (xxxxxxxxxxxxxxxxxxx).

• Verify if there are related instances to the message_id and check their status. There should be at least one successful EmailProcessingQueue associated with the message_id and linked recap_documents.

  • If such a record is found, the email was properly processed.
  • If no successful EmailProcessingQueue related to the message_id is found, it will be necessary to create an EmailProcessingQueue using the metadata related to the message: court, message_id, and destination email.

• Use the admin to process it manually.