Implementation: Automated TI <> RS tests

[scleary1cs]

Story

As a Intermediary Developer, so that I can prevent potential errors in the message flow, I need to identify potential problems in RS before they go into Production.

Pre-conditions

• We can use RS staging environment

Acceptance Criteria

• Sample messages are scheduled to be submitted to RS staging daily
• Tests are scheduled to run daily
• All tests pass on the sample messages

Tasks

We have decided to implement Solution 5.1

• Document expectations for final HL7 in order to create test assertions: documented here
• Set up senders and receives in RS staging for the full test flow (we can reuse the existing sender and receiver for the RS => TI => RS part of the flow)
  • Sender (Github Action) for first leg of the flow
    • Create org settings for sender: automated-staging-test-sender
    • Create public/private key pair for sender (use RSA 2048)
      • Ask Peter to add the private key to a Github Secret
  • Receiver (Azure Blob Storage) for second leg of the flow
    • Create org settings for receiver: automated-staging-test-receiver
    • Define jurisdictionalFilter: use receiver routing filter. Choose a unique ID for this
    • Load Azure Blob Storage credentials in RS
  • Create and merge PR in RS [PR]
• Create Azure Blob Storage account and a bucket to store final HL7 files [PR]
• Create Github Workflow, to be run daily, that will submit a request to RS. It will send all HL7 files downloaded from an Azure storage account container [PR]
• Integration test framework in TI
  • Retrieve final HL7 files from Azure container
  • Retrieve initial HL7 files from repository
  • Determine a reliable way to pair initial and final HL7 files
  • Implement Assertion Rule Engine
    • Create a definitions file with all the assertions documented
    • Implement rules engine that will load the definitions file and apply assertions to an output message (and its input message)
    • Design and implement an HL7 statement parser and evaluator that will validate the rules in the definitions file
    • Add unit tests
  • Create test that will loop through all the input/output HL7 message pairs and run the assertions using the rules engine
  • Add workflow to schedule test runs
• Add sample HL7 files to test in specified folder above. Update receiver (MSH-6.2) to match filter in RS org settings
• Add documentation in readme on how to set-up a test file in examples/Test/Automated/ and how to set-up assertions
• Add assertions for remaining transformations
  • [ ] convertToOmlOrder: when input.MSH-9.2 = 'O01', then output.MSH-9.2 = 'O21' (we don't have the capability to have conditions on input files yet. Leaving for future improvement)
  • ucsdOruUpdateReceivingFacilityWithOrderingFacilityIdentifier: ORC-21.10 = MSH-6
  • ucsdOruRemoveObservationRequests: OBR-4.1 = '54089-8'
  • ucsdOruMapLocalObservationCodes: ?
  • [ ] ucsdOruRemoveAccessionNumberObservation: UCSD ORU and OBX-3.4 = '99717-5' and OBX-3.6 = 'L'. then remove OBX (we need an assertion like OBX-3.4 != '99717-5' or OBX-3.6 != 'L', but we don't have the capability at the moment)

Definition of Done

• Documentation tasks completed
  • Documentation and diagrams created or updated
    • ADRs (/adr folder)
    • Main README.md
    • Other READMEs in the repo
    • If applicable, update the ReportStream Setup section in README.md
  • Threat model updated
  • API documentation updated
• Code quality tasks completed
  • Code refactored for clarity and no design/technical debt
  • Adhere to separation of concerns; code is not tightly coupled, especially to 3rd party dependencies
  • Code is reviewed or developed by pair; 1 approval is needed but consider requiring an outside-the-pair reviewer
  • Code quality checks passed
• Security & Privacy tasks completed
  • Security & privacy gates passed
• Testing tasks completed
  • Load tests passed
  • Unit test coverage of our code >= 90%
• Build & Deploy tasks completed
  • Build process updated
  • API(s) are versioned
  • Feature toggles created and/or deleted. Document the feature toggle
  • Source code is merged to the main branch

Research Questions

• Optional: Any initial questions for research

Decisions

• Optional: Any decisions we've made while working on this story

Notes

• Create an automated end to end test(s) that includes specific callouts to/from TI/RS while making sure SFTP ingestion functionality is accounted for.