CDMS-144 handle unmatched CHED

README.md

@@ -7,22 +7,23 @@ Core delivery C# ASP.NET backend template.
 * [Testing](#testing)
 * [Running](#running)
 * [Dependabot](#dependabot)
+* [Test Data](#test-data)
 
-### Install MongoDB
+# Install MongoDB
 - Install [MongoDB](https://www.mongodb.com/docs/manual/tutorial/#installation) on your local machine
 - Start MongoDB:
 ```bash
 sudo mongod --dbpath ~/mongodb-cdp
 ```
 
-### Inspect MongoDB
+# Inspect MongoDB
 
 To inspect the Database and Collections locally:
 ```bash
 mongosh
 ```
 
-### Testing
+# Testing
 
 Run the tests with:
 
@@ -33,18 +34,126 @@ Tests do not use mocking of any sort and read and write from the in-memory datab
 dotnet test
 ````
 
-### Running
+# Running
 
 Run CDP-Deployments application:
 ```bash
-dotnet run --project Btms.Backend --launch-profile Development
+dotnet run --project Btms.Backend --launch-profile Btms.Backend
 ```
 
-### SonarCloud
+# SonarCloud
 
 Example SonarCloud configuration are available in the GitHub Action workflows.
 
-### Dependabot
+# Dependabot
 
 We have added an example dependabot configuration file to the repository. You can enable it by renaming
 the [.github/example.dependabot.yml](.github/example.dependabot.yml) to `.github/dependabot.yml`
+
+# Test data
+We are able to obtain test data to use in our tests a few ways.
+
+## Canned test data
+This is data that we created by hand based on examples of messages that we had obtained. Canned test data can be found in [`Btms.Backend.IntegrationTests/Fixtures/SmokeTest`](Btms.Backend.IntegrationTests/Fixtures/SmokeTest). In there you will find relevant folders for the messages that you want to simulate:
+* ALVS - Custom Record notifications from CDS
+* DECISIONS - Decision that is made by BTMS.
+* GVMSAPIRESPONSE - ??
+* IPAFFS - CHED notifications from IPAFFS
+  * CHEDA
+  * CHEDD
+  * CHEDP
+  * CHEDPP
+
+## Using the Test Data Generator
+The Test Data Generator can be found in the `tools` project ([`tools/TestDataGenerator`](TestDataGenerator/TestDataGenerator.csproj)). The test data is generated based on specifications provided in a scenario e.g. [`ChedASimpleMatchScenarioGenerator.cs`](TestDataGenerator/Scenarios/ChedASimpleMatchScenarioGenerator.cs). A scenario should container at least a `GetNotificationBuilder` or `GetClearanceRequestBuilder`.
+
+Example usage of `GetNotificationBuilder`
+```csharp
+var notification = GetNotificationBuilder("cheda-one-commodity")
+    .WithCreationDate(entryDate)
+    .WithRandomArrivalDateTime(config.ArrivalDateRange)
+    .WithReferenceNumber(ImportNotificationTypeEnum.Cveda, scenario, entryDate, item)
+    .ValidateAndBuild();
+```
+
+Example usage of `GetClearanceRequestBuilder`
+```csharp
+var clearanceRequest = GetClearanceRequestBuilder("cr-one-item")
+    .WithCreationDate(entryDate)
+    .WithArrivalDateTimeOffset(notification.PartOne!.ArrivalDate, notification.PartOne!.ArrivalTime)
+    .WithReferenceNumber(notification.ReferenceNumber!)
+    .ValidateAndBuild();
+```
+Note: 
+* Both the Notification Builder and Clearance Request Builder both take a sample file which it uses as a basis to create the test data. The sample file is located in [`Scenarios/Samples`](TestDataGenerator/Scenarios/Samples). 
+
+After creating your scenario your will need to add it to `ConfigureTestGenerationServices` in [`BuilderExtensions.cs`](TestDataGenerator/Helpers/BuilderExtensions.cs). 
+
+Next, you will need to create a dataset that's specified in [`Program.cs`](TestDataGenerator/Program.cs).
+Example dataset:
+```csharp
+ var datasets = new[]
+    {
+        new
+        {
+            Dataset = "All-CHED-No-Match",
+            RootPath = "GENERATED-ALL-CHED-NO-MATCH",
+            Scenarios = new[] { app.CreateScenarioConfig<AllChedsNoMatchScenarioGenerator>(1, 1) }
+        },
+        ...
+    }
+```
+
+* Dataset - Name of the dataset 
+* RootPath - Folder where the data will be created in. The folder will be in [`TestDataGenerator/.test-data-generator`](TestDataGenerator/.test-data-generator). 
+* Scenarios - List of scenarios to create test data for. The CreateScenarioConfig generates scenarios based on the Scenario types from the [`Scenarios`](TestDataGenerator/Scenarios) folder.
+
+And finally, in order to trigger the data creation you will need to add some configuration to [`Properties/launchSettings.json`](TestDataGenerator/Properties/launchSettings.json):
+```json
+{
+    "profiles": {
+        "Generate All CHED no match": {
+            "commandName": "Project",
+            "commandLineArgs": "All-CHED-No-Match",
+            "environmentVariables": {
+                "DMP_ENVIRONMENT": "dev",
+                "DMP_SERVICE_BUS_NAME": "DEVTREINFSB1001",
+                "DMP_BLOB_STORAGE_NAME": "devdmpinfdl1001",
+                "DMP_SLOT": "1003",
+                "AZURE_TENANT_ID": "c9d74090-b4e6-4b04-981d-e6757a160812"
+            }
+        }
+    }
+}
+```
+
+* Give the profile a name. This can be free text.
+* commandLineArgs - The name of the new dataset.
+* The rest of the configuration can be copied from the other profiles.
+
+### Creating new sample file
+You may want to a new sample if you are creating data for a new scenario. To do this:
+* Place the new sample file with the relevant JSON in the [`Samples`](TestDataGenerator/Scenarios/Samples) folder.
+* Change the Properties of the file in Rider:
+  * Build action: **Content**
+  * Copy to output directory: **Copy if newer**
+
+## Using data from Blob Storage
+We have imported redacted production data that is stored in a Blob Storage. We can use BTMS Backend to import this data.
+
+* Update [`local.env`](Btms.Backend/Properties/local.env):
+  * For one day dataset - `BusinessOptions:DmpBlobRootFolder=PRODREDACTED-20241204`
+  * For one month dataset - `BusinessOptions:DmpBlobRootFolder=PRODREDACTED-202411`
+  * When importing the data set the following:
+    ```dotenv
+    BlobServiceOptions:CacheWriteEnabled=true
+    # BlobServiceOptions:CacheReadEnabled=true
+    ```
+  * After importing update the config to be the following so the data doesn't get imported again when you call the `initialise` API (http://0.0.0.0:5002/mgmt/initialise?syncPeriod=All)
+    ```dotenv
+    # BlobServiceOptions:CacheWriteEnabled=true
+    BlobServiceOptions:CacheReadEnabled=true
+    ```
+  * Once the config has been updated, start BTMS Backend and call the `initialise` API (http://0.0.0.0:5002/mgmt/initialise?syncPeriod=All). Note that a large amount of data will be loaded, particularly the full month dataset. It is also advisable to run Backend from a standalone terminal rather than from Rider as it struggles running this task.
+
+

TestDataGenerator/Config/Datasets.cs

@@ -25,7 +25,8 @@ public static Dataset[] GetDatasets(IHost app)
             ds.LoadTest,
             ds.LoadTest90Dx1,
             ds.LoadTestCondensed,
-            ds.LoadTest90Dx10k
+            ds.LoadTest90Dx10k,
+            ds.AllChedNoMatch
         ];
     }
 
@@ -118,6 +119,14 @@ public static Dataset[] GetDatasets(IHost app)
             }
     };
 
+    public readonly Dataset AllChedNoMatch = new()
+    {
+        Name = "All-CHED-No-Match",
+        Description = "LIM TODO",
+        RootPath = "GENERATED-ALL-CHED-NO-MATCH",
+        Scenarios = new[] { app.CreateScenarioConfig<AllChedsNoMatchScenarioGenerator>(1, 1) }
+    };
+
     // public readonly Dataset Pha = new()
     // {
     //     Name = "PHA",

TestDataGenerator/Extensions/BuilderExtensions.cs

@@ -23,6 +23,7 @@ public static IServiceCollection ConfigureTestGenerationServices(this IServiceCo
         services.AddSingleton<ChedANoMatchScenarioGenerator>();
         services.AddSingleton<CrNoMatchScenarioGenerator>();
         services.AddSingleton<ChedPMultiStepScenarioGenerator>();
+        services.AddSingleton<AllChedsNoMatchScenarioGenerator>();
 
         var blobOptionsValidatorDescriptor = services.Where(d => 
             d.ServiceType == typeof(IValidateOptions<BlobServiceOptions>));

TestDataGenerator/Properties/launchSettings.json

@@ -1,6 +1,17 @@
 {
   "$schema": "https://json.schemastore.org/launchsettings.json",
   "profiles": {
+    "Generate All CHED no match": {
+      "commandName": "Project",
+      "commandLineArgs": "All-CHED-No-Match",
+      "environmentVariables": {
+        "DMP_ENVIRONMENT": "dev",
+        "DMP_SERVICE_BUS_NAME": "DEVTREINFSB1001",
+        "DMP_BLOB_STORAGE_NAME": "devdmpinfdl1001",
+        "DMP_SLOT": "1003",
+        "AZURE_TENANT_ID": "c9d74090-b4e6-4b04-981d-e6757a160812"
+      }
+    },
     "Generate EndToEnd-IBM": {
       "commandName": "Project",
       "commandLineArgs": "EndToEnd-IBM",

TestDataGenerator/Scenarios/AllChedsNoMatchScenarioGenerator.cs

@@ -0,0 +1,48 @@
+using Btms.Types.Ipaffs;
+using Microsoft.Extensions.Logging;
+
+namespace TestDataGenerator.Scenarios;
+
+public class AllChedsNoMatchScenarioGenerator(ILogger<AllChedsNoMatchScenarioGenerator> logger) : ScenarioGenerator
+{
+    public override GeneratorResult Generate(int scenario, int item, DateTime entryDate, ScenarioConfig config)
+    {
+        var chedANotification = GetNotificationBuilder("cheda-one-commodity")
+            .WithCreationDate(entryDate)
+            .WithRandomArrivalDateTime(config.ArrivalDateRange)
+            .WithReferenceNumber(ImportNotificationTypeEnum.Cveda, scenario, entryDate, item)
+            .ValidateAndBuild();
+
+        logger.LogInformation("Created {NotificationReferenceNumber}", 
+            chedANotification.ReferenceNumber);
+
+        var chedPNotification = GetNotificationBuilder("chedp-one-commodity")
+            .WithCreationDate(entryDate)
+            .WithRandomArrivalDateTime(config.ArrivalDateRange)
+            .WithReferenceNumber(ImportNotificationTypeEnum.Cvedp, scenario, entryDate, item)
+            .ValidateAndBuild();
+
+        logger.LogInformation("Created {NotificationReferenceNumber}", 
+            chedPNotification.ReferenceNumber);
+
+        var chedDNotification = GetNotificationBuilder("chedd-one-commodity")
+            .WithCreationDate(entryDate)
+            .WithRandomArrivalDateTime(config.ArrivalDateRange)
+            .WithReferenceNumber(ImportNotificationTypeEnum.Ced, scenario, entryDate, item)
+            .ValidateAndBuild();
+
+        logger.LogInformation("Created {NotificationReferenceNumber}", 
+            chedDNotification.ReferenceNumber);
+
+        var chedPPNotification = GetNotificationBuilder("chedpp-multiple-commodity")
+            .WithCreationDate(entryDate)
+            .WithRandomArrivalDateTime(config.ArrivalDateRange)
+            .WithReferenceNumber(ImportNotificationTypeEnum.Chedpp, scenario, entryDate, item)
+            .ValidateAndBuild();
+
+        logger.LogInformation("Created {NotificationReferenceNumber}", 
+            chedPPNotification.ReferenceNumber);
+
+        return new GeneratorResult([chedANotification, chedPNotification, chedDNotification, chedPPNotification]);
+    }
+}

TestDataGenerator/Scenarios/ChedANoMatchScenarioGenerator.cs

@@ -13,7 +13,7 @@ public override GeneratorResult Generate(int scenario, int item, DateTime entryD
             .WithReferenceNumber(ImportNotificationTypeEnum.Cveda, scenario, entryDate, item)
             .WithVersionNumber()
             .ValidateAndBuild();
-
+        
         logger.LogInformation("Created {NotificationReferenceNumber}", 
             notification.ReferenceNumber);