Skip to content

Commit

Permalink
Develop (#172)
Browse files Browse the repository at this point in the history
* further fixes for UI tests

* trialing removal of #dualMapMode

* add reconcile tab

* trialing ngDefaultControl

* trivy fixes

* fix up miscopy of fixed version

* further version changes required to make trivy happy

* making queries work with mysql in dev environment (fingers crossed)

* fix typo

* blindmapflag logic is in the view

* fix up equivalent logic

* allow same target code to be used twice in dual mapping

* make dual mapping work for same concept in the details view

* don't show review tab unless in dual map mode

* only add reconcile column if in dual map mode
reconcile is only a task type if in dual map mode

* colour task in drop down list

* prevent multiple reconciler columns

* further changes necessary to support reconciler column

* make reconcile task table show rows

* always display map type (not just hidden behind edit button)

* fix tests

* implement default sorting for dual map view
implement sorting for dbmapview

* fix up available status choices in reconcile task

* fix up icon for reconcile

* sort reconcile task table

* remove annoying console log

* add reconcile task type

* adding in translations for dual / single map mode indicator on table screens

* make query do a left outer join so we can see rows even when there is no second author yet

* reinstate sibiling row author now that left outer join is working again

* Undo reference to snapagogo

* fix note delete tooltip

* fix typo

* fleshing out reconcile task

* use nativeQuery to write a query to get around both mysql and hibernate quirks / incompatibilities

* sourceIndex was incorrect, is actually source id but that is not available from MapView

* missed translation from earlier commit

* id needs to come from the api, only have index locally

* fix tests

* fix issue that prevents user bubble from showing up for both entries

* finishing off reconcile task

* fix the tab that activates when a reconcile task is created

* fix limiting of author task to 2

* further fixes for the reconcile task creation / delete / update

* reviewed

* fix view so it doesn't join different maps together

* prevent "unmapped | draft | mapped | in review .." showing up when the second dual map is put into mapped mode

* remove full stops, added automatically

* upgrade mockserver

* removing entry as upgraded mock-server

* work around trivy issue with guava until mock-server updates dependency

* - create a note on each row to keep track of the dual mapper
- merge notes
- merge targets on reconcile

* fix issue where selecting no map in a dual map reconcile would not do anything due to a server side error complaining about the map row target not being found

* category for system notes
some changes to get table editing of reconcile working better

* fixing tests

* Test commit

* make map copy work for dual maps

* further changes to handle reconcile in the table

* prevent editing of reconcile table

* fix typo

* don't disable checkbox in details view

* bump version in hopes of a trivvy resolution

* don't go into mapped state if problem exist in reconcile

* fix status message logic .. was returning too early

* changing due to drop being prevented in the table

* change code to deal with reconcile state

* create a lastAuthor for a MapRowTarget so we can keep track of each target .. before the only mechanism was on the maprow which was overwritten as soon a change to any MapRowTarget was made.

* remove debug logs

* fix tests

* add missing migration

* initial commit.  Discussion to be held with how to handle the workflow of displaying / accepting replacement and if what is proposed is enough.

* getting versions up to date on develop branch

* updates to fix trivy issues

* trivy issues

* updating hapi version to get rid of okio and its trivy issue

* updating hapi version to get rid of okio and its trivy issue

* fix merge problems

* fix expected test output and copy-paste error

* implement in details view. makes sure concept is active and in scope

* sort entries in a rolegroup

* remove update inactives button from view screen

* remove code that didn't end up getting used

* fix tests broken by refactoring

* fix issue due to refactoring

* remove placeholder

* add copyright notice

* add in the dual mapping sql migration files (and move the migration file for this change) as the db migration has already happened in develop for dual mapping

* fix order

* add copyright notices

* set a default because it is the easiest way to deal with the confusion of develop having migrations from a branch that is yet to be merged to the develop branch

* change file name after adding an extra dual mapping migration

* renamed

* format and return cards to their original order

* remove code tidy as it introduced an error where selection is only possible if the row has a target

* change to formControlGroup to keep tests happy

* disable the ability to import into a dual map

* make export work for dual mapping

* make oos count update when table view contents altered.
make oos count update when entries removed from detailed view

* make sort work again for view / author / reconcile

* don't show import warning message in dual mode where you can't import a file

* update copyright date displayed in app

* fixing up sort and filter so it works for the view and map_row

* api support for new version of a dual map

* new version dual map ui changes

* changes to bulk change UI for dual mapping

* missed a required file for bulk change ui

* another missed file

* tweaks to labels

* add reconcile status info message

* info to only show for dual mapping

* tweak following feedback

* after discussion, don't reset reconcile on a new version

* ensure blind targets are not exposed in REVIEW task

* fix missing targets in certain task views

* fix subscription bugs - subscription inside callback; redundant backend requests

* remove temporary logging change from April 2022

* remove circle as looks like a radio button and not overly useful in this context anyway

* bulk change / maintenance through reconcile / prevent export of deleted comments

* fix builder issues with null

* make bulk updates work for rows in reconcile state from dual mapping

* fix up issues with selecting custom ECL

* fix issue where one author selecting no map would remove the mapping the other author had done

* add userguide link in footer

* add translate capability to test

* clear the error message in the details screen on a details screen close or next/previous

* add in HttpClientTestingModule import

* change over to the translation

* fixing issues raised by SI testing

* Fix problem with creating new version of the map where if one reconcile row was blank, it would result in the other reconcile row being duplicated

* prevent reviewer from getting inactive suggestions and point them in the right direction

* fix logic changes to allow for sibling rows to have the no map flag or target

* cater for case where two targets get moved into the same relationship

* properly fix two issues that were previously "fixed" but has introduced futher errors

* code cleanup

* stop next skipping a row in reconcile after placing a row into the mapped state

* fixing issues:
prevent edit and drag in reconcile task after changed into mapping status
only check for duplicated target if target is not null

* adding info

* Bump hashicorp/aws from 4.58.0 to 5.23.1 in /terraform

Bumps [hashicorp/aws](https://github.com/hashicorp/terraform-provider-aws) from 4.58.0 to 5.23.1.
- [Release notes](https://github.com/hashicorp/terraform-provider-aws/releases)
- [Changelog](https://github.com/hashicorp/terraform-provider-aws/blob/main/CHANGELOG.md)
- [Commits](hashicorp/terraform-provider-aws@v4.58.0...v5.23.1)

---
updated-dependencies:
- dependency-name: hashicorp/aws
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>

* Bump com.github.eirslett:frontend-maven-plugin in /ui

Bumps [com.github.eirslett:frontend-maven-plugin](https://github.com/eirslett/frontend-maven-plugin) from 1.12.1 to 1.14.2.
- [Changelog](https://github.com/eirslett/frontend-maven-plugin/blob/master/CHANGELOG.md)
- [Commits](eirslett/frontend-maven-plugin@frontend-plugins-1.12.1...frontend-plugins-1.14.2)

---
updated-dependencies:
- dependency-name: com.github.eirslett:frontend-maven-plugin
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Bump org.yaml:snakeyaml from 2.0 to 2.2 in /ui

Bumps [org.yaml:snakeyaml](https://bitbucket.org/snakeyaml/snakeyaml) from 2.0 to 2.2.
- [Commits](https://bitbucket.org/snakeyaml/snakeyaml/branches/compare/snakeyaml-2.2..snakeyaml-2.0)

---
updated-dependencies:
- dependency-name: org.yaml:snakeyaml
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Bump org.mariadb.jdbc:mariadb-java-client from 3.1.2 to 3.2.0 in /api

Bumps [org.mariadb.jdbc:mariadb-java-client](https://github.com/mariadb-corporation/mariadb-connector-j) from 3.1.2 to 3.2.0.
- [Release notes](https://github.com/mariadb-corporation/mariadb-connector-j/releases)
- [Changelog](https://github.com/mariadb-corporation/mariadb-connector-j/blob/master/CHANGELOG.md)
- [Commits](mariadb-corporation/mariadb-connector-j@3.1.2...3.2.0)

---
updated-dependencies:
- dependency-name: org.mariadb.jdbc:mariadb-java-client
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Merge branch 'develop' into feature/SNOMED-4-dual-authoring-process

* Revert "Merge branch 'develop' into feature/SNOMED-4-dual-authoring-process"

This reverts commit f1c4aeb.

* Bump sentry.version from 6.16.0 to 6.32.0 in /api

Bumps `sentry.version` from 6.16.0 to 6.32.0.

Updates `io.sentry:sentry-spring-boot-starter` from 6.16.0 to 6.32.0
- [Release notes](https://github.com/getsentry/sentry-java/releases)
- [Changelog](https://github.com/getsentry/sentry-java/blob/main/CHANGELOG.md)
- [Commits](getsentry/sentry-java@6.16.0...6.32.0)

Updates `io.sentry:sentry-logback` from 6.16.0 to 6.32.0
- [Release notes](https://github.com/getsentry/sentry-java/releases)
- [Changelog](https://github.com/getsentry/sentry-java/blob/main/CHANGELOG.md)
- [Commits](getsentry/sentry-java@6.16.0...6.32.0)

---
updated-dependencies:
- dependency-name: io.sentry:sentry-spring-boot-starter
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: io.sentry:sentry-logback
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Bump org.springframework.security:spring-security-core in /api

Bumps [org.springframework.security:spring-security-core](https://github.com/spring-projects/spring-security) from 6.0.5 to 6.1.5.
- [Release notes](https://github.com/spring-projects/spring-security/releases)
- [Changelog](https://github.com/spring-projects/spring-security/blob/main/RELEASE.adoc)
- [Commits](spring-projects/spring-security@6.0.5...6.1.5)

---
updated-dependencies:
- dependency-name: org.springframework.security:spring-security-core
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Bump com.google.guava:guava from 32.1.1-jre to 32.1.3-jre in /api

Bumps [com.google.guava:guava](https://github.com/google/guava) from 32.1.1-jre to 32.1.3-jre.
- [Release notes](https://github.com/google/guava/releases)
- [Commits](https://github.com/google/guava/commits)

---
updated-dependencies:
- dependency-name: com.google.guava:guava
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* Bump org.apache.maven.plugins:maven-resources-plugin from 3.3.0 to 3.3.1

Bumps [org.apache.maven.plugins:maven-resources-plugin](https://github.com/apache/maven-resources-plugin) from 3.3.0 to 3.3.1.
- [Release notes](https://github.com/apache/maven-resources-plugin/releases)
- [Commits](apache/maven-resources-plugin@maven-resources-plugin-3.3.0...maven-resources-plugin-3.3.1)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-resources-plugin
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* Bump software.aws.rds:aws-mysql-jdbc from 1.1.4 to 1.1.10

Bumps [software.aws.rds:aws-mysql-jdbc](https://github.com/awslabs/aws-mysql-jdbc) from 1.1.4 to 1.1.10.
- [Release notes](https://github.com/awslabs/aws-mysql-jdbc/releases)
- [Changelog](https://github.com/awslabs/aws-mysql-jdbc/blob/main/CHANGELOG.md)
- [Commits](awslabs/aws-mysql-jdbc@1.1.4...1.1.10)

---
updated-dependencies:
- dependency-name: software.aws.rds:aws-mysql-jdbc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* Bump org.apache.commons:commons-text from 1.10.0 to 1.11.0 in /api

Bumps org.apache.commons:commons-text from 1.10.0 to 1.11.0.

---
updated-dependencies:
- dependency-name: org.apache.commons:commons-text
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Bump com.google.cloud.tools:jib-maven-plugin from 3.3.1 to 3.4.0 in /api

Bumps [com.google.cloud.tools:jib-maven-plugin](https://github.com/GoogleContainerTools/jib) from 3.3.1 to 3.4.0.
- [Release notes](https://github.com/GoogleContainerTools/jib/releases)
- [Commits](https://github.com/GoogleContainerTools/jib/commits)

---
updated-dependencies:
- dependency-name: com.google.cloud.tools:jib-maven-plugin
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Bump swagger-request-validator.version from 2.34.0 to 2.38.1

Bumps `swagger-request-validator.version` from 2.34.0 to 2.38.1.

Updates `com.atlassian.oai:swagger-request-validator-springmvc` from 2.34.0 to 2.38.1
- [Changelog](https://bitbucket.org/atlassian/swagger-request-validator/src/master/RELEASE-NOTES.md)
- [Commits](https://bitbucket.org/atlassian/swagger-request-validator/branches/compare/swagger-request-validator-2.38.1..swagger-request-validator-2.34.0)

Updates `com.atlassian.oai:swagger-request-validator-mockmvc` from 2.34.0 to 2.38.1
- [Changelog](https://bitbucket.org/atlassian/swagger-request-validator/src/master/RELEASE-NOTES.md)
- [Commits](https://bitbucket.org/atlassian/swagger-request-validator/branches/compare/swagger-request-validator-2.38.1..swagger-request-validator-2.34.0)

---
updated-dependencies:
- dependency-name: com.atlassian.oai:swagger-request-validator-springmvc
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: com.atlassian.oai:swagger-request-validator-mockmvc
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* terraform commands failing in azure .. rolling back version of hashicorp/aws

* fix issue with select all not working for bulk dual mapping resets

* fix issues with bulk data dialog ..
 1) make the right statues show up for dual maps (view table)
 2) don't allow reset in table view, should be the normal dialog

* refresh out of scope count on map version change

* Bump semver from 5.7.1 to 5.7.2 in /ui/snapclient

Bumps [semver](https://github.com/npm/node-semver) from 5.7.1 to 5.7.2.
- [Release notes](https://github.com/npm/node-semver/releases)
- [Changelog](https://github.com/npm/node-semver/blob/v5.7.2/CHANGELOG.md)
- [Commits](npm/node-semver@v5.7.1...v5.7.2)

---
updated-dependencies:
- dependency-name: semver
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>

* Revert "Update README.md"

This reverts commit 936242a.

* Revert "Update pom.xml"

This reverts commit 977802a.

* Revert "Update env.prod"

This reverts commit 6590aba.

* Revert "Update pom.xml"

This reverts commit 8056dbc.

* Revert "Update azure-pipelines.yml"

This reverts commit 34d623f.

* Revert "Update pom.xml"

This reverts commit 724c43e.

* Revert "Update README.md"

This reverts commit dc3f876.

* Set DB log level to info by default

* include mapid in the view

* less calls to refresh page

* wip .. improving union performance

* revert refresh improvements until can investigate why new maps didn't work with these changes

* finishing filter and sort for dual map native query

* fix spelling mistake

* addressing trivy issue CVE-2023-6378

* returning to original version as it did not address the trivy issue .. adding to trivy ignore as no current fix

* Bump com.github.eirslett:frontend-maven-plugin in /ui

Bumps [com.github.eirslett:frontend-maven-plugin](https://github.com/eirslett/frontend-maven-plugin) from 1.14.2 to 1.15.0.
- [Changelog](https://github.com/eirslett/frontend-maven-plugin/blob/master/CHANGELOG.md)
- [Commits](eirslett/frontend-maven-plugin@frontend-plugins-1.14.2...frontend-plugins-1.15.0)

---
updated-dependencies:
- dependency-name: com.github.eirslett:frontend-maven-plugin
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* stop maps with additional columns showing up duplicated
sort numerical additional columns in a numerical way

* Add Lambda promtail zip

* updating sentry-logback to address trivy error

* updating spring-boot-starter-parent to address trivy issue with dependency tomcat-embed-core

* update problem-spring-web version to get rid of an obsolete trivy ignore

update spring-core.version as much as I can, we can't go to 6.  May need to add a trivy ignore if it still triggers.

* Revert "update problem-spring-web version to get rid of an obsolete trivy ignore"

This reverts commit 939288f.

* further addressing CVE-2023-6481 and removing obsolete fixes

* see if this is still an issue after changes for other trivy issues around logback

* still an issue, adding back to trivy ignores

* Bump software.aws.rds:aws-mysql-jdbc from 1.1.10 to 1.1.13 in /api

Bumps [software.aws.rds:aws-mysql-jdbc](https://github.com/awslabs/aws-mysql-jdbc) from 1.1.10 to 1.1.13.
- [Release notes](https://github.com/awslabs/aws-mysql-jdbc/releases)
- [Changelog](https://github.com/awslabs/aws-mysql-jdbc/blob/1.1.13/CHANGELOG.md)
- [Commits](awslabs/aws-mysql-jdbc@1.1.10...1.1.13)

---
updated-dependencies:
- dependency-name: software.aws.rds:aws-mysql-jdbc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* Bump org.apache.poi:poi-ooxml from 5.2.3 to 5.2.5

Bumps org.apache.poi:poi-ooxml from 5.2.3 to 5.2.5.

---
updated-dependencies:
- dependency-name: org.apache.poi:poi-ooxml
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* fix terraform error in deploy

* initial commit

* fixing test

* including additional column option in tests

* missed usage

* addressing trivy issues

* addressing trivy issues

* trivy issue

* fix extra identified issues

* initial addition of fix

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Deanne Ukovich <[email protected]>
Co-authored-by: Michael Lawley <[email protected]>
Co-authored-by: Attila Edelenyi <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
  • Loading branch information
5 people authored Mar 12, 2024
1 parent e85eb62 commit 2379522
Show file tree
Hide file tree
Showing 17 changed files with 171 additions and 76 deletions.
1 change: 1 addition & 0 deletions .trivyignore
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,5 @@ CVE-2022-45143
CVE-2016-1000027

# dependency of spring-boot-starter-actuator with no available fix
# still an issue 2024/01/24
CVE-2023-6378
41 changes: 23 additions & 18 deletions api/pom.xml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Copyright © 2022 SNOMED International
~ Copyright © 2022-24 SNOMED International
~
~ Licensed under the Apache License, Version 2.0 (the "License");
~ you may not use this file except in compliance with the License.
Expand Down Expand Up @@ -33,33 +33,44 @@
<java.version>17</java.version>
<docker.repository>aehrc/snap2snomed</docker.repository>
<docker.registry.host>quay.io</docker.registry.host>
<sentry.version>6.34.0</sentry.version>
<sentry.version>7.2.0</sentry.version>
<jacoco.version>0.8.8</jacoco.version>
<commons-csv.version>1.10.0</commons-csv.version>
<aws-mysql-jdbc.version>1.1.10</aws-mysql-jdbc.version>
<aws-mysql-jdbc.version>1.1.13</aws-mysql-jdbc.version>
<org.springdoc.version>1.6.15</org.springdoc.version>
<jib.version>3.3.1</jib.version>
<jib.version>3.4.0</jib.version>
<apt-maven-plugin.version>1.1.3</apt-maven-plugin.version>
<problem-spring-web.version>0.27.0</problem-spring-web.version>
<swagger-request-validator.version>2.34.0</swagger-request-validator.version>
<apache-poi.version>5.2.3</apache-poi.version>
<swagger-request-validator.version>2.38.1</swagger-request-validator.version>
<apache-poi.version>5.2.5</apache-poi.version>
<rest-assured.version>4.5.1</rest-assured.version>
<jjwt.version>0.11.5</jjwt.version>
<tc-mysql.version>1.17.2</tc-mysql.version>
<mockserver.version>5.15.0</mockserver.version>
<hamcrest-test.version>1.3</hamcrest-test.version>
<hamcrest-date.version>2.0.8</hamcrest-date.version>
<hapi.version>6.6.2</hapi.version>
<spring-security-core.version>6.1.5</spring-security-core.version>
<spring-security-core.version>6.2.2</spring-security-core.version>
<spring-core.version>5.3.29</spring-core.version>
<snomed-utilities.version>2.1.0</snomed-utilities.version>
</properties>

<dependencyManagement>
<dependencies>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-core</artifactId>
<version>1.2.13</version> <!-- force version for CVE-2023-6481 ; remove when spring-boot-starter-actuator updates to have a dependency >= 1.2.13 -->
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-web</artifactId>
<version>5.3.32</version> <!-- force version for CVE-2024-22243; remove when spring-boot-starter-web updates to this (currently 2.7.18) -->
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-text</artifactId>
<version>1.10.0</version> <!-- force version for CVE-2022-42889 ; remove when mockserver-netty updates -->
<artifactId>commons-compress</artifactId>
<version>1.26.0</version> <!-- force version for CVE-2024-25710 & CVE-2024-26308; remove when org.apache.poi:poi-ooxml updates to this (currently 5.2.3)-->
</dependency>
</dependencies>
</dependencyManagement>
Expand Down Expand Up @@ -256,7 +267,8 @@
<version>${mockserver.version}</version>
<scope>test</scope>
</dependency>
<!-- included to work around trivy issue with mock-server dependency CVE-2023-2976 in 31.1-jre .. remove when mock-server updates dependency -->
<!-- included to work around trivy issue with mock-server dependency CVE-2023-2976 in 31.1-jre .. remove when mock-server updates dependency
01/2024: still an issue -->
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
Expand Down Expand Up @@ -289,14 +301,7 @@
<dependency>
<groupId>org.ihtsdo.snomed.util</groupId>
<artifactId>snomed-utilities</artifactId>
<version>1.4.1</version>
</dependency>
<!-- included to work around snomed-utilities which includes (transitively) an older version
PR open on SNOMED International to update snomed-utilities and this can be removed -->
<dependency>
<groupId>commons-validator</groupId>
<artifactId>commons-validator</artifactId>
<version>1.7</version>
<version>${snomed-utilities.version}</version>
</dependency>

</dependencies>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ public class Snap2snomedConfiguration {
String defaultLanguage = "en";

int maximumImportedCodeSetRows = 200000;
int importBatchSize = 10000;

@URL
String userRegistrationUrl = "http://snomed.org/account-apply";
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
import javax.persistence.Column;
import javax.persistence.ElementCollection;
import javax.persistence.Entity;
import javax.persistence.FetchType;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;
Expand Down Expand Up @@ -52,6 +53,7 @@ public class ImportedCode implements Snap2SnomedEntity {

public static final int DISPLAY_SIZE_LIMIT = 2048;
public static final int CODE_SIZE_LIMIT = 50;
public static final int ADDITIONAL_COLUMN_SIZE_LIMIT = 1000;

@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
Expand Down Expand Up @@ -79,7 +81,7 @@ public class ImportedCode implements Snap2SnomedEntity {
String display;

@ReadOnlyProperty
@ElementCollection
@ElementCollection(fetch = FetchType.LAZY)
@OrderColumn(name = "collection_order")
@CollectionTable(
name="IMPORTED_CODE_ADDITIONAL_COLUMNS",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@
import org.apache.commons.csv.CSVParser;
import org.apache.commons.csv.CSVRecord;
import org.hibernate.Session;
import org.hibernate.exception.GenericJDBCException;
import org.hibernate.jdbc.Work;
import org.snomed.snap2snomed.config.Snap2snomedConfiguration;
import org.snomed.snap2snomed.controller.dto.ImportDetails;
Expand Down Expand Up @@ -117,6 +118,11 @@ public class CodeSetImportService {
private static class InsertCodeWork implements Work {

private final List<ImportedCode> codes = new ArrayList<>();
private int importBatchSize;

public InsertCodeWork(int importBatchSize) {
this.importBatchSize = importBatchSize;
}

public void add(ImportedCode code) {
codes.add(code);
Expand All @@ -138,20 +144,26 @@ public void execute(Connection connection) throws SQLException {
final PreparedStatement additionalColumnsStatement2 = connection.prepareStatement("insert into imported_code_additional_columns (imported_code_id, value, collection_order) values (?, ?, ?)");
final ResultSet generatedKeys2 = statement.getGeneratedKeys();

for (final ImportedCode code : codes) {
generatedKeys2.next();
for (int i=0; i < code.getAdditionalColumns().size(); i++) {
final String additionalColumnVal = code.getAdditionalColumns().get(i).getValue();
additionalColumnsStatement2.setLong(1, generatedKeys2.getLong(1));
additionalColumnsStatement2.setString(2, additionalColumnVal);
additionalColumnsStatement2.setInt(3, i);
additionalColumnsStatement2.addBatch();
}
int batchCount = 0;
for (final ImportedCode code : codes) {
generatedKeys2.next();
for (int i=0; i < code.getAdditionalColumns().size(); i++) {
final String additionalColumnVal = code.getAdditionalColumns().get(i).getValue();
additionalColumnsStatement2.setLong(1, generatedKeys2.getLong(1));
additionalColumnsStatement2.setString(2, additionalColumnVal);
additionalColumnsStatement2.setInt(3, i);
additionalColumnsStatement2.addBatch();
batchCount++;
}

}
//}
if (batchCount >= importBatchSize) {
additionalColumnsStatement2.executeLargeBatch();
batchCount = 0;
}

additionalColumnsStatement2.executeLargeBatch();
}

additionalColumnsStatement2.executeLargeBatch(); // flush the last few records.

}
}
Expand Down Expand Up @@ -298,7 +310,7 @@ public ImportedCodeSet importCodeSet(
final Set<String> importedCodes = new HashSet<>();
final MessageDigest md = MessageDigest.getInstance("MD5");

final InsertCodeWork insertCodeWork = new InsertCodeWork();
final InsertCodeWork insertCodeWork = new InsertCodeWork(configuration.getImportBatchSize());
try (CSVParser parser = format.parse(reader)) {
importedCodeSet = importedCodeSetRepository.save(importDetails.toImportedCodeSetEntity(parser.getHeaderNames()));

Expand Down Expand Up @@ -365,7 +377,7 @@ public ImportedCodeSet importCodeSet(
}
}

validateRecord(importedCodes, code, display, recordNumber);
validateRecord(importedCodes, code, display, recordNumber, additionalColumnValues, importDetails);

final ImportedCode importedCode = new ImportedCode(null, code, importedCodeSet, recordNumber, display, additionalColumnValues);

Expand All @@ -380,12 +392,14 @@ public ImportedCodeSet importCodeSet(
throw new CodeSetImportProblem("invalid-column", "Invalid column specfied", e.getLocalizedMessage());
} catch (final NoSuchAlgorithmException e ) {
log.error("Unecpected error creating MD5 hash for code", e);
throw Problem.valueOf(Status.INTERNAL_SERVER_ERROR, "Unecpected error creating MD5 hash for code");
throw Problem.valueOf(Status.INTERNAL_SERVER_ERROR, "Unexpected error creating MD5 hash for code");
} catch (final IOException e) {
log.error("Failed reading code set from an import request", e);
throw Problem.valueOf(Status.INTERNAL_SERVER_ERROR, "Unable to read the file in the request due to an I/O error");
} catch (IllegalStateException|IllegalArgumentException e) {
throw new CodeSetImportProblem("invalid-file", "Invalid data encountered in source file", e.getLocalizedMessage());
} catch (final GenericJDBCException e) {
throw Problem.valueOf(Status.INTERNAL_SERVER_ERROR, "JDBC Exception:" + e.getLocalizedMessage());
}
entityManager.flush();
entityManager.clear();
Expand Down Expand Up @@ -441,10 +455,12 @@ private void validateRecord(Set<String> importedCodes, String code, String targe
checkStatus(targetCode, status);
}

private void validateRecord(Set<String> importedCodes, String code, String display, long recordNumber) {
private void validateRecord(Set<String> importedCodes, String code, String display, long recordNumber, List<AdditionalCodeValue> additionalColumnValues,
ImportDetails importDetails) {
checkCode(code, recordNumber);
checkDuplicateCodes(importedCodes, code);
checkDisplay(display, recordNumber);
checkAdditionalColumn(additionalColumnValues, importDetails.getAdditionalColumnIndexes(), recordNumber);
}

private void checkCode(String code, long recordNumber) {
Expand All @@ -453,7 +469,7 @@ private void checkCode(String code, long recordNumber) {
"The code value for record " + recordNumber + " is blank in the import file");
} else if (code.length() > ImportedCode.CODE_SIZE_LIMIT) {
throw new CodeSetImportProblem("code-size", "Code in the import file is too large",
"The code value for record " + recordNumber + " is " + code.length() + " which longer than the allowed limit of "
"The code value for record " + recordNumber + " is too long: " + code.length() + " is greater than the limit of "
+ ImportedCode.CODE_SIZE_LIMIT + " " + code);
}
}
Expand All @@ -471,8 +487,7 @@ private void checkDisplay(String display, long recordNumber) {
"The display value for record " + recordNumber + " is blank in the import file");
} else if (display.length() > ImportedCode.DISPLAY_SIZE_LIMIT) {
throw new CodeSetImportProblem("display-size", "Display in the import file is too large",
"The display value for record " + recordNumber + " is " + display.length() + " which is longer than the allowed limit of "
+ ImportedCode.DISPLAY_SIZE_LIMIT);
"The display value of record " + recordNumber + " is too long: " + display.length() + " is greater than the limit of " + ImportedCode.DISPLAY_SIZE_LIMIT);
}
}

Expand All @@ -495,6 +510,19 @@ private void checkStatus(String targetCode, MapStatus status) {
}
}

private void checkAdditionalColumn(List<AdditionalCodeValue> additionalColumnValues, List<Integer> additionalColumnIndexes, long recordNumber) {
Integer additionalColumnCount = 1;
for (AdditionalCodeValue additionalColumnValue : additionalColumnValues)
{
if (additionalColumnValue.getValue().length() > ImportedCode.ADDITIONAL_COLUMN_SIZE_LIMIT) {
throw new CodeSetImportProblem("additional-column-size", "Additional Column in the import file is too large",
"Column " + (additionalColumnIndexes.get(additionalColumnCount-1)+1) + " of record " + recordNumber + " is too long: " + additionalColumnValue.getValue().length() + " is greater than the limit of "
+ ImportedCode.ADDITIONAL_COLUMN_SIZE_LIMIT);
}
additionalColumnCount++;
}
}

protected CSVFormat detectFileFormat(Character delimiter, MultipartFile file, BufferedReader reader, Integer columnIndex)
throws IOException, HttpMediaTypeNotAcceptableException {
final CSVFormat format = CSVFormat.DEFAULT;
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
/*
* Copyright © 2024 SNOMED International
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

alter table imported_code_additional_columns modify value varchar(1000);
alter table imported_code_additional_columns_aud modify value varchar(1000);
Original file line number Diff line number Diff line change
Expand Up @@ -394,7 +394,8 @@ public long createImportedCodeSet(String subject, String name, String version, i
}

public void expectCreateImportedCodeSetFail(String name, String version, int codeColumnIndex, int displayColumnIndex, boolean hasHeader,
String delimiter, File file, String fileType, int status, String errorUri)
String[] additionalColumnIndexes, String[] additionalColumnTypes,
String delimiter, File file, String fileType, int status, String errorUri)
throws JsonProcessingException {
final java.util.Map<String, Object> map = new HashMap<>();
map.put("name", name);
Expand All @@ -403,6 +404,10 @@ public void expectCreateImportedCodeSetFail(String name, String version, int cod
map.put("displayColumnIndex", displayColumnIndex);
map.put("hasHeader", hasHeader);
map.put("delimiter", delimiter);
if (null != additionalColumnIndexes) {
map.put("additionalColumnIndexes", additionalColumnIndexes);
map.put("additionalColumnTypes", additionalColumnTypes);
}

given().headers(
"Authorization", "Bearer " + createAccessToken(IntegrationTestBase.DEFAULT_TEST_USER_SUBJECT,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -226,9 +226,9 @@ public void failCreateEntityInvalidCsv() throws Exception {
*/
@Test
public void failCreateEntityInvalidTsv() throws Exception {
restClient.expectCreateImportedCodeSetFail("badAAA", "2", 0, 2, true, "\t",
restClient.expectCreateImportedCodeSetFail("badAAA", "2", 0, 2, true, null, null, "\t",
new ClassPathResource("AAA_invalid_csv_mixeddelimiters.csv").getFile(), "text/tsv", 400, null);
restClient.expectCreateImportedCodeSetFail("badAAA", "2", 0, 2, true, ",",
restClient.expectCreateImportedCodeSetFail("badAAA", "2", 0, 2, true, null, null, ",",
new ClassPathResource("AAA_invalid_csv_doublequotes.tsv").getFile(), "text/tsv", 400, null);
}

Expand Down
Loading

0 comments on commit 2379522

Please sign in to comment.