BlueButton 2 RIF Exporter #770

hadleynet · 2020-08-19T19:05:03Z

This pull request features a number changes in order to support the RIF data dictionary required for the CMS BFD project.

Related pull requests:

codecov · 2020-08-19T22:25:03Z

Codecov Report

Merging #770 (a74506b) into master (e801d58) will increase coverage by 1%.
The diff coverage is 93%.

@@            Coverage Diff             @@
##             master    #770     +/-   ##
==========================================
+ Coverage        80%     81%     +1%     
- Complexity     2886    3145    +259     
==========================================
  Files           113     116      +3     
  Lines         17459   21105   +3646     
  Branches       2474    2800    +326     
==========================================
+ Hits          14077   17279   +3202     
- Misses         2593    2985    +392     
- Partials        789     841     +52

Impacted Files	Coverage Δ
.../java/org/mitre/synthea/export/BB2RIFExporter.java	`59% <ø> (ø)`
...tre/synthea/world/concepts/ClinicianSpecialty.java	`3% <1%> (-64%)`	⬇️
...java/org/mitre/synthea/world/agents/Clinician.java	`79% <40%> (-5%)`	⬇️
...in/java/org/mitre/synthea/world/agents/Person.java	`85% <50%> (+<1%)`	⬆️
.../world/agents/behaviors/ProviderFinderNearest.java	`95% <50%> (ø)`
src/main/java/org/mitre/synthea/engine/State.java	`87% <66%> (+<1%)`	⬆️
...rc/main/java/org/mitre/synthea/helpers/Config.java	`69% <66%> (-2%)`	⬇️
...c/main/java/org/mitre/synthea/export/Exporter.java	`65% <70%> (+5%)`	⬆️
.../java/org/mitre/synthea/world/agents/Provider.java	`87% <87%> (+<1%)`	⬆️
...n/java/org/mitre/synthea/world/concepts/Claim.java	`93% <87%> (+6%)`	⬆️
... and 43 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e801d58...a74506b. Read the comment docs.

eedrummer

Some comments on the Java side of things. Still need to run things locally and check the GMF.

src/main/java/org/mitre/synthea/export/BB2RIFExporter.java

src/main/resources/modules/hospice_treatment.json

build.gradle

src/main/java/org/mitre/synthea/engine/State.java

dehall · 2021-09-27T21:31:30Z

src/main/java/org/mitre/synthea/export/BB2RIFExporter.java

+    }
+  }
+
+  private String trimToLength(String str, int maxLength) {


I think this is equivalent to org.apache.commons.lang3.StringUtils.truncate(str, maxLength) -- StringUtils is already in the repo via one or more existing dependencies so we don't have to add anything new

src/main/java/org/mitre/synthea/export/BB2RIFExporter.java

src/test/java/org/mitre/synthea/world/concepts/LossOfCareHealthRecordTest.java

eedrummer · 2021-09-28T17:00:04Z

Is there a file that is missing? I'm getting errors when I run Synthea with BFD export turned on:

java.lang.NoClassDefFoundError: Could not initialize class org.mitre.synthea.export.BB2RIFExporter$SingletonHolder
        at org.mitre.synthea.export.BB2RIFExporter.getInstance(BB2RIFExporter.java:4096)
        at org.mitre.synthea.export.Exporter.exportRecord(Exporter.java:259)
        at org.mitre.synthea.export.Exporter.export(Exporter.java:159)
        at org.mitre.synthea.engine.Generator.generatePerson(Generator.java:530)
        at org.mitre.synthea.engine.Generator.lambda$run$3(Generator.java:357)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
java.lang.ExceptionInInitializerError
        at org.mitre.synthea.export.BB2RIFExporter.getInstance(BB2RIFExporter.java:4096)
        at org.mitre.synthea.export.Exporter.exportRecord(Exporter.java:259)
        at org.mitre.synthea.export.Exporter.export(Exporter.java:159)
        at org.mitre.synthea.engine.Generator.generatePerson(Generator.java:530)
        at org.mitre.synthea.engine.Generator.lambda$run$3(Generator.java:357)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.util.MissingResourceException: Unable to read code map file: export/condition_code_map.json
        at org.mitre.synthea.export.BB2RIFExporter$CodeMapper.<init>(BB2RIFExporter.java:4130)
        at org.mitre.synthea.export.BB2RIFExporter.<init>(BB2RIFExporter.java:147)
        at org.mitre.synthea.export.BB2RIFExporter.<init>(BB2RIFExporter.java:84)
        at org.mitre.synthea.export.BB2RIFExporter$SingletonHolder.<clinit>(BB2RIFExporter.java:4087)
        ... 9 more

hadleynet · 2021-09-28T17:25:55Z

Set exporter.bfd.require_code_maps to false in your src/main/resources/synthea.properties file. Its set to true by default so that folks will see an error if they haven't installed the code mapping files. We can't check in the code mapping files because they contain codes from proprietary code systems.

eedrummer · 2021-09-28T17:35:28Z

@hadleynet - That worked. Thanks.

…kstyle warnings.

…eficiary table

…% of benes change their current plan each Jan.

…iciary together.

…F IDs (BENE_ID, MBI etc) that increment per patient or claim. This allows a subsequent run of Synthea to pick up where the previous one finished by using the -c flag to import the new starting point.

… existing file.

…or UPDATES. This avoids an issue in the RifLoader related to batching where an UPDATE could precede an INSERT.

…r the final (chronologically) updates (1 per bene) and one for all interim updates between the initial insert and the final update. Also includes some refactoring and renaming with the aim of aligning with BFD type names.

…ounter skip criteria for outpatient and inpatient files to include encounters with coded reasons even if no diagnoses or procedures are recorded for that encounter.

…re present

…d to ensure fields are supported across all appropriate file types.

…es and between spreadsheet and code

… File.

…ermination.

…istory

…rnings

dehall

Just a few minor comments that I want to submit before I lose them. Will finish reviewing tomorrow

src/main/java/org/mitre/synthea/engine/State.java

src/main/java/org/mitre/synthea/helpers/Utilities.java

src/main/java/org/mitre/synthea/world/agents/Provider.java

src/main/resources/modules/home_health_treatment.json

src/main/java/org/mitre/synthea/modules/LifecycleModule.java

dehall · 2021-12-14T01:51:58Z

src/main/java/org/mitre/synthea/export/BB2RIFExporter.java

+   * @param stopTime end time of simulation
+   * @throws IOException if something goes wrong
+   */
+  private void exportSNF(Person person, long stopTime) throws IOException {


I generated a sample 100 patient population, and in the output I had a snf.csv file that was totally empty. Is that the intended functionality? (ie, should the file be blank instead of just a header row?) I did confirm the file gets written to correctly if there are rows in it.

Do you recall what params you used? In my testing I don't see a snf.csv file being created if there are no rows (vs creating an empty file).

I tried it again a couple times and I haven't see any instances where the file is empty at the end of generation. Maybe I jumped the gun and looked at the file in the editor before generation was complete one time.

One other thing I do notice though is that if you generate one population where a file such as snf.csv is present, then a second population where nothing gets written to the file, the original file stays there. This is ok (the manifest doesn't refer to the file, and the same thing can happen with the CSV exporter if you change the included/excluded files) but it might be nice to document somehow. (Though I'm not sure where such a note would best fit)

The BB RIF exporter is designed so that you can run Synthea multiple times to build up a population. We have a script that runs Synthea for each of the states and the resulting single sets of files then contain a national population.

dehall

Overall looks really good. I focused less on the new exporter itself (as I don't expect the broader userbase to make heavy use of that) and more on the surrounding changes. Of which there are a surprising number.
There is only one change which I think is necessary, which is the default setting for the few new config items. Anything else is a suggestion/recommendation not a requirement.

src/main/java/org/mitre/synthea/world/agents/Provider.java

src/main/resources/modules/home_health_treatment.json

dehall · 2021-12-14T15:58:54Z

src/main/resources/modules/home_health_treatment.json

+      "distributed_transition": [
+        {
+          "distribution": 0.74,
+          "transition": "Physical_Therapy"
+        },
+        {
+          "distribution": 0.18,
+          "transition": "Occupational_Therapy"
+        },
+        {
+          "distribution": 0.08,
+          "transition": "Speech_Language_Therapy"


This may be too big to easily fix, but this randomized approach could lead to weird results, for example a patient could come from knee replacement but then get speech language therapy.

Not fixed, I think we should create an issue to revisit this module in a separate PR after merge.

src/main/java/org/mitre/synthea/world/concepts/Costs.java

…lue of an attribue is null

hadleynet force-pushed the bb-export branch from ff8bcaf to 427017c Compare September 3, 2020 17:46

hadleynet force-pushed the bb-export branch 3 times, most recently from 641b037 to f214bba Compare February 4, 2021 15:53

jawalonoski mentioned this pull request Apr 8, 2021

Add new encounter classes for home, hospice, and SNF. synthetichealth/module-builder#291

Merged

jawalonoski mentioned this pull request May 11, 2021

Bfd 458 test synthea data 2 CMSgov/beneficiary-fhir-data#590

Merged

5 tasks

jawalonoski mentioned this pull request Jun 24, 2021

Generating patients in Hospice care #834

Open

hadleynet force-pushed the bb-export branch from b606692 to cbe6487 Compare July 9, 2021 00:47

hadleynet force-pushed the bb-export branch from c387111 to a225518 Compare September 20, 2021 18:57

hadleynet marked this pull request as ready for review September 21, 2021 17:09

hadleynet requested review from dehall and eedrummer September 21, 2021 17:11

hadleynet mentioned this pull request Sep 21, 2021

Synthea Team Updates for BFD-458 (Test with Synthea Data) CMSgov/beneficiary-fhir-data#775

Closed

5 tasks

eedrummer reviewed Sep 23, 2021

View reviewed changes

dehall reviewed Sep 27, 2021

View reviewed changes

hadleynet and others added 11 commits December 9, 2021 16:36

Initial skeleton BB2 exporter

3164b24

Add beneficiary details. Add inpatient and outpatient files. Fix chec…

53612dc

…kstyle warnings.

Initial BB2 Inpatient claims. Required fields only.

7a7fee5

Fix broken unit tests.

c3237ab

Fix claim costs.

a85f4b3

Export BB2 race code

e9d37e8

Fix FHIR export with cost changes.

e729c5e

Add missing properties.

fdc0adf

Add reference year and age at end of reference year fields to the ben…

3e33942

…eficiary table

Initial outpatient file for bb2.

c71555d

Add carrier claims to BB2.

e6edb8b

hadleynet and others added 22 commits December 9, 2021 17:07

Address PR review comments. Adjust part d contract history so only 20…

c672335

…% of benes change their current plan each Jan.

Only generate a single beneficiary file but group rows for each benef…

ce49709

…iciary together.

Improved Medicare Part D output for RIF exporter.

32a3624

Additional missing AB2D field.

4f27c19

Export a properties file that captures the end state of all of the RI…

f6d67e4

…F IDs (BENE_ID, MBI etc) that increment per patient or claim. This allows a subsequent run of Synthea to pick up where the previous one finished by using the -c flag to import the new starting point.

Only write RIF file headers if the file is empty, otherwise append to…

6c0fe8d

… existing file.

Add new option to SetAttribute for valueAttribute.

3c3e65a

Add RIF export for FI_DOC_CLM_CNTL_NUM and CARR_CLM_CNTL_NUM fields

2bfa21c

Separate BFD bene export into two files: one for INSERTS, the other f…

3f080f8

…or UPDATES. This avoids an issue in the RifLoader related to batching where an UPDATE could precede an INSERT.

DME updates for supplies.

e503ec3

Move wellness claims from outpatient file to carrier file. Adjust enc…

02c953b

…ounter skip criteria for outpatient and inpatient files to include encounters with coded reasons even if no diagnoses or procedures are recorded for that encounter.

Only test certain bb export assertions when corresponding code maps a…

b5ce580

…re present

Assorted additional BFD fields and tweaks for consistency across files.

05764cb

Fix BFD TSV error

c3cc699

Assorted updates to align BFD field coverage spreadsheet with code an…

2175bce

…d to ensure fields are supported across all appropriate file types.

Another batch of small changes to reconcile field coverage across fil…

e2c320c

…es and between spreadsheet and code

Update cost adjustments with 2019 CMS Geographic Variation Public Use…

ac3ea24

… File.

Re-add end_stage_renal_disease attribute for Medicare eligibility det…

af8931b

…ermination.

Remove empty assign_to_attribute fields.

7bdefc5

Add betos codes to carrier and dme, add ESRD codes to bene and bene_h…

c0d0866

…istory

Post rebase updates to restore function and address new checkstyle wa…

0e597ff

…rnings

hadleynet force-pushed the bb-export branch from 16a83fe to 0e597ff Compare December 10, 2021 15:06

dehall reviewed Dec 14, 2021

View reviewed changes

hadleynet added 3 commits December 16, 2021 13:19

Address PR review comments. Rework low income indicator field.

cb92986

Undo a change that introduces slightly different behavior when the va…

65b0403

…lue of an attribue is null

Trying to track down a couple of CI-only test failures

a74506b

dehall approved these changes Dec 21, 2021

View reviewed changes

hadleynet merged commit af4a81e into master Dec 21, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BlueButton 2 RIF Exporter #770

BlueButton 2 RIF Exporter #770

hadleynet commented Aug 19, 2020 •

edited by jawalonoski

Loading

codecov bot commented Aug 19, 2020 •

edited

Loading

eedrummer left a comment

dehall Sep 27, 2021

eedrummer commented Sep 28, 2021

hadleynet commented Sep 28, 2021

eedrummer commented Sep 28, 2021

dehall left a comment

dehall Dec 14, 2021

hadleynet Dec 16, 2021

dehall Dec 16, 2021

hadleynet Dec 16, 2021

dehall left a comment

dehall Dec 14, 2021

hadleynet Dec 16, 2021

BlueButton 2 RIF Exporter #770

BlueButton 2 RIF Exporter #770

Conversation

hadleynet commented Aug 19, 2020 • edited by jawalonoski Loading

codecov bot commented Aug 19, 2020 • edited Loading

Codecov Report

eedrummer left a comment

Choose a reason for hiding this comment

dehall Sep 27, 2021

Choose a reason for hiding this comment

eedrummer commented Sep 28, 2021

hadleynet commented Sep 28, 2021

eedrummer commented Sep 28, 2021

dehall left a comment

Choose a reason for hiding this comment

dehall Dec 14, 2021

Choose a reason for hiding this comment

hadleynet Dec 16, 2021

Choose a reason for hiding this comment

dehall Dec 16, 2021

Choose a reason for hiding this comment

hadleynet Dec 16, 2021

Choose a reason for hiding this comment

dehall left a comment

Choose a reason for hiding this comment

dehall Dec 14, 2021

Choose a reason for hiding this comment

hadleynet Dec 16, 2021

Choose a reason for hiding this comment

hadleynet commented Aug 19, 2020 •

edited by jawalonoski

Loading

codecov bot commented Aug 19, 2020 •

edited

Loading