[GEN-1468] overwrite tier1 variable #156

danlu1 · 2024-10-18T19:42:18Z

Problem:

The tier1a variables (race, sex, ethnicity, sample_type, seq_date) in BPC tables need to be replaced with values extracted from Main GENIE.

Solution:

Updated get_main_genie_clinical_sample_file to allow it more generic to pull both patient and sample release files.
Added update_tier1a and overwrite_tier1a functions to update BPC tables allowing to update for all cohorts or a specific cohort.

Testing:

Unit tests have been added.

…erwrite_tier1_variable merge changes from GEN-1516-table_update_cohort_specific branch

scripts/table_updates/Dockerfile

thomasyu888 · 2024-10-22T02:30:43Z

scripts/table_updates/config.json

@@ -19,7 +19,7 @@
        "CRC2": "syn52943210",
        "RENAL": "syn59474249"
    },
-    "main_genie_release_version": "16.6-consortium",
+    "main_genie_release_version": "17.4-consortium",


@Chelsea-Na do we want to keep this at 17.2?

Good question! If we are ready to test the output, we should test it out on 17.4-consortium. We will eventually need to also test on 17.6-consortium once its out.

Do we know what happens if a main GENIE value is missing? Or if the sample/patient is missing? Or if it adds to a log if there is a mismatch between the upload and the replaced value?

@Chelsea-Na This is a hard replacement so I didn't check GENIE value missingness. I checked with Rixing that we think all BPC samples/patients should be in Main GENIE referring to BPC project description. I'm happy to add a check for sample/patient missingness. Moreover, it doesn't log for the discrepancies between uploaded and replaced values. I'm happy to add a function for it if it's helpful.

I forgot to address your question: "Do we know what happens if a main GENIE value is missing? Or if the sample/patient is missing?
If the main GENIE sample/patient is missing, the values will be filled with NaN.
If the GENIE value is missing, then the final dataframe shows missingness since we use all main GENIE values to hard replace BPC. However, this might not be the case because no missingness has been found in target columns in the sample clinical table and patient clinical table.

@Chelsea-Na Two functions have been added to check for sample/patient missingness and log the discrepancies between uploaded and replaced values.

Let's push the version to 17.6-consortium

scripts/table_updates/update_data_table.py

scripts/table_updates/utilities.py

rxu17

Just did a first pass, had some comments

scripts/table_updates/utilities.py

scripts/table_updates/tests/test_utilities.py

scripts/table_updates/utilities.py

scripts/table_updates/tests/test_utilities.py

rxu17 · 2024-10-23T19:59:57Z

scripts/table_updates/tests/test_utilities.py

+        mock_logger.assert_not_called()
+
+
+@pytest.mark.parametrize(


nit: there's a lot of parameters here so it makes it really hard to read when using pytest.mark.parametrize (usually once I have more than 3 parameters, I'd use a different method). I'd recommend something like this

rxu17 · 2024-10-24T18:13:01Z

scripts/table_updates/utilities.py

+    """
+    # check the validity of bpc_column_list
+    valid_col = column_mapping_table.loc[column_mapping_table["prissmm_form"] == form,].prissmm_element.tolist()


Why are we using prissm_form? Could there be somewhere in the function docstring describing why we are pulling our column list for both bpc and main genie from here?

The reason is the prissmm_form matches form column in the Data Table information table. I can update the doctring.

rxu17 · 2024-10-24T18:15:20Z

scripts/table_updates/utilities.py

+        )
+    else: 
+        main_genie_table = main_genie_table[main_genie_column_list + ["SAMPLE_ID"]]


I'm not sure what the original code looked like but was there ever a handling of potential duplicates before (when we first query by cohort) and after merging here?

The original code doesn't handle duplicate.

rxu17 · 2024-10-24T18:21:25Z

scripts/table_updates/utilities.py

+            how="left",
+            left_on="cpt_genie_sample_id",
+            right_on="SAMPLE_ID",


I see Chelsea's concern here: what happens if there isn't a 1:1 merge between bpc and main genie (BPC has sample/patients not present in clinical). How did the code previously handle the merge?

The original code doesn't handle duplicate. See here: https://github.com/Sage-Bionetworks/genie-bpc-pipeline/blob/1bc58ec5c7415ba5b989dbb5a0de39b4839a1b0b/scripts/table_updates/update_data_table.py#L346C1-L357C6

…er1a replacement

scripts/table_updates/update_data_table.py

thomasyu888

🔥 Great work here! I'll defer to @rxu17 for a final review!

thomasyu888 · 2025-01-07T11:30:59Z

scripts/table_updates/config.json

        "OVARIAN": "syn64042773"
    },
-    "main_genie_release_version": "16.6-consortium",
+    "tier1a_replacement_mapping":{
+        "patient_characteristics_tier1a_replacement_mapping_table": "syn64331052",


May I ask what these tables are for?

The patient_characteristics_tier1a_replacement_mapping_table and cancer_panel_test_tier1a_replacement_mapping_table track how original bpc codes are mapped to NAACCR code. This is added to address Chelsea's request: "adds to a log if there is a mismatch between the upload and the replaced value".

rxu17

LGTM overall! Some comments/questions

scripts/table_updates/update_data_table.py

danlu1 added 2 commits October 18, 2024 07:43

add function to overwrite tier1a variables

a473710

add test script for overwrite_tier1a

1435a93

danlu1 marked this pull request as draft October 18, 2024 19:42

danlu1 added 2 commits October 18, 2024 22:06

Merge branch 'GEN-1516-table_update_cohort_specific' into GEN-1468_ov…

ca3d92a

…erwrite_tier1_variable merge changes from GEN-1516-table_update_cohort_specific branch

separate tier1a data update and table update

886d754

danlu1 marked this pull request as ready for review October 21, 2024 16:59

thomasyu888 reviewed Oct 22, 2024

View reviewed changes

scripts/table_updates/Dockerfile Outdated Show resolved Hide resolved

thomasyu888 reviewed Oct 22, 2024

View reviewed changes

scripts/table_updates/update_data_table.py Outdated Show resolved Hide resolved

thomasyu888 reviewed Oct 22, 2024

View reviewed changes

scripts/table_updates/utilities.py Outdated Show resolved Hide resolved

toggle on cohort specific custom_fix_for_tier1a_variable

39e04ee

danlu1 requested a review from a team as a code owner October 23, 2024 19:12

rxu17 reviewed Oct 23, 2024

View reviewed changes

rxu17 reviewed Oct 24, 2024

View reviewed changes

danlu1 added 3 commits December 7, 2024 00:06

add function to check join keys betweem bpc and main Genie and log ti…

fbe906c

…er1a replacement

fix typo

38592da

add function to check join keys

2396bbd

danlu1 requested a review from rxu17 January 2, 2025 19:22

thomasyu888 reviewed Jan 7, 2025

View reviewed changes

scripts/table_updates/update_data_table.py Outdated Show resolved Hide resolved

thomasyu888 reviewed Jan 7, 2025

View reviewed changes

scripts/table_updates/update_data_table.py Outdated Show resolved Hide resolved

thomasyu888 approved these changes Jan 7, 2025

View reviewed changes

thomasyu888 reviewed Jan 7, 2025

View reviewed changes

rxu17 approved these changes Jan 7, 2025

View reviewed changes

scripts/table_updates/update_data_table.py Show resolved Hide resolved

scripts/table_updates/update_data_table.py Show resolved Hide resolved

scripts/table_updates/update_data_table.py Outdated Show resolved Hide resolved

scripts/table_updates/update_data_table.py Show resolved Hide resolved

danlu1 added 5 commits January 10, 2025 01:10

use most recent main GENIE release

4d95807

remove assert, duplicate code, make bpc_column_list as param

c9472ed

moved test scripts to module folder

d94d535

add test script for update_data_table.py

e8a3a0c

wipe table before saving and add test case for mapping table

02117cc

danlu1 merged commit 95e96c4 into develop Jan 11, 2025
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[GEN-1468] overwrite tier1 variable #156

[GEN-1468] overwrite tier1 variable #156

danlu1 commented Oct 18, 2024 •

edited

Loading

thomasyu888 Oct 22, 2024

Chelsea-Na Oct 23, 2024 •

edited

Loading

danlu1 Oct 23, 2024 •

edited

Loading

danlu1 Dec 4, 2024 •

edited

Loading

danlu1 Dec 7, 2024

thomasyu888 Jan 7, 2025

rxu17 left a comment

rxu17 Oct 23, 2024

danlu1 Dec 7, 2024

rxu17 Oct 24, 2024

danlu1 Oct 24, 2024

rxu17 Oct 24, 2024 •

edited

Loading

danlu1 Dec 3, 2024

rxu17 Oct 24, 2024

danlu1 Oct 24, 2024

thomasyu888 left a comment

thomasyu888 Jan 7, 2025

danlu1 Jan 7, 2025 •

edited

Loading

rxu17 left a comment

		mock_logger.assert_not_called()


		@pytest.mark.parametrize(

[GEN-1468] overwrite tier1 variable #156

[GEN-1468] overwrite tier1 variable #156

Conversation

danlu1 commented Oct 18, 2024 • edited Loading

Choose a reason for hiding this comment

Chelsea-Na Oct 23, 2024 • edited Loading

Choose a reason for hiding this comment

danlu1 Oct 23, 2024 • edited Loading

Choose a reason for hiding this comment

danlu1 Dec 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rxu17 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rxu17 Oct 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thomasyu888 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danlu1 Jan 7, 2025 • edited Loading

Choose a reason for hiding this comment

rxu17 left a comment

Choose a reason for hiding this comment

danlu1 commented Oct 18, 2024 •

edited

Loading

Chelsea-Na Oct 23, 2024 •

edited

Loading

danlu1 Oct 23, 2024 •

edited

Loading

danlu1 Dec 4, 2024 •

edited

Loading

rxu17 Oct 24, 2024 •

edited

Loading

danlu1 Jan 7, 2025 •

edited

Loading