Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tdb/elife_upload: Update assay_date parsing #146

Draft
wants to merge 5 commits into
base: master
Choose a base branch
from
Draft

tdb/elife_upload: Update assay_date parsing #146

Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
23a3248
tdb/elife_upload: Refactor assay_date parsing
joverlee521 Nov 15, 2023
37f9f3b
tdb/elife_upload: Only parse assay_date once
joverlee521 Nov 15, 2023
124508a
tdb/elife_upload: Update `parse_assay_date_from_filename`
joverlee521 Nov 15, 2023
e510854
tdb/elife_upload: Only use fstem assay date if record does not have one
joverlee521 Nov 15, 2023
c3cbfed
tdb/elife_upload: Raise Exception if filename contains multiple dates
joverlee521 Nov 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 38 additions & 12 deletions tdb/elife_upload.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ def format_measurements(self, measurements, **kwargs):
self.HI_ref_name_abbrev =self.define_strain_fixes(self.HI_ref_name_abbrev_fname)
self.define_location_label_fixes("source-data/flu_fix_location_label.tsv")
self.define_countries("source-data/geo_synonyms.tsv")
fstem_assay_date = self.parse_assay_date_from_filename(kwargs['fstem'])
for meas in measurements:
meas['virus_strain'], meas['original_virus_strain'] = self.fix_name(self.HI_fix_name(meas['virus_strain'], serum=False))
meas['serum_strain'], meas['original_serum_strain'] = self.fix_name(self.HI_fix_name(meas['serum_strain'], serum=True))
Expand All @@ -49,18 +50,8 @@ def format_measurements(self, measurements, **kwargs):
self.format_subtype(meas)
self.format_assay_type(meas)
self.format_date(meas)
tmp = kwargs['fstem'].split('-')[0]
if len(tmp) > 8:
tmp = tmp[:(8-len(tmp))]
elif len(tmp) < 8:
meas['assay_date'] = "XXXX-XX-XX"
else:
if tmp[0:2] == '20':
meas['assay_date'] = "{}-{}-{}".format(tmp[0:4],tmp[4:6],tmp[6:8])
else:
meas['assay_date'] = "XXXX-XX-XX"
if 'assay_date' not in meas.keys() or meas['assay_date'] is None:
meas['assay_date'] = "XXXX-XX-XX"
if meas.get('assay_date') is None:
meas['assay_date'] = fstem_assay_date
self.format_passage(meas, 'serum_passage', 'serum_passage_category')
self.format_passage(meas, 'virus_passage', 'virus_passage_category')
self.format_ref(meas)
Expand All @@ -78,6 +69,41 @@ def format_measurements(self, measurements, **kwargs):
self.disambiguate_sources(measurements)
return measurements


def parse_assay_date_from_filename(self, fstem):
"""
Parse assay date from the *fstem*.
*fstem* is expected to be formatted as `YYYYMMDD*`

If unable to parse date from *fstem*, then return masked assay date as `XXXX-XX-XX`.
If there are multiple valid dates, then return the latest date.
"""
assay_date = "XXXX-XX-XX"
valid_dates = set()
for potential_date in re.findall(r"\d{8}", fstem):
# Check if the dates are valid
try:
date = datetime.datetime.strptime(potential_date, '%Y%m%d')
except ValueError:
continue

# Date is only a valid assay date if it's earlier than the current datetime!
if date < datetime.datetime.now():
valid_dates.add(date)

if len(valid_dates) == 0:
print(f"Failed to parse assay date from filename {fstem!r}")
elif len(valid_dates) == 1:
assay_date = datetime.datetime.strftime(valid_dates.pop(), '%Y-%m-%d')
else:
sorted_dates = list(map(lambda x: datetime.datetime.strftime(x, '%Y-%m-%d'), sorted(valid_dates)))
assay_date = sorted_dates[-1]
print(f"Found multiple potential assay dates in filename {fstem!r}: {sorted_dates}.",
f"Using the last valid date as the assay date: {assay_date!r}")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this case ever happen? Are we sure that we want to pick the latest date as the assay date when we do find multiple dates or do we want to throw an error so the user has a chance to fix the file name before the upload happens?

As a minor aesthetic note, you could also sort the dates with a list comprehension to avoid using list, map, and lambda:

sorted_dates = [datetime.datetime.strftime(valid_date, '%Y-%m-%d') for valid_date in sorted(valid_dates)]

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this case ever happen? Are we sure that we want to pick the latest date as the assay date when we do find multiple dates or do we want to throw an error so the user has a chance to fix the file name before the upload happens?

Not yet 😄 You're right that it would probably be better to throw an error here and manually fix the filename. I was thinking we would want the latest date if the file had an updated date appended, but better to be explicit about having one date in the filename.


return assay_date


def disambiguate_sources(self, measurements):
'''
Add counter to sources so that create_index still creates unique identifiers for each
Expand Down