Duplicate Files in Migrated Items

[sec122]

Duplicate Files in Migrated Items

Expected behavior

Only one copy of each file should be present in migrated objects.

Actual behavior

Some files are duplicated in migrated datasets, easily noticed by filenames with a prefix of either "dataspace" or "globus"

These duplicated file cases fall into three groups:
a) Only the README files are duplicated - 22 cases
b) All files are duplicated (and there are TAR files) - 6 cases In a separate ticket #1920
^ Matt has more info about how we want to handle these cases
c) All files are duplicated (and there are no TAR files) - 1 case

Steps to replicate

View the full list of items (color coded in red) on the NEEDS ATTENTION tab of the "Copy of RDOS Records in DataSpace" google sheet https://docs.google.com/spreadsheets/d/130B7RMhnqSeTIKPFBdDsrSVbC1C_PCdZp0qwTucR0QA/edit?usp=sharing

Issue type = "Duplicate Files beyond Readmes" and "Duplicate Readmes Only" for specific examples and links to the records.

Impact of this bug

We cannot approve these datasets until the issue is fixed. Therefore, these records remain in DataSpace until the issue is resolved.

Honeybadger link and code snippet, if applicable

Implementation notes, if any

I believe @carolyncole may already have a script to take care of these issues - since she has fixed a very similar issue for us earlier in the migration. Unsure if it requires making a new similar script or rerunning the existing one though.

Acceptance criteria

• Create a list of ids that will be updated
• Run rake task on each id
  • https://datacommons.princeton.edu/describe/works/347
  • https://datacommons.princeton.edu/describe/works/353 No files matched the globus_
  • https://datacommons.princeton.edu/describe/works/367
  • https://datacommons.princeton.edu/describe/works/38 moved to Write Code to pull down 6 tar files explode the tar files, checksum each file in the tar and produce a report for the curators #1920
  • https://datacommons.princeton.edu/describe/works/383
  • https://datacommons.princeton.edu/describe/works/384
  • https://datacommons.princeton.edu/describe/works/389
  • https://datacommons.princeton.edu/describe/works/392
  • https://datacommons.princeton.edu/describe/works/394
  • https://datacommons.princeton.edu/describe/works/395
  • https://datacommons.princeton.edu/describe/works/403
  • https://datacommons.princeton.edu/describe/works/412
  • https://datacommons.princeton.edu/describe/works/413
  • https://pdc-describe-prod.princeton.edu/describe/works/425 moved to Write Code to pull down 6 tar files explode the tar files, checksum each file in the tar and produce a report for the curators #1920
  • https://pdc-describe-prod.princeton.edu/describe/works/429 moved to Write Code to pull down 6 tar files explode the tar files, checksum each file in the tar and produce a report for the curators #1920
  • https://pdc-describe-prod.princeton.edu/describe/works/431 moved to Write Code to pull down 6 tar files explode the tar files, checksum each file in the tar and produce a report for the curators #1920
  • https://pdc-describe-prod.princeton.edu/describe/works/441
  • https://pdc-describe-prod.princeton.edu/describe/works/442
  • https://pdc-describe-prod.princeton.edu/describe/works/467
  • https://pdc-describe-prod.princeton.edu/describe/works/468
  • https://pdc-describe-prod.princeton.edu/describe/works/473
  • https://pdc-describe-prod.princeton.edu/describe/works/475
  • https://pdc-describe-prod.princeton.edu/describe/works/479
• Report back if there are any checksum miss matches

[carolyncole]

Hey team! Please add your planning poker estimate with Zenhub @bess @hectorcorrea @JaymeeH @leefaisonr

[carolyncole]

Here is the output of the rake task showing that there are some actual checksum miss-matches between globus and datspace.

@sec122 the curators will need to look into these further.

442.txt
479.txt

[matthewjchandler]

@sec122 In the case of 479 mentioned above, I see the only difference is with the README file, and I can see in the one starting with "globus_" shows some character encoding errors when viewed in a browser (for me, at least). If you don't see any substantial differences between the two in terms of content, then I'd recommend we go with the one starting "data_space_".

[matthewjchandler]

@sec122 As for 442, that's more mysterious. Here the options I see:

1. Leave all of the files in PDC for now, flag it for later review, and move on with the migration
2. Make a judgment call to preserve the file set closest to what was originally uploaded (those starting with "data_space_" I believe)
3. Go through all of the .nc files, figure out why the checksums don't match, and make a confident decision about what to keep and what to delete

One way or another, RDSS will need clear guidance from PRDS about what to keep and what to delete (if anything).

[sec122]

@carolyncole Here are a set of items that just need the duplicate files and prefixes removed (see notes in our spreadsheet under the column "actions remaining" for specifics about each):

• https://pdc-describe-prod.princeton.edu/describe/works/479
• https://datacommons.princeton.edu/describe/works/513
• https://datacommons.princeton.edu/describe/works/511
• https://datacommons.princeton.edu/describe/works/501
• https://datacommons.princeton.edu/describe/works/498
• https://datacommons.princeton.edu/describe/works/494
• https://datacommons.princeton.edu/describe/works/519
• https://datacommons.princeton.edu/describe/works/515
• https://pdc-describe-prod.princeton.edu/describe/works/425
• https://pdc-describe-prod.princeton.edu/describe/works/429
• https://pdc-describe-prod.princeton.edu/describe/works/431
• https://pdc-describe-prod.princeton.edu/describe/works/416
• https://datacommons.princeton.edu/describe/works/514
• https://pdc-describe-prod.princeton.edu/describe/works/442

[carolyncole]

@sec122 Those updates are completed now.