Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Collection iteration optimization #115

Merged
merged 40 commits into from
Apr 14, 2024
Merged

Collection iteration optimization #115

merged 40 commits into from
Apr 14, 2024

Commits on Feb 6, 2024

  1. bump peak memory estimation factor

    @alexdunnjpl
    alexdunnjpl committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    82392b4 View commit details
    Browse the repository at this point in the history

Commits on Mar 27, 2024

  1. dump chunks every ten minutes 

    this works out to ~1000 dump files if iteration takes one week, which seems sane
    @alexdunnjpl
    alexdunnjpl committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    3f0bde5 View commit details
    Browse the repository at this point in the history

  2. add debug log

    @alexdunnjpl
    alexdunnjpl committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    56750ce View commit details
    Browse the repository at this point in the history

  3. switch ancestry from json dump to pickle dump 

    also fixes a bug in the test where expected content was compared with itself rather than the actual output
    @alexdunnjpl
    alexdunnjpl committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    2eb33cd View commit details
    Browse the repository at this point in the history

  4. add run-configs to .gitig

    @alexdunnjpl
    alexdunnjpl committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    5b3a4f7 View commit details
    Browse the repository at this point in the history

Commits on Apr 2, 2024

  1. back to manual management of chunk size

    @alexdunnjpl
    alexdunnjpl committed Apr 2, 2024
    Configuration menu
    Copy the full SHA
    48db330 View commit details
    Browse the repository at this point in the history

  2. improve query progress logging 

    does not update deprecated scroll-based implementation
    @alexdunnjpl
    alexdunnjpl committed Apr 2, 2024
    Configuration menu
    Copy the full SHA
    8f0cebb View commit details
    Browse the repository at this point in the history

  3. remove deprecated non-chunked ancestry processing

    @alexdunnjpl
    alexdunnjpl committed Apr 2, 2024
    Configuration menu
    Copy the full SHA
    a001d52 View commit details
    Browse the repository at this point in the history

  4. fix consumable iterator, as null-ancestry metadata still needs to be … 

    …written for bundles
    @alexdunnjpl
    alexdunnjpl committed Apr 2, 2024
    Configuration menu
    Copy the full SHA
    b4dbdc2 View commit details
    Browse the repository at this point in the history

  5. implement utils.misc.bin_elements()

    @alexdunnjpl
    alexdunnjpl committed Apr 2, 2024
    Configuration menu
    Copy the full SHA
    53e1d66 View commit details
    Browse the repository at this point in the history

  6. overhaul ancestry with collection-iterative approach

    @alexdunnjpl
    alexdunnjpl committed Apr 2, 2024
    Configuration menu
    Copy the full SHA
    2da05ba View commit details
    Browse the repository at this point in the history

  7. Revert "remove deprecated non-chunked ancestry processing" 

    This reverts commit a001d52.
    @alexdunnjpl
    alexdunnjpl committed Apr 2, 2024
    Configuration menu
    Copy the full SHA
    c813e0e View commit details
    Browse the repository at this point in the history

Commits on Apr 3, 2024

  1. squash! overhaul ancestry with collection-iterative approach

    @alexdunnjpl
    alexdunnjpl committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    410aee1 View commit details
    Browse the repository at this point in the history

  2. [fatally broken] improve handling of registry-refs pages where collec… 

    …tion does not exist in registry
    @alexdunnjpl
    alexdunnjpl committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    fe3ecbb View commit details
    Browse the repository at this point in the history

  3. implement intentional skipping of secondary-collection registry-refs … 

    …pages during ancestry
    @alexdunnjpl
    alexdunnjpl committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    c2c70aa View commit details
    Browse the repository at this point in the history

  4. squash! overhaul ancestry with collection-iterative approach

    @alexdunnjpl
    alexdunnjpl committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    691769e View commit details
    Browse the repository at this point in the history

  5. squash! [fatally broken] improve handling of registry-refs pages wher… 

    …e collection does not exist in registry
    @alexdunnjpl
    alexdunnjpl committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    53c5de8 View commit details
    Browse the repository at this point in the history

  6. increment ancestry version

    @alexdunnjpl
    alexdunnjpl committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    8f9c27d View commit details
    Browse the repository at this point in the history

  7. remove alternate_ids ancestry test case 

    reverts 9548917
alternate_ids are not considered reliable for this purpose, and the relevant feature requires reimplementation
    @alexdunnjpl
    alexdunnjpl committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    60fd01d View commit details
    Browse the repository at this point in the history

Commits on Apr 9, 2024

  1. back to memory-based dumping for chunked behaviour

    @alexdunnjpl
    alexdunnjpl committed Apr 9, 2024
    Configuration menu
    Copy the full SHA
    b167982 View commit details
    Browse the repository at this point in the history

  2. clean up cruft

    @alexdunnjpl
    alexdunnjpl committed Apr 9, 2024
    Configuration menu
    Copy the full SHA
    ee2a112 View commit details
    Browse the repository at this point in the history

Commits on Apr 10, 2024

  1. minor additions

    @alexdunnjpl
    alexdunnjpl committed Apr 10, 2024
    Configuration menu
    Copy the full SHA
    fdfea21 View commit details
    Browse the repository at this point in the history

  2. implement ancestry.utils.update.from_record()

    @alexdunnjpl
    alexdunnjpl committed Apr 10, 2024
    Configuration menu
    Copy the full SHA
    1a483a7 View commit details
    Browse the repository at this point in the history

  3. implement ancestry partial-record deferral/handling

    @alexdunnjpl
    alexdunnjpl committed Apr 10, 2024
    Configuration menu
    Copy the full SHA
    31d974b View commit details
    Browse the repository at this point in the history

Commits on Apr 11, 2024

  1. add query stubs for pre-existing ancestry tests

    @alexdunnjpl
    alexdunnjpl committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    e57d6cc View commit details
    Browse the repository at this point in the history

  2. add tests for record/update count 

    previously, nil records/updates would not cause failure of some test cases
    @alexdunnjpl
    alexdunnjpl committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    3ae1db2 View commit details
    Browse the repository at this point in the history

  3. update existing ancestry tests for overhaul

    @alexdunnjpl
    alexdunnjpl committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    c10f268 View commit details
    Browse the repository at this point in the history

  4. generate deferred updates only after non-deferred updates have been w… 

    …ritten to db
    @alexdunnjpl
    alexdunnjpl committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    482146a View commit details
    Browse the repository at this point in the history

  5. update log messages

    @alexdunnjpl
    alexdunnjpl committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    bff5810 View commit details
    Browse the repository at this point in the history

  6. fix bug in json structure

    @alexdunnjpl
    alexdunnjpl committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    5360632 View commit details
    Browse the repository at this point in the history

  7. bump default request timeout 

    bulk updates are often taking longer than 90 seconds
    @alexdunnjpl
    alexdunnjpl committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    e753e28 View commit details
    Browse the repository at this point in the history

  8. fix assignment of bundle history to LID collection references 

    aliases are not working, and have been removed
    @alexdunnjpl
    alexdunnjpl committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    ab8bd2b View commit details
    Browse the repository at this point in the history

  9. excise behaviour related to aliases/alternate_ids, as this property i… 

    …s not reliable and should not be used
    @alexdunnjpl
    alexdunnjpl committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    c81f2ed View commit details
    Browse the repository at this point in the history

  10. minor fix

    @alexdunnjpl
    alexdunnjpl committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    dc65c84 View commit details
    Browse the repository at this point in the history

Commits on Apr 12, 2024

  1. add type validation to AncestryRecord.lidvid

    @alexdunnjpl
    alexdunnjpl committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    303b9db View commit details
    Browse the repository at this point in the history

  2. demote log

    @alexdunnjpl
    alexdunnjpl committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    c112482 View commit details
    Browse the repository at this point in the history

  3. handle case where registry-refs page erroneously includes collection … 

    …in list of its own references
    @alexdunnjpl
    alexdunnjpl committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    083b3fe View commit details
    Browse the repository at this point in the history

  4. increment SWEEPERS_ANCESTRY_VERSION

    @alexdunnjpl
    alexdunnjpl committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    a8a1b86 View commit details
    Browse the repository at this point in the history

Commits on Apr 14, 2024

  1. Merge pull request #116 from NASA-PDS/deferred-partial-updates 

    Deferred partial updates
    @alexdunnjpl
    alexdunnjpl authored Apr 14, 2024
    Configuration menu
    Copy the full SHA
    7c4c1ed View commit details
    Browse the repository at this point in the history

  2. tqdm pin tweak

    @alexdunnjpl
    alexdunnjpl committed Apr 14, 2024
    Configuration menu
    Copy the full SHA
    1b09ce6 View commit details
    Browse the repository at this point in the history