Implement reindexer sweeper #149
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… or as "keyword" if not present there Manual tests against docker registry: - logged counts are correct - missing mappings are added - types of missing mappings change according to the resolved typename - metadata updates are written to db - metadata updates are sufficient to trigger re-index, causing previously-unsearchable properties to become searchable. - presence of metadata attribute excludes document from document set on subsequent runs
…weep do not erroneously get flagged as processed
alexdunnjpl
requested review from
tloubrieu-jpl,
nutjob4life and
collinss-jpl
as code owners
|
per @jordanpadams review/live-test postponed until next week, after the current site demos. |
…o the resolution logic
alexdunnjpl
force-pushed
the
index-mapping-repair
branch
from
9339ede
to
3090b5f
Compare
this should prevent unintended continuous looping
|
@jordanpadams regarding the "needs to work with I suggest punting those. If any of your comments amount to "this should be free-text and not keyword", then I can make those updates. |
…riate this is because the retry would pass the consumed iterator to subsequent calls
…pings protection against race condition is provided by harvest-time constraint to LT sweeper execution timestamp
alexdunnjpl
force-pushed
the
index-mapping-repair
branch
from
5ca906b
to
0e9d6b2
Compare
|
Status: final testing in progress |
dateutil is the official third-party library for parsing
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
🗒️ Summary
Implements a sweeper with the following behaviour:
registryindex which haven't been swept yet by this sweeper and were not been harvested since the time the sweeper began execution (to ensure consistency of queries)registry-ddindexregistry-ddindex if available, else defaults tokeyword_mappingsfor all missing propertiesops:Provenance/ops:reindexed_at, which is used to identify when documents were checked/fixed, and which triggers a whole-document reindex operation, ensuring that any just-added mappings become searchable for that document.@jordanpadams @tloubrieu-jpl could you please review these requirements to ensure that they accurately reflect the desired behaviour?
⚙️ Test Data and/or Report
Manually tested:
@jordanpadams @tloubrieu-jpl having tested locally on the I&T db, would you like me to run the sweeper against one/some of the MCP indices, after the code has been reviewed?
♻️ Related Issues
resolves #148 once run/deployed
related to NASA-PDS/registry#230