Static data imports #370
Draft
Static data imports #370
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Using the concer. Previously they just had `has_paper_trail`, which meant changes were recorded with paper trail but there was no `revision_comment`.
joeroe
commented
Dec 17, 2024
| END | ||
|
|
||
| PaperTrail.request(whodunnit: admin_user_id) do | ||
| ActiveRecord::Base.transaction do |
There was a problem hiding this comment.
Note everything is wrapped in a transaction, so if there are any errors the whole import is aborted.
|
Agreed, and in that case I will proceed in the same way for the dendro imports. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR represents a rethinking of the import pipeline with the aim of getting data into XRONOS sooner. Basically, I dropped the idea of creating a generalised architecture or UI for imports, which greatly complicated the problem and have been holding up imports for a long time.
The new approach is simply to write a rake task for each resource to be imported. Advantages of this approach:
db/importand the rake script shows exactly how it was importedThe major disadvantage is of course that bulk imports can only be done be sysadmins. However, I am no longer convinced that this was ever a viable option. Even with a relatively small and clean dataset like the one below, there are too many potential problems to guard against and a poorly-specified import can great a huge mess.
I have written a proof-of-concept of this approach with Wang et al.'s (2014) database of Chinese radiocarbon dates. These are already in the production version of XRONOS via p3k14c, and so this will create more duplicates, but it illustrate the advantages of the scripted approach because we pick up richer data on the context and sample, as well as full(ish) bibliographic references. If you agree with this approach @MartinHinz, I'll add more scripts for other static resources (https://github.com/xronos-ch/xronos-import/issues/5, prioritising https://github.com/xronos-ch/xronos-import/issues/54) before finalising the PR and running them in production.
To perform the import you need to set the environment variable
ADMIN_USER_IDand then runbin/rails import:wang_et_al-2014.