-
Notifications
You must be signed in to change notification settings - Fork 84
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[dvc][doc] Create MVP for DaVinciRecordTransformer #1087
Open
kvargha
wants to merge
69
commits into
linkedin:main
Choose a base branch
from
kvargha:dvrt-mvp
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+1,763
−319
Open
Changes from all commits
Commits
Show all changes
69 commits
Select commit
Hold shift + click to select a range
8dc58f0
Create MVP for record transformer
kvargha 88b62fe
Fix docs and imports
kvargha fc856b7
Don't make serializers and deserializers global
kvargha 03f0453
Dont check for boolean val
kvargha 45ba862
Delete classHash if it exists
kvargha e8fa547
Assert file deletion is true
kvargha b3f3e0a
Improve code coverage
kvargha f3cacff
Add tests for blocking record transformer
kvargha d2532b3
Created AvroGenericDaVinciClient for record transformer test
kvargha 4d3ee2b
Make sure getRecordTransformer is valid
kvargha 189127c
Make sure getRecordTransformer isn't null
kvargha 9995758
Merge branch 'main' into dvrt-mvp
kvargha 2c421a6
Reorganize testRecordTransformerClient and fix key schema
kvargha 68d2be8
Fix TestStringRecordTransformer to work with Avro objects and update doc
kvargha 9ffbf41
Merge pull request #1 from kvargha/dvrt-mvp
kvargha fc0b188
Refactor onRecovery and add javadoc for DaVinciRecordTransformer's co…
kvargha 05c22b4
Reset offset if we need to bootstrap from VT
kvargha 1ef7c20
Merge branch 'main' into dvrt-mvp
kvargha 6b56ff8
Delete classHash after running tests
kvargha 2cbd14f
Throw an error if a user tries to use blob transfer with record trans…
kvargha 1ed816b
Make previous public methods private, remove subscribe call inside on…
kvargha 6162604
Correctly pass DVRT functional interface to initBackend, add todo to …
kvargha 968d357
Fix spotbugs
kvargha e78340e
Init DVRT inside SIT, and move DVRT recovery to SIT
kvargha 42485d9
Modify checkout action
kvargha 22b57e9
Undo
kvargha 5f4da22
Merge branch 'linkedin:main' into dvrt-mvp
kvargha e57caeb
Merge branch 'main' into dvrt-mvp
kvargha 3c7ec6c
Fix compilation
kvargha 36926d5
Cache deserializer/serializer
kvargha ef6ac83
Create utility class for record transformer
kvargha 14fef31
Create AbstractStorageIterator and a RocksDB implementation
kvargha 80dd551
Delete classHash file. Compare value classes and don't override value…
kvargha 2dd00a8
Merge branch 'linkedin:main' into dvrt-mvp
kvargha a4227a5
Remove compareCacheConfig mock
kvargha 91ba185
Wrap access modifier with doPrivileged
kvargha cf7169e
Merge branch 'main' into dvrt-mvp
kvargha 4d958a2
Fix spotless error
kvargha 4017da6
Merge branch 'main' into dvrt-mvp
kvargha d15f437
Remove unused variables
kvargha 7e49576
Added a ToDo to make chunking with record transformer lazy, and make …
kvargha f175b10
Created a config for record transformer that's passed into the DaVinc…
kvargha 849ebce
Add message envelope for DVRT
kvargha 3f44da3
Merge branch 'main' into dvrt-mvp
kvargha ccf1940
Fix spotless issue
kvargha 54ba9ee
Fix test
kvargha 12b012c
update docs
kvargha e747f70
Dvrt mvp (#4)
kvargha 33690b7
Add another test, and update docs
kvargha 1bb889b
Cleanup code
kvargha 9462bc7
Cleanup docs
kvargha ac165e6
Merge branch 'main' into dvrt-mvp
kvargha edbd3ca
Cleanup
kvargha 0b74a7a
Cleanup
kvargha 39201bc
Add integration test for onrecovery. And fix iterator deserializer issue
kvargha 4a8e44a
Throw error if II and DVRT are enabled together
kvargha fdd0ef0
Cleanup
kvargha b09a3c9
Compress transformed value
kvargha feb73cf
Add test for chunking
kvargha 71ffec7
Cleanup
kvargha 6b8bab1
Cleanup
kvargha a94b196
cleanup
kvargha be07be4
cleanup
kvargha 609c06d
Add versionIngestion to onStart/onEnd function names
kvargha 4215967
Cleanup
kvargha beb821f
Cleanup
kvargha bb25b73
Address review comments
kvargha 83a88bd
Reorganize
kvargha bf85c1e
Add overrides
kvargha File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's do a test case through these, I think looking back at the integration test cases the transformed type is always the same.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have a couple of unit tests that go over this. Unless you mean changing the types? Ex: Int -> String?