Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[dvc][doc] Create MVP for DaVinciRecordTransformer #1087

Open
wants to merge 69 commits into
base: main
Choose a base branch
from

Commits on Jul 29, 2024

  1. Configuration menu
    Copy the full SHA
    8dc58f0 View commit details
    Browse the repository at this point in the history
  2. Fix docs and imports

    kvargha committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    88b62fe View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    fc856b7 View commit details
    Browse the repository at this point in the history
  4. Dont check for boolean val

    kvargha committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    03f0453 View commit details
    Browse the repository at this point in the history
  5. Delete classHash if it exists

    kvargha committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    45ba862 View commit details
    Browse the repository at this point in the history
  6. Assert file deletion is true

    kvargha committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    e8fa547 View commit details
    Browse the repository at this point in the history

Commits on Jul 30, 2024

  1. Improve code coverage

    kvargha committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    b3f3e0a View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    f3cacff View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    d2532b3 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    4d3ee2b View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    189127c View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    9995758 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    2c421a6 View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2024

  1. Configuration menu
    Copy the full SHA
    68d2be8 View commit details
    Browse the repository at this point in the history
  2. Merge pull request #1 from kvargha/dvrt-mvp

    Dvrt mvp
    kvargha authored Jul 31, 2024
    Configuration menu
    Copy the full SHA
    9ffbf41 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    fc0b188 View commit details
    Browse the repository at this point in the history

Commits on Aug 1, 2024

  1. Configuration menu
    Copy the full SHA
    05c22b4 View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2024

  1. Configuration menu
    Copy the full SHA
    1ef7c20 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    6b56ff8 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    2cbd14f View commit details
    Browse the repository at this point in the history

Commits on Aug 13, 2024

  1. Configuration menu
    Copy the full SHA
    1ed816b View commit details
    Browse the repository at this point in the history
  2. Correctly pass DVRT functional interface to initBackend, add todo to …

    …store classHash in rocksDB
    kvargha committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    6162604 View commit details
    Browse the repository at this point in the history
  3. Fix spotbugs

    kvargha committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    968d357 View commit details
    Browse the repository at this point in the history

Commits on Aug 14, 2024

  1. Configuration menu
    Copy the full SHA
    e78340e View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2024

  1. Modify checkout action

    kvargha committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    42485d9 View commit details
    Browse the repository at this point in the history
  2. Undo

    kvargha committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    22b57e9 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    5f4da22 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    e57caeb View commit details
    Browse the repository at this point in the history
  5. Fix compilation

    kvargha committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    3c7ec6c View commit details
    Browse the repository at this point in the history

Commits on Aug 16, 2024

  1. Cache deserializer/serializer

    kvargha committed Aug 16, 2024
    Configuration menu
    Copy the full SHA
    36926d5 View commit details
    Browse the repository at this point in the history

Commits on Aug 19, 2024

  1. Configuration menu
    Copy the full SHA
    ef6ac83 View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2024

  1. Configuration menu
    Copy the full SHA
    14fef31 View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2024

  1. Delete classHash file. Compare value classes and don't override value…

    … class in AvroGenericDaVinciClient
    kvargha committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    80dd551 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2dd00a8 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    a4227a5 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    91ba185 View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2024

  1. Configuration menu
    Copy the full SHA
    cf7169e View commit details
    Browse the repository at this point in the history
  2. Fix spotless error

    kvargha committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    4d958a2 View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2024

  1. Configuration menu
    Copy the full SHA
    4017da6 View commit details
    Browse the repository at this point in the history
  2. Remove unused variables

    kvargha committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    d15f437 View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2024

  1. Added a ToDo to make chunking with record transformer lazy, and make …

    …the return type of delete void. Proceed with deletion based off of storeRecordsInDaVinci
    kvargha committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    7e49576 View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2024

  1. Created a config for record transformer that's passed into the DaVinc…

    …iConfig to remove the need of instantiating a dummy record transformer, updated docs, and simplified function names
    kvargha committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    f175b10 View commit details
    Browse the repository at this point in the history

Commits on Oct 19, 2024

  1. Add message envelope for DVRT

    kvargha committed Oct 19, 2024
    Configuration menu
    Copy the full SHA
    849ebce View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    3f44da3 View commit details
    Browse the repository at this point in the history
  3. Fix spotless issue

    kvargha committed Oct 19, 2024
    Configuration menu
    Copy the full SHA
    ccf1940 View commit details
    Browse the repository at this point in the history
  4. Fix test

    kvargha committed Oct 19, 2024
    Configuration menu
    Copy the full SHA
    54ba9ee View commit details
    Browse the repository at this point in the history
  5. update docs

    kvargha committed Oct 19, 2024
    Configuration menu
    Copy the full SHA
    12b012c View commit details
    Browse the repository at this point in the history
  6. Dvrt mvp (#4)

    * Create MVP for record transformer
    
    * Fix docs and imports
    
    * Don't make serializers and deserializers global
    
    * Dont check for boolean val
    
    * Delete classHash if it exists
    
    * Assert file deletion is true
    
    * Improve code coverage
    
    * Add tests for blocking record transformer
    
    * Created AvroGenericDaVinciClient for record transformer test
    
    * Make sure getRecordTransformer is valid
    
    * Make sure getRecordTransformer isn't null
    
    * Reorganize testRecordTransformerClient and fix key schema
    
    * Fix TestStringRecordTransformer to work with Avro objects and update doc
    
    * Refactor onRecovery and add javadoc for DaVinciRecordTransformer's constructor
    
    * Reset offset if we need to bootstrap from VT
    
    * Delete classHash after running tests
    
    * Throw an error if a user tries to use blob transfer with record transformer
    
    * Make previous public methods private, remove subscribe call inside onRecovery
    
    * Correctly pass DVRT functional interface to initBackend, add todo to store classHash in rocksDB
    
    * Fix spotbugs
    
    * Init DVRT inside SIT, and move DVRT recovery to SIT
    
    * Modify checkout action
    
    * Undo
    
    * Fix compilation
    
    * Cache deserializer/serializer
    
    * Create utility class for record transformer
    
    * Create AbstractStorageIterator and a RocksDB implementation
    
    * Delete classHash file. Compare value classes and don't override value class in AvroGenericDaVinciClient
    
    * Remove compareCacheConfig mock
    
    * Wrap access modifier with doPrivileged
    
    * Fix spotless error
    
    * Remove unused variables
    
    * Added a ToDo to make chunking with record transformer lazy, and make the return type of delete void. Proceed with deletion based off of storeRecordsInDaVinci
    
    * Created a config for record transformer that's passed into the DaVinciConfig to remove the need of instantiating a dummy record transformer, updated docs, and simplified function names
    
    * Add message envelope for DVRT
    
    * Fix spotless issue
    
    * Fix test
    
    * update docs
    kvargha authored Oct 19, 2024
    Configuration menu
    Copy the full SHA
    e747f70 View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2024

  1. Configuration menu
    Copy the full SHA
    33690b7 View commit details
    Browse the repository at this point in the history
  2. Cleanup code

    kvargha committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    1bb889b View commit details
    Browse the repository at this point in the history
  3. Cleanup docs

    kvargha committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    9462bc7 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    ac165e6 View commit details
    Browse the repository at this point in the history
  5. Cleanup

    kvargha committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    edbd3ca View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2024

  1. Cleanup

    kvargha committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    0b74a7a View commit details
    Browse the repository at this point in the history

Commits on Oct 23, 2024

  1. Configuration menu
    Copy the full SHA
    39201bc View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    4a8e44a View commit details
    Browse the repository at this point in the history
  3. Cleanup

    kvargha committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    fdd0ef0 View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2024

  1. Compress transformed value

    kvargha committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    b09a3c9 View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2024

  1. Add test for chunking

    kvargha committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    feb73cf View commit details
    Browse the repository at this point in the history
  2. Cleanup

    kvargha committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    71ffec7 View commit details
    Browse the repository at this point in the history
  3. Cleanup

    kvargha committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    6b8bab1 View commit details
    Browse the repository at this point in the history
  4. cleanup

    kvargha committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    a94b196 View commit details
    Browse the repository at this point in the history
  5. cleanup

    kvargha committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    be07be4 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    609c06d View commit details
    Browse the repository at this point in the history
  7. Cleanup

    kvargha committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    4215967 View commit details
    Browse the repository at this point in the history
  8. Cleanup

    kvargha committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    beb821f View commit details
    Browse the repository at this point in the history

Commits on Oct 28, 2024

  1. Address review comments

    kvargha committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    bb25b73 View commit details
    Browse the repository at this point in the history
  2. Reorganize

    kvargha committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    83a88bd View commit details
    Browse the repository at this point in the history
  3. Add overrides

    kvargha committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    bf85c1e View commit details
    Browse the repository at this point in the history