Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(WIP) add admin delete API endpoint to clear test bank data #4721

Open
wants to merge 10 commits into
base: master
Choose a base branch
from

Conversation

rkovalik-raft
Copy link
Contributor

Curl command to delete 2020 submission data for Bank0:

curl --location --request DELETE 'http://localhost:8081/delete/B90YWS6AFX2LGWOXJ1LD/year/2020' \
--header 'Content-Type: application/CSV' \
--data '[object Object]'

TODO:
Test in Postman.

Note: Even when I set the keyspaces hmda2_journal and hmda2_snapshot and respective tables, I still sometimes see this error: Caused by: com.datastax.oss.driver.api.core.servererrors.InvalidQueryException: Keyspace hmda2_snapshot does not exist

@rkovalik-raft
Copy link
Contributor Author

Updated to delete all data rows for "Submission", "EditDetail", "HmdaRawData", "HmdaValidationError" persistence id prefixes.

Tested with local cassandra 4 and local platform. Resulting queries in cassandra for delete:

Type: single-query
Query start time: 1699569131753
Protocol version: 5
Generated timestamp:-9223372036854775808
Generated nowInSeconds:1699569131
Query: 
      INSERT INTO hmda2_journal.metadata (persistence_id, deleted_to)
      VALUES ( ?, ? )
    
Values: 
00000000 48 6D 64 61 56 61 6C 69  64 61 74 69 6F 6E 45 72 HmdaVali dationEr
00000010 72 6F 72 2D 42 39 30 59  57 53 36 41 46 58 32 4C ror-B90Y WS6AFX2L
00000020 47 57 4F 58 4A 31 4C 44  2D 32 30 32 30 2D 39    GWOXJ1LD -2020-9 
-----
00000000 00 00 00 00 00 00 00 0D                          ········         
-----

Type: single-query
Query start time: 1699569131757
Protocol version: 5
Generated timestamp:-9223372036854775808
Generated nowInSeconds:1699569131
Query: 
      DELETE FROM hmda2_journal.journal WHERE
        persistence_id = ? AND
        partition_nr = ? AND
        sequence_nr >= 0 AND
        sequence_nr <= ?
    
Values: 
00000000 48 6D 64 61 56 61 6C 69  64 61 74 69 6F 6E 45 72 HmdaVali dationEr
00000010 72 6F 72 2D 42 39 30 59  57 53 36 41 46 58 32 4C ror-B90Y WS6AFX2L
00000020 47 57 4F 58 4A 31 4C 44  2D 32 30 32 30 2D 39    GWOXJ1LD -2020-9 
-----
00000000 00 00 00 00 00 00 00 01                          ········         
-----
00000000 00 00 00 00 00 00 00 0D                          ········         
-----

Type: single-query
Query start time: 1699569131757
Protocol version: 5
Generated timestamp:-9223372036854775808
Generated nowInSeconds:1699569131
Query: 
      DELETE FROM hmda2_journal.journal WHERE
        persistence_id = ? AND
        partition_nr = ? AND
        sequence_nr >= 0 AND
        sequence_nr <= ?
    
Values: 
00000000 48 6D 64 61 56 61 6C 69  64 61 74 69 6F 6E 45 72 HmdaVali dationEr
00000010 72 6F 72 2D 42 39 30 59  57 53 36 41 46 58 32 4C ror-B90Y WS6AFX2L
00000020 47 57 4F 58 4A 31 4C 44  2D 32 30 32 30 2D 39    GWOXJ1LD -2020-9 
-----
00000000 00 00 00 00 00 00 00 00                          ········         
-----
00000000 00 00 00 00 00 00 00 0D                          ········         
-----

Type: single-query
Query start time: 1699569131761
Protocol version: 5
Generated timestamp:-9223372036854775808
Generated nowInSeconds:1699569131
Query: 
    DELETE FROM hmda2_snapshot.snapshot
    WHERE persistence_id = ?
    AND sequence_nr >= ?
    AND sequence_nr <= ?
  
Values: 
00000000 48 6D 64 61 56 61 6C 69  64 61 74 69 6F 6E 45 72 HmdaVali dationEr
00000010 72 6F 72 2D 42 39 30 59  57 53 36 41 46 58 32 4C ror-B90Y WS6AFX2L
00000020 47 57 4F 58 4A 31 4C 44  2D 32 30 32 30 2D 39    GWOXJ1LD -2020-9 
-----
00000000 00 00 00 00 00 00 00 00                          ········         
-----
00000000 7F FF FF FF FF FF FF FF                          ········         
-----

Type: single-query
Query start time: 1699569131765
Protocol version: 5
Generated timestamp:-9223372036854775808
Generated nowInSeconds:1699569131
Query: 
       SELECT * from hmda2_journal.tag_write_progress WHERE
       persistence_id = ?
     
Values: 
00000000 48 6D 64 61 56 61 6C 69  64 61 74 69 6F 6E 45 72 HmdaVali dationEr
00000010 72 6F 72 2D 42 39 30 59  57 53 36 41 46 58 32 4C ror-B90Y WS6AFX2L
00000020 47 57 4F 58 4A 31 4C 44  2D 32 30 32 30 2D 39    GWOXJ1LD -2020-9 
-----

@rkovalik-raft
Copy link
Contributor Author

With neverUsePersistenceIdAgain parameter set to true for cleanup.deletaAll(), if the deletion endpoint is called twice, no persistence ids need to be deleted the second time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant