Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SNOW-1757822: Allow JDBC to handle ZSTD decompression #1932

Open
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

sfc-gh-dheyman-1
Copy link
Contributor

Overview

SNOW-1757822

Pre-review self checklist

  • PR branch is updated with all the changes from master branch
  • The code is correctly formatted (run mvn -P check-style validate)
  • New public API is not unnecessary exposed (run mvn verify and inspect target/japicmp/japicmp.html)
  • The pull request name is prefixed with SNOW-XXXX:
  • Code is in compliance with internal logging requirements

External contributors - please answer these questions before submitting a pull request. Thanks!

  1. What GitHub issue is this PR addressing? Make sure that there is an accompanying issue to your PR.

    Issue: #NNNN

  2. Fill out the following pre-review checklist:

    • I am adding a new automated test(s) to verify correctness of my new code
    • I am adding new logging messages
    • I am modifying authorization mechanisms
    • I am adding new credentials
    • I am modifying OCSP code
    • I am adding a new dependency or upgrading an existing one
    • I am adding new public/protected component not marked with @SnowflakeJdbcInternalApi (note that public/protected methods/fields in classes marked with this annotation are already internal)
  3. Please describe how your code solves the related issue.

    Please write a short description of how your code change solves the related issue.

@sfc-gh-dheyman-1 sfc-gh-dheyman-1 marked this pull request as ready for review October 25, 2024 10:15
@sfc-gh-dheyman-1 sfc-gh-dheyman-1 requested a review from a team as a code owner October 25, 2024 10:15
}

@Test
public void testDetectContentEncodingAndGetInputStream_Zstd() throws Exception {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem with such approach is that if we compress and decompress using the same tool, which contains some bug, we will silently pass a test. But there is probably no better approach now unless we use a fixed file which is compressed somewhere else.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Possibly we could use some system utility like command line zstd in a subprocess to have this 3rd party library for verification? We'd need to ensure it's available on test environments though

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think for now we should wait for full backend support of ZSTD chunks compression and then properly test the integration, because the most important thing is whether we are able to decompress what backend compresses. 3rd party tools are of smaller importance here IMO

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants