HADOOP-19385. S3A: Add iceberg bulk delete test #7316
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
steveloughran
changed the title
Add Iceberg core to the hadoop-aws test classpath. Iceberg is java17+ only, so this adds * A new test source path src/test/java17 * A new profile "java-17-or-later" which includes this and declares the dependency on iceberg-core. The new test is ITestIcebergBulkDelete; it is parameterized Iceberg bulk delete enabled/disabled and s3a multipart delete enabled/disabled. There is a superclass contract test org.apache.fs.test.formats.AbstractIcebergDeleteTest To support future stores which implement bulk delete. This is currently a minimal superclass; all tests are currently in ITestIcebergBulkDelete Change-Id: Ica8682f4efdd0cb04ca7f4762b2e47d396552910
steveloughran
force-pushed
the
s3/HADOOP-19388-TestIcebergBulkDelete
branch
from
09b4682 to
d37310c
Compare
|
🎊 +1 overall
This message was automatically generated. |
steveloughran
commented
Jan 23, 2025
| @@ -787,7 +787,17 @@ | |||
| <version>${hadoop.version}</version> | |||
| <type>test-jar</type> | |||
| </dependency> | |||
|
|
|||
| <dependency> | |||
steveloughran
commented
Jan 23, 2025
| @@ -1521,6 +1521,13 @@ private Constants() { | |||
| */ | |||
| public static final String FS_S3A_PERFORMANCE_FLAGS = | |||
| "fs.s3a.performance.flags"; | |||
|
|
|||
| /** | |||
| * All performance flags in the enumeration. | |||
| * @return a configuration for subclasses to extend | ||
| */ | ||
|
|
||
| @Override |
There was a problem hiding this comment.
cut for now, or make this suite parameterized on bulk delete enabled
| * Format tests. | ||
| */ | ||
|
|
||
| package org.apache.fs.test.formats; |
|
|
||
| private static final int DELETE_FILE_COUNT = 7; | ||
|
|
||
| @Parameterized.Parameters(name = "multiobjectdelete-{0}-usebulk-{1}") |
There was a problem hiding this comment.
flip params and add a javadoc
| * The classic invocation mechanism reports a failure. | ||
| */ | ||
| @Test | ||
| public void testDeleteDirectory() throws Throwable { |
There was a problem hiding this comment.
expand detail in name directory does not happen
| public void testDeleteManyFiles() throws Throwable { | ||
| Path path = methodPath(); | ||
| final FileSystem fs = getFileSystem(); | ||
| final List<Path> files = createFiles(fs, path, 1, DELETE_FILE_COUNT, 0); |
There was a problem hiding this comment.
+add a local temp file to show mixing of filesystems
| assertSuccessfulBulkDelete(bulkDelete_delete(getFileSystem(), basePath, paths)); | ||
| } | ||
|
|
||
| public static List<String> stringList(List<Path> files) { |
There was a problem hiding this comment.
javadocs and explain why not .toURI()
| } | ||
|
|
||
| @Test | ||
| public void testDeleteManyFiles() throws Throwable { |
There was a problem hiding this comment.
comment that this is how it is expected to be used
| /** Is bulk delete enabled on hadoop runtimes with API support: {@value}. */ | ||
| public static final String ICEBERG_BULK_DELETE_ENABLED = "iceberg.hadoop.bulk.delete.enabled"; | ||
|
|
||
| private static final int DELETE_PAGE_SIZE = 3; |
There was a problem hiding this comment.
explain choice of this and the delete file count
| final FileSystem fs = getFileSystem(); | ||
| final List<Path> files = createFiles(fs, path, 1, DELETE_FILE_COUNT, 0); | ||
| try (HadoopFileIO fileIO = createFileIO()) { | ||
| fileIO.deleteFiles(stringList(files)); |
There was a problem hiding this comment.
add iostats assertions
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add Iceberg core to the hadoop-aws test classpath to verify that it is correctly
invoked through the HadoopFileIO component when enabled in iceberg
Iceberg is java17+ only, so this adds
The new test is ITestIcebergBulkDelete; it is parameterized Iceberg bulk delete enabled/disabled and s3a multipart delete enabled/disabled.
There is a superclass contract test
This is to support future stores which implement bulk delete. This is currently a minimal superclass; all tests are currently in
ITestIcebergBulkDeleteHow was this patch tested?
Fun!
Once we have iceberg PR 10233 merged in, we can merge this. Until then it doesn't currently compile as for testing unless we move to DynMethods there (maybe I should)
Important: Yetus cannot test this yet!
Yetus doesn't have a java17 run, so the new test is not executed. Even if the Iceberg Jar was there
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?