Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make view metadata path configurable by write.metadata.path #12017

Merged
merged 4 commits into from
Jan 30, 2025
Merged

Make view metadata path configurable by write.metadata.path #12017

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
bf58c0e
Make view metadata path configurable
tomtongue Jan 21, 2025
57a76bb
Add metadata location tests and format the parameter table
tomtongue Jan 29, 2025
26700e6
Add Spark 3.4 test
tomtongue Jan 29, 2025
2a189c3
Nit: fix the parameter description
tomtongue Jan 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 9 additions & 3 deletions core/src/main/java/org/apache/iceberg/view/BaseViewOperations.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,9 +163,15 @@ private String newMetadataFilePath(ViewMetadata metadata, int newVersion) {
}

private String metadataFileLocation(ViewMetadata metadata, String filename) {
return String.format(
"%s/%s/%s",
LocationUtil.stripTrailingSlash(metadata.location()), METADATA_FOLDER_NAME, filename);
String metadataLocation = metadata.properties().get(ViewProperties.WRITE_METADATA_LOCATION);

if (metadataLocation != null) {
return String.format("%s/%s", LocationUtil.stripTrailingSlash(metadataLocation), filename);
} else {
return String.format(
"%s/%s/%s",
LocationUtil.stripTrailingSlash(metadata.location()), METADATA_FOLDER_NAME, filename);
}
}

protected void refreshFromMetadataLocation(String newLocation) {
Expand Down
1 change: 1 addition & 0 deletions core/src/main/java/org/apache/iceberg/view/ViewProperties.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ public class ViewProperties {

public static final String METADATA_COMPRESSION = "write.metadata.compression-codec";
public static final String METADATA_COMPRESSION_DEFAULT = "gzip";
public static final String WRITE_METADATA_LOCATION = "write.metadata.path";
public static final String COMMENT = "comment";
public static final String REPLACE_DROP_DIALECT_ALLOWED = "replace.drop-dialect.allowed";
public static final boolean REPLACE_DROP_DIALECT_ALLOWED_DEFAULT = false;
Expand Down
32 changes: 32 additions & 0 deletions core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -239,6 +239,38 @@ public void completeCreateView() {
assertThat(catalog().viewExists(identifier)).as("View should not exist").isFalse();
}

@Test
public void createViewWithCustomMetadataLocation() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you also please add a test to TestViews with a custom metadata location. You should be able to do a DESCRIBE in that test and verify that the metadata locations match

Copy link
Contributor Author

@tomtongue tomtongue Jan 29, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! Added the test in the TestViews.

TableIdentifier identifier = TableIdentifier.of("ns", "view");

if (requiresNamespaceCreate()) {
catalog().createNamespace(identifier.namespace());
}

assertThat(catalog().viewExists(identifier)).as("View should not exist").isFalse();

String location = Paths.get(tempDir.toUri().toString()).toString();
String customLocation = Paths.get(tempDir.toUri().toString(), "custom-location").toString();

View view =
catalog()
.buildView(identifier)
.withSchema(SCHEMA)
.withDefaultNamespace(identifier.namespace())
.withDefaultCatalog(catalog().name())
.withQuery("spark", "select * from ns.tbl")
.withProperty(ViewProperties.WRITE_METADATA_LOCATION, customLocation)
.withLocation(location)
.create();

assertThat(view).isNotNull();
assertThat(catalog().viewExists(identifier)).as("View should exist").isTrue();
assertThat(view.properties()).containsEntry("write.metadata.path", customLocation);
assertThat(((BaseView) view).operations().current().metadataFileLocation())
.isNotNull()
.startsWith(customLocation);
}

@Test
public void createViewErrorCases() {
TableIdentifier identifier = TableIdentifier.of("ns", "view");
Expand Down
11 changes: 6 additions & 5 deletions docs/docs/view-configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,12 @@ title: "Configuration"
Iceberg views support properties to configure view behavior. Below is an overview of currently available view properties.


| Property | Default | Description |
|--------------------------------------------|---------|------------------------------------------------------------------------------------|
| write.metadata.compression-codec | gzip | Metadata compression codec: `none` or `gzip` |
| version.history.num-entries | 10 | Controls the number of `versions` to retain |
| replace.drop-dialect.allowed | false | Controls whether a SQL dialect is allowed to be dropped during a replace operation |
| Property | Default | Description |
|----------------------------------|---------------------------|------------------------------------------------------------------------------------|
| write.metadata.compression-codec | gzip | Metadata compression codec: `none` or `gzip` |
| write.metadata.path | view location + /metadata | Base location for metadata files |
| version.history.num-entries | 10 | Controls the number of `versions` to retain |
| replace.drop-dialect.allowed | false | Controls whether a SQL dialect is allowed to be dropped during a replace operation |


### View behavior properties
Expand Down
20 changes: 20 additions & 0 deletions spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.assertj.core.api.Assumptions.assumeThat;

import java.io.IOException;
import java.util.List;
import java.util.Locale;
import java.util.Map;
Expand Down Expand Up @@ -2089,6 +2090,25 @@ public void createViewWithRecursiveCycleInSubqueryExpression() {
String.format("Recursive cycle in view detected: %s (cycle: %s)", view1, cycle));
}

@Test
public void createViewWithCustomMetadataLocation() throws IOException {
String viewName = viewName("v");
String customMetadataLocation = temp.newFolder("custom-metadata-location").toString();
sql(
"CREATE VIEW %s TBLPROPERTIES ('%s'='%s') AS SELECT * FROM %s",
viewName, ViewProperties.WRITE_METADATA_LOCATION, customMetadataLocation, tableName);
String location = viewCatalog().loadView(TableIdentifier.of(NAMESPACE, viewName)).location();

assertThat(sql("DESCRIBE EXTENDED %s", viewName))
.contains(
row(
"View Properties",
String.format(
"['format-version' = '1', 'location' = '%s', 'provider' = 'iceberg', 'write.metadata.path' = '%s']",
location, customMetadataLocation),
""));
}

private void insertRows(int numRows) throws NoSuchTableException {
List<SimpleRecord> records = Lists.newArrayListWithCapacity(numRows);
for (int i = 1; i <= numRows; i++) {
Expand Down
21 changes: 21 additions & 0 deletions spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.assertj.core.api.Assumptions.assumeThat;

import java.nio.file.Paths;
import java.util.List;
import java.util.Locale;
import java.util.Random;
Expand Down Expand Up @@ -2093,6 +2094,26 @@ public void createViewWithRecursiveCycleInSubqueryExpression() {
String.format("Recursive cycle in view detected: %s (cycle: %s)", view1, cycle));
}

@TestTemplate
public void createViewWithCustomMetadataLocation() {
String viewName = viewName("v");
String customMetadataLocation =
Paths.get(temp.toUri().toString(), "custom-metadata-location").toString();
sql(
"CREATE VIEW %s TBLPROPERTIES ('%s'='%s') AS SELECT * FROM %s",
viewName, ViewProperties.WRITE_METADATA_LOCATION, customMetadataLocation, tableName);
String location = viewCatalog().loadView(TableIdentifier.of(NAMESPACE, viewName)).location();

assertThat(sql("DESCRIBE EXTENDED %s", viewName))
.contains(
row(
"View Properties",
String.format(
"['format-version' = '1', 'location' = '%s', 'provider' = 'iceberg', 'write.metadata.path' = '%s']",
location, customMetadataLocation),
""));
}

private void insertRows(int numRows) throws NoSuchTableException {
List<SimpleRecord> records = Lists.newArrayListWithCapacity(numRows);
for (int i = 1; i <= numRows; i++) {
Expand Down