Make view metadata path configurable by `write.metadata.path` #12017

tomtongue · 2025-01-21T10:37:43Z

Overview

This commit makes the View's metadata file location configurable by write.metadata.path as the table configuration, it and provides more flexible way to configure the metadata file path for Iceberg Views.

Background

Currently, View's metadata file location respects its relevant database location first. If the database doesn't have the location, the View's metadata location is set to a warehouse location.

For now, when you customize the View's metadata location, you need to set the warehouse parameter. However, if your database has the configured location in its property such as location: /db/table/path, you need to change the database location (this is not always easy).

Example

-- SparkSQL
CREATE VIEW db.view TBLPROPERTIES('write.metadata.path'='s3://bucket/path/new-metadata-loc') 
AS SELECT year, count(*) as cnt FROM db.tbl GROUP BY year

DESCRIBE EXTENDED db.view
+---------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
|col_name                   |data_type                                                                                                                                                               |comment|
+---------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
|year                       |int                                                                                                                                                                     |       |
|cnt                        |bigint                                                                                                                                                                  |       |
|                           |                                                                                                                                                                        |       |
|# Detailed View Information|                                                                                                                                                                        |       |
|Comment                    |                                                                                                                                                                        |       |
|View Catalog and Namespace |hive_catalog.db                                                                                                                                                         |       |
|View Query Output Columns  |[year, cnt]                                                                                                                                                             |       |
|View Properties            |['format-version' = '1', 'location' = 's3://bucket/path/database-location', 'provider' = 'iceberg', 'write.metadata.path' = 's3://bucket/path/new-metadata-loc']        |       |
|Created By                 |Spark 3.5.3-amzn-0                                                                                                                                                      |       |
+---------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+

tomtongue · 2025-01-28T05:15:45Z

@nastra I believe you implemented the relevant part. If possible, can I ask you to review this change?

nastra · 2025-01-28T16:10:30Z

docs/docs/view-configuration.md

@@ -28,6 +28,7 @@ Iceberg views support properties to configure view behavior. Below is an overvie
 | Property                                   | Default | Description                                                                        |
 |--------------------------------------------|---------|------------------------------------------------------------------------------------|
 | write.metadata.compression-codec           | gzip    | Metadata compression codec: `none` or `gzip`                                       |
+| write.metadata.path              | table location + /metadata | Base location for metadata files                                          |


nit: could you align the formatting of the table please?

nastra · 2025-01-28T16:15:26Z

core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java

+    assertThat(view).isNotNull();
+    assertThat(catalog().viewExists(identifier)).as("View should exist").isTrue();
+    assertThat(view.properties()).containsEntry("write.metadata.path", customLocation);
+    assertThat(((BaseView) view).operations().current().metadataFileLocation()).isNotNull();


maybe also add a .startsWith(customLocation) check at the end

Sure, thank you!

nastra · 2025-01-28T16:15:48Z

core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java

+            .withDefaultNamespace(identifier.namespace())
+            .withDefaultCatalog(catalog().name())
+            .withQuery("spark", "select * from ns.tbl")
+            .withProperty("write.metadata.path", customLocation)


let's use the statically defined property from ViewProperties here

nastra · 2025-01-28T16:17:08Z

core/src/test/java/org/apache/iceberg/view/ViewCatalogTests.java

@@ -239,6 +239,36 @@ public void completeCreateView() {
    assertThat(catalog().viewExists(identifier)).as("View should not exist").isFalse();
  }

+  @Test
+  public void createViewWithCustomMetadataLocation() {


can you also please add a test to TestViews with a custom metadata location. You should be able to do a DESCRIBE in that test and verify that the metadata locations match

Thank you! Added the test in the TestViews.

tomtongue · 2025-01-28T17:36:44Z

Thanks for the review! I'm working on it, and will fix them.

tomtongue · 2025-01-29T12:27:30Z

@nastra @amogh-jahagirdar Reflected the review by adding tests. Could you review the new changes?

nastra · 2025-01-29T17:17:48Z

docs/docs/view-configuration.md

+| Property                         | Default                    | Description                                                                        |
+|----------------------------------|----------------------------|------------------------------------------------------------------------------------|
+| write.metadata.compression-codec | gzip                       | Metadata compression codec: `none` or `gzip`                                       |
+| write.metadata.path              | table location + /metadata | Base location for metadata files                                                   |


Oh I missed this, let me fix now. Thank you!

nastra

LGTM with one remaining comment, thanks @tomtongue. I'll also leave this open for a bit so that @amogh-jahagirdar has a chance to review this too

tomtongue · 2025-01-29T17:49:53Z

Sure. Thanks so much for the review!

amogh-jahagirdar

Thanks @tomtongue!

tomtongue · 2025-01-30T04:51:15Z

Thanks so much for the quick review!

Make view metadata path configurable

bf58c0e

github-actions bot added core docs labels Jan 21, 2025

nastra reviewed Jan 28, 2025

View reviewed changes

nastra requested a review from amogh-jahagirdar January 28, 2025 16:17

Add metadata location tests and format the parameter table

57a76bb

github-actions bot added the spark label Jan 29, 2025

Add Spark 3.4 test

26700e6

nastra reviewed Jan 29, 2025

View reviewed changes

nastra approved these changes Jan 29, 2025

View reviewed changes

Nit: fix the parameter description

2a189c3

amogh-jahagirdar approved these changes Jan 30, 2025

View reviewed changes

nastra merged commit 2a0d5e8 into apache:main Jan 30, 2025
46 checks passed

tomtongue deleted the view-metadata-path branch January 30, 2025 16:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make view metadata path configurable by `write.metadata.path` #12017

Make view metadata path configurable by `write.metadata.path` #12017

tomtongue commented Jan 21, 2025 •

edited

Loading

tomtongue commented Jan 28, 2025 •

edited

Loading

nastra Jan 28, 2025

nastra Jan 28, 2025

tomtongue Jan 29, 2025

nastra Jan 28, 2025

nastra Jan 28, 2025

tomtongue Jan 29, 2025 •

edited

Loading

tomtongue commented Jan 28, 2025

tomtongue commented Jan 29, 2025

nastra Jan 29, 2025

tomtongue Jan 29, 2025

nastra left a comment

tomtongue commented Jan 29, 2025

amogh-jahagirdar left a comment

tomtongue commented Jan 30, 2025

	\| write.metadata.path \| table location + /metadata \| Base location for metadata files \|
	\| write.metadata.path \| view location + /metadata \| Base location for metadata files \|

Make view metadata path configurable by write.metadata.path #12017

Make view metadata path configurable by write.metadata.path #12017

Conversation

tomtongue commented Jan 21, 2025 • edited Loading

Overview

Background

Example

tomtongue commented Jan 28, 2025 • edited Loading

nastra Jan 28, 2025

Choose a reason for hiding this comment

nastra Jan 28, 2025

Choose a reason for hiding this comment

tomtongue Jan 29, 2025

Choose a reason for hiding this comment

nastra Jan 28, 2025

Choose a reason for hiding this comment

nastra Jan 28, 2025

Choose a reason for hiding this comment

tomtongue Jan 29, 2025 • edited Loading

Choose a reason for hiding this comment

tomtongue commented Jan 28, 2025

tomtongue commented Jan 29, 2025

nastra Jan 29, 2025

Choose a reason for hiding this comment

tomtongue Jan 29, 2025

Choose a reason for hiding this comment

nastra left a comment

Choose a reason for hiding this comment

tomtongue commented Jan 29, 2025

amogh-jahagirdar left a comment

Choose a reason for hiding this comment

tomtongue commented Jan 30, 2025

Make view metadata path configurable by `write.metadata.path` #12017

Make view metadata path configurable by `write.metadata.path` #12017

tomtongue commented Jan 21, 2025 •

edited

Loading

tomtongue commented Jan 28, 2025 •

edited

Loading

tomtongue Jan 29, 2025 •

edited

Loading