Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The materialized view partition ttl property has being lost after restarting fe #52093

Open
guanxianchun opened this issue Oct 18, 2024 · 2 comments
Labels
type/bug Something isn't working

Comments

@guanxianchun
Copy link

guanxianchun commented Oct 18, 2024

  1. create materialized view
    CREATE MATERIALIZED VIEW mv_hive_order (dt, product_id, gmv)
    PARTITION BY (str2date(dt, '%Y-%m-%d'))
    DISTRIBUTED BY HASH(product_id)
    REFRESH MANUAL
    PROPERTIES (
    "replicated_storage" = "true",
    "partition_refresh_number" = "1",
    "session.enable_spill" = "true",
    "replication_num" = "1",
    "session.query_timeout" = "3600",
    "partition_ttl" = "2 DAY",
    "storage_medium" = "HDD"
    )
    AS SELECT hive_order.dt, hive_order.product_id, sum(hive_order.price) AS gmv
    FROM hive_hms_catalog.ods.hive_order

  2. restart fe

  3. show create materialized view
    | mv_hive_order | CREATE MATERIALIZED VIEW mv_hive_order (dt, product_id, gmv)
    PARTITION BY (str2date(dt, '%Y-%m-%d'))
    DISTRIBUTED BY HASH(product_id)
    REFRESH MANUAL
    PROPERTIES (
    "replicated_storage" = "true",
    "partition_refresh_number" = "1",
    "session.enable_spill" = "true",
    "replication_num" = "1",
    "session.query_timeout" = "3600",
    "storage_medium" = "HDD"
    )
    AS SELECT hive_order.dt, hive_order.product_id, sum(hive_order.price) AS gmv
    FROM hive_hms_catalog.ods.hive_order
    GROUP BY hive_order.dt, hive_order.product_id;

fix partition ttl lost url : #51194
It was removed when calling the PropertyAnalyzer.analyzePartitionTTL method. Therefore, the properties need to be copied before calling the method and the copied object is used as a parameter. such code :
Map<String, String> props = new HashMap(properties);
Pair<String, PeriodDuration> ttlDuration = PropertyAnalyzer.analyzePartitionTTL(props);

@guanxianchun guanxianchun added the type/bug Something isn't working label Oct 18, 2024
@guanxianchun guanxianchun changed the title The problem of materialized view partition ttl property being lost after restarting fe The materialized view partition ttl property has being lost after restarting fe Oct 18, 2024
@rohitrs1983
Copy link
Contributor

rohitrs1983 commented Oct 20, 2024

@guanxianchun this is fixed as part of PR #51028
it seems your version does not have the fix.

@guanxianchun
Copy link
Author

guanxianchun commented Oct 21, 2024

@guanxianchun this is fixed as part of PR #51028 it seems your version does not have the fix.

I am using the latest code from branch 3.1, which already includes the bug fix code. This is a new issue that arose after using the analyzePartitionTTL method.

It was removed when calling the PropertyAnalyzer.analyzePartitionTTL method:

public static Pair<String, PeriodDuration> analyzePartitionTTL(Map<String, String> properties) {
if (properties != null && properties.containsKey(PROPERTIES_PARTITION_TTL)) {
String ttlStr = properties.get(PROPERTIES_PARTITION_TTL);
PeriodDuration duration;
try {
duration = TimeUtils.parseHumanReadablePeriodOrDuration(ttlStr);
} catch (NumberFormatException e) {
throw new SemanticException(String.format("illegal %s: %s", PROPERTIES_PARTITION_TTL, e.getMessage()));
}
properties.remove(PROPERTIES_PARTITION_TTL);
return Pair.create(ttlStr, duration);
}
return Pair.create(null, PeriodDuration.ZERO);
}

In the analyzePartitionTTL method, the PROPERTIES_PARTITION_TTL property was removed from properties. the code as follow:

properties.remove(PROPERTIES_PARTITION_TTL);

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants