Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add GraalVM Reachability Metadata and corresponding nativeTest for Iceberg table in HiveServer2 #31526

Merged
merged 1 commit into from
Aug 16, 2024

Conversation

linghengqian
Copy link
Member

@linghengqian linghengqian commented Jun 2, 2024

For #29052.

Changes proposed in this pull request:


Before committing this PR, I'm sure that I have checked the following options:

  • My code follows the code of conduct of this project.
  • I have self-reviewed the commit code.
  • I have (or in comment I request) added corresponding labels for the pull request.
  • I have passed maven check locally : ./mvnw clean install -B -T1C -Dmaven.javadoc.skip -Dmaven.jacoco.skip -e.
  • I have made corresponding changes to the documentation.
  • I have added corresponding unit tests for my changes.

@linghengqian linghengqian force-pushed the hive-native-test branch 18 times, most recently from 99f7063 to 02e7c95 Compare June 5, 2024 14:17
@linghengqian linghengqian force-pushed the hive-native-test branch 3 times, most recently from 7640e07 to e8bac17 Compare June 6, 2024 09:37
@linghengqian linghengqian changed the title Add GraalVM Reachability Metadata and corresponding nativeTest for Hive Add GraalVM Reachability Metadata and corresponding nativeTest for HiveServer2 Jun 6, 2024
Copy link
Member Author

@linghengqian linghengqian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

@linghengqian linghengqian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

@linghengqian linghengqian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

@linghengqian linghengqian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • We needed a transactional HiveServer2 for our unit tests, but the options provided by apache/hive were a bit overkill. Compared to manually creating ACID tables,
set metastore.compactor.initiator.on=true;
set metastore.compactor.cleaner.on=true;
set metastore.compactor.worker.threads=5;

set hive.support.concurrency=true;
set hive.exec.dynamic.partition.mode=nonstrict;
set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;

CREATE TABLE IF NOT EXISTS t_order
(
    order_id   BIGINT,
    order_type INT,
    user_id    INT    NOT NULL,
    address_id BIGINT NOT NULL,
    status     VARCHAR(50),
    PRIMARY KEY (order_id) disable novalidate
) CLUSTERED BY (order_id) INTO 2 BUCKETS STORED AS ORC TBLPROPERTIES ('transactional' = 'true');
  • I prefer to create Iceberg tables directly. Whether it is an ACID table or an iceberg table, from the perspective of the apache/shardingsphere parser, it only provides support for DML statements, and any transaction rollback operations are not supported.
set iceberg.mr.schema.auto.conversion=true;

CREATE TABLE IF NOT EXISTS t_order
(
    order_id   BIGINT,
    order_type INT,
    user_id    INT    NOT NULL,
    address_id BIGINT NOT NULL,
    status     VARCHAR(50),
    PRIMARY KEY (order_id) disable novalidate
) STORED BY ICEBERG STORED AS ORC TBLPROPERTIES ('format-version' = '2');

@linghengqian linghengqian changed the title Add GraalVM Reachability Metadata and corresponding nativeTest for HiveServer2 Add GraalVM Reachability Metadata and corresponding nativeTest for Iceberg table in HiveServer2 Aug 16, 2024
@linghengqian linghengqian force-pushed the hive-native-test branch 2 times, most recently from 4262004 to f4b2af0 Compare August 16, 2024 08:47
@linghengqian linghengqian marked this pull request as ready for review August 16, 2024 09:58
@strongduanmu strongduanmu merged commit 61c6207 into apache:master Aug 16, 2024
142 checks passed
@strongduanmu strongduanmu added this to the 5.5.1 milestone Aug 16, 2024
@linghengqian linghengqian deleted the hive-native-test branch August 16, 2024 10:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants