Skip to content

Commit

Permalink
Fixes the issue where HiveMetaDataLoader creates an additional embe…
Browse files Browse the repository at this point in the history
…dded HiveServer2 when using embedded Hive Metastore Server
  • Loading branch information
linghengqian committed Aug 16, 2024
1 parent 6f4cac7 commit 9205fb1
Show file tree
Hide file tree
Showing 8 changed files with 183 additions and 410 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -25,13 +25,6 @@ ShardingSphere JDBC 要求在如下或更高版本的 `GraalVM CE` 完成构建
```shell
com.sun.beans.introspect.ClassInfo was unintentionally initialized at build time. To see why com.sun.beans.introspect.ClassInfo got initialized use --trace-class-initialization=com.sun.beans.introspect.ClassInfo
java.beans.Introspector was unintentionally initialized at build time. To see why java.beans.Introspector got initialized use --trace-class-initialization=java.beans.Introspector
com.sun.beans.util.Cache$Kind$2 was unintentionally initialized at build time. To see why com.sun.beans.util.Cache$Kind$2 got initialized use --trace-class-initialization=com.sun.beans.util.Cache$Kind$2
com.sun.beans.TypeResolver was unintentionally initialized at build time. To see why com.sun.beans.TypeResolver got initialized use --trace-class-initialization=com.sun.beans.TypeResolver
java.beans.ThreadGroupContext was unintentionally initialized at build time. To see why java.beans.ThreadGroupContext got initialized use --trace-class-initialization=java.beans.ThreadGroupContext
com.sun.beans.util.Cache$Kind was unintentionally initialized at build time. To see why com.sun.beans.util.Cache$Kind got initialized use --trace-class-initialization=com.sun.beans.util.Cache$Kind
com.sun.beans.introspect.MethodInfo was unintentionally initialized at build time. To see why com.sun.beans.introspect.MethodInfo got initialized use --trace-class-initialization=com.sun.beans.introspect.MethodInfo
com.sun.beans.util.Cache$Kind$1 was unintentionally initialized at build time. To see why com.sun.beans.util.Cache$Kind$1 got initialized use --trace-class-initialization=com.sun.beans.util.Cache$Kind$1
com.sun.beans.util.Cache$Kind$3 was unintentionally initialized at build time. To see why com.sun.beans.util.Cache$Kind$3 got initialized use --trace-class-initialization=com.sun.beans.util.Cache$Kind$3
```

### Maven 生态
Expand Down Expand Up @@ -87,7 +80,7 @@ com.sun.beans.util.Cache$Kind$3 was unintentionally initialized at build time. T

使用者需要主动使用 GraalVM Reachability Metadata 中央仓库。
如下配置可供参考,以配置项目额外的 Gradle Tasks,以 GraalVM Native Build Tools 的文档为准。
由于 Gradle 8.6 的限制,用户需要通过 Maven 依赖的形式引入 Metadata Repository 的 JSON 文件。
由于 https://github.com/gradle/gradle/issues/17559 的限制,用户需要通过 Maven 依赖的形式引入 Metadata Repository 的 JSON 文件。
参考 https://github.com/graalvm/native-build-tools/issues/572

```groovy
Expand Down Expand Up @@ -346,7 +339,7 @@ ShardingSphere 定义了 `shardingsphere-test-native` 的 Maven Module 用于为

ShardingSphere 定义了 `nativeTestInShardingSphere` 的 Maven Profile 用于为 `shardingsphere-test-native` 模块执行 nativeTest 。

假设贡献者处于新的 Ubuntu 22.04.3 LTS 实例下,其可通过如下 bash 命令通过 SDKMAN! 管理 JDK 和工具链,
假设贡献者处于新的 Ubuntu 22.04.4 LTS 实例下,其可通过如下 bash 命令通过 SDKMAN! 管理 JDK 和工具链,
并为 `shardingsphere-test-native` 子模块执行 nativeTest。

你必须安装 Docker Engine 以执行 `testcontainers-java` 相关的单元测试。
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,13 +26,6 @@ GraalVM CE only supports AWT for GraalVM CE For JDK22 and higher versions.
```shell
com.sun.beans.introspect.ClassInfo was unintentionally initialized at build time. To see why com.sun.beans.introspect.ClassInfo got initialized use --trace-class-initialization=com.sun.beans.introspect.ClassInfo
java.beans.Introspector was unintentionally initialized at build time. To see why java.beans.Introspector got initialized use --trace-class-initialization=java.beans.Introspector
com.sun.beans.util.Cache$Kind$2 was unintentionally initialized at build time. To see why com.sun.beans.util.Cache$Kind$2 got initialized use --trace-class-initialization=com.sun.beans.util.Cache$Kind$2
com.sun.beans.TypeResolver was unintentionally initialized at build time. To see why com.sun.beans.TypeResolver got initialized use --trace-class-initialization=com.sun.beans.TypeResolver
java.beans.ThreadGroupContext was unintentionally initialized at build time. To see why java.beans.ThreadGroupContext got initialized use --trace-class-initialization=java.beans.ThreadGroupContext
com.sun.beans.util.Cache$Kind was unintentionally initialized at build time. To see why com.sun.beans.util.Cache$Kind got initialized use --trace-class-initialization=com.sun.beans.util.Cache$Kind
com.sun.beans.introspect.MethodInfo was unintentionally initialized at build time. To see why com.sun.beans.introspect.MethodInfo got initialized use --trace-class-initialization=com.sun.beans.introspect.MethodInfo
com.sun.beans.util.Cache$Kind$1 was unintentionally initialized at build time. To see why com.sun.beans.util.Cache$Kind$1 got initialized use --trace-class-initialization=com.sun.beans.util.Cache$Kind$1
com.sun.beans.util.Cache$Kind$3 was unintentionally initialized at build time. To see why com.sun.beans.util.Cache$Kind$3 got initialized use --trace-class-initialization=com.sun.beans.util.Cache$Kind$3
```

### Maven Ecology
Expand Down Expand Up @@ -90,7 +83,7 @@ and the documentation of GraalVM Native Build Tools shall prevail.
Users need to actively use the GraalVM Reachability Metadata central repository.
The following configuration is for reference to configure additional Gradle Tasks for the project,
and the documentation of GraalVM Native Build Tools shall prevail.
Due to the limitations of Gradle 8.6,
Due to the limitations of https://github.com/gradle/gradle/issues/17559 ,
users need to introduce the JSON file of Metadata Repository through Maven dependency.
Reference https://github.com/graalvm/native-build-tools/issues/572 .

Expand Down Expand Up @@ -362,7 +355,7 @@ This subset of unit tests avoids the use of third-party libraries such as Mockit

ShardingSphere defines the Maven Profile of `nativeTestInShardingSphere` for executing nativeTest for the `shardingsphere-test-native` module.

Assuming that the contributor is under a new Ubuntu 22.04.3 LTS instance, Contributors can manage the JDK and tool chain through
Assuming that the contributor is under a new Ubuntu 22.04.4 LTS instance, Contributors can manage the JDK and tool chain through
`SDKMAN!` through the following bash command, and execute nativeTest for the `shardingsphere-test-native` submodule.

You must install Docker Engine to execute `testcontainers-java` related unit tests.
Expand All @@ -386,7 +379,7 @@ on https://github.com/oracle/graalvm-reachability-metadata . ShardingSphere acti
some third-party libraries in the `shardingsphere-infra-reachability-metadata` submodule.

If nativeTest execution fails, preliminary GraalVM Reachability Metadata should be generated for unit tests,
and manually adjust the contents of the `META-INF/native-image/org.apache.shardingsphere/shardingsphere-infra-reachability-metadata` folder on the classpath of the `shardingsphere-infra-reachability-metadata` submodule to fix nativeTest.
and manually adjust the contents of the `META-INF/native-image/org.apache.shardingsphere/shardingsphere-infra-reachability-metadata/` folder on the classpath of the `shardingsphere-infra-reachability-metadata` submodule to fix nativeTest.
If necessary,
use the `org.junit.jupiter.api.condition.DisabledInNativeImage` annotation or the `org.graalvm.nativeimage.imagecode` System Property blocks some unit tests from running under GraalVM Native Image.

Expand Down
10 changes: 10 additions & 0 deletions infra/database/type/hive/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,16 @@
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-jdbc</artifactId>
<version>${hive.version}</version>
<scope>provided</scope>
<optional>true</optional>
</dependency>
<dependency>
<groupId>io.github.linghengqian</groupId>
<artifactId>hive-server2-jdbc-driver-thin</artifactId>
<version>${hive-server2-jdbc-driver-thin.version}</version>
<scope>provided</scope>
<optional>true</optional>
</dependency>

<dependency>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
import org.apache.hadoop.hive.metastore.api.Table;
import org.apache.shardingsphere.infra.database.core.metadata.data.loader.DialectMetaDataLoader;
import org.apache.shardingsphere.infra.database.core.metadata.data.loader.MetaDataLoaderMaterial;
import org.apache.shardingsphere.infra.database.core.metadata.data.loader.type.TableMetaDataLoader;
import org.apache.shardingsphere.infra.database.core.metadata.data.model.ColumnMetaData;
import org.apache.shardingsphere.infra.database.core.metadata.data.model.SchemaMetaData;
import org.apache.shardingsphere.infra.database.core.metadata.data.model.TableMetaData;
Expand All @@ -31,7 +32,9 @@

import javax.sql.DataSource;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.sql.Types;
import java.util.Collection;
import java.util.Collections;
Expand All @@ -45,13 +48,27 @@ public final class HiveMetaDataLoader implements DialectMetaDataLoader {

private static final String HIVE_METASTORE_URIS = "hive.metastore.uris";

@SuppressWarnings("SqlNoDataSourceInspection")
@Override
public Collection<SchemaMetaData> load(final MetaDataLoaderMaterial material) throws SQLException {
String hiveMetastoreUris;
try (Statement statement = material.getDataSource().getConnection().createStatement()) {
ResultSet resultSet = statement.executeQuery("SET hive.metastore.uris");
resultSet.next();
hiveMetastoreUris = resultSet.getString("set");
}
if ("hive.metastore.uris is undefined".equals(hiveMetastoreUris)) {
Collection<TableMetaData> tableMetaData = new LinkedList<>();
for (String each : material.getActualTableNames()) {
TableMetaDataLoader.load(material.getDataSource(), each, material.getStorageType()).ifPresent(tableMetaData::add);
}
return Collections.singletonList(new SchemaMetaData(material.getDefaultSchemaName(), tableMetaData));
}
HiveMetaStoreClient storeClient = null;
try {
// TODO Support set hive.metastore uris when register storage unit.
HiveConf hiveConf = new HiveConf();
hiveConf.set(HIVE_METASTORE_URIS, "");
hiveConf.set(HIVE_METASTORE_URIS, hiveMetastoreUris);
storeClient = new HiveMetaStoreClient(hiveConf);
return Collections.singletonList(new SchemaMetaData(material.getDefaultSchemaName(),
getTableMetaData(storeClient.getAllTables(material.getDefaultSchemaName()), storeClient, material)));
Expand Down
Loading

0 comments on commit 9205fb1

Please sign in to comment.