Add GraalVM Reachability Metadata and corresponding nativeTest for Hi…

…veServer2
apache · Jun 10, 2024 · 9a81922 · 9a81922
1 parent e1b9bc9
commit 9a81922
Show file tree

Hide file tree

Showing 18 changed files with 676 additions and 52 deletions.
diff --git a/...ument/content/user-manual/shardingsphere-jdbc/graalvm-native-image/_index.cn.md b/...ument/content/user-manual/shardingsphere-jdbc/graalvm-native-image/_index.cn.md
@@ -295,6 +295,67 @@ Caused by: java.io.UnsupportedEncodingException: Codepage Cp1252 is not supporte
 
 ClickHouse 不支持 ShardingSphere 集成级别的本地事务，XA 事务和 Seata AT 模式事务，更多讨论位于 https://github.com/ClickHouse/clickhouse-docs/issues/2300 。
 
+7. 当需要通过 ShardingSphere JDBC 使用 Hive 方言时，受未关闭的 https://github.com/apache/logging-log4j2/issues/1539 影响，
+用户应排除 HiveServer2 JDBC Driver 中的 apache/logging-log4j2 和 apache/logging-log4j1 相关的库，
+并使用 qos-ch/slf4j 的其他实现来完成可能需要的日志记录。受 https://issues.apache.org/jira/browse/HIVE-28308 影响，
+用户不应该使用 `classifier` 为 `standalone` 的 `org.apache.hive:hive-jdbc:4.0.0`，以避免依赖冲突。
+若用户使用 qos-ch/logback 作为 qos-ch/slf4j 实现，则可能的配置例子如下，
+```xml
+<project>
+    <dependencies>
+      <dependency>
+         <groupId>org.apache.shardingsphere</groupId>
+         <artifactId>shardingsphere-jdbc</artifactId>
+         <version>${shardingsphere.version}</version>
+      </dependency>
+       <dependency>
+          <groupId>org.apache.shardingsphere</groupId>
+          <artifactId>shardingsphere-parser-sql-hive</artifactId>
+          <version>${shardingsphere.version}</version>
+      </dependency>
+       <dependency>
+          <groupId>org.apache.hive</groupId>
+          <artifactId>hive-jdbc</artifactId>
+          <version>4.0.0</version>
+       </dependency>
+       <dependency>
+          <groupId>org.apache.hive</groupId>
+          <artifactId>hive-service</artifactId>
+          <version>4.0.0</version>
+          <exclusions>
+             <exclusion>
+                <groupId>org.apache.logging.log4j</groupId>
+                <artifactId>log4j-slf4j-impl</artifactId>
+             </exclusion>
+             <exclusion>
+                <groupId>org.slf4j</groupId>
+                <artifactId>slf4j-reload4j</artifactId>
+             </exclusion>
+             <exclusion>
+                <groupId>org.apache.logging.log4j</groupId>
+                <artifactId>log4j-api</artifactId>
+             </exclusion>
+          </exclusions>
+       </dependency>
+       <dependency>
+          <groupId>org.apache.hadoop</groupId>
+          <artifactId>hadoop-client-runtime</artifactId>
+          <version>3.3.6</version>
+       </dependency>
+    </dependencies>
+</project>
+```
+受 https://github.com/grpc/grpc-java/issues/10601 影响，用户如果在项目中引入了 `org.apache.hive:hive-service`，
+则需要在项目的 classpath 的 `META-INF/native-image/io.grpc/grpc-netty-shaded` 文件夹下创建包含如下内容的文件 `native-image.properties`，
+```properties
+Args=--initialize-at-run-time=\
+  io.grpc.netty.shaded.io.netty.channel,\
+  io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise,\
+  io.grpc.netty.shaded.io.netty.util.internal.MacAddressUtil,\
+  io.grpc.netty.shaded.io.netty.util.internal.SystemPropertyUtil,\
+  io.grpc.netty.shaded.io.netty.util.NetUtilInitializations
+```
+
 ## 贡献 GraalVM Reachability Metadata
 
 ShardingSphere 对在 GraalVM Native Image 下的可用性的验证，是通过 GraalVM Native Build Tools 的 Maven Plugin 子项目来完成的。

diff --git a/...ument/content/user-manual/shardingsphere-jdbc/graalvm-native-image/_index.en.md b/...ument/content/user-manual/shardingsphere-jdbc/graalvm-native-image/_index.en.md
@@ -110,6 +110,69 @@ graalvmNative {
 }
 ```
 
+7. When using the Hive dialect via ShardingSphere JDBC, users are affected by the unresolved issue at https://github.com/apache/logging-log4j2/issues/1539,
+necessitating the exclusion of libraries related to `apache/logging-log4j2` and `apache/logging-log4j1` from the HiveServer2 JDBC Driver.
+Instead, they should employ alternative implementations from `qos-ch/slf4j` for any required logging functionalities.
+Further, due to the impact of https://issues.apache.org/jira/browse/HIVE-28308,
+users must refrain from using the `classifier` `standalone` version of `org.apache.hive:hive-jdbc:4.0.0` to circumvent dependency conflicts. 
+If opting for `qos-ch/logback` as the `qos-ch/slf4j` implementation, a possible configuration example could be structured as follows,
+```xml
+<project>
+    <dependencies>
+      <dependency>
+         <groupId>org.apache.shardingsphere</groupId>
+         <artifactId>shardingsphere-jdbc</artifactId>
+         <version>${shardingsphere.version}</version>
+      </dependency>
+       <dependency>
+          <groupId>org.apache.shardingsphere</groupId>
+          <artifactId>shardingsphere-parser-sql-hive</artifactId>
+          <version>${shardingsphere.version}</version>
+      </dependency>
+       <dependency>
+          <groupId>org.apache.hive</groupId>
+          <artifactId>hive-jdbc</artifactId>
+          <version>4.0.0</version>
+       </dependency>
+       <dependency>
+          <groupId>org.apache.hive</groupId>
+          <artifactId>hive-service</artifactId>
+          <version>4.0.0</version>
+          <exclusions>
+             <exclusion>
+                <groupId>org.apache.logging.log4j</groupId>
+                <artifactId>log4j-slf4j-impl</artifactId>
+             </exclusion>
+             <exclusion>
+                <groupId>org.slf4j</groupId>
+                <artifactId>slf4j-reload4j</artifactId>
+             </exclusion>
+             <exclusion>
+                <groupId>org.apache.logging.log4j</groupId>
+                <artifactId>log4j-api</artifactId>
+             </exclusion>
+          </exclusions>
+       </dependency>
+       <dependency>
+          <groupId>org.apache.hadoop</groupId>
+          <artifactId>hadoop-client-runtime</artifactId>
+          <version>3.3.6</version>
+       </dependency>
+    </dependencies>
+</project>
+```
+Affected by https://github.com/grpc-java/issues/10601, should users incorporate `org.apache.hive:hive-service` into their project, 
+it is imperative to create a file named `native-image.properties` within the directory `META-INF/native-image/io.grpc/grpc-netty-shaded` of the classpath, 
+containing the following content,
+```properties
+Args=--initialize-at-run-time=\
+  io.grpc.netty.shaded.io.netty.channel,\
+  io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise,\
+  io.grpc.netty.shaded.io.netty.util.internal.MacAddressUtil,\
+  io.grpc.netty.shaded.io.netty.util.internal.SystemPropertyUtil,\
+  io.grpc.netty.shaded.io.netty.util.NetUtilInitializations
+```
+
 ### For build tools such as sbt that are not supported by GraalVM Native Build Tools
 
 Such requirements require opening additional issues at https://github.com/graalvm/native-build-tools 

diff --git a/.../native-image/org.apache.shardingsphere/generated-reachability-metadata/proxy-config.json b/.../native-image/org.apache.shardingsphere/generated-reachability-metadata/proxy-config.json
@@ -6,5 +6,9 @@
   {
     "condition":{"typeReachable":"org.apache.shardingsphere.mode.repository.standalone.jdbc.JDBCRepository"},
     "interfaces":["java.sql.Connection"]
+  },
+  {
+    "condition":{"typeReachable":"org.apache.shardingsphere.mode.metadata.MetaDataContextsFactory"},
+    "interfaces":["org.apache.hive.service.rpc.thrift.TCLIService$Iface"]
   }
 ]
diff --git a/...ative-image/org.apache.shardingsphere/generated-reachability-metadata/reflect-config.json b/...ative-image/org.apache.shardingsphere/generated-reachability-metadata/reflect-config.json
@@ -16,7 +16,31 @@
   "name":"[Lcom.github.dockerjava.api.model.VolumesFrom;"
 },
 {
-  "condition":{"typeReachable":"org.apache.shardingsphere.infra.datasource.pool.CatalogSwitchableDataSource"},
+  "condition":{"typeReachable":"org.apache.shardingsphere.infra.database.DatabaseTypeEngine"},
+  "name":"[Lcom.zaxxer.hikari.util.ConcurrentBag$IConcurrentBagEntry;"
+},
+{
+  "condition":{"typeReachable":"org.apache.shardingsphere.infra.database.core.metadata.data.loader.type.TableMetaDataLoader"},
+  "name":"[Lcom.zaxxer.hikari.util.ConcurrentBag$IConcurrentBagEntry;"
+},
+{
+  "condition":{"typeReachable":"org.apache.shardingsphere.infra.database.h2.metadata.data.loader.H2MetaDataLoader"},
+  "name":"[Lcom.zaxxer.hikari.util.ConcurrentBag$IConcurrentBagEntry;"
+},
+{
+  "condition":{"typeReachable":"org.apache.shardingsphere.infra.database.mysql.metadata.data.loader.MySQLMetaDataLoader"},
+  "name":"[Lcom.zaxxer.hikari.util.ConcurrentBag$IConcurrentBagEntry;"
+},
+{
+  "condition":{"typeReachable":"org.apache.shardingsphere.infra.database.opengauss.metadata.data.loader.OpenGaussMetaDataLoader"},
+  "name":"[Lcom.zaxxer.hikari.util.ConcurrentBag$IConcurrentBagEntry;"
+},
+{
+  "condition":{"typeReachable":"org.apache.shardingsphere.infra.database.postgresql.metadata.data.loader.PostgreSQLMetaDataLoader"},
+  "name":"[Lcom.zaxxer.hikari.util.ConcurrentBag$IConcurrentBagEntry;"
+},
+{
+  "condition":{"typeReachable":"org.apache.shardingsphere.infra.database.sqlserver.metadata.data.loader.SQLServerMetaDataLoader"},
   "name":"[Lcom.zaxxer.hikari.util.ConcurrentBag$IConcurrentBagEntry;"
 },
 {
@@ -360,6 +384,18 @@
   "condition":{"typeReachable":"org.apache.shardingsphere.infra.database.core.type.DatabaseTypeFactory"},
   "name":"org.apache.shardingsphere.infra.database.h2.type.H2DatabaseType"
 },
+{
+  "condition":{"typeReachable":"org.apache.shardingsphere.infra.datasource.pool.creator.DataSourcePoolReflection"},
+  "name":"org.apache.shardingsphere.infra.database.hive.connector.HiveConnectionPropertiesParser"
+},
+{
+  "condition":{"typeReachable":"org.apache.shardingsphere.infra.database.core.type.DatabaseTypeRegistry"},
+  "name":"org.apache.shardingsphere.infra.database.hive.metadata.database.HiveDatabaseMetaData"
+},
+{
+  "condition":{"typeReachable":"org.apache.shardingsphere.infra.database.core.type.DatabaseTypeFactory"},
+  "name":"org.apache.shardingsphere.infra.database.hive.type.HiveDatabaseType"
+},
 {
   "condition":{"typeReachable":"org.apache.shardingsphere.infra.database.core.type.DatabaseTypeFactory"},
   "name":"org.apache.shardingsphere.infra.database.mariadb.type.MariaDBDatabaseType"
@@ -580,7 +616,7 @@
   "queryAllDeclaredMethods":true
 },
 {
-  "condition":{"typeReachable":"org.apache.shardingsphere.mode.subsciber.EventSubscriberRegistry$$Lambda/0x00007fce3b398818"},
+  "condition":{"typeReachable":"org.apache.shardingsphere.mode.subsciber.EventSubscriberRegistry$$Lambda/0x00007f36af398208"},
   "name":"org.apache.shardingsphere.infra.util.eventbus.EventSubscriber"
 },
 {
@@ -947,7 +983,7 @@
   "name":"org.apache.shardingsphere.mode.manager.cluster.coordinator.subscriber.ResourceMetaDataChangedSubscriber"
 },
 {
-  "condition":{"typeReachable":"org.apache.shardingsphere.mode.manager.cluster.coordinator.listener.MetaDataChangedListener$$Lambda/0x00007fce3b8eaca0"},
+  "condition":{"typeReachable":"org.apache.shardingsphere.mode.manager.cluster.coordinator.listener.MetaDataChangedListener$$Lambda/0x00007f36af90f640"},
   "name":"org.apache.shardingsphere.mode.manager.cluster.coordinator.subscriber.ResourceMetaDataChangedSubscriber"
 },
 {
@@ -1599,6 +1635,11 @@
   "condition":{"typeReachable":"org.apache.shardingsphere.sql.parser.core.database.cache.ParseTreeCacheBuilder"},
   "name":"org.apache.shardingsphere.sql.parser.core.database.cache.ParseTreeCacheLoader"
 },
+{
+  "condition":{"typeReachable":"org.apache.shardingsphere.sql.parser.core.database.visitor.SQLStatementVisitorFactory"},
+  "name":"org.apache.shardingsphere.sql.parser.hive.visitor.statement.HiveStatementVisitorFacade",
+  "methods":[{"name":"<init>","parameterTypes":[] }]
+},
 {
   "condition":{"typeReachable":"org.apache.shardingsphere.sql.parser.core.SQLParserFactory"},
   "name":"org.apache.shardingsphere.sql.parser.mysql.parser.MySQLLexer",
@@ -1694,6 +1735,16 @@
   "name":"org.apache.shardingsphere.sql.parser.sql.dialect.statement.clickhouse.dml.ClickHouseSelectStatement",
   "methods":[{"name":"<init>","parameterTypes":[] }]
 },
+{
+  "condition":{"typeReachable":"org.apache.shardingsphere.infra.binder.statement.dml.DeleteStatementBinder"},
+  "name":"org.apache.shardingsphere.sql.parser.sql.dialect.statement.hive.dml.HiveDeleteStatement",
+  "methods":[{"name":"<init>","parameterTypes":[] }]
+},
+{
+  "condition":{"typeReachable":"org.apache.shardingsphere.driver.jdbc.core.statement.ShardingSpherePreparedStatement"},
+  "name":"org.apache.shardingsphere.sql.parser.sql.dialect.statement.hive.dml.HiveSelectStatement",
+  "methods":[{"name":"<init>","parameterTypes":[] }]
+},
 {
   "condition":{"typeReachable":"org.apache.shardingsphere.infra.binder.statement.dml.DeleteStatementBinder"},
   "name":"org.apache.shardingsphere.sql.parser.sql.dialect.statement.mysql.dml.MySQLDeleteStatement",

diff --git a/...tive-image/org.apache.shardingsphere/generated-reachability-metadata/resource-config.json b/...tive-image/org.apache.shardingsphere/generated-reachability-metadata/resource-config.json
@@ -2766,6 +2766,9 @@
   }, {
     "condition":{"typeReachable":"org.apache.shardingsphere.infra.url.classpath.ClassPathURLLoader"},
     "pattern":"\\Qtest-native/yaml/databases/clickhouse.yaml\\E"
+  }, {
+    "condition":{"typeReachable":"org.apache.shardingsphere.infra.url.classpath.ClassPathURLLoader"},
+    "pattern":"\\Qtest-native/yaml/databases/hive.yaml\\E"
   }, {
     "condition":{"typeReachable":"org.apache.shardingsphere.infra.url.classpath.ClassPathURLLoader"},
     "pattern":"\\Qtest-native/yaml/databases/mysql.yaml\\E"

diff --git a/.../org.apache.shardingsphere/shardingsphere-infra-reachability-metadata/reflect-config.json b/.../org.apache.shardingsphere/shardingsphere-infra-reachability-metadata/reflect-config.json
@@ -53,5 +53,25 @@
   "condition":{"typeReachable":"org.apache.shardingsphere.infra.util.eventbus.EventBusContext"},
   "name":"org.apache.shardingsphere.mode.manager.cluster.coordinator.subscriber.StateChangedSubscriber",
   "methods":[{"name":"renew","parameterTypes":["org.apache.shardingsphere.mode.manager.cluster.coordinator.registry.status.cluster.event.ClusterStateEvent"] }]
+},
+{
+  "condition":{"typeReachable":"org.apache.shardingsphere.sql.parser.hive.parser.HiveLexer"},
+  "name":"org.apache.shardingsphere.sql.parser.hive.parser.HiveLexer",
+  "methods":[{"name":"<init>","parameterTypes":["org.antlr.v4.runtime.CharStream"] }]
+},
+{
+  "condition":{"typeReachable":"org.apache.shardingsphere.sql.parser.hive.parser.HiveParser"},
+  "name":"org.apache.shardingsphere.sql.parser.hive.parser.HiveParser",
+  "methods":[{"name":"<init>","parameterTypes":["org.antlr.v4.runtime.TokenStream"] }]
+},
+{
+  "condition":{"typeReachable":"org.apache.shardingsphere.sql.parser.hive.visitor.statement.type.HiveDMLStatementVisitor"},
+  "name":"org.apache.shardingsphere.sql.parser.hive.visitor.statement.type.HiveDMLStatementVisitor",
+  "methods":[{"name":"<init>","parameterTypes":[] }]
+},
+{
+  "condition":{"typeReachable":"org.apache.shardingsphere.sql.parser.sql.dialect.statement.hive.dml.HiveInsertStatement"},
+  "name":"org.apache.shardingsphere.sql.parser.sql.dialect.statement.hive.dml.HiveInsertStatement",
+  "methods":[{"name":"<init>","parameterTypes":[] }]
 }
 ]
diff --git a/pom.xml b/pom.xml
@@ -126,6 +126,8 @@
         <opengauss.version>3.1.0-og</opengauss.version>
         <mariadb-java-client.version>2.4.2</mariadb-java-client.version>
         <clickhouse-jdbc.version>0.6.0-patch5</clickhouse-jdbc.version>
+        <hive.version>4.0.0</hive.version>
+        <hadoop.version>3.3.6</hadoop.version>
 
         <hikari-cp.version>4.0.3</hikari-cp.version>
 
@@ -479,6 +481,24 @@
                 <classifier>http</classifier>
                 <scope>test</scope>
             </dependency>
+            <dependency>
+                <groupId>org.apache.hive</groupId>
+                <artifactId>hive-jdbc</artifactId>
+                <version>${hive.version}</version>
+                <scope>test</scope>
+            </dependency>
+            <dependency>
+                <groupId>org.apache.hive</groupId>
+                <artifactId>hive-service</artifactId>
+                <version>${hive.version}</version>
+                <scope>test</scope>
+            </dependency>
+            <dependency>
+                <groupId>org.apache.hadoop</groupId>
+                <artifactId>hadoop-client-runtime</artifactId>
+                <version>${hadoop.version}</version>
+                <scope>test</scope>
+            </dependency>
 
             <dependency>
                 <groupId>com.zaxxer</groupId>
@@ -970,7 +990,8 @@
                             <artifactId>maven-surefire-plugin</artifactId>
                             <configuration>
                                 <trimStackTrace>false</trimStackTrace>
-                                <argLine>--add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.lang.reflect=ALL-UNNAMED</argLine>
+                                <!-- For `org.apache.hive:hive-common:4.0.0`-->
+                                <argLine>--add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED</argLine>
                             </configuration>
                         </plugin>
                     </plugins>

diff --git a/test/native/native-image-filter/extra-filter.json b/test/native/native-image-filter/extra-filter.json
@@ -2,14 +2,16 @@
   "rules": [
     {"includeClasses": "**"},
 
-    {"excludeClasses": "com.sun.crypto.provider.**"},
-    {"excludeClasses": "com.sun.org.apache.xerces.internal.jaxp.**"},
+    {"excludeClasses": "com.sun.**"},
     {"excludeClasses": "java.**"},
     {"includeClasses": "java.util.Properties"},
+    {"excludeClasses": "javax.script.**"},
     {"excludeClasses": "javax.security.auth.x500.**"},
     {"excludeClasses": "javax.smartcardio.**"},
+    {"excludeClasses": "javax.swing.**"},
     {"excludeClasses": "javax.xml.stream.**"},
     {"excludeClasses": "jdk.internal.misc.**"},
+    {"excludeClasses": "org.w3c.dom.**"},
     {"excludeClasses": "sun.misc.**"},
     {"excludeClasses": "sun.nio.ch.**"},
     {"excludeClasses": "sun.security.**"},
@@ -39,6 +41,8 @@
     {"excludeClasses": "net.bytebuddy.**"},
     {"excludeClasses": "org.apache.calcite.**"},
     {"excludeClasses": "org.apache.commons.logging.**"},
+    {"excludeClasses": "org.apache.derby.jdbc.**"},
+    {"excludeClasses": "org.apache.hive.**"},
     {"excludeClasses": "org.apache.logging.log4j.**"},
     {"excludeClasses": "org.apache.log4j.**"},
     {"excludeClasses": "org.apache.zookeeper.**"},