Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevent use of GraalVM Native Build Tools Maven Plugin in parallel builds of project #32574

Merged
merged 1 commit into from
Aug 18, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/workflows/graalvm.yml
Original file line number Diff line number Diff line change
Expand Up @@ -54,5 +54,6 @@ jobs:
distribution: 'graalvm-community'
github-token: ${{ secrets.GITHUB_TOKEN }}
cache: 'maven'
native-image-job-reports: 'true'
- name: Run nativeTest with GraalVM CE for ${{ matrix.java-version }}
run: ./mvnw -PnativeTestInShardingSphere -T1C -e clean test
run: ./mvnw -PnativeTestInShardingSphere -e clean test
9 changes: 2 additions & 7 deletions .github/workflows/nightly-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -123,13 +123,8 @@ jobs:
java-version: '22.0.2'
distribution: 'graalvm-community'
github-token: ${{ secrets.GITHUB_TOKEN }}
- uses: actions/cache@v4
with:
path: ~/.m2/repository
key: ${{ needs.global-environment.outputs.GLOBAL_CACHE_PREFIX }}-maven-third-party-cache-${{ github.sha }}
restore-keys: |
${{ needs.global-environment.outputs.GLOBAL_CACHE_PREFIX }}-maven-third-party-cache-
${{ needs.global-environment.outputs.GLOBAL_CACHE_PREFIX }}-maven-third-party-
cache: 'maven'
native-image-job-reports: 'true'
- uses: docker/login-action@v3
with:
registry: ${{ env.HUB }}
Expand Down
11 changes: 3 additions & 8 deletions .github/workflows/nightly-ci-dynamic.yml
Original file line number Diff line number Diff line change
Expand Up @@ -129,12 +129,7 @@ jobs:
java-version: '22.0.2'
distribution: 'graalvm-community'
github-token: ${{ secrets.GITHUB_TOKEN }}
- uses: actions/cache@v4
with:
path: ~/.m2/repository
key: ${{ needs.global-environment.outputs.GLOBAL_CACHE_PREFIX }}-maven-third-party-cache-${{ github.sha }}
restore-keys: |
${{ needs.global-environment.outputs.GLOBAL_CACHE_PREFIX }}-maven-third-party-cache-
${{ needs.global-environment.outputs.GLOBAL_CACHE_PREFIX }}-maven-third-party-
cache: 'maven'
native-image-job-reports: 'true'
- name: Run nativeTest with GraalVM CE
run: ./mvnw -PnativeTestInShardingSphere -T1C -e clean test
run: ./mvnw -PnativeTestInShardingSphere -e clean test
11 changes: 3 additions & 8 deletions .github/workflows/nightly-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -118,12 +118,7 @@ jobs:
java-version: '22.0.2'
distribution: 'graalvm-community'
github-token: ${{ secrets.GITHUB_TOKEN }}
- uses: actions/cache@v4
with:
path: ~/.m2/repository
key: ${{ needs.global-environment.outputs.GLOBAL_CACHE_PREFIX }}-maven-third-party-cache-${{ github.sha }}
restore-keys: |
${{ needs.global-environment.outputs.GLOBAL_CACHE_PREFIX }}-maven-third-party-cache-
${{ needs.global-environment.outputs.GLOBAL_CACHE_PREFIX }}-maven-third-party-
cache: 'maven'
native-image-job-reports: 'true'
- name: Run nativeTest with GraalVM CE
run: ./mvnw -PnativeTestInShardingSphere -T1C -e clean test
run: ./mvnw -PnativeTestInShardingSphere -e clean test
2 changes: 1 addition & 1 deletion distribution/proxy/src/main/release-docs/LICENSE
Original file line number Diff line number Diff line change
Expand Up @@ -240,7 +240,7 @@ The text of each license is the standard Apache 2.0 license.
failsafe 2.4.4: https://github.com/jhalterman/failsafe, Apache 2.0
failureaccess 1.0.1: https://github.com/google/guava, Apache 2.0
freemarker 2.3.31: https://freemarker.apache.org/, Apache 2.0
groovy 4.0.19: https://groovy.apache.org/, Apache 2.0
groovy 4.0.22: https://groovy.apache.org/, Apache 2.0
grpc-api 1.65.1: https://github.com/grpc/grpc-java, Apache 2.0
grpc-context 1.65.1: https://github.com/grpc/grpc-java, Apache 2.0
grpc-core 1.65.1: https://github.com/grpc/grpc-java, Apache 2.0
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -393,7 +393,7 @@ ClickHouse 不支持 ShardingSphere 集成级别的本地事务,XA 事务和 S
<dependency>
<groupId>io.github.linghengqian</groupId>
<artifactId>hive-server2-jdbc-driver-thin</artifactId>
<version>1.0.0</version>
<version>1.2.0</version>
<exclusions>
<exclusion>
<groupId>com.fasterxml.woodstox</groupId>
Expand Down Expand Up @@ -442,8 +442,7 @@ Args=--initialize-at-run-time=\
用户应当考虑在 ShardingSphere JDBC 中仅使用支持 ACID 的表。`apache/hive` 提供了多种事务解决方案。

第1种选择是使用 ACID 表,可能的建表流程如下。
如果用户在 ACID 表上频繁更新和删除数据,用户可能不得不在 DML 语句执行前后进行等待,
以让 HiveServer2 完成低效的 DML 操作,因为 ACID 表中数据是在文件夹级别跟踪。
由于其过时的基于目录的表格式,用户可能不得不在 DML 语句执行前后进行等待,以让 HiveServer2 完成低效的 DML 操作。

```sql
set metastore.compactor.initiator.on=true;
Expand All @@ -465,7 +464,9 @@ CREATE TABLE IF NOT EXISTS t_order
) CLUSTERED BY (order_id) INTO 2 BUCKETS STORED AS ORC TBLPROPERTIES ('transactional' = 'true');
```

第2种选择是使用 Iceberg 表,可能的建表流程如下。`apache/iceberg` 表格式有望在未来几年取代传统的 Hive 表格式。
第2种选择是使用 Iceberg 表,可能的建表流程如下。
Apache Iceberg 表格式有望在未来几年取代传统的 Hive 表格式,
参考 https://blog.cloudera.com/from-hive-tables-to-iceberg-tables-hassle-free/ 。

```sql
set iceberg.mr.schema.auto.conversion=true;
Expand Down Expand Up @@ -510,7 +511,7 @@ sudo apt-get install build-essential zlib1g-dev -y

git clone [email protected]:apache/shardingsphere.git
cd ./shardingsphere/
./mvnw -PnativeTestInShardingSphere -T1C -e clean test
./mvnw -PnativeTestInShardingSphere -e clean test
```

当贡献者发现缺少与 ShardingSphere 无关的第三方库的 GraalVM Reachability Metadata 时,应当在
Expand Down Expand Up @@ -539,5 +540,10 @@ Reachability Metadata 位于 `shardingsphere-infra-reachability-metadata` 子模
```bash
git clone [email protected]:apache/shardingsphere.git
cd ./shardingsphere/
./mvnw -PgenerateMetadata -DskipNativeTests -e -T1C clean test native:metadata-copy
./mvnw -PgenerateMetadata -DskipNativeTests -e clean test native:metadata-copy
```

在使用 GraalVM Native Build Tools 的 Maven Plugin 时,
贡献者应避免使用 Maven 的并行构建功能。
GraalVM Native Build Tools 的 Maven Plugin并不是线程安全的,
它与 https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3 不兼容。
Original file line number Diff line number Diff line change
Expand Up @@ -409,7 +409,7 @@ An example of a possible configuration is as follows,
<dependency>
<groupId>io.github.linghengqian</groupId>
<artifactId>hive-server2-jdbc-driver-thin</artifactId>
<version>1.0.0</version>
<version>1.2.0</version>
<exclusions>
<exclusion>
<groupId>com.fasterxml.woodstox</groupId>
Expand Down Expand Up @@ -459,8 +459,8 @@ In order to be able to use DML SQL statements such as `delete`, when connecting
users should consider using only ACID-supported tables in ShardingSphere JDBC. `apache/hive` provides a variety of transaction solutions.

The first option is to use ACID tables, and the possible table creation process is as follows.
If users frequently update and delete data on ACID tables, users may have to wait before and after the execution of DML statements,
to allow HiveServer2 to complete inefficient DML operations, because data in ACID tables is tracked at the folder level.
Due to its outdated catalog-based table format,
users may have to wait before and after DML statement execution to let HiveServer2 complete the inefficient DML operations.

```sql
set metastore.compactor.initiator.on=true;
Expand All @@ -482,8 +482,9 @@ CREATE TABLE IF NOT EXISTS t_order
) CLUSTERED BY (order_id) INTO 2 BUCKETS STORED AS ORC TBLPROPERTIES ('transactional' = 'true');
```

The second option is to use Iceberg table. The possible table creation process is as follows.
The `apache/iceberg` table format is expected to replace the traditional Hive table format in the next few years.
The second option is to use Iceberg table. The possible table creation process is as follows.
Apache Iceberg table format is poised to replace the traditional Hive table format in the coming years,
see https://blog.cloudera.com/from-hive-tables-to-iceberg-tables-hassle-free/ .

```sql
set iceberg.mr.schema.auto.conversion=true;
Expand Down Expand Up @@ -530,7 +531,7 @@ sudo apt-get install build-essential zlib1g-dev -y

git clone [email protected]:apache/shardingsphere.git
cd ./shardingsphere/
./mvnw -PnativeTestInShardingSphere -T1C -e clean test
./mvnw -PnativeTestInShardingSphere -e clean test
```

When contributors discover that GraalVM Reachability Metadata is missing for a third-party library not related to ShardingSphere,
Expand Down Expand Up @@ -560,5 +561,8 @@ contributors should place it on the classpath of the `shardingsphere-test-native
```bash
git clone [email protected]:apache/shardingsphere.git
cd ./shardingsphere/
./mvnw -PgenerateMetadata -DskipNativeTests -e -T1C clean test native:metadata-copy
./mvnw -PgenerateMetadata -DskipNativeTests -e clean test native:metadata-copy
```

Contributors should avoid using Maven's parallel builds feature when using the Maven Plugin for GraalVM Native Build Tools.
The Maven Plugin for GraalVM Native Build Tools is not thread-safe and is incompatible with https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3 .
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,8 @@
import org.apache.shardingsphere.infra.expr.spi.InlineExpressionParser;
import org.apache.shardingsphere.infra.spi.type.typed.TypedSPILoader;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.condition.EnabledForJreRange;
import org.junit.jupiter.api.condition.EnabledIfSystemProperty;
import org.junit.jupiter.api.condition.EnabledOnOs;
import org.junit.jupiter.api.condition.JRE;
import org.junit.jupiter.api.condition.OS;

import java.util.Collections;
Expand All @@ -34,7 +33,7 @@
import static org.hamcrest.MatcherAssert.assertThat;
import static org.junit.jupiter.api.Assertions.assertThrows;

@EnabledForJreRange(min = JRE.JAVA_22)
@EnabledIfSystemProperty(named = "java.vm.vendor", matches = "GraalVM Community", disabledReason = "Github Actions device performance is too low")
@EnabledOnOs(value = OS.LINUX, architectures = "amd64", disabledReason = "See https://www.graalvm.org/jdk21/reference-manual/java-on-truffle/faq/#does-java-running-on-truffle-run-on-hotspot-too")
class EspressoInlineExpressionParserTest {

Expand Down
6 changes: 3 additions & 3 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@
<json-smart.version>2.4.10</json-smart.version>
<accessors-smart.version>2.4.9</accessors-smart.version>
<asm.version>9.3</asm.version>
<groovy.version>4.0.19</groovy.version>
<groovy.version>4.0.22</groovy.version>
<freemarker.version>2.3.31</freemarker.version>
<bytebuddy.version>1.14.13</bytebuddy.version>

Expand Down Expand Up @@ -126,7 +126,7 @@
<mariadb-java-client.version>2.4.2</mariadb-java-client.version>
<clickhouse-jdbc.version>0.6.3</clickhouse-jdbc.version>
<hive.version>4.0.0</hive.version>
<hive-server2-jdbc-driver-thin.version>1.0.0</hive-server2-jdbc-driver-thin.version>
<hive-server2-jdbc-driver-thin.version>1.2.0</hive-server2-jdbc-driver-thin.version>
<presto.version>0.282</presto.version>

<hikari-cp.version>4.0.3</hikari-cp.version>
Expand All @@ -140,7 +140,7 @@
<hamcrest.version>2.2</hamcrest.version>
<mockito.version>4.11.0</mockito.version>
<awaitility.version>4.2.0</awaitility.version>
<testcontainers.version>1.20.0</testcontainers.version>
<testcontainers.version>1.20.1</testcontainers.version>
<commons-csv.version>1.9.0</commons-csv.version>

<graal-sdk.version>24.0.2</graal-sdk.version>
Expand Down