Skip to content

Commit

Permalink
GH-44950: [C++] Bump minimum CMake version to 3.25
Browse files Browse the repository at this point in the history
Try installing CMake 3.25 manually on Ubuntu

Install cmake in linux-apt-lint.dockerfile and update install_cmake.sh to not require platform argument

Install CMake 3.25.0 on several Ubuntu jobs and Windows jobs that require the existing version to be bumped

Several fixes to install_cmake

Some more fixes and upgrade minimal examples to use Ubuntu 24.04

Some more fixes for R

Update from CMAKE to CMAKE_VERSION

Ensure that versions returned are strings, bump cmake download

Try forcing MINGW package to pull a newer CMake

Revert trying to pull newer CMake

Try to completely remove mingw CMake from CRAN

/c/cmake/bin/cmake.exe should be present and in path, try with .exe

Try with full path for CMake

Try removing manual installation of CMake and installing CMake via pacman before setting up RTools

Remove manual Installation of CMake from pacman not present

More tries

Temporarily try using msys2 GH action to install cmake

Remove CMake from PKGBUILD

CMake should be in the path

Debug new CMake location

Try with C:\Program Files\CMake\bin\cmake.exe

Just testing at this point

Bundle AWSSDK under RTools (#90)

Check for mingw

Check Windows compiler

Try modifying patch due to redefinition

Try fixing patch

AWS is bundled on Windows R now, do not use MSYS libraries

Fix license header on patch + add CMake condition for AWS patch

Update configure.win to remove specific AWS CPP libs

Update r/configure.win

Co-authored-by: Bryce Mecum <[email protected]>

Update r/configure.win

Test turning ccache back on

Update ci/scripts/r_windows_build.sh

Co-authored-by: Neal Richardson <[email protected]>

Update r/configure.win

Co-authored-by: Neal Richardson <[email protected]>

Review comments

Remove MSYS2_ARG_CONV_EXCL

Testing with manually adding missing rtools40

Add correct slash for path

Add comment on why MSYS2_ARG_CONV_EXCL is required

Use MSYS2_ARG_CONV_EXCL on the same CMake command

Update tutorial examples to use a newer Ubuntu

Add CMAKE_VERSION to cuda jobs

Upgrade minimal Ubuntu to use 24.04 instead of 22.04

Fix minimal build for Ubuntu 24.04

ubuntu-cuda-cpp requires cmake argument

Add cmake to docker-compose.yml missing images

Add missing cmake to some more images

Some more fixing mainly for R binary jobs

Set CMAKE to newly installed path

Revert "Try modifying patch due to redefinition"

This reverts commit b772b23.

Reapply "Try modifying patch due to redefinition"

This reverts commit 1d71375.

Hardcode enum value

I tried to find a proper way to include-guard this but
everything I tried didn't work.

Remove outdated refs to aws libs

Since we are bundling these we don't have these to copy

Move _WIN32_WINNT def into {c,cxx}flags

Ignore warnings-as-errors in cmake
  • Loading branch information
raulcd committed Jan 21, 2025
1 parent f41f590 commit 99d010f
Show file tree
Hide file tree
Showing 36 changed files with 516 additions and 59 deletions.
1 change: 1 addition & 0 deletions .env
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ UBUNTU=22.04

# Default versions for various dependencies
CLANG_TOOLS=14
CMAKE_VERSION=3.25.0
CUDA=11.2.2
DASK=latest
DOTNET=8.0
Expand Down
10 changes: 10 additions & 0 deletions .github/workflows/cpp.yml
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,11 @@ jobs:
with:
fetch-depth: 0
submodules: recursive
- name: Install CMake 3.25.0
shell: bash
run: |
ci/scripts/install_cmake.sh 3.25.0 /tmp/local/
echo "/tmp/local/bin" >> $GITHUB_PATH
- name: Check CMake presets
run: |
cd cpp
Expand Down Expand Up @@ -352,6 +357,11 @@ jobs:
with:
fetch-depth: 0
submodules: recursive
- name: Install CMake 3.25.0
shell: bash
run: |
ci/scripts/install_cmake.sh 3.25.0 /c/cmake
echo "c:\\cmake\\bin" >> $GITHUB_PATH
- name: Download Timezone Database
shell: bash
run: ci/scripts/download_tz_database.sh
Expand Down
5 changes: 5 additions & 0 deletions .github/workflows/dev.yml
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,11 @@ jobs:
uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
with:
python-version: 3.12
- name: Install CMake 3.25.0
shell: bash
run: |
ci/scripts/install_cmake.sh 3.25.0 /tmp/local/
echo "/tmp/local/bin" >> $GITHUB_PATH
- name: Install pre-commit
run: |
python -m pip install pre-commit
Expand Down
6 changes: 6 additions & 0 deletions .github/workflows/r.yml
Original file line number Diff line number Diff line change
Expand Up @@ -282,6 +282,12 @@ jobs:
restore-keys: |
r-${{ matrix.config.rtools }}-ccache-mingw-${{ matrix.config.arch }}-${{ hashFiles('cpp/src/**/*.cc','cpp/src/**/*.h)') }}-
r-${{ matrix.config.rtools }}-ccache-mingw-${{ matrix.config.arch }}-
- name: Install CMake via MSYS2
uses: msys2/setup-msys2@v2
with:
msystem: UCRT64
update: true
install: mingw-w64-ucrt-x86_64-cmake
- uses: r-lib/actions/setup-r@v2
with:
# Note: RTools must be 40 here because RTools40 + ucrt is how we build the Arrow C++
Expand Down
7 changes: 5 additions & 2 deletions ci/docker/centos-7-cpp.dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,6 @@ RUN \
-e 's/mirror\.centos\.org/vault.centos.org/' \
/etc/yum.repos.d/CentOS-SCLo-scl*.repo && \
yum install -y \
cmake3 \
curl \
devtoolset-8 \
diffutils \
Expand All @@ -49,9 +48,13 @@ RUN \
wget \
which

ARG cmake
COPY ci/scripts/install_cmake.sh /arrow/ci/scripts/
RUN /arrow/ci/scripts/install_cmake.sh ${cmake} /usr/local/

COPY ci/scripts/install_sccache.sh /arrow/ci/scripts/
RUN bash /arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin

ENV \
ARROW_R_DEV=TRUE \
CMAKE=/usr/bin/cmake3
CMAKE=/usr/local/bin/cmake
4 changes: 4 additions & 0 deletions ci/docker/linux-r.dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,10 @@ RUN /arrow/ci/scripts/r_docker_configure.sh
COPY ci/scripts/install_sccache.sh /arrow/ci/scripts/
RUN /arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin

ARG cmake
COPY ci/scripts/install_cmake.sh /arrow/ci/scripts/
RUN /arrow/ci/scripts/install_cmake.sh ${cmake} /usr/local/

COPY ci/scripts/r_deps.sh /arrow/ci/scripts/
COPY r/DESCRIPTION /arrow/r/
RUN /arrow/ci/scripts/r_deps.sh /arrow
2 changes: 1 addition & 1 deletion ci/docker/python-wheel-manylinux.dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ ENV PATH=/opt/python/${CPYTHON_VERSION}-${CPYTHON_VERSION}/bin:${PATH}
# Install CMake
ARG cmake=3.29.2
COPY ci/scripts/install_cmake.sh arrow/ci/scripts/
RUN /arrow/ci/scripts/install_cmake.sh ${arch} linux ${cmake} /usr/local
RUN /arrow/ci/scripts/install_cmake.sh ${cmake} /usr/local

# Install Ninja
ARG ninja=1.10.2
Expand Down
5 changes: 4 additions & 1 deletion ci/docker/ubuntu-20.04-cpp-minimal.dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,6 @@ RUN apt-get update -y -q && \
apt-get install -y -q \
build-essential \
ccache \
cmake \
curl \
gdb \
git \
Expand Down Expand Up @@ -68,6 +67,10 @@ RUN latest_system_llvm=10 && \
apt-get clean && \
rm -rf /var/lib/apt/lists*

ARG cmake
COPY ci/scripts/install_cmake.sh /arrow/ci/scripts/
RUN /arrow/ci/scripts/install_cmake.sh ${cmake} /usr/local/

COPY ci/scripts/install_minio.sh /arrow/ci/scripts/
RUN /arrow/ci/scripts/install_minio.sh latest /usr/local

Expand Down
5 changes: 4 additions & 1 deletion ci/docker/ubuntu-20.04-cpp.dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,6 @@ RUN apt-get update -y -q && \
autoconf \
ca-certificates \
ccache \
cmake \
curl \
g++ \
gcc \
Expand Down Expand Up @@ -120,6 +119,10 @@ RUN apt-get update -y -q && \
apt-get clean && \
rm -rf /var/lib/apt/lists*

ARG cmake
COPY ci/scripts/install_cmake.sh /arrow/ci/scripts/
RUN /arrow/ci/scripts/install_cmake.sh ${cmake} /usr/local/

COPY ci/scripts/install_minio.sh /arrow/ci/scripts/
RUN /arrow/ci/scripts/install_minio.sh latest /usr/local

Expand Down
4 changes: 4 additions & 0 deletions ci/docker/ubuntu-22.04-cpp-minimal.dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,10 @@ RUN latest_system_llvm=14 && \
apt-get clean && \
rm -rf /var/lib/apt/lists*

ARG cmake
COPY ci/scripts/install_cmake.sh /arrow/ci/scripts/
RUN /arrow/ci/scripts/install_cmake.sh ${cmake} /usr/local/

COPY ci/scripts/install_minio.sh /arrow/ci/scripts/
RUN /arrow/ci/scripts/install_minio.sh latest /usr/local

Expand Down
5 changes: 4 additions & 1 deletion ci/docker/ubuntu-22.04-cpp.dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,6 @@ RUN apt-get update -y -q && \
ceph \
ceph-fuse \
ceph-mds \
cmake \
curl \
gdb \
git \
Expand Down Expand Up @@ -167,6 +166,10 @@ RUN if [ "${gcc}" = "" ]; then \
# make sure zlib is cached in the EMSDK folder
RUN source ~/emsdk/emsdk_env.sh && embuilder --pic build zlib

ARG cmake
COPY ci/scripts/install_cmake.sh /arrow/ci/scripts/
RUN /arrow/ci/scripts/install_cmake.sh ${cmake} /usr/local/

COPY ci/scripts/install_minio.sh /arrow/ci/scripts/
RUN /arrow/ci/scripts/install_minio.sh latest /usr/local

Expand Down
54 changes: 54 additions & 0 deletions ci/rtools/BuildAwsCCommon.patch
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

diff --git a/cmake/AwsCFlags.cmake b/cmake/AwsCFlags.cmake
index b717bca..5aa8ac9 100644
--- a/cmake/AwsCFlags.cmake
+++ b/cmake/AwsCFlags.cmake
@@ -120,6 +120,10 @@ function(aws_set_common_properties target)
list(APPEND AWS_C_FLAGS -Wno-strict-aliasing)
endif()

+ if(CMAKE_C_IMPLICIT_LINK_LIBRARIES MATCHES "mingw32")
+ list(APPEND AWS_C_FLAGS -D__USE_MINGW_ANSI_STDIO=1 -Wno-unused-local-typedefs)
+ endif()
+
# -moutline-atomics generates code for both older load/store exclusive atomics and also
# Arm's Large System Extensions (LSE) which scale substantially better on large core count systems.
#
diff --git a/include/aws/common/byte_order.inl b/include/aws/common/byte_order.inl
index 1204be0..0abd9cb 100644
--- a/include/aws/common/byte_order.inl
+++ b/include/aws/common/byte_order.inl
@@ -13,7 +13,7 @@
# include <stdlib.h>
#else
# include <netinet/in.h>
-#endif /* _MSC_VER */
+#endif /* _WIN32 */

AWS_EXTERN_C_BEGIN

@@ -39,7 +39,7 @@ AWS_STATIC_IMPL uint64_t aws_hton64(uint64_t x) {
uint64_t v;
__asm__("bswap %q0" : "=r"(v) : "0"(x));
return v;
-#elif defined(_MSC_VER)
+#elif defined(_WIN32)
return _byteswap_uint64(x);
#else
uint32_t low = x & UINT32_MAX;
88 changes: 88 additions & 0 deletions ci/rtools/aws_c_io_ep.patch
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

diff --git a/source/windows/host_resolver.c b/source/windows/host_resolver.c
index 59fbb85..ad4a99e 100644
--- a/source/windows/host_resolver.c
+++ b/source/windows/host_resolver.c
@@ -4,6 +4,7 @@
*/

/* don't move this below the Windows.h include!!!!*/
+#define _WIN32_WINNT 0x0601
#include <winsock2.h>
#include <ws2tcpip.h>

diff --git a/source/windows/iocp/iocp_event_loop.c b/source/windows/iocp/iocp_event_loop.c
index 9ccce30..5cbbef7 100644
--- a/source/windows/iocp/iocp_event_loop.c
+++ b/source/windows/iocp/iocp_event_loop.c
@@ -12,6 +12,7 @@

#include <aws/io/logging.h>

+#define _WIN32_WINNT 0x0601
#include <Windows.h>

/* The next set of struct definitions are taken directly from the
diff --git a/source/windows/secure_channel_tls_handler.c b/source/windows/secure_channel_tls_handler.c
index 50caf02..29fe850 100644
--- a/source/windows/secure_channel_tls_handler.c
+++ b/source/windows/secure_channel_tls_handler.c
@@ -2,6 +2,7 @@
* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
* SPDX-License-Identifier: Apache-2.0.
*/
+#define _WIN32_WINNT 0x0601
#define SECURITY_WIN32

#include <aws/io/tls_channel_handler.h>
@@ -35,6 +36,25 @@
# pragma warning(disable : 4306) /* Identifier is type cast to a larger pointer. */
#endif

+#ifndef SP_PROT_TLS1_0_SERVER
+#define SP_PROT_TLS1_0_SERVER SP_PROT_TLS1_SERVER
+#endif
+#ifndef SP_PROT_TLS1_0_CLIENT
+#define SP_PROT_TLS1_0_CLIENT SP_PROT_TLS1_CLIENT
+#endif
+#ifndef SP_PROT_TLS1_1_SERVER
+#define SP_PROT_TLS1_1_SERVER 0x00000100
+#endif
+#ifndef SP_PROT_TLS1_1_CLIENT
+#define SP_PROT_TLS1_1_CLIENT 0x00000200
+#endif
+#ifndef SCH_USE_STRONG_CRYPTO
+#define SCH_USE_STRONG_CRYPTO 0x00400000
+#endif
+#ifndef SECBUFFER_ALERT
+#define SECBUFFER_ALERT 0x11
+#endif
+
#define KB_1 1024
#define READ_OUT_SIZE (16 * KB_1)
#define READ_IN_SIZE READ_OUT_SIZE
@@ -456,7 +476,7 @@ static int s_fillin_alpn_data(

*extension_length += sizeof(uint32_t) + sizeof(uint16_t);

- *extension_name = SecApplicationProtocolNegotiationExt_ALPN;
+ *extension_name = 2;
/*now add the protocols*/
for (size_t i = 0; i < protocols_count; ++i) {
struct aws_byte_cursor *protocol_ptr = NULL;
Loading

0 comments on commit 99d010f

Please sign in to comment.