Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Infra: Migrate to gradle #783

Open
wants to merge 18 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -42,4 +42,7 @@ build/
*.tgz

/docker/*.override.yaml
out/
node_modules/
.gradle
/e2e-tests/allure-results/
130 changes: 130 additions & 0 deletions api/build.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@

plugins {
id 'antlr'
id 'checkstyle'
alias(libs.plugins.spring.boot)
alias(libs.plugins.git.properties)
alias(libs.plugins.jib)
}

dependencies {
implementation project(":contract")
if (prod) {
implementation project(":frontend")
}
implementation project(":serde-api")
implementation libs.spring.starter.webflux
implementation libs.spring.starter.security
implementation libs.spring.starter.actuator
implementation libs.spring.starter.logging
implementation libs.spring.starter.oauth2.client
implementation libs.spring.boot.actuator
compileOnly libs.spring.boot.devtools

implementation libs.spring.security.ldap

implementation libs.kafka.clients

implementation libs.apache.avro
implementation libs.apache.commons
implementation libs.apache.commons.pool2
implementation libs.apache.datasketches

implementation libs.confluent.schema.registry.client
implementation libs.confluent.avro.serializer
implementation libs.confluent.protobuf.serializer
implementation libs.confluent.json.schema.serializer

implementation libs.aws.msk.auth
implementation (libs.azure.identity) {
exclude group: 'io.netty', module: 'netty-tcnative-boringssl-static'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's please explain why we are excluding this

}

implementation libs.jackson.databind.nullable
implementation libs.cel
antlr libs.antlr
implementation libs.antlr.runtime

implementation libs.opendatadiscovery.oddrn
implementation (libs.opendatadiscovery.client) {
exclude group: 'org.springframework.boot', module: 'spring-boot-starter-webflux'
exclude group: 'io.projectreactor', module: 'reactor-core'
exclude group: 'io.projectreactor.ipc', module: 'reactor-netty'
}

runtimeOnly libs.micrometer.registry.prometheus

// CVE Fixes
implementation libs.apache.commons.compress
implementation libs.okhttp3.logging.intercepter

// Annotation processors
implementation libs.lombok
implementation libs.mapstruct
annotationProcessor libs.lombok
annotationProcessor libs.mapstruct.processor
annotationProcessor libs.spring.boot.configuration.processor
testAnnotationProcessor libs.lombok
testAnnotationProcessor libs.mapstruct.processor

// Tests
testImplementation libs.spring.starter.test
testImplementation libs.reactor.test
testImplementation libs.testcontainers
testImplementation libs.testcontainers.kafka
testImplementation libs.testcontainers.jupiter
testImplementation libs.junit.jupiter.engine
testImplementation libs.mockito.core
testImplementation libs.mockito.jupiter
testImplementation libs.bytebuddy
testImplementation libs.assertj
testImplementation libs.jsonschemavalidator

testImplementation libs.okhttp3
testImplementation libs.okhttp3.mockwebserver
}

generateGrammarSource {
maxHeapSize = "64m"
arguments += ["-package", "ksql"]
}

sourceSets {
main {
antlr {
srcDirs = ["src/main/antlr4"]
}
java {
srcDirs += generateGrammarSource.outputDirectory
}
}
}

tasks.withType(Checkstyle) {
exclude '**/ksql/**'
}

checkstyle {
toolVersion '10.3.1'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this be added to the catalog?

configFile = rootProject.file('etc/checkstyle/checkstyle.xml')
ignoreFailures = false
maxWarnings = 0
maxErrors = 0
}

test {
useJUnitPlatform()
}

jib {
from {
image = 'azul/zulu-openjdk-alpine:21.0.5-jre-headless'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see a few major issues with using jib:

  1. Currently, in our Dockerfile we have a few additional packages installed:
  • gcompat required for snappy
  • tzdata as requested by someone in the past to be able to configure timezones properly in the container
  1. We're creating a /etc/kafkaui/ dir for a dynamic config. I am unsure what will happen with the current implementation, dynamic config operations might fail.
  2. We're running the app as a non-root user there, not sure what's the default behavior with jib?
  3. Currently dependabot can properly bump the base image version present in dockerfile, however, it won't be able to work with this version located here, in api/build.gradle.

Overall, I prefer dockerfile build with a mvn/gradle plugin rather than jib in favor of more granular customization possibilities.

}
Comment on lines +120 to +122
Copy link
Collaborator

@yeikel yeikel Jan 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this be delegated to a Dockerfile or read from the existing Dockerfile?

The main downside of setting this here is that dependabot won't get any visibility and we will need to update it manually

to {
image = 'ghcr.io/kafbat/kafka-ui'
}
container {
user = "kafkaui"
jvmFlags = ['--add-opens java.rmi/javax.rmi.ssl=ALL-UNNAMED']
}
}
4 changes: 2 additions & 2 deletions api/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -443,10 +443,10 @@
<goal>copy-resources</goal>
</goals>
<configuration>
<outputDirectory>${basedir}/target/classes/static</outputDirectory>
<outputDirectory>${basedir}/target/classes/</outputDirectory>
<resources>
<resource>
<directory>../frontend/build</directory>
<directory>../frontend/build/vite/static</directory>
</resource>
</resources>
</configuration>
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
package io.kafbat.ui.config;

import io.kafbat.ui.exception.ValidationException;
import javax.annotation.PostConstruct;
import jakarta.annotation.PostConstruct;
import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Configuration;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
package io.kafbat.ui.config.auth;

import io.kafbat.ui.model.rbac.Role;
import jakarta.annotation.PostConstruct;
import java.util.ArrayList;
import java.util.List;
import javax.annotation.PostConstruct;
import org.springframework.boot.context.properties.ConfigurationProperties;

@ConfigurationProperties("rbac")
Expand Down
48 changes: 48 additions & 0 deletions build.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
plugins {
alias(libs.plugins.nexus.publish.plugin)
}

subprojects {
apply plugin: "java"

repositories {
mavenCentral()
maven {
name = 'confluent'
url = 'https://packages.confluent.io/maven/'
}
}

group = 'io.kafbat'
version = version

java {
sourceCompatibility = JavaVersion.VERSION_21
targetCompatibility = JavaVersion.VERSION_21
}
}

boolean resolveBooleanProperty(String propertyName, boolean defaultValue = false) {
def propertyValueStr = findProperty(propertyName)
return propertyValueStr == null ? defaultValue : propertyValueStr.toBoolean();
}

ext {
prod = resolveBooleanProperty("prod")
}

if (prod) {
nexusPublishing {
repositories {
sonatype {
nexusUrl = uri("https://s01.oss.sonatype.org/service/local/")
snapshotRepositoryUrl = uri("https://s01.oss.sonatype.org/content/repositories/snapshots/")

username = sonatypeUsername
password = sonatypePassword
}
}
}
}


116 changes: 116 additions & 0 deletions contract/build.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
import org.openapitools.generator.gradle.plugin.tasks.GenerateTask

plugins {
id "java-library"
alias(libs.plugins.openapi.generator)
}


def specDir = project.layout.projectDirectory.dir("src/main/resources/swagger/")
def targetDir = project.layout.buildDirectory.dir("generated").get()

dependencies {
implementation libs.spring.starter.webflux
implementation libs.spring.starter.validation
api libs.swagger.integration.jakarta
api libs.jackson.databind.nullable
api libs.annotation.api
compileOnly libs.lombok
annotationProcessor libs.lombok
}

tasks.register('generateUiClient', GenerateTask) {
generatorName = "java"
inputSpec = specDir.file("kafbat-ui-api.yaml").asFile.absolutePath
outputDir = targetDir.dir("kafbat-ui-client").asFile.absolutePath
apiPackage = "io.kafbat.ui.api.api"
invokerPackage = "io.kafbat.ui.api"
modelPackage = "io.kafbat.ui.api.model"

configOptions = [library : 'webclient',
asyncNative : 'true',
useBeanValidation: 'true',
dateLibrary : 'java8',
useJakartaEe : 'true',]
}

tasks.register('generateBackendApi', GenerateTask) {
generatorName = "spring"
inputSpec = specDir.file("kafbat-ui-api.yaml").asFile.absolutePath
outputDir = targetDir.dir("api").asFile.absolutePath
apiPackage = "io.kafbat.ui.api"
invokerPackage = "io.kafbat.ui.api"
modelPackage = "io.kafbat.ui.model"
modelNameSuffix = "DTO"

additionalProperties = [removeEnumValuePrefix: "false"]

configOptions = [reactive : "true",
interfaceOnly : "true",
skipDefaultInterface : "true",
useTags : "true",
useSpringBoot3 : "true",
dateLibrary : "java8",
generateConstructorWithAllArgs : "false",
generatedConstructorWithRequiredArgs: "false",
additionalModelTypeAnnotations : """
@lombok.experimental.SuperBuilder
@lombok.AllArgsConstructor
@lombok.NoArgsConstructor
"""]
}

tasks.register('generateConnectClient', GenerateTask) {
generatorName = "java"
inputSpec = specDir.file("kafka-connect-api.yaml").asFile.absolutePath
outputDir = targetDir.dir("kafka-connect-client").asFile.absolutePath
generateApiTests = false
generateModelTests = false
apiPackage = "io.kafbat.ui.connect.api"
modelPackage = "io.kafbat.ui.connect.model"
invokerPackage = "io.kafbat.ui.connect"


configOptions = [asyncNative : "true",
library : "webclient",
useJakartaEe : "true",
useBeanValidation: "true",
dateLibrary : "java8",]
}

tasks.register('generateSchemaRegistryClient', GenerateTask) {
generatorName = "java"
inputSpec = specDir.file("kafka-sr-api.yaml").asFile.absolutePath
outputDir = targetDir.dir("kafka-sr-client").asFile.absolutePath
generateApiTests = false
generateModelTests = false
apiPackage = "io.kafbat.ui.sr.api"
invokerPackage = "io.kafbat.ui.sr"
modelPackage = "io.kafbat.ui.sr.model"

configOptions = [asyncNative : "true",
library : "webclient",
useJakartaEe : "true",
useBeanValidation: "true",
dateLibrary : "java8",]
}

sourceSets {
main {
java {
srcDir targetDir.dir("api/src/main/java")
srcDir targetDir.dir("kafka-connect-client/src/main/java")
srcDir targetDir.dir("kafbat-ui-client/src/main/java")
srcDir targetDir.dir("kafka-sr-client/src/main/java")
}

resources {
srcDir specDir
}
}
}

compileJava.dependsOn generateUiClient, generateBackendApi, generateConnectClient, generateSchemaRegistryClient
processResources.dependsOn generateUiClient, generateBackendApi, generateConnectClient, generateSchemaRegistryClient


48 changes: 48 additions & 0 deletions e2e-tests/build.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
plugins {
id 'java'
id 'checkstyle'
alias(libs.plugins.allure)
}

ext {
kafkaVersion = "3.8.0"
aspectjVersion = "1.9.21"
allureVersion = "2.27.0"
Comment on lines +8 to +10
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same for these

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, this version does not seem correct. It should be from the confluent distribution

}

dependencies {
implementation project(":contract")
implementation "org.apache.kafka:kafka_2.13:${kafkaVersion}"
implementation "org.aspectj:aspectjweaver:${aspectjVersion}"
implementation "commons-io:commons-io:2.16.1"
implementation "org.testng:testng:7.10.0"
implementation "com.codeborne:selenide:7.2.3"
Comment on lines +17 to +19
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's please add this to the catalog as well

implementation "io.qameta.allure:allure-testng:${allureVersion}"
implementation "io.qameta.allure:allure-selenide:${allureVersion}"
implementation "io.github.bonigarcia:webdrivermanager:5.8.0"
implementation libs.spring.starter.webflux
implementation "io.netty:netty-resolver-dns-native-macos:4.1.116.Final"

implementation libs.lombok

annotationProcessor libs.lombok
testAnnotationProcessor libs.lombok
}

checkstyle {
toolVersion '10.3.1'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This version should ideally be shared with the other module

configFile = rootProject.file('etc/checkstyle/checkstyle-e2e.xml')
ignoreFailures = false
maxWarnings = 10
maxErrors = 0
}


test {
useTestNG()
}

tasks.named('test') {
enabled = prod
}

Loading
Loading