feat: "spark-submit jar on k8s" related label and resource expansion #4784

lenoxzhao · 2023-07-19T01:58:40Z

What is the purpose of the change

Enable k8s cluster recognition and k8s resource management.

Brief change log

Expand UniversalCmdTemplate for recognizing "-k8sCluster";
Modify ClusterLabel extension;
Add k8s resource related classes.

Checklist

I have read the Contributing Guidelines on pull requests.
I have explained the need for this PR and the problem it solves
I have explained the changes or the new features added to this PR
I have added tests corresponding to this change
I have updated the documentation to reflect this change
I have verified that this change is backward compatible (If not, please discuss on the Linkis mailing list first)
If this is a code change: I have written unit tests to fully verify the new behavior.

aiceflower · 2023-07-19T06:47:14Z

Please handle the third dependency check problem.

lenoxzhao · 2023-07-19T11:14:49Z

Please handle the third dependency check problem.

Thanks. Already fix and please check.

peacewong · 2023-07-20T14:06:57Z

@guoshupei

guoshupei · 2023-07-22T01:30:59Z

.../main/java/org/apache/linkis/manager/rm/external/kubernetes/KubernetesResourceRequester.java

+                .withClientCertData(k8sClientCertData)
+                .withClientKeyData(k8sClientKeyData)
+                .withCaCertData(k8sCaCertData)
+                .build());


I was wondering if DefaultKubernetesClient needs to be initialized every time? Does creating a client for each requestResourceInfo affect the K8s cluster?

guoshupei · 2023-07-22T01:50:22Z

...in/java/org/apache/linkis/manager/rm/service/impl/DriverAndKubernetesReqResourceService.java

+      throws RMWarnException {
+    //        if (!super.canRequest(labelContainer, resource)) {
+    //          return false;
+    //        }


Is the configuration resource not validated?

guoshupei · 2023-07-22T02:02:02Z

...-label-common/src/main/java/org/apache/linkis/manager/label/entity/cluster/ClusterLabel.java

-public class ClusterLabel extends GenericLabel implements EMNodeLabel {
+import static org.apache.linkis.manager.label.errorcode.LabelCommonErrorCodeSummary.LABEL_ERROR_CODE;
+
+public class ClusterLabel extends GenericLabel implements EngineNodeLabel, UserModifiable {

  public ClusterLabel() {
    setLabelKey(LabelKeyConstant.YARN_CLUSTER_KEY);


Whether the LabelKey can be changed

guoshupei · 2023-07-22T02:22:48Z

.../main/java/org/apache/linkis/manager/rm/external/kubernetes/KubernetesResourceRequester.java

+public class KubernetesResourceRequester implements ExternalResourceRequester {
+  private static final Logger logger = LoggerFactory.getLogger(KubernetesResourceRequester.class);
+  private ExternalResourceProvider provider = null;
+  private String k8sMasterUrl;


How to support different namespaces and different queues?

guoshupei · 2023-07-22T02:35:44Z

.../main/java/org/apache/linkis/manager/rm/external/kubernetes/KubernetesResourceRequester.java

+    for (Node item : client.nodes().list().getItems()) {
+      allocatableMemory +=
+          Long.parseLong(item.getStatus().getAllocatable().get("memory").getAmount());
+      allocatableCPU += Long.parseLong(item.getStatus().getAllocatable().get("cpu").getAmount());


Whether the reference usedCPU is correct?

guoshupei · 2023-07-22T02:41:35Z

...-label-common/src/main/java/org/apache/linkis/manager/label/entity/cluster/ClusterLabel.java

-public class ClusterLabel extends GenericLabel implements EMNodeLabel {
+import static org.apache.linkis.manager.label.errorcode.LabelCommonErrorCodeSummary.LABEL_ERROR_CODE;
+
+public class ClusterLabel extends GenericLabel implements EngineNodeLabel, UserModifiable {


The column label_key of linkis_cg_manager_label needs to be extended to 50

guoshupei · 2023-07-26T02:42:31Z

.../main/java/org/apache/linkis/manager/rm/external/kubernetes/KubernetesResourceRequester.java

+        || !StringUtils.equals(
+            k8sMasterUrl, (String) provider.getConfigMap().get("k8sMasterUrl"))) {
+      reloadExternalResourceAddress(provider);
+    }


KubernetesResourceRequester is singleton, there are problems with member variables of client and provider

guoshupei · 2023-07-26T02:56:50Z

...ation-manager/src/main/java/org/apache/linkis/manager/rm/service/RequestResourceService.java

+      KubernetesResource kubernetesResource = (KubernetesResource) requestResource;
+      KubernetesResource kubernetesResourceAvailable = (KubernetesResource) availableResource;
+      KubernetesResource kubernetesResourceMax = (KubernetesResource) maxResource;
+      if (kubernetesResource.getCores() > kubernetesResource.getCores()) {


The right thing to do is kubernetesResource.getCores() > kubernetesResourceAvailable.getCores()

guoshupei · 2023-07-26T03:12:38Z

...ommon/src/main/java/org/apache/linkis/manager/common/entity/resource/KubernetesResource.java

+  }
+
+  public KubernetesResource() {
+    this(Long.MAX_VALUE, Long.MAX_VALUE);


add default value of namespace

guoshupei · 2023-07-26T03:17:58Z

...main/scala/org/apache/linkis/engineplugin/spark/factory/SparkEngineConnResourceFactory.scala

+      engineResourceRequest: EngineResourceRequest
+  ): Resource = {
+    val clusterLabel = LabelUtil.getLabelFromList[ClusterLabel](engineResourceRequest.labels)
+    if (clusterLabel != null && clusterLabel.getClusterType == "K8S") {


change '==' to equals， and use toUpperCase to be compatible with string matching

guoshupei · 2023-07-26T03:18:48Z

...main/scala/org/apache/linkis/engineplugin/spark/factory/SparkEngineConnResourceFactory.scala

+      engineResourceRequest: EngineResourceRequest
+  ): Resource = {
+    val clusterLabel = LabelUtil.getLabelFromList[ClusterLabel](engineResourceRequest.labels)
+    if (clusterLabel != null && clusterLabel.getStringValue.startsWith("K8S")) {


use toUpperCase to be compatible with string matching

guoshupei · 2023-07-26T03:19:58Z

...main/scala/org/apache/linkis/engineplugin/spark/factory/SparkEngineConnResourceFactory.scala

+      executorMemoryWithUnit
+    ) * executorNum + ByteTimeUtils.byteStringAsBytes(driverMemoryWithUnit)
+    val totalExecutorCores = executorCores * executorNum + driverCores
+    // logger.info(s"总需求内存:$totalExecutorMemory, 总需求核数$totalExecutorCores")


Use English

…l to 50

guoshupei · 2023-08-02T08:48:44Z

...ation-manager/src/main/java/org/apache/linkis/manager/rm/service/RequestResourceService.java

+                    kubernetesResource.getCores(),
+                    kubernetesResourceAvailable.getCores(),
+                    kubernetesResourceMax.getCores()));
+      } else if (kubernetesResource.getMemory() > kubernetesResource.getMemory()) {


change to "kubernetesResource.getMemory() > kubernetesResourceAvailable.getMemory()"

guoshupei

LGTM

peacewong · 2023-08-04T07:16:29Z

linkis-computation-governance/linkis-manager/linkis-application-manager/pom.xml

+    <dependency>
+      <groupId>io.fabric8</groupId>
+      <artifactId>kubernetes-model-core</artifactId>
+      <version>5.4.1</version>


can use props value?

peacewong · 2023-08-04T07:51:40Z

linkis-dist/package/db/upgrade/1.4.0_schema/mysql/linkis_ddl.sql

+ALTER TABLE `linkis_cg_ec_resource_info_record` MODIFY COLUMN metrics TEXT DEFAULT NULL COMMENT 'ec metrics';
+ALTER TABLE `linkis_cg_manager_label` MODIFY COLUMN label_key varchar(50);


need to move 1.5.0_schema

peacewong · 2023-08-04T07:52:10Z

linkis-engineconn-plugins/spark/pom.xml

@@ -418,7 +418,6 @@
      <artifactId>kubernetes-model-core</artifactId>
      <version>5.4.1</version>


can use props value?

peacewong · 2023-08-04T07:56:42Z

.../src/main/scala/org/apache/linkis/engineplugin/spark/config/SparkResourceConfiguration.scala

@@ -31,4 +31,22 @@ object SparkResourceConfiguration {
  val LINKIS_SPARK_EXECUTOR_INSTANCES = CommonVars[Int]("spark.executor.instances", 3)
  val LINKIS_QUEUE_NAME = CommonVars[String]("wds.linkis.rm.yarnqueue", "default")

+  val LINKIS_SPARK_KUBERNETES_NAMESPACE =
+    CommonVars[String]("spark.kubernetes.namespace", "default")


need to be consistent

peacewong · 2023-08-04T07:57:37Z

...main/scala/org/apache/linkis/engineplugin/spark/factory/SparkEngineConnResourceFactory.scala

+  ): Resource = {
+    val clusterLabel = LabelUtil.getLabelFromList[ClusterLabel](engineResourceRequest.labels)
+    if (
+        clusterLabel != null && StringUtils.equals(clusterLabel.getClusterType.toUpperCase(), "K8S")


"k8s" should defined constant

peacewong

LGTM.

peacewong

LGTM.

lenoxzhao added 3 commits July 9, 2023 15:09

feat: add k8sCluster label recognition

adf7ae5

feat: add kubernetes resource related basic classes

05f03e7

feat: add kubernetes resource related classes

f445f4a

lenoxzhao added 2 commits July 19, 2023 18:14

fix: supplement third dependency

ad669b1

Merge branch 'master' into master-spark-dev

61c77e6

lenoxzhao closed this Jul 19, 2023

lenoxzhao force-pushed the master-spark-dev branch 2 times, most recently from 3456d77 to d349093 Compare July 19, 2023 10:35

lenoxzhao reopened this Jul 19, 2023

guoshupei reviewed Jul 22, 2023

View reviewed changes

feat: k8s resource support different namespace

53a7dcd

guoshupei reviewed Jul 26, 2023

View reviewed changes

lenoxzhao added 3 commits July 26, 2023 11:57

fix: fix some basic problems

e4d16ed

feat: extend the length of column label_key of linkis_cg_manager_labe…

e1df7e2

…l to 50

fix: format code according to scala style

ad60b9e

guoshupei suggested changes Aug 2, 2023

View reviewed changes

lenoxzhao added 2 commits August 2, 2023 17:01

feat: KubernetesResourceRequester supports multiple providers

f85510e

fix: fix wrong variable use

0dddd69

guoshupei reviewed Aug 2, 2023

View reviewed changes

peacewong reviewed Aug 4, 2023

View reviewed changes

lenoxzhao added 3 commits August 4, 2023 21:17

feat: adjust constants

0ec91aa

feat: adjust constants

4d8ea2f

Merge branch 'master' into master-spark-dev

67f440a

peacewong previously approved these changes Aug 4, 2023

View reviewed changes

fix: format code according to scala style

0c7ff06

lenoxzhao dismissed peacewong’s stale review via 0c7ff06 August 4, 2023 16:09

peacewong approved these changes Aug 5, 2023

View reviewed changes

peacewong merged commit 958c494 into apache:master Aug 5, 2023
12 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: "spark-submit jar on k8s" related label and resource expansion #4784

feat: "spark-submit jar on k8s" related label and resource expansion #4784

lenoxzhao commented Jul 19, 2023

aiceflower commented Jul 19, 2023

lenoxzhao commented Jul 19, 2023

peacewong commented Jul 20, 2023

guoshupei Jul 22, 2023

guoshupei Jul 22, 2023

guoshupei Jul 22, 2023

guoshupei Jul 22, 2023

guoshupei Jul 22, 2023

guoshupei Jul 22, 2023

guoshupei Jul 26, 2023

guoshupei Jul 26, 2023

guoshupei Jul 26, 2023

guoshupei Jul 26, 2023

guoshupei Jul 26, 2023

guoshupei Jul 26, 2023

guoshupei Aug 2, 2023

guoshupei left a comment

peacewong Aug 4, 2023

peacewong Aug 4, 2023

peacewong Aug 4, 2023

peacewong Aug 4, 2023

peacewong Aug 4, 2023

peacewong left a comment

peacewong left a comment

		ALTER TABLE `linkis_cg_ec_resource_info_record` MODIFY COLUMN metrics TEXT DEFAULT NULL COMMENT 'ec metrics';
		ALTER TABLE `linkis_cg_manager_label` MODIFY COLUMN label_key varchar(50);

		@@ -418,7 +418,6 @@
		<artifactId>kubernetes-model-core</artifactId>
		<version>5.4.1</version>

feat: "spark-submit jar on k8s" related label and resource expansion #4784

feat: "spark-submit jar on k8s" related label and resource expansion #4784

Conversation

lenoxzhao commented Jul 19, 2023

What is the purpose of the change

Brief change log

Checklist

aiceflower commented Jul 19, 2023

lenoxzhao commented Jul 19, 2023

peacewong commented Jul 20, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guoshupei left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peacewong left a comment

Choose a reason for hiding this comment

peacewong left a comment

Choose a reason for hiding this comment