Skip to content

Releases: kubeflow/spark-operator

v2.1.0

11 Dec 06:06
@github-actions github-actions
v2.1.0
664b9d0
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v2.1.0 Latest
Latest

New Features

  • Upgrade to Spark 3.5.3 (#2202 by @jacobsalway)
  • feat: support archives param for spark-submit (#2256 by @kaka-zb)
  • Allow --ingress-class-name to be specified in chart (#2278 by @jacobsalway)
  • Update default container security context (#2265 by @ChenYi015)
  • Support pod template for Spark 3.x applications (#2141 by @ChenYi015)
  • Allow setting automountServiceAccountToken (#2298 by @Aranch)
  • Allow the Controller and Webhook Containers to run with the securityContext: readOnlyRootfilesystem: true (#2282 by @npgretz)
  • Use NSS_WRAPPER_PASSWD instead of /etc/passwd as in spark-operator image entrypoint.sh (#2312 by @Aakcht)

Bug Fixes

Misc

Full Changelog

Assets 7
Loading

v2.1.0-rc.0

24 Oct 09:09
@github-actions github-actions
v2.1.0-rc.0
1ff61a4
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v2.1.0-rc.0 Pre-release
Pre-release

New Features

Bug Fixes

Misc

Full Changelog

Assets 7
Loading

v2.0.2

11 Oct 02:18
@github-actions github-actions
v2.0.2
ef9a2a1
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v2.0.2

Bug Fixes

Misc

What's Changed

Full Changelog

More Details

  • By removing setcap 'cap_net_bind_service=+ep' from the docker build, the container can run with all capabilities dropped on a non-privileged port. If you want to listen on port <1024, you could either run as root or add back the NET_BIND_SERVICE capability or build your own image with this flag on the binary.

Contributors

ChenYi015
Assets 3
Loading

v2.0.1

26 Sep 03:05
@github-actions github-actions
v2.0.1
94cdc52
Compare
Choose a tag to compare
v2.0.1

New Features

  • FEATURE: build operator image as non-root (#2171 by @ImpSy)

Bug Fixes

  • Update controller RBAC for ConfigMap and PersistentVolumeClaim (#2187 by @ChenYi015)

Misc

Full Changelog: v2.0.0...v2.0.1

Assets 3
Loading

v2.0.0

23 Sep 04:05
@github-actions github-actions
v2.0.0
f1b8d46
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v2.0.0

Breaking Changes

  • Use controller-runtime to reconsturct spark operator (#2072 by @ChenYi015)
  • feat: support driver and executor pod use different priority (#2146 by @Kevinz857)

New Features

Bug Fixes

  • fix: Add default values for namespaces to match usage descriptions (#2128 by @snappyyouth)
  • Fix: Spark role binding did not render properly when setting spark service account name (#2135 by @ChenYi015)
  • fix: unable to set controller/webhook replicas to zero (#2147 by @ChenYi015)
  • Adding support for setting spark job namespaces to all namespaces (#2123 by @ChenYi015)
  • Fix: e2e test failes due to webhook not ready (#2149 by @ChenYi015)
  • fix: webhook not working when settings spark job namespaces to empty (#2163 by @ChenYi015)
  • fix: The logger had an odd number of arguments, making it panic (#2166 by @tcassaert)
  • fix the make kind-delete-custer to avoid accidental kubeconfig deletion (#2172 by @ImpSy)
  • Add specific error in log line when failed to create web UI service (#2170 by @tcassaert)
  • Account for spark.executor.pyspark.memory in Yunikorn gang scheduling (#2178 by @jacobsalway)
  • Fix: spark application does not respect time to live seconds (#2165 by @ChenYi015)

Misc

What's Changed

Full Changelog: v1beta2-1.6.2-3.5.0...v2.0.0

More details

This release is a major refactoring of the Spark operator and the Helm chart, includes:

  • Use controller-runtime to reconsturct spark operator (#547). It will significantly improve the maintenance and performance of spark operator.

  • Support multiple namespaces (#507, #2052). For example, if you set spark.jobNamespaces to ["default", "spark-operator"] (please make sure these spark job namespaces already exist before the installation), then the controller and the webhook server will only watch and handle SparkApplications in these spark job namespaces:

    helm install spark-operator spark-operator/spark-operator \
    --version 2.0.0 \
    --create-namespace \
    --namespace spark-operator \
    --set 'spark.jobNamespaces={default,spark-operator}'

  • Support all namespaces. If you want to watch SparkApplications in all namespaces, you can set spark.jobNamespaces to []:

    helm install spark-operator spark-operator/spark-operator \
    --version 2.0.0 \
    --create-namespace \
    --namespace spark-operator \
    --set 'spark.jobNamespaces={}'

    But this will not create RBAC resources for Spark in any namespaces. If we want to watch all namespaces and also create RBAC resources for Spark in namespace default and spark-operator, then we can do as follows:

    helm install spark-operator spark-operator/spark-operator \
    --version 2.0.0 \
    --create-namespace \
    --namespace spark-operator \
    --set 'spark.jobNamespaces={,default,spark-operator}'

  • Support multiple instances. Deploy several spark operator in the same namespace or different namespaces. For example, install two spark operator both in the spark-operator namespace. One with name spark-operator and handles namespace default, another one with name spark-operator-2 and handles namespace spark-operator (please make sure these instances have different release names and handle different spark job namespaces so that they are not conflicting with each other):

    helm install spark-operator spark-operator/spark-operator \
    --version 2.0.0 \
    --create-namespace \
    --namespace spark-operator \
    --set 'spark.jobNamespaces={default}'

helm install spark-operator-2 spark-operator/spark-operator \
    --version 2.0.0 \
    --create-namespace \
    --namespace spark-operator \
    --set 'spark.jobNamespaces={spark-operator}'

  • Support Yunikorn and Kube scheduler as batch schedulers.

  • Leader election is enabled by default and cannot be disabled, it can make sure only one controller instance will be handling SparkApplications during the install/upgrade/rollback process.

  • Webhook server is enabled by default. It will be used to default/validate SparkApplications and mutate Spark pods.

  • Webhook secret will be populated and handled properly during the install/upgrade/rollback process. It will be created and updated by the controller. If the secret is empty, then new certificates will be generated to populate it, otherwise, controller will sync certificates to local disk.

  • Change the default of webhook failurePolicy from Ignore to Fail. Change the default of webhook timeoutSeconds from 30 to 10. There are many issues related to webhook, e.g. environments variables dropped, volumes not mounted. And these issues can be solv...

Read more

Contributors

ChenYi015
Assets 3
Loading

v2.0.0-rc.0

12 Aug 15:17
@vara-bonthu vara-bonthu
v2.0.0-rc.0
74b345a
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v2.0.0-rc.0 Pre-release
Pre-release

This is the Spark Operator v2.0.0-rc.0 pre-release.

Breaking Changes

  • Use controller-runtime to reconsturct spark operator (#2072 by @ChenYi015)

Misc

What's Changed

Full Changelog: spark-operator-chart-1.4.3...v2.0.0-rc.0

More details

This pre-release is a major refactoring of the Spark operator and the Helm chart, includes:

  • Use controller-runtime to reconsturct spark operator (#547). It will significantly improve the maintenance and performance of spark operator.

  • Support multiple namespaces (#507, #2052). For example, if you set spark.jobNamespaces to [default, spark-operator] (please make sure these spark job namespaces already exist before the installation), then the controller and the webhook server will only watch and handle SparkApplications in these spark job namespaces:

helm install spark-operator spark-operator/spark-operator \
    --version 2.0.0-rc.0 \
    --create-namespace \
    --namespace spark-operator \
    --set 'spark.jobNamespaces={default,spark-operator}'
  • Support multiple instances. Deploy several spark operator in the same namespace or different namespaces. For example, install two spark operator both in the spark-operator namespace. One with name spark-operator and handles namespace default, another one with name spark-operator-2 and handles namespace spark-operator (please make sure these instances have different release names and handle different spark job namespaces so that they are not conflicting with each other):
helm install spark-operator spark-operator/spark-operator \
    --version 2.0.0-rc.0 \
    --create-namespace \
    --namespace spark-operator \
    --set 'spark.jobNamespaces={default}'

helm install spark-operator-2 spark-operator/spark-operator \
    --version 2.0.0-rc.0 \
    --create-namespace \
    --namespace spark-operator \
    --set 'spark.jobNamespaces={spark-operator}'

  • Leader election is enabled by default and cannot be disabled, it can make sure only one controller instance will be handling SparkApplications during the install/upgrade/rollback process.

  • Webhook server is enabled by default and cannot be disabled. It will be used to default/validate SparkApplications and mutate Spark pods.

  • Webhook secret will be populated and handled properly during the install/upgrade/rollback process. It will be created and updated by the controller. If the secret is empty, then new certificates will be generated to populate it, otherwise, controller will sync certificates to local disk.

  • Change the default of webhook failurePolicy from Ignore to Fail. Change the default of webhook timeoutSeconds from 30 to 10. There are many issues related to webhook, e.g. environments variables dropped, volumes not mounted. And these issues can be solved by setting webhook.failurePolicy to Failure, webhook server will admit spark pods creation only when there is no error.

  • Controller and webhook server are deployed in different k8s deployments and can be scaled independently. When deploying spark applications at a very large scale, the webhook server can be a performance bottleneck. This can be solved by increasing the replicas of webhook server:

helm install spark-operator spark-operator/spark-operator \
    --version 2.0.0-rc.0 \
    --create-namespace \
    --namespace spark-operator \
    --set webhook.replicas=5

Some Helm values renamings:

  • Change imagePullSecrets to image.pullSecrets.
  • All controller configurations are prefixed with controller e.g. controller.replicas and controller.workers
  • All webhook configurations are prefixed with webhook e.g. webhook.replicas and webhook.failurePolicy.
  • All monitoring configurations are prefixed with promethues e.g. promethues.metrics and promethues.podMonitor.
  • The update strategy of controller/webhook deployment will be the rolling update, not recreate.
  • Change the default spark job namespace from [] to ["default], thus the SparkApplication under examples directory can be running directly without creating rbac resources manually.
  • Service account are configured with controller.serviceAccount, webhook.serviceAccount and spark.serviceAccount respectively.
  • RBAC resources are configured with controller.rbac, webhook.rbac and spark.rbac respectively.
  • logLevel will be one of info, debug and error.

If you want to try this new pre-release with Helm, do as follows:

helm repo add spark-operator https://kubeflow.github.io/spark-operator

helm repo update

helm install spark-operator spark-operator/spark-operator \
    --version 2.0.0-rc.0 \
    --create-namespace \
    --namespace spark-operator

Or upgrade from chart 1.4.6 :

helm upgrade spark-operator spark-operator/spark-operator \
    --version 2.0.0-rc.0 \
    --namespace spark-operator

Contributors

ChenYi015
Assets 3
Loading

spark-operator-chart-1.4.6

26 Jul 08:40
@github-actions github-actions
spark-operator-chart-1.4.6
4108f54
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
spark-operator-chart-1.4.6

A Helm chart for Spark on Kubernetes operator

Assets 3
Loading

spark-operator-chart-1.4.5

22 Jul 05:10
@github-actions github-actions
spark-operator-chart-1.4.5
779ea3d
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
spark-operator-chart-1.4.5

A Helm chart for Spark on Kubernetes operator

Assets 3
Loading

spark-operator-chart-1.4.4

22 Jul 04:03
@github-actions github-actions
spark-operator-chart-1.4.4
eca3fc8
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
spark-operator-chart-1.4.4

A Helm chart for Spark on Kubernetes operator

Assets 3
Loading

spark-operator-chart-1.4.3

03 Jul 07:36
@github-actions github-actions
spark-operator-chart-1.4.3
b8c9013
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
spark-operator-chart-1.4.3

A Helm chart for Spark on Kubernetes operator

Assets 3
Loading