how to deploy a dolt cluster in k8s?

[chicco785]

hi!
this project looks interesting, and as far as I understood help to manage ha with active / passive deployment of a dolt cluster. I couldn't find though any information on how to actually deploy the dolt cluster :) any hint will be welcome!

[reltuk]

Hi @chicco785! Thanks for your interest in dolt.

We don't currently have published guidance on running a cluster in kubernetes. I can provide some guidance on how we've typically gone about it recently, but I can't guarantee it will meet all your use cases.

We typically deploy it as a statically sized StatefulSet, with a unique rendered config file for each pod instance. Those config files each configure the that pod to replicate to the other pods in the StatefulSet.

We use three Services to address and route traffic to the pods:

1. Is a headless Service which allows the pods themselves to address each other directly using hostnames of the form POD_NAME.INTERNAL_SERVICE_NAME.NAMESPACE.... It is used in the cluster replication URLs which dolt is configured with.

2. Is a ClusterIP or LoadBalancer Service which targets the pod in the StatefulSet which is labeled as primary.

3. Is a ClusterIP or LoadBalancer Service which targets the pod(s) in the StatefulSet which are labeled 'standby'.

Currently dolt itself does not integrate with the kubernetes control plane, but traffic routing in the above setup relies on unique labels being maintained on the Pods within the cluster. So an external agent needs to run to apply the correct labels, and it needs to run to update the labels anytime there is a failover or a pod is deleted and recreated by the StatefulSet controller, for example.

I've scratched together some example manifests for possible deployment in a gist here: https://gist.github.com/reltuk/8a97771e8b46dd32e47c80ef0c3645f7

You can also see how some kubernetes API objects come together for some test deployments in the e2e tests of this project. For example, here is where we create a typical simple StatefulSet of pods which will replicate amongst each other:

doltclusterctl/e2e/statefulset_test.go

Line 185 in 09d3026

return &appsv1.StatefulSet{

Very happy to answer any further questions you have here.

As cluster replication becomes more mature, we hope to add more guidance to how to use it ergonomically from within a Kubernetes to our official documentation.

[chicco785]

Hi @chicco785! Thanks for your interest in dolt.

We don't currently have published guidance on running a cluster in kubernetes. I can provide some guidance on how we've typically gone about it recently, but I can't guarantee it will meet all your use cases.

We typically deploy it as a statically sized StatefulSet, with a unique rendered config file for each pod instance. Those config files each configure the that pod to replicate to the other pods in the StatefulSet.

We use three Services to address and route traffic to the pods:

1. Is a headless Service which allows the pods themselves to address each other directly using hostnames of the form POD_NAME.INTERNAL_SERVICE_NAME.NAMESPACE.... It is used in the cluster replication URLs which dolt is configured with.
2. Is a ClusterIP or LoadBalancer Service which targets the pod in the StatefulSet which is labeled as primary.
3. Is a ClusterIP or LoadBalancer Service which targets the pod(s) in the StatefulSet which are labeled 'standby'.

Currently dolt itself does not integrate with the kubernetes control plane, but traffic routing in the above setup relies on unique labels being maintained on the Pods within the cluster. So an external agent needs to run to apply the correct labels, and it needs to run to update the labels anytime there is a failover or a pod is deleted and recreated by the StatefulSet controller, for example.

I've scratched together some example manifests for possible deployment in a gist here: https://gist.github.com/reltuk/8a97771e8b46dd32e47c80ef0c3645f7

You can also see how some kubernetes API objects come together for some test deployments in the e2e tests of this project. For example, here is where we create a typical simple StatefulSet of pods which will replicate amongst each other:

doltclusterctl/e2e/statefulset_test.go

Line 185 in 09d3026

return &appsv1.StatefulSet{

Very happy to answer any further questions you have here.

As cluster replication becomes more mature, we hope to add more guidance to how to use it ergonomically from within a Kubernetes to our official documentation.

More than enough to start :) in case we are going to use dolt in our products we may have resources to deliver an helm chart to the community. Would that be of interest for you guys?

thx a lot!

[reltuk]

More than enough to start :) in case we are going to use dolt in our products we may have resources to deliver an helm chart to the community. Would that be of interest for you guys?

Absolutely! We would love to collaborate on some Helm charts which could help people get started with Dolt in Kubernetes faster.