Feature: Porting workloads from legacy clusters

[RainbowMango]

Given a scenario that workloads already running in some clusters, how do we import these workloads to Karmada?

[GitHubxsy]

One possible solution: Maybe, we need a tool to help users convert single-cluster applications to mutil-cluster applications.
Step:

1. Obtaining deployment templates from one Cluster
2. Parsing deployment dependent objects(e.g.: configmap\secret\pvc)
3. Generated propagationpolicy
4. Create deployment 、propagationpolicy、and other dependent objects. karmada should takeover deployment with the same name.
   

[RainbowMango]

I'm thinking to introduce a strategy to control the behavior(adopt or not).

The strategy could be an anootation, or a field in Work.

[lonelyCZ]

I also have the same idea, but there is a question.

If users have migrated a workload from ClusterA, and they want to edit automatically generated PropagationPolicy to take over a workload of the same name in ClusterB. This behavior can't trigger karmadactl migrate, so how to judge whether to take over the workload of ClusterB.

Or we only add an annotation or field to specify this Work can takeover all workloads, if no this annotation, the Work can`t takeover workload,.

[pigletfly]

related issue #593

[pigletfly]

I'm thinking to introduce a strategy to control the behavior(adopt or not).

The strategy could be an anotation, or a field in Work.

I'd prefer a new strategy in Propagation CRD other than a karmadactl plugin or command.Since users may interact with karmada using apis,e.g.,a Golang karmada client.

[pigletfly]

possible strategy:

type AdoptExsitObjectStrategyType string
const (
    // AdoptIfPossibleStrategyType indicates that if a PropagationPolicy matches a object but
    // the object has been created in member cluster,then karmada try to adopt it if possible
    AdoptIfPossibleStrategyType AdoptExsitObjectStrategyType = "AdoptIfPossible"
    // IgnoreStrategyType means if a PropagationPolicy matches a object but
    // the object has been created in member cluster,then karmada will do nothing.
   IgnoreStrategyType AdoptExsitObjectStrategyType = "Ignore"
)

[mrlihanbo]

IMHO, for the first stage, we can achieve the basic features:

• users should specify resource group and version, like: karmadactl migrate deployment.v1.apps
• if --cluster-kubeconfig is provided, use the provided cluster kubeconfig to connect to member cluster, otherwise try to get cluster info from karmada apiserver(cluster should be joined first)
• show raw data of converted resource template and propagation policy if user specify the output format: karmadactl migrate deployment.v1.apps nginx -oyaml
• default to create converted resource in karmada controller plan, but can be controlled by --auto-commit flag.

karmadactl migrate 
flags:
-C, --cluster string              the name of legacy cluster (eg -C=member1)
    --cluster-context string      Context name of legacy cluster in kubeconfig. Only works when there are multiple contexts in the kubeconfig.
    --cluster-kubeconfig string   Path of the legacy cluster's kubeconfig.
-h, --help                        help for migrate
    --karmada-context string      Name of the cluster context in control plane kubeconfig file.
-n, --namespace string            -n=namespace or -n namespace (default "default")    
-o, --output string               Output format. One of:json|yaml
    --auto-commit                 If true, create converted resource in karmada controller plan, eg: resource template, propagation policy...

for the second stage, it should be more automatically and intelligently:

• like kubectl, if users only specify resource, karmadactl needs to automatically detect the gvk.(need to fetch the list of APIs installed in the member cluster)
• karmadactl migrate cmd can automatically use template of previous workload and add some OverridePolicy.
• karmadactl migrate cmd can parse the dependencies of resource, and migrate the dependencies together, such as a deployment refers configmaps and secrets.

[mrlihanbo]

I agree you, the phased development allows the feature to be used earlier by users so that more user feedback can be obtained as quickly as possible.

We can get template and propagationpolicy by --output. If necessary, we can firstly execute karmadactl migrate deployment.v1.apps nginx -oyaml >file.yaml, then edit file.yaml to realize more advanced strategies, which can reduce users' copy-paste work.

@lonelyCZ Hi, are you still working on this issue, expect to receive your feedback

[lonelyCZ]

Yes, We are working on enhancing the functionality of this command. I think that we will be able to demonstrate the functionality of first stage before the next weekly meeting.

[lonelyCZ]

default to create converted resource in karmada controller plan, but can be controlled by --auto-commit flag.

Hi @mrlihanbo, I consider --auto-commit can be replaced by --dry-run, it exists in all commands representing that run the command in dry-run mode, without making any server requests.

[mrlihanbo]

default to create converted resource in karmada controller plan, but can be controlled by --auto-commit flag.

Hi @mrlihanbo, I consider --auto-commit can be replaced by --dry-run, it exists in all commands representing that run the command in dry-run mode, without making any server requests.

Hi @lonelyCZ Would it be weird for users to use although they have the same effect? @RainbowMango @GitHubxsy How do you say?

[GitHubxsy]

I prefer to use --dry-run

[iawia002]

For the following scenario, can we take over the existing workloads directly? eg:

I have three clusters A, B and C, there is a deployment named nginx on both A and C. Now A, B and C are all managed by karmada, I created a deployment named nginx in karmada apiserver, I hope to distribute it to clusters A and B, the existing nginx deployment in cluster A will be taken over by karmada (we will update it and add some labels to it), but nginx on cluster C will not be managed.

[iawia002]

the takeover policy is added.

Thanks for pointing out, somehow I missed your discussion about the takeover policy.

[lonelyCZ]

For this scenario, if we don't set cluster C in PropagationPolicy, the nginx of cluster C will not be managed. Of course, in order to protect workload that users create PropagationPolicy by mistake leading to takeover it incorrectly, so we need to add a takeover policy to caught the attention of users.
@iawia002 @RainbowMango @GitHubxsy

[iawia002]

I'm interested in this feature and I am also doing some takeover tests locally yesterday. I attended the community meeting just now and I know what you want to achieve, can I implement this feature? @lonelyCZ @RainbowMango

[lonelyCZ]

Surely, thanks. We firstly decide what the annotation format is.

[iawia002]

We need to decide whether to introduce this policy first, we can discuss it here #1039 (comment)

[lonelyCZ]

We have realized karmadactl migrate of first stage.

karmadactl migrate

[root@master67 karmada]# karmadactl migrate -h
Migrate resources from legacy clusters to karmada control plane.

Usage:
  karmadactl migrate <RESOURCE_TYPE> <RESOURCE_NAME> -n <NAME_SPACE> -c <CLUSTER_NAME> [flags]

Examples:

# Migrating deployment(default/nginx) from cluster1 to karmada.
karmadactl migrate deployment nginx -n default -c cluster1

# Only printing the converted resource, not to create them in karmada.
karmadactl migrate deployment nginx -n default -c cluster1 -o yaml|json

# Migrating deployment((default/nginx) with gvk from cluster1 to karmada.
karmadactl migrate deployment.v1.apps nginx -n default -c cluster1

# Migrating secret(default/default-token) from cluster1 to karmada.
karmadactl migrate secret default-token -n default -c cluster1

# If it's cluster of pull mode, you can use --cluster-kubeconfig and --cluster-context
karmadactl migrate deployment nginx -n default -c cluster1 --cluster-kubeconfig=<CLUSTER_KUBECONFIG_PATH> --cluster-context=<CLUSTER_CONTEXT>


Flags:
  -c, --cluster string              the name of legacy cluster (eg -c=member1)
      --cluster-context string      Context name of legacy cluster in kubeconfig. Only works when there are multiple contexts in the kubeconfig.
      --cluster-kubeconfig string   Path of the legacy cluster's kubeconfig.
      --cluster-namespace string    Namespace in the control plane where member cluster are stored. (default "karmada-cluster")
      --dry-run                     Run the command in dry-run mode, without making any server requests.
  -h, --help                        help for migrate
      --karmada-context string      Name of the cluster context in control plane kubeconfig file.
  -n, --namespace string            -n=namespace or -n namespace (default "default")
  -o, --output string               Output format. One of: json|yaml

--output

[root@master67 karmada]# karmadactl migrate deployment nginx -c=bj-47  -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"nginx"},"name":"nginx","namespace":"default"},"spec":{"replicas":1,"selector":{"matchLabels":{"app":"nginx"}},"template":{"metadata":{"labels":{"app":"nginx"}},"spec":{"containers":[{"image":"nginx","name":"nginx"}]}}}}
  creationTimestamp: "2021-12-06T16:32:57Z"
  generation: 1
  labels:
    app: nginx
  name: nginx
  namespace: default
  uid: 0bdb00b5-c162-464a-9845-831293aa4164
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: nginx
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        imagePullPolicy: Always
        name: nginx
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30

---
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  creationTimestamp: null
  name: nginx-propagation
  namespace: default
spec:
  placement:
    clusterAffinity:
      clusterNames:
      - bj-47
  resourceSelectors:
  - apiVersion: apps/v1
    kind: Deployment
    name: nginx

@dddddai and I collaborated on this code. https://github.com/lonelyCZ/karmada/blob/karmadactl-migrate/pkg/karmadactl/migrate.go

Next

we need to consider to import adopt strategy that need more detail discuss.

[GitHubxsy]

Good job @lonelyCZ @dddddai

[lonelyCZ]

Thanks @GitHubxsy

The takeover policy has been added, we can discuss the next functions.

And now, the first stage, there are several results when we use karmadactl migrate

1. When there aren't template and Propagation, we can directly create them in karmada control plane.
2. If there have been template, we give a prompt and set the takeover stragy of template to overwrite

[root@master67 karmada]# karmadactl migrate deploy nginx -c bj-47
the resource "apps/v1, Resource=deployments"(default/nginx) already exist in karmada control plane, you can edit PropagationPolicy and OverridePolicy to propagate it.

Next, we if implement OverridePolicy for all field by Plaintext, or only including ImageOverrider, CommandOverrider and ArgsOverrider.

[dddddai]

Ah......IMHO, it's unnecessary to introduce a policy, if users don't wanna adopt the resource, they could update the propagation policy to control this(e.g. ExcludeClusters). Users are supposed to take the responsibility to make the propagation policy meet their requirements, aren't they?
If users declared a propagation policy which propagates the resource to a cluster, it literally means they want karmada to adopt it

I believe the premise of problem is that the user's behavior is correct, otherwise it seems pointless

[iawia002]

I agree with you, so my initial idea was to add a global feature gate to control whether to directly take over the resource with the same name.

[lonelyCZ]

It's right. But it'is necessary to import a switch to control this function for making no changes to the current situation, only if users want to takeover workload, they need to turn this switch on.

[GitHubxsy]

Agree +1 . Karmada takes over applications with the same name by default.

[dddddai]

After discussed with @RainbowMango and @mrlihanbo, we probably need to be cautious(e.g. kubectl create does not create a resource which already exists), I'd let them give the final word

[RainbowMango]

Proposing an annotation:

const (
	// ResourceConflictResolutionAnnotation is added to the resource template to specify how to resolve the conflict
	// in case of resource already existing in member clusters.
	// The valid value is:
	// - overwrite: always overwrite the resource if already exist. The resource will be overwritten with the
	//   configuration from control plane.
	//   Note: Propagation of the resource template without this annotation will fail in case of already exists.
	ResourceConflictResolutionAnnotation = "work.karmada.io/conflict-resolution"
)

As discussed at the last community meeting, we are going to use a light-way annotation instead of extending current APIs to define the adoption behavior.

@pigletfly How do you think?
If no objections, @lonelyCZ can you help file a PR for this?
This annotation makes it possible for people to migrate workloads from member clusters by manual.

After that, we can back to this discussion and work out an auto migrate solution.

[RainbowMango]

Hold on.
Maybe we can introduce a flag to karmada-controller-manager and karmada-agent instead of the annotation.

@iawia002 I don't think we need a feature-gate here, feature-gate controllers on-off, but not a strategy.

[iawia002]

I think this is actually not a strategy, it's just a switch: on or off, will we have other strategy values?

[lonelyCZ]

If there is only a flag of karmada-controller-manager and karmada-agent, we can't control per deployment. In order to use migrate, we only set the flag to true.

Adding a takeover strategy, the main purpose is avoiding to damage existing workload. Sometime the template is different of existing workload, we need to add an OverridePolicy to keep the behavior of existing workload.

If users directly create a template and a PropagationPolicy to deploy application and there have been a same name workload in one of the clusters, we can remind them that there have been a same name workload and whether to takeover it, they can inspect the spec field and add an OverridePolicy, rather than directly overriding the workload.

[RainbowMango]

So, @lonelyCZ what's your preference?

[lonelyCZ]

I think firstly try to import annotation to control takeover behavior that is more light and fine grained for per resource template. It is also easily to be achieved and that don't influence original behaviors of users.

[lonelyCZ]

Ok, @iawia002 is interested in this feature #1039 (reply in thread), would you like to file this pr?

[iawia002]

Yes, I will submit a PR.

[lonelyCZ]

Perhaps, we can define a new command name. I think migrate is more suited for this scenario #1097.

we can use adopt, porting, takeover or others.

Which name do you think is better？

[RainbowMango]

Perhaps, we can define a new command name. I think migrate is more suited for this scenario #1097.

That makes sense.

[RainbowMango]

Propose more alternatives: promote、elevate

[GitHubxsy]

Maybe，we need help from @kevin-wangzefeng.