[Security] A potential risk of kubecost makes a worker node get the token of any Service Account

[sparkEchooo]

Kubecost Helm Chart Version

1.107

Kubernetes Version

s

Kubernetes Platform

GKE

Description

Summary

  The Kubecost in GKE gave excessive authority when defining Service Account named "kubecost-1-cost-analyzer-serviceaccount-name-dff5" "kubecost-1-cost-analyzer-prometheus-serviceaccounts-server-name-e82d" and "kubecost-1-deployer-kvsqj". Besides, these Service Accounts are mounted into pod, witch makes it possible for attackers to raise rights to administrators.
 

Detailed Analysis

• I deployed Kubecost in the marketplace of Google's GKE cluster by default.

• The clusterrole named "default:kubecost-1:cost-analyzer.serviceAccount.name-r0" defines the "*" verb of "pods, deployments, replicationcontrollers and nodes". And this clusterrole is bound to the Service Account named "kubecost-1-cost-analyzer-serviceaccount-name-dff5". The Service Account is mounted into the pod named "kubecost-1-cost-analyzer-789fc48778-xgpkg".

• The clusterrole named "default:kubecost-1:cost-analyzer.prometheus.serviceAccounts.server.name-r0" defines the "*" verb of "pods, jobs, deployments, statefulsets, replicationcontrollers and nodes". And this clusterrole is bound to the Service Account named "kubecost-1-cost-analyzer-prometheus-serviceaccounts-server-name-e82d". The Service Account is mounted into the pod named "kubecost-1-prometheus-server-6f9d5c9989-l972j".

• The clusterrole named "default:kubecost-1:deployerServiceAccount-r0" defines the "*" verb of "clusterroles and clusterrolebindings". And this clusterrole is bound to the Service Account named "kubecost-1-deployer-sa". The Service Account is mounted into the pod named "kubecost-1-deployer-kvsqj".
   

Attacking Strategy

  If a malicious user controls a specific worker node which has the pod mentioned above, or steals one of the SA token mentioned above.He/She can raise permissions to administrator level and control the whole cluster.
For example,

• With the "*" verb of "clusterroles and clusterrolebindings", attacker can elevate privileges by creating a clusterrolebinding resource and binding cluster-admin to their own Service Account.

• With the "*" verb of "pods, jobs, deployments, statefulsets, replicationcontrollers", attacker can elevate privileges by creating a pod to mount and steal any Service Account he/she want.

• With the "*" verb of nodes, attacker can hijack other components and steal token by adding a "NoExecute" taint to other nodes.

Mitigation Discussion

• Developer could use the rolebinding instead of the clusterrolebinding to restrict permissions to namespace.
• Developers could define precise permissions for workload resources, including pods, deployments, jobs, statefulsets, replicationcontrollers , rather than using wildcard (*).
• The "kubecost-1-deployer" appears to be used for initialization, and developers can delete resources such as the corresponding pod or Service Account after they are no longer needed.

A few questions

• Is it a real issue in Kubecost?
• If it's a real issue, can Kubecost mitigate the risks following my suggestions discussed in the "mitigation discussion"?
• If it's a real issue, does Kubecost plan to fix this issue?

Reporter list

• Xingyu Liu([email protected], me)
• Nanzi Yang([email protected]/[email protected])
• Xunqi Liu([email protected])
• Xin Guo([email protected])
• Wenbo Shen([email protected])
• Jinku Li([email protected])

Looking forward to your reply. Regards Xingyu Liu

Steps to reproduce

1. Deploy the kubecost by default in GKE.
2. Use kubectl get sa to get the list of service accounts.
3. Use kubectl get rolebinding,clusterrolebinding --all-namespaces -o jsonpath='{range .items[?(@.subjects[0].name=="SERVICE_ACCOUNT_NAME")]}[{.roleRef.kind},{.roleRef.name}]{end}' to get the clusterrole related to the service account, and view the permission definition.

Expected behavior

This is a configuration error.

The Service Accounts mentioned above are given excessive authority, witch makes it possible for attackers to raise rights to administrators.

Impact

No response

Screenshots

No response

Logs

No response

Slack discussion

No response

Troubleshooting

• I have read and followed the issue guidelines and this is a bug impacting only the Helm chart.
• I have searched other issues in this repository and mine is not recorded.

[AjayTripathy]

@sparkEchooo could you share the exact command you used to install kubecost? I don't see any "kubecost-1-deployer" service accounts in the base helm chart.

[sparkEchooo]

Hi @AjayTripathy, thanks for your reply!

Install Kubecost

I install kubecost from the marketplace of Google GKE.
Like this,

Pods

And this is the result of command kubectl get pods

ClusterRoleBinding & ClusterRole

This is the ClusterRoleBinding and the ClusterRole related to the service account "kubecost-1-deployer-sa".

Discussion

If Kubecost don't defines the service account named "kubecost-1-deployer-sa", it may be defined by GKE. I will report this to Google.

However, the high authority of "kubecost-1-cost-analyzer-serviceaccount-name-dff5" and "kubecost-1-cost-analyzer-prometheus-serviceaccounts-server-name-e82d" defined by Kubecost is still worth noting.

Thanks again.

[AjayTripathy]

I believe that pod is defined by GKE. You can quickly verify that with a helm install

helm install kubecost cost-analyzer \
--repo https://kubecost.github.io/cost-analyzer/ \
--namespace kubecost --create-namespace \
--set kubecostToken="YWpheUBrdWJlY29zdC5jb20=xm343yadf98"

and diffing the created service accounts.

[sparkEchooo]

You are right.

When I deployed Kubecost locally, I found that the latest version and v1.107.1 (used by GKE) had no problems.

Some questions

The version of kubecost in GKE is v1.107.1. I deployed this version locally, and found that the all permission verbs are "get list watch".
But in GKE, the permissions given to the two Service Accounts, "cost-analyzer" and "cost-analyzer-prometheus", are both "*"

Is v1.107.1 in github patched?

[chipzoller]

This issue is related to the Kubecost listing on GCP Marketplace hence transferred to its parent repo. The configuration is found here which is specific to the Marketplace listing and, as pointed out earlier, not part of the main chart. This may have been a requirement for the deployment to succeed in GCP because it's the only place where the differences appear.

[AjayTripathy]

whoops. sorry for the accidental close. I think this is appropriately filed now but there's not much we can do on our side.

[sparkEchooo]

Dear kubecost maintainers,

I kindly request that you issue a relevant security advisory to alert and assist those who, like us, have installed and are using Kubecost on GKE with the default settings. This is a real issue, and we believe there are many others who may encounter this potential security problem without being aware of it beforehand. If possible, please express gratitude for our report in the advisory.

Thank you for your understanding and cooperation.

Regards,
Xingyu Liu

[AjayTripathy]

Hi @sparkEchooo as noted this is not related to kubecost and is a function of the GCP marketplace.

[dwbrown2]

Is there a place to file this against GCP marketplace or a related repo?

[AjayTripathy]

@alexkubecost is there someone on the GCP Marketplace team we could share this with?

[AjayTripathy]

@chipzoller actually, do you know how to quickly test whether that's actually a requirement on the marketplace?

[sparkEchooo]

Hi there,
If you're looking to update Kubecost on the GCP Marketplace, this might help: https://cloud.google.com/marketplace/docs/partners/kubernetes/maintaining-product

[AjayTripathy]

Thank you @sparkEchooo we will look at updating this in https://github.com/kubecost/gcp-marketplace/blob/main/kubecost/schema.yaml#L84-L158

[AjayTripathy]

cc @cliffcolvin let's file a ticket for internal tracking