however the current tenant '***' is not authorized to access linked subscription '109a5e88-712a-48ae-9078-9ca8b3c81345

[lissff]

What happened:
Starting from 2020-08-22:
AKS uses shared image gallery(https://docs.microsoft.com/en-us/azure/virtual-machines/windows/shared-image-galleries) for some new clusters which means the image reference in the vmss is "/subscriptions/109a5e88-712a-48ae-9078-9ca8b3c81345/resourceGroups/AKS-Ubuntu/providers/Microsoft.Compute/galleries/AKSUbuntu/images/1604/versions/2020.08.13", if user performs any write operation on the vmss, it will be denied since the user/tenant doesn't have access to the aks image sub.
The error message looks like:
"however the current tenant ''
is not authorized to access linked subscription '109a5e88-712a-48ae-9078-9ca8b3c81345'.""*

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):
cluster created after 2020-8-22 that using 3rd party storage, e.g. portworx as the storage option

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
• Size of cluster (how many worker nodes are in the cluster?)
• General description of workloads in the cluster (e.g. HTTP microservices, Java app, Ruby on Rails, machine learning, etc.)
• Others:

[ghost]

Hi lissff, AKS bot here 👋
Thank you for posting on the AKS Repo, I'll do my best to get a kind human from the AKS team to assist you.

I might be just a bot, but I'm told my suggestions are normally quite good, as such:

1. If this case is urgent, please open a Support Request so that our 24/7 support team may help you faster.
2. Please abide by the AKS repo Guidelines and Code of Conduct.
3. If you're having an issue, could it be described on the AKS Troubleshooting guides or AKS Diagnostics?
4. Make sure your subscribed to the AKS Release Notes to keep up to date with all that's new on AKS.
5. Make sure there isn't a duplicate of this issue already reported. If there is, feel free to close this one and '+1' the existing issue.

[yangl900]

hi @lissff , yes we are aware of this change. You can still do updates to the VMSS, as long as the PUT request payload does not contain the SIG image reference (a PATCH instead of full PUT). That is how aad-pod-identity works today. And that's in general a good practice too to avoid overwritting properties from other clients.

[palma21]

To @yangl900's point, which action were you trying to perform?

[amjadaljunaidi]

Azure/aad-pod-identity#753

[whuchenlei]

i don't understand, below cmd also got the same error:

 az vmss extension set  \
--resource-group myResourceGroup\
--vmss-name aks-agentpool-*-vmss \
--name VMAccessForLinux \
--publisher Microsoft.OSTCExtensions \
--version 1.4 \
--protected-settings "{\"username\":\"azureuser\", \"ssh_key\":\"$(cat ~/.ssh/id_rsa.pub)\"}"

The client has permission to perform action 'Microsoft.Compute/galleries/images/versions/read' on scope '/subscriptions/*', however the current tenant '**' is not authorized to access linked subscription '109a5e88-712a-48ae-9078-9ca8b3c81345'.

[nileshpadmagiriwar]

I am getting the same error when I try to add AKS VMSS in the Application Gateway backend pool -
... however the current tenant '*' is not authorized to access linked subscription '109a5e88-712a-48ae-9078-9ca8b3c81345'.

[erplsf]

Have the same issue when I try to enable boot diagnostics on AKS nodes via: az vm boot-diagnostics enable --storage https://${var.storage}.blob.core.windows.net/ --ids "$ids":

The client has permission to perform action 'Microsoft.Compute/galleries/images/versions/read' on scope '/subscriptions/***/resourceGroups/<>/providers/Microsoft.Compute/virtualMachines/<>', however the current tenant '***' is not authorized to access linked subscription '109a5e88-712a-48ae-9078-9ca8b3c81345'.

[r-b-g-b]

I have the same issue when I try to attach a disk to the AKS node VM:

AZURE_CLUSTER_RESOURCE_GROUP=$(az aks show --resource-group $AZURE_RESOURCE_GROUP_NAME --name $AZURE_CLUSTER_NAME --query nodeResourceGroup -o tsv)
AZURE_CLUSTER_VM_NAME=$(az vm list --resource-group $AZURE_CLUSTER_RESOURCE_GROUP -o tsv --query '[0].name')

az vm disk attach \
   -g $AZURE_CLUSTER_RESOURCE_GROUP \
   --vm-name $AZURE_CLUSTER_VM_NAME \
   --name $AZURE_DISK_NAME

The client has permission to perform action 'Microsoft.Compute/galleries/images/versions/read' on scope '/subscriptions/aaa-bbb-ccc/resourceGroups/<AZURE_CLUSTER_RESOURCE_GROUP>/providers/Microsoft.Compute/virtualMachines/AZURE_CLUSTER_VM_NAME', however the current tenant 'xxx-yyy-zzz' is not authorized to access linked subscription '109a5e88-712a-48ae-9078-9ca8b3c81345'.

[amadav]

Ran into this in production. We are accessing the scale-sets created by AKS in the node-resource group to attach disks to each VM instance programmatically. Any workarounds?

[yangl900]

Hi @amadav , I'm looking into this. Could you share how you attach disks? I'm aware Azure CLI will hit this issue because it does a full property PUT on VM. I'm working with the CLI owner to figure out a solution.

if you are calling the API directly, the fix would be drop the "osProfile.storageProfile" property completely from request body.

[palma21]

@whuchenlei that command is deprecated and will be removed from the docs soon, please use something like https://github.com/kvaps/kubectl-node-shell as we work to provide a similarly integrated experience in CLI and portal.
In relation to this though it seems the vmss CLI is not doing the request correctly and we should open this on the azure CLI repo for a fix.

@amadav, @r-b-g-b, @erplsf while we work on that. Just making sure you are aware that those are unsupported operations?
Is there a reason you can't use static disk provisioning via kubernetes for the disk attach cases? Can you talk a bit more about your individual scenarios?
https://docs.microsoft.com/en-us/azure/aks/support-policies#user-customization-of-worker-nodes

[yangl900]

Want to give an update on the disk attach / detach workaround:

For now you can use az resource update command instead to make the update. I have tested following and it works for me to attach a disk to VMSS instance. The key here is to have --remove properties.storageProfile.imageReference so the CLI don't send the image reference as an update, and won't trigger the permission check.

az resource update --ids /subscriptions/xxxx/resourceGroups/MC_xxx/providers/Microsoft.Compute/virtualMachineScaleSets/aks-agentpool-xxxx-vmss/virtualmachines/0 --add properties.storageProfile.dataDisks "{\"lun\": 0, \"createOption\": \"Attach\", \"managedDisk\": { \"id\": \"/subscriptions/xxxx/resourcegroups/MC_xxx/providers/Microsoft.Compute/disks/testdisk\" } }" --remove properties.storageProfile.imageReference

[amadav]

Thanks for the quick comments @yangl900 and @palma21 and the support.

I understand that changing resources in infrastructure resource group is not supported, but it has been a requirement for us from IT and customers alike.

One such scenario is:
Since AKS does not propagate user defined tags to individual resources in the node resource-group, cost based automation may perform remediation steps (kill, make it unusable, alert) which may not be pleasantly tolerated. We therefore list the scale-sets in each of these node resource-groups and update the tags so that costbased reports can account for them.

Regarding disk attachments, we have requirement to attach more disks for HostPath volume based on node metrics of disk usage since we have applications using HostPath for persistent storage and making the move to use PVC may require more cycles. Is it possible to do the same (as mentioned with az-cli) via java sdk? We already have a dependency on azure sdk 1.27.2 and would prefer if we can continue to keep using that for the said purpose.

[yangl900]

hi @amadav , AKS now support propagating tags into VMSS, hope that makes your management tasks easier :) See doc: https://docs.microsoft.com/en-us/azure/aks/use-multiple-node-pools#setting-nodepool-azure-tags

You can achieve the same in java SDK, if you are going to update VMSS, before you make CreateOrUpdate() request, set the properties.virtualMachineProfile.storageProfile to null. And if you are going to update individual VM, set properties.storageProfile.imageReference to null. This is to tell the API that "there is no change in the OS disk images for this request". Hope this helps.

[amadav]

Thanks for the pointers @yangl900 . I tried setting

scaleSet.inner().virtualMachineProfile().withStorageProfile(null);

but sdk version bundled, throws an NPE here. I believe it requires the storage-profile populated and there seems to be no direct way to set it to null just before VirtualMachineScaleSetImpl.updateResourceAsync()

[lissff]

How about Portworx users? any workaround @yangl900

[erplsf]

@palma21 if I can do it through the UI now, why I can't do it via the CLI interface? (setting boot diagnostics for AKS VMs). They are still VMs, and should have the same capabilities as other VMs.
And to that point, It was working a month or two ago via CLI too.
EDIT: Just read the link that you provided. We want to do it, because it's a customer request, but I'll see if I can persuade them as it turns out it's an unsupported scenario.

[yangl900]

@lissff sorry for the inconvenience, we are working with Portworx too. A new version 2.5.7 will mitigate the issue. It should be released this week.

@erplsf yes, CLI should work too, it's an issue in CLI that we are working on a fix. They are still VMs but a little bit special in the OS disk image, since the image is managed by AKS service.

[palma21]

@erplsf yes to the comment I made on the un-supported nature of those operations. It's the same for portal or cli. They should work, as Anders mentions, but any IaaS direct operation is not supported by the managed service policy. You can only interact with AKS and k8s APIs (or AKS Portal/CLI) to interact with the cluster resources.

[TheCloudScout]

I encountered the same error message while trying to update a key for SSH access to the nodesas per Microsoft doc https://docs.microsoft.com/nl-nl/azure/aks/ssh

az vmss extension set
--resource-group
--vmss-name
--name VMAccessForLinux
--publisher Microsoft.OSTCExtensions
--version 1.4
--protected-settings "{"username":"azureuser", "ssh_key":"$(cat ~/.ssh/id_rsa.pub)"}"

Is there any alternative for this as well?

[amjadaljunaidi]

try this
az resource update --ids /subscriptions/xxxx/resourceGroups/MC_xxx/providers/Microsoft.Compute/virtualMachineScaleSets/aks-agentpool-xxxx-vmss
--remove properties.virtualMachineProfile.storageProfile
--add properties.extensionProfile.extensions "{ "name": "VMAccessForLinux", "properties": { "publisher": "Microsoft.OSTCExtensions", "type": "VMAccessForLinux", "typeHandlerVersion": "1.5", "autoUpgradeMinorVersion": true }}"

[TheCloudScout]

Thanks I already found a way in with kubectl node-shell

https://github.com/kvaps/kubectl-node-shell

[leesutcliffe]

I have this issue,.
AAD MIC Pod logs these errors when attempting to apply bindings to azure-ingress controller.

1 mic.go:848] Applying binding ingress-azure node ** for pod ** resulted in error compute.VirtualMachineScaleSetsClient#CreateOrUpdate: Failure sending request: StatusCode=403 -- Original Error: Code="LinkedAuthorizationFailed" Message="The client has permission to perform action 'Microsoft.Compute/galleries/images/versions/read' on scope '/subscriptions/**/resourceGroups/**infra-rg/providers/Microsoft.Compute/virtualMachineScaleSets/aks-**-vmss', however the current tenant 'd481b612-8d2a-409f-9738-9508185d1a50' is not authorized to access linked subscription '109a5e88-712a-48ae-9078-9ca8b3c81345'."

Had the following response from MS

FYI: ISSUE
AKS rolled out the change to use Shared Image Gallery image to provision VMs (previously using PIR a.k.a platform images). This will make the customer VM hold an image reference back to AKS owned subscription. If you do az vmss show you can see following in storageProfile:

"imageReference": {
"exactVersion": "2020.08.13",
"id": "/subscriptions/XXXXXXXXXXXXXXXX/resourceGroups/AKS-Ubuntu/providers/Microsoft.Compute/galleries/AKSUbuntu/images/1604/versions/2020.08.13",
"resourceGroup": "AKS-Ubuntu"
},

When client like CLI making a PUT request to VMSS to perform an update, the request body cannot contain imageReference property, otherwise it will be treat as an update and triggers ARM linked access check to ensure user has permission on the SIG resource. Apparently no user would have that access, thus the failure.

NOTE: Only new node pools / clusters will be using SIG, existing clusters won't be upgraded automatically.

[palma21]

@leesutcliffe Please use Pod Identity version 1.6 and above that has that issue fixed.

[leesutcliffe]

@palma21 Thanks for the advice, worked perfectly

[markpenner34]

Also running into the same issue following the below documentation.

https://docs.microsoft.com/en-us/azure/aks/ssh#:~:text=to%20that%20node.-,Use%20the%20az%20aks%20show%20command%20to%20get%20the%20resource,of%20your%20cluster's%20Linux%20node.&text=To%20add%20your%20SSH%20keys%20to%20the%20node%2C%20use,az%20vm%20user%20update%20command.

az vmss extension set
--resource-group $CLUSTER_RESOURCE_GROUP
--vmss-name $SCALE_SET_NAME
--name VMAccessForLinux
--publisher Microsoft.OSTCExtensions
--version 1.4
--protected-settings "{"username":"azureuser", "ssh_key":"$(cat ~/.ssh/id_rsa.pub)

Any updates on the CLI fix @yangl900

or atleast a work around so i can SSH onto the node?

[lissff]

@yangl900 Can you share the release plan or release note here?

[palma21]

@markpenner34 You can enter into a node using something like: https://github.com/kvaps/kubectl-node-shell

We will be providing a similar experience to avoid interacting with that command.

The fix for that command was merged in the CLI and will be in the next core CLI release train.

@lissff Portworx already shipped a fix for versions above 2.5.7, you should be able to use that.
https://docs.portworx.com/reference/release-notes/portworx/

[lissff]

@palma21 is the change gonna shown in AKS's release note?

[ghost]

Action required from @Azure/aks-pm

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[xuto2]

@lissff If you were having issue when using AZ CLI to update the VMSS, the issue should be resolved now by CLI team. Please update to the latest CLI version.

[ghost]

Thanks for reaching out. I'm closing this issue as it was marked with "Fix released" and it hasn't had activity for 7 days.

[sean-keane25]

I ran into an issue using kubectl node shell, and opened up a bug in that repo.

kvaps/kubectl-node-shell#20