AKS Unable to Access IMDS

Question

AKS Unable to Access IMDS

Ivan Webber 20 Microsoft Employee

I have a simple FastAPI mock service deployed on an old AKS cluster setup a few years ago that uses the Python Azure SDK to access azure storage. It uses MI with Storage Blob Data Contributor with client_id specified via environment variable. It's based on the Azure documentation tutorials.

E.g.:


    try:
        print(os.environ.get("AZURE_CLIENT_ID"))
        client_identity = BlobServiceClient(
            account_url=account_url,
            credential=ManagedIdentityCredential(
                # must be in kwargs passed to IMDS
                client_id=os.environ.get("AZURE_CLIENT_ID")
            )
        )
        containers = client_identity.list_containers()
        print("Connect to Azure Storage succeeded. Find {} containers".format(len(list(containers))), file=sys.stderr)
    except Exception as e:
        print("Connect to Azure Storage failed: {}".format(e), file=sys.stderr)

This was working, but I wanted to improve security and have dedicated resources for this new service, so I made a new resource group and AKS cluster with the following differences:

I chose to use a vnet so that I would be able to create NSG and rules to control ingress and egress. I plan to use an internal load balancer and vnet peering. Unlike the workload of the old cluster, I don't want this one accessible from the internet.
I chose "private cluster" instead of "public cluster".
I decided to use a system-assigned MI because it would be only used for this one thing.
I used separate agentpool and userpool whereas the older cluster had only agentpool.

However, I am now getting the following error:

ManagedIdentityCredential: ManagedIdentityCredential authentication unavailable, no response from the IMDS endpoint. invalid_request

I tried ensuring there was an allow rule to IMDS, but I got the same response. I'm very new to networking, but I'm wondering if one of the other differences is to blame for the error. I appreciate the help.

LISBOA-4826 85 Reputation points

2025-03-19T15:00:04.1566667+00:00

Hello Ivan Webber •

Welcome to the Microsoft Q&A Platform! Thank you for asking your question here.

I understand that you are experiencing issue after change your AKS cluster to private.

Below you can find detailed documentation regarding this topic and how to setup the access, DNS, Endpoints and other resources you may need to configure to reach your goals.

https://learn.microsoft.com/en-us/azure/aks/private-clusters?tabs=default-basic-networking%2Cazure-portal

Also please have a look here: https://learn.microsoft.com/en-us/azure/aks/private-clusters?tabs=default-basic-networking%2Cazure-portal#options-for-connecting-to-the-private-cluster

Additional Links: Azure Instance Metadata Service endpoint - Managed identity

What are managed identities for Azure resources?

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

If you have any other questions or are still running into more issues, let me know in the "comments" and I would be happy to help you.

Thank You.

Lisboa
Ivan Webber 20 Reputation points Microsoft Employee

2025-03-19T20:40:02.1033333+00:00

I tried using "Service Connector" from the Azure portal and switching to user-assigned MI. The validation passed, but I still get the same error.
Ivan Webber 20 Reputation points Microsoft Employee

2025-03-19T21:30:48.5533333+00:00

I am basing off: https://learn.microsoft.com/en-us/azure/service-connector/tutorial-python-aks-storage-workload-identity?tabs=azure-portal
Ivan Webber 20 Reputation points Microsoft Employee

2025-03-19T22:18:14.7033333+00:00
The issue was that the identity was not getting associated correctly so the call to IMDS was failing. It would've been helpful if the error message indicated this.

I made sure my MI had read on the subnet, but I still had the problem.

I looked back at the service connector and decided to alter my deployment YAML (even though it worked as-is on my old cluster) to match the snippet. I only added the following two keys because the other gave me the below error:

#... azure.workload.identity/use: "true" #... serviceAccountName: sc-account-asdf #...

The Deployment "sc-sample-deployment" is invalid: spec.template.spec.containers[0].envFrom[0].secretRef.name: Invalid value: "sc-mystorageblob_asdf-secret": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')

With those lines I was able to list the containers.
Ivan Webber 20 Reputation points Microsoft Employee

2025-03-19T22:27:30.29+00:00

The old cluster didn't require those additional keys (or use Service Connector), but it was able to work with the MI to access blob storage. My best guess is that I setup pod identity or something when I set it up a couple years ago. I would like to understand the differences better, but there is a lot of documentation to filter though.

Accepted answer

0 additional answers

Your answer

LISBOA-4826 85 Reputation points

2025-03-19T15:00:04.1566667+00:00

Hello Ivan Webber •

Welcome to the Microsoft Q&A Platform! Thank you for asking your question here.

I understand that you are experiencing issue after change your AKS cluster to private.

Below you can find detailed documentation regarding this topic and how to setup the access, DNS, Endpoints and other resources you may need to configure to reach your goals.

https://learn.microsoft.com/en-us/azure/aks/private-clusters?tabs=default-basic-networking%2Cazure-portal

Also please have a look here: https://learn.microsoft.com/en-us/azure/aks/private-clusters?tabs=default-basic-networking%2Cazure-portal#options-for-connecting-to-the-private-cluster

Additional Links: Azure Instance Metadata Service endpoint - Managed identity

What are managed identities for Azure resources?

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

If you have any other questions or are still running into more issues, let me know in the "comments" and I would be happy to help you.

Thank You.

Lisboa
Ivan Webber 20 Reputation points Microsoft Employee

2025-03-19T20:40:02.1033333+00:00

I tried using "Service Connector" from the Azure portal and switching to user-assigned MI. The validation passed, but I still get the same error.
Ivan Webber 20 Reputation points Microsoft Employee

2025-03-19T21:30:48.5533333+00:00

I am basing off: https://learn.microsoft.com/en-us/azure/service-connector/tutorial-python-aks-storage-workload-identity?tabs=azure-portal
Ivan Webber 20 Reputation points Microsoft Employee

2025-03-19T22:18:14.7033333+00:00

The issue was that the identity was not getting associated correctly so the call to IMDS was failing. It would've been helpful if the error message indicated this.

I made sure my MI had read on the subnet, but I still had the problem.

I looked back at the service connector and decided to alter my deployment YAML (even though it worked as-is on my old cluster) to match the snippet. I only added the following two keys because the other gave me the below error:

#... azure.workload.identity/use: "true" #... serviceAccountName: sc-account-asdf #...

The Deployment "sc-sample-deployment" is invalid: spec.template.spec.containers[0].envFrom[0].secretRef.name: Invalid value: "sc-mystorageblob_asdf-secret": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')

With those lines I was able to list the containers.
Ivan Webber 20 Reputation points Microsoft Employee

2025-03-19T22:27:30.29+00:00

The old cluster didn't require those additional keys (or use Service Connector), but it was able to work with the MI to access blob storage. My best guess is that I setup pod identity or something when I set it up a couple years ago. I would like to understand the differences better, but there is a lot of documentation to filter though.

Answer 1

Hi Ivan Webber,

We noticed your feedback that the above answer was not helpful. Thank you for taking time to share feedback.

Your understanding is spot on—the old cluster most likely used Pod Identity, which is why authentication worked without additional configuration. The new cluster follows Workload Identity, which requires explicit configuration in the deployment YAML.

Old Cluster (Using Pod Identity)

A few years ago, Azure AD Pod Identity was the recommended way to assign Managed Identities to pods.https://github.com/Azure/aad-pod-identity
It worked by deploying an agent (DaemonSet) in AKS that:
- Intercepted requests to IMDS (Instance Metadata Service).
- Redirected them to use a Kubernetes-assigned identity.
The FastAPI app in the old cluster relied on this Pod Identity mechanism to authenticate without needing Service Connector or extra annotations.

New Cluster (Using Workload Identity) workload-identity

Microsoft deprecated Pod Identity and replaced it with Azure Workload Identity for better performance and security.
IMDS is no longer needed for authentication in Workload Identity.
Instead, authentication now works by:
- Using OIDC (OpenID Connect).
- The Kubernetes Service Account is linked directly to an Azure Managed Identity.
- The pod uses an environment variable (AZURE_CLIENT_ID) and federated credentials to authenticate.
  https://learn.microsoft.com/en-us/azure/aks/workload-identity-migrate-from-pod-identity

Workload Identity requires explicitly linking the Kubernetes Service Account to the Managed Identity.

The additional YAML keys (azure.workload.identity/use: "true" and serviceAccountName) were needed to enable Workload Identity authentication.

As per my understanding, your resolution follows Microsoft's recommended approach for enabling Workload Identity without code changes.

On a cluster that is already running a pod-managed identity, you can configure it to use workload identity one of two ways. The first option allows you to use the same configuration that you've implemented for pod-managed identity. You can annotate the service account within the namespace with the identity to enable Microsoft Entra Workload ID and inject the annotations into the pods.

https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview?tabs=dotnet#how-to-migrate-to-microsoft-entra-workload-id

Hope this helps!

Let me know if you have any further queries!

If you find the answer heplful, please consider Accepting answer and upvote.

Share via

AKS Unable to Access IMDS

0 additional answers

Your answer