All posts by Thiyagu

Azure APIM POST Request Caching: Cut Backend Load by 70% With a Custom Cache Key

Most engineers assume POST requests cannot be cached. That assumption is costing their teams unnecessary backend load, slower response times, and wasted infrastructure spend — every single day. When I was working on a high-traffic travel booking API on Azure API Management, our backend was getting hammered by identical POST requests with the same payload, dozens of times per minute. The backend processing was heavy. The response never changed for the same input. We were rebuilding the same result over and over.

The fix was custom POST request caching in Azure APIM — using cache-lookup-value and cache-store-value policies with a dynamically built cache key from the request body. After implementing it, backend calls for repeated payloads dropped by over 70%. Response times for cache hits went from 400ms to under 10ms.

In this guide, you will learn exactly how to build a complete Azure APIM POST request caching solution — handling all three real-world content types, building a reliable cache key, and knowing precisely when to use this pattern and when to avoid it.

Why POST Request Caching in Azure APIM Is Worth Your Attention

Azure APIM’s built-in cache-lookup and cache-response policies work out of the box for GET requests — they use the URL as the cache key. However, POST requests send their data in the request body, not the URL. As a result, APIM cannot cache POST responses automatically.

Furthermore, most tutorials only show caching for application/json content types. In real production environments, APIs receive POST requests in three different formats — and your caching policy needs to handle all of them reliably. Specifically, the three content types you will encounter are:

  • application/json — the most common modern API format. Payload arrives as a JSON body.
  • application/x-www-form-urlencoded — used by legacy clients, HTML forms, and many third-party integrations.
  • multipart/form-data — used when clients send both JSON data and file uploads in the same request.

💡 Key principle: APIM never compares request payloads directly. Instead, you extract the fields that drive your response, build a unique string called a cache key, and APIM uses that string as the lookup identifier. Same input fields = same key = cache hit.

When Should You Cache a POST Request in Azure APIM?

Before writing a single line of policy XML, confirm your POST endpoint actually qualifies for caching. Not every POST call should be cached — and caching the wrong one causes serious problems.

Cache these POST calls:

  • Read-heavy search or filter operations sent as POST because the query is too complex for a URL query string — for example, flight searches, product filters, or report generation.
  • Heavy computation endpoints where identical inputs always produce identical outputs — pricing calculations, eligibility checks, or recommendation engines.
  • Third-party API proxies where the upstream call is slow and expensive but the response is stable for a known time window.

Never cache these POST calls:

  • State-changing operations — anything that creates, updates, or deletes data. Caching these means repeated identical requests silently skip the backend, and your data is never written.
  • Payment or transaction endpoints — retrying a cached payment response causes silent double-charge risks or missed payments.
  • Authentication or token endpoints — one-time tokens, OTPs, and session credentials must never be served from cache.
  • Real-time or per-user personalised responses — if the same payload returns different results for different users, a shared cache key will return the wrong user’s data.

⚠️ Rule of thumb: if you could safely call the same POST endpoint 100 times with the same payload and always expect the same response — it is safe to cache. If the 100th call should behave differently from the first — never cache it.

How Azure APIM POST Request Caching Works

Understanding the flow before writing the policy saves a lot of debugging time. Here is what happens on the first call (cache miss) and the second call (cache hit):

First call — cache miss: The request arrives at APIM. The inbound policy reads the request body, extracts the relevant fields, and builds a unique cacheKey string. Next, cache-lookup-value checks whether that key exists in the cache. Because it is the first call, nothing is found. Therefore, APIM forwards the request to the backend. The backend returns a response. In the outbound policy, cache-store-value saves the response body against the cacheKey for a set duration.

Second call — cache hit: The same payload arrives. APIM builds the identical cacheKey string. This time, cache-lookup-value finds the stored response. As a result, APIM immediately returns the cached response using return-response — without ever touching the backend. The entire round-trip is eliminated.

Understanding the Cache Key Design

Your cache key is the most critical part of the entire solution. Get it wrong and you either serve stale data to the wrong callers or fragment your cache so badly that you never get a hit. Here is the key design used in the complete policy below:

cacheKey = "v2_mycustomcache-" + all JSON fields sorted alphabetically + joined by "-"
// Example payload: { "Channel": "DESKTOP", "Type": "TRAIN", "Unit": "BU-UNIT" }
// Result: v2_mycustomcache-DESKTOP-BU-UNIT-TRAIN

Three important design decisions are built into this key. First, the v2_ prefix acts as a version tag — when you need to invalidate all cached responses after a backend change, simply change this prefix and all existing keys instantly become misses. Second, properties are sorted alphabetically with .OrderBy(p => p.Name) — this ensures the key is identical regardless of the order fields arrive in the payload. Third, all field values are joined into a single flat string — making the key readable in logs and easy to debug.

The Complete Azure APIM POST Request Caching Policy

This policy handles all three content types in a single implementation. Copy it into your APIM policy editor and adjust the cache key prefix and duration to match your use case:

<policies>
    <inbound>
        <base />
        <!-- Step 1: Clear cookies to prevent cache pollution -->
        <set-header name="Cookie" exists-action="override">
            <value />
        </set-header>
        <!-- Step 2: Read and preserve the full request body -->
        <set-variable name="requestBodyRaw"
            value="@(context.Request.Body.As<string>(preserveContent: true))" />
        <set-variable name="isCacheable" value="@(false)" />
        <!-- Step 3: Parse body by Content-Type -->
        <choose>
            <!-- Handler 1: application/x-www-form-urlencoded -->
            <when condition="@(context.Request.Headers
                    .GetValueOrDefault('Content-Type','')
                    .ToLower().Contains('application/x-www-form-urlencoded'))">
                <set-variable name="isCacheable" value="@(true)" />
                <set-variable name="jsonEncoded"
                    value="@(((string)context.Variables['requestBodyRaw'])
                        .Substring(((string)context.Variables['requestBodyRaw'])
                        .IndexOf('=') + 1))" />
                <set-variable name="jsonDecoded"
                    value="@(System.Net.WebUtility.UrlDecode(
                        (string)context.Variables['jsonEncoded']))" />
                <set-variable name="innerRequest"
                    value="@((JObject)Newtonsoft.Json.JsonConvert
                        .DeserializeObject(
                            (string)context.Variables['jsonDecoded']))" />
            </when>
            <!-- Handler 2: multipart/form-data -->
            <when condition="@(context.Request.Headers
                    .GetValueOrDefault('Content-Type','')
                    .ToLower().StartsWith('multipart/form-data'))">
                <set-variable name="isCacheable" value="@(true)" />
                <set-variable name="boundary" value="@{
                    string ct = context.Request.Headers
                        .GetValueOrDefault('Content-Type', '');
                    int idx = ct.IndexOf('boundary=');
                    if (idx >= 0)
                        return ct.Substring(idx + 9).Split(';')[0].Trim();
                    return '';
                }" />
                <set-variable name="jsonPart" value="@{
                    string body = (string)context.Variables['requestBodyRaw'];
                    string boundary = (string)context.Variables['boundary'];
                    if (string.IsNullOrEmpty(boundary)) return '';
                    string delim = '--' + boundary;
                    int start = body.IndexOf('name=\'jsonRequest\'');
                    if (start >= 0) {
                        int cs = body.IndexOf('\r\n\r\n', start) + 4;
                        int ce = body.IndexOf(delim, cs);
                        if (ce == -1) ce = body.IndexOf(delim + '--', cs);
                        if (cs > 3 && ce > cs)
                            return body.Substring(cs, ce - cs).Trim();
                    }
                    return '';
                }" />
                <set-variable name="innerRequest" value="@{
                    string json = (string)context.Variables['jsonPart'];
                    if (!string.IsNullOrEmpty(json)) {
                        try { return (JObject)Newtonsoft.Json.JsonConvert
                                .DeserializeObject(json); }
                        catch { return new JObject(); }
                    }
                    return new JObject();
                }" />
            </when>
            <!-- Handler 3: application/json -->
            <when condition="@(context.Request.Headers
                    .GetValueOrDefault('Content-Type','')
                    .ToLower().Contains('application/json'))">
                <set-variable name="isCacheable" value="@(true)" />
                <set-variable name="requestBody"
                    value="@(context.Request.Body
                        .As<JObject>(preserveContent: true))" />
                <set-variable name="jsonRequestString"
                    value="@((string)((JObject)context.Variables['requestBody'])
                        ['jsonRequest'])" />
                <set-variable name="jsonRequestUnescaped"
                    value="@(System.Text.RegularExpressions.Regex
                        .Unescape((string)context.Variables
                            ['jsonRequestString']))" />
                <set-variable name="innerRequest" value="@{
                    try { return (JObject)Newtonsoft.Json.JsonConvert
                            .DeserializeObject((string)context.Variables
                                ['jsonRequestUnescaped']); }
                    catch { return new JObject(); }
                }" />
            </when>
        </choose>
        <!-- Step 4: Build cache key and look up cache -->
        <choose>
            <when condition="@((bool)context.Variables['isCacheable'])">
                <set-variable name="cacheKey" value="@('v2_mycustomcache-' +
                    string.Join('-',
                        ((JObject)context.Variables['innerRequest'])
                            ?.Properties()
                            .OrderBy(p => p.Name)
                            .Select(p => p.Value?.ToString() ?? '')
                        ?? new string[] { })
                )" />
                <cache-lookup-value
                    key="@((string)context.Variables['cacheKey'])"
                    variable-name="cacheResponse" />
                <choose>
                    <when condition="@(context.Variables
                            .ContainsKey('cacheResponse')
                            && context.Variables['cacheResponse'] != null)">
                        <return-response>
                            <set-header name="Content-Type"
                                exists-action="override">
                                <value>application/json</value>
                            </set-header>
                            <set-body>
                                @((string)context.Variables['cacheResponse'])
                            </set-body>
                        </return-response>
                    </when>
                </choose>
            </when>
        </choose>
    </inbound>
    <backend>
        <base />
    </backend>
    <outbound>
        <base />
        <!-- Step 5: Store response in cache on 200 OK -->
        <choose>
            <when condition="@((bool)context.Variables['isCacheable']
                    && context.Response != null
                    && context.Response.Body != null
                    && context.Response.StatusCode == 200)">
                <cache-store-value
                    key="@((string)context.Variables['cacheKey'])"
                    value="@(context.Response.Body
                        .As<string>(preserveContent: true))"
                    duration="172800" />
            </when>
        </choose>
    </outbound>
    <on-error>
        <base />
    </on-error>
</policies>

Policy Walkthrough: What Each Section Does

Step 1 — Clear the Cookie Header

Cookies are cleared from the inbound request before any caching logic runs. This prevents session cookies from polluting the cache key or causing different users to share cached responses they should not see.

Step 2 — Read and Preserve the Request Body

The request body is read once and stored in requestBodyRaw with preserveContent: true. This is critical — in APIM, reading a request body consumes it by default. Without preserveContent: true, the backend receives an empty body. Furthermore, the isCacheable flag starts as false — it only becomes true if a supported Content-Type is detected.

Step 3 — Content-Type Detection and Azure APIM POST Caching Payload Parsing

The choose block detects which Content-Type was sent and parses the payload accordingly. Each handler extracts the same end result — a JObject named innerRequest containing the relevant fields. As a result, Step 4 (key building) works identically regardless of how the payload arrived.

Step 4 — Build Cache Key and Azure APIM POST Cache Lookup

The cache key is built by sorting all properties of innerRequest alphabetically and joining their values with a hyphen. Next, cache-lookup-value checks whether this exact key string exists in the cache. If a hit is found, return-response immediately returns the cached body — the backend is never called.

Step 5 — Store the Response After a Backend Call

In the outbound policy, if the backend returned a 200 OK and the request was cacheable, cache-store-value saves the response body against the cache key for 172,800 seconds — that is 48 hours. Consequently, any identical request within the next 48 hours gets served from cache instantly.

💡 Adjust the duration value to match your data freshness requirements. For real-time pricing, use 300 seconds (5 minutes). For stable reference data, 172800 seconds (48 hours) or more is appropriate.

Common Mistakes That Break Azure APIM POST Request Caching

  • Not using preserveContent: true when reading the body. This is the most common mistake. Without it, APIM reads and discards the request body — the backend receives nothing and returns a 400 or 500 error. Always set preserveContent: true on every Body.As<>() call.
  • Building a cache key that is too broad. If your key only uses one field from a five-field payload, requests with different values for the other four fields will all hit the same cache entry and get wrong responses. Include every field that meaningfully changes the response.
  • Building a cache key that is too granular. Conversely, including timestamp or session ID fields in the key means you never get a cache hit. Only include stable, response-determining fields.
  • Caching without checking the status code. The outbound cache-store-value must check context.Response.StatusCode == 200 before storing. Otherwise, error responses get cached and served to subsequent callers — causing hard-to-diagnose intermittent failures.
  • Not versioning the cache key prefix. When your backend response schema changes, you need a way to invalidate all cached entries immediately. Without a version prefix like v2_, old cached responses continue being served. Incrementing the prefix instantly invalidates the entire cache without touching any infrastructure.

Internal vs External Redis Cache for Azure APIM POST Caching

The policy above works with both APIM’s built-in internal cache and an external Azure Cache for Redis. Here is when to use each:

  • Internal APIM cache — suitable for development, staging, and single-region deployments. Simple to set up with no extra infrastructure. However, in the classic APIM tiers, internal cache contents do not persist across service updates. The v2 tiers provide persistent built-in cache.
  • External Azure Cache for Redis — recommended for production, multi-region, or high-throughput deployments. Provides persistence, higher capacity, and cache sharing across multiple APIM units. For production workloads, connecting an external Azure Cache for Redis is usually the better option — create a Redis instance in the same region as your APIM instance, then go to APIM → External cache → Add and provide the Redis connection string.

Conclusion: Azure APIM POST Request Caching Done Right

Azure APIM POST request caching is one of the highest-impact performance optimisations available in the APIM policy toolkit — and one of the most underused. By combining careful payload parsing across all three content types with a well-designed cache key, you can transform heavy POST endpoints into near-instant responses for repeated inputs. Furthermore, the versioned key prefix gives you a clean invalidation mechanism whenever your backend schema changes.

In summary, the five things that make this work reliably are: always preserve the request body, handle all three content types, sort your key fields alphabetically, only store on 200 OK responses, and version your cache key prefix from day one. Apply this pattern only to idempotent read-heavy POST operations — and never to state-changing or security-sensitive endpoints.

Quick Reference

  • Read body: always use Body.As<string>(preserveContent: true)
  • Build key: extract fields → sort alphabetically → join with hyphen → prefix with version tag
  • Cache lookup: <cache-lookup-value key="..." variable-name="cacheResponse" />
  • Cache store: <cache-store-value key="..." value="..." duration="172800" />
  • Invalidate all: increment the version prefix in the cache key

Frequently Asked Questions

Q: Does APIM automatically compare the full POST payload to decide on a cache hit?
No. APIM never compares payload content directly. It only checks whether the exact cache key string you generate already exists in the cache store. Your key design entirely determines what counts as a cache hit or miss.

Q: What happens if a field in my payload changes by even one character?
A new cache key is generated, resulting in a cache miss. APIM forwards the request to the backend, gets a fresh response, and stores it under the new key. The old cached entry remains until it expires.

Q: How do I invalidate the cache immediately without waiting for the duration to expire?
The fastest way is to increment the version prefix in your cache key — for example, change v2_mycustomcache- to v3_mycustomcache-. All existing v2 keys instantly become orphaned and new requests build fresh v3 entries. Alternatively, use the APIM Management REST API to delete specific cache entries by key.

Q: Can I use this pattern with Azure Cache for Redis instead of the internal cache?
Yes. The cache-lookup-value and cache-store-value policies work with both APIM’s internal cache and an external Redis cache. No policy changes are needed — APIM automatically uses the external cache when one is configured, falling back to the internal cache if the external cache is unavailable.

Q: The cached response is being returned but the Content-Type header is wrong. How do I fix it?
In the return-response block, always explicitly set the Content-Type header using set-header. When APIM returns a response directly from cache without hitting the backend, it does not automatically carry forward the original response headers — you must set them manually in the policy.

Related Articles

Stop Rebuilding the Same App Twice — Download Azure DevOps Pipeline Artifacts Across Projects in 5 Minutes

Downloading Azure DevOps pipeline artifacts across projects sounds straightforward — until you actually try it. The first time our team needed to share build outputs between two separate Azure DevOps projects, we spent nearly half a day hitting permission errors, wrong project GUIDs, and silent failures that gave us zero useful output in the pipeline logs. The Microsoft docs show you the YAML. They do not show you the five things that go wrong when you actually run it.

This guide walks through exactly how the DownloadPipelineArtifact@2 task works for downloading Azure DevOps pipeline artifacts from another project — including the real-world configuration, how to find the project GUID and pipeline definition ID, the permissions you must set up first, and the mistakes that cause silent failures that the official docs never mention.

Why Download Pipeline Artifacts Across Projects?

Before diving into configuration, it helps to understand the real scenarios where downloading Azure DevOps pipeline artifacts across projects saves significant time. In our experience working on large enterprise Azure DevOps setups, three situations come up constantly:

  • Multi-team deployments. A backend team builds and publishes API binaries in their project. The frontend team’s deployment pipeline needs those binaries to run integration tests — without triggering a full rebuild. Consequently, the frontend pipeline downloads the artifact directly from the backend project and proceeds immediately.
  • Shared infrastructure components. A platform team maintains a base Docker image or shared configuration artifact. Multiple product teams across different projects consume it in their deployment pipelines. Furthermore, when the platform team publishes a new version, all downstream pipelines pull it automatically.
  • Avoiding redundant builds. Rebuilding the same application three times — once for dev, once for staging, once for prod — wastes agent minutes and introduces inconsistency risk. Instead, build once, publish the artifact, and download it in each environment-specific pipeline. As a result, you guarantee that exactly the same binary reaches every environment.

💡 Key insight: The DownloadPipelineArtifact@2 task downloads artifacts that were published with the PublishPipelineArtifact@1 task. If the source pipeline used the older PublishBuildArtifacts@1 task, you need DownloadBuildArtifacts@0 instead — not DownloadPipelineArtifact@2. This mismatch is the most common source of silent failures.

Prerequisites: Permissions First

This is the step the official docs bury — and it is the one that blocks most teams. Before your pipeline can download Azure DevOps pipeline artifacts from another project, you must configure cross-project permissions in both directions. Without this, the task fails silently or throws a vague 403 error.

In the source project (the one that published the artifact), go to Project Settings → Pipelines → Settings and enable “Allow pipelines from other projects to access this project’s resources”. In addition, grant the build service account of the consuming project the Reader role on the source pipeline. Specifically, go to Pipelines → [Source Pipeline] → Edit → Security and add the consuming project’s build service as a Reader.

⚠️ If you skip the permissions step, the DownloadPipelineArtifact@2 task will either fail with “Pipeline not found” or silently download nothing — with no useful error message in the logs.

How to Find Your Project GUID and Pipeline Definition ID

The two values most engineers struggle to find are the project GUID and the definition ID. Here is exactly where to get both.

Finding the Project GUID

Call the Azure DevOps Projects List API — replace [YOUR_ORGANIZATION_NAME] with your actual org name:

https://dev.azure.com/[YOUR_ORGANIZATION_NAME]/_apis/projects?api-version=5.0
# Example
https://dev.azure.com/demojev/_apis/projects?api-version=5.0

Open this URL in your browser while logged into Azure DevOps. The response is a JSON array of all projects in your organisation. Each project has an id field — that is your project GUID. Copy the one that matches the source project name.

Finding the Pipeline Definition ID

Navigate to the source pipeline in Azure DevOps. Look at the URL in your browser — the definitionId value in the URL is your pipeline definition ID. For example:

https://dev.azure.com/demojev/MyProject/_build?definitionId=96
# In this example, the definition ID is 96

YAML Configuration: DownloadPipelineArtifact@2

Here is the complete YAML configuration for downloading Azure DevOps pipeline artifacts from another project. This is the configuration that works in production — not just in a demo environment:

- task: DownloadPipelineArtifact@2
  displayName: 'Download Pipeline Artifacts from Source Project'
  inputs:
    buildType: specific
    project: 'e7586412-0126-45b1-92e4-465fb1710baf'   # source project GUID
    definition: 96                                      # source pipeline definition ID
    buildVersionToDownload: latest                      # always pull the latest successful build
    allowFailedBuilds: true                             # also allow artifacts from failed builds
    artifactName: 'dist'                                # name of the artifact to download
    targetPath: '$(Build.Repository.LocalPath)/dist'   # local path on the build agent

Every Parameter Explained

  • buildType: specific — tells the task to pull from a specific pipeline rather than the current one. Use current when downloading artifacts from an earlier stage in the same pipeline.
  • project — the GUID of the source project (not the project name). Always use the GUID — project names can change, GUIDs never do.
  • definition — the pipeline definition ID from the source project’s pipeline URL.
  • buildVersionToDownload: latest — downloads the most recent successful build. Alternatively, use specific and set buildId to pin to an exact build run.
  • allowFailedBuilds: true — allows the task to download artifacts even from failed pipeline runs. Useful when you need to debug a deployment using a partially-built artifact.
  • artifactName — the exact name of the artifact as it was published by PublishPipelineArtifact@1 in the source pipeline. This is case-sensitive.
  • targetPath — the directory on the build agent where the artifact files land. Use $(Pipeline.Workspace) as the base path rather than $(Build.Repository.LocalPath) when the agent workspace is outside the repo folder.

Real-World Example: Frontend Pipeline Consuming a Backend Build Artifact

Here is a complete real-world pipeline example. The backend team publishes API binaries as a pipeline artifact. The frontend team’s deployment pipeline downloads those binaries and uses them for integration testing — all without triggering a backend rebuild:

trigger:
  - main
pool:
  vmImage: 'ubuntu-latest'
stages:
  - stage: IntegrationTest
    displayName: 'Integration Test Stage'
    jobs:
      - job: RunTests
        displayName: 'Download Backend Artifact and Run Tests'
        steps:
          # Step 1: Download the backend API artifact from the backend project
          - task: DownloadPipelineArtifact@2
            displayName: 'Download Backend API Binaries'
            inputs:
              buildType: specific
              project: 'e7586412-0126-45b1-92e4-465fb1710baf'
              definition: 96
              buildVersionToDownload: latest
              artifactName: 'api-binaries'
              targetPath: '$(Pipeline.Workspace)/api'
          # Step 2: Verify the artifact downloaded correctly
          - script: |
              echo "Downloaded artifact contents:"
              ls -la $(Pipeline.Workspace)/api
            displayName: 'Verify Artifact Download'
          # Step 3: Run integration tests against the downloaded binaries
          - script: |
              cd $(Pipeline.Workspace)/api
              dotnet test ./IntegrationTests.dll
            displayName: 'Run Integration Tests'

Common Mistakes That Cause Silent Failures

Based on real production experience, here are the mistakes that waste the most time when working with DownloadPipelineArtifact@2:

  • Using project name instead of project GUID. The project field requires the GUID — not the display name. Using the display name causes a silent failure with no clear error. Always use the API call above to get the exact GUID.
  • Mismatched artifact task versions. If the source pipeline used PublishBuildArtifacts@1 (the old task), you must use DownloadBuildArtifacts@0 to consume it — not DownloadPipelineArtifact@2. Mixing these two pairs is the single most common source of “artifact not found” errors.
  • Case-sensitive artifact names. The artifactName field is case-sensitive. If the source pipeline published an artifact named Dist but you specify dist in your download task, it will fail. Always copy the artifact name exactly as it appears in the source pipeline’s publish step.
  • Skipping cross-project permissions. As mentioned above, without explicitly granting the consuming project’s build service account access to the source pipeline, the task either silently downloads nothing or throws a vague permissions error. Set this up before you write a single line of YAML.
  • Wrong targetPath base variable. Using $(Build.ArtifactStagingDirectory) instead of $(Pipeline.Workspace) as the base path causes confusion when subsequent steps look for the files. Use $(Pipeline.Workspace) consistently for downloaded artifacts.

💡 Pro tip: Always add a verification script step immediately after DownloadPipelineArtifact@2 that lists the target directory contents. This one-liner — echo “$(Pipeline.Workspace)/dist” && ls -la — saves enormous debugging time when something goes wrong silently.

Downloading the Latest vs a Specific Build Run

One decision every team faces is whether to always download the latest artifact or pin to a specific build run. Each approach has clear trade-offs:

  • buildVersionToDownload: latest — always pulls the most recent successful build. Good for continuous deployment pipelines where you always want the freshest artifact. However, this means your pipeline can break if the source team publishes a breaking change.
  • buildVersionToDownload: specific + buildId — pins to an exact build run by its numeric ID. Good for release pipelines where you need to guarantee a specific tested version reaches production. Furthermore, this approach makes your deployments fully reproducible — the same build ID always produces the same result.
# Pin to a specific build run by ID
- task: DownloadPipelineArtifact@2
  inputs:
    buildType: specific
    project: 'e7586412-0126-45b1-92e4-465fb1710baf'
    definition: 96
    buildVersionToDownload: specific
    buildId: 1088  # exact build run ID from the pipeline URL
    artifactName: 'dist'
    targetPath: '$(Pipeline.Workspace)/dist'

Conclusion

Downloading Azure DevOps pipeline artifacts across projects with DownloadPipelineArtifact@2 is one of the most powerful ways to eliminate redundant builds and share outputs cleanly between teams. The configuration itself is straightforward — but only once you have the right project GUID, the correct pipeline definition ID, and the cross-project permissions set up correctly beforehand.

In summary, get the project GUID from the Projects List API, get the definition ID from the pipeline URL, set cross-project permissions before writing any YAML, match your publish and download task versions carefully, and always add a verification step after the download. Follow these five steps and the task works reliably every time.

Quick Reference

  • Get project GUID: https://dev.azure.com/[ORG]/_apis/projects?api-version=5.0
  • Get definition ID: from pipeline URL — definitionId=XX
  • Set permissions: Source project → Project Settings → Pipelines → Settings → allow cross-project access
  • Match task versions: PublishPipelineArtifact@1DownloadPipelineArtifact@2
  • Verify download: add ls -la $(Pipeline.Workspace)/[artifactName] after the task

Frequently Asked Questions

Q: Can I download artifacts from a pipeline in a completely different Azure DevOps organisation?
No. The DownloadPipelineArtifact@2 task only works within the same Azure DevOps organisation. To share artifacts across organisations, you need to use Azure Artifacts feeds or an external storage solution like Azure Blob Storage.

Q: What is the difference between DownloadPipelineArtifact@2 and DownloadBuildArtifacts@0?
They consume different artifact types. DownloadPipelineArtifact@2 downloads artifacts published with PublishPipelineArtifact@1. DownloadBuildArtifacts@0 downloads artifacts published with the older PublishBuildArtifacts@1 task. Using the wrong download task for a given artifact type results in a silent failure or “artifact not found” error.

Q: Can I download multiple artifacts from the same source pipeline in one step?
Yes. Leave the artifactName field empty — the task will download all artifacts published by that pipeline run into separate subdirectories under your targetPath.

Q: The task runs but the target directory is empty. What is wrong?
The most common causes are a case-sensitive artifact name mismatch, using a project display name instead of the GUID, or missing cross-project permissions. Add a ls -la script step after the task and check that the targetPath directory exists and contains files.

Q: Does DownloadPipelineArtifact@2 work on self-hosted agents?
Yes — but the self-hosted agent must have outbound network access to dev.azure.com and *.vsblob.vsassets.io (the Azure Artifacts CDN). Firewall restrictions on self-hosted agents are a common cause of artifact download failures in corporate environments.

Related Articles

Your AKS Pod Says “Running” — But Your App Is Dying. Here’s the PVC Disk Full – The Fix Takes 15 Minutes

An AKS PVC disk full condition is one of the most deceptive problems in Kubernetes operations. When a PersistentVolumeClaim (PVC) fills to 100% on Azure Kubernetes Service (AKS), the pod using it does not crash immediately — and that is what makes it so dangerous. The pod keeps showing Running in kubectl. No alert fires. Everything looks fine. Then suddenly your application starts throwing cryptic errors that seem completely unrelated to disk space.

This guide covers the fix for any workload running on AKS — whether you are running Solr, PostgreSQL, MongoDB, Elasticsearch, Redis, or any other stateful application that writes data to a PVC. The kubectl commands and the recovery steps are identical regardless of what is running inside the pod.

In our case, the workload was Sitecore with Solr on AKS. The disk-full condition showed up as this IndexWriter is closed — a misleading Lucene error that buried the real cause: java.io.IOException: No space left on device. But whether you see that error or a Postgres could not write to file, a MongoDB No space left on device, or an Elasticsearch flood stage disk watermark exceeded — the fix is exactly the same.

In this guide, you will learn how to confirm the root cause, safely expand your Solr PVC on AKS, recover the application, and rebuild the Sitecore index — all without losing any data. In my environment, Sitecore and Solr are deployed as custom Docker images running on AKS.

Prerequisites

  • kubectl configured and connected to your AKS cluster.
  • Permissions to manage PVCs, StatefulSets, Deployments, and pods in your Kubernetes namespace.
  • Solr deployed on AKS using the Azure managed-premium storage class.
  • Access to the Sitecore Control Panel for index management (for the Sitecore-specific recovery step).

Which Workloads Does This Affect?

Any stateful pod that writes data to a PVC can hit this problem. The AKS PVC disk full fix is identical for all of them. What changes is only the application-level error message and the final recovery step. Here are the most common workloads and the errors each one throws when disk space runs out:

  • Solr / Elasticsearch: java.io.IOException: No space left on devicethis IndexWriter is closed
  • PostgreSQL: could not write to file base/pgsql_tmp: No space left on device
  • MongoDB: No space left on device: couldn't open file for writing
  • MySQL / MariaDB: ERROR 3 (HY000): Error writing file '/tmp/...' (Errcode: 28 - No space left on device)
  • Redis: MISCONF Redis is configured to save RDB snapshots, but it's currently unable to persist on disk
  • Any custom application pod: write failures, silent data loss, or application-level errors that bury the real No space left on device cause deep in the logs.

💡 The kubectl fix is identical for all workloads — expand the PVC, restart the pod, verify recovery. Only the final application-level recovery step differs per workload.

Why the Error Message Is Always Misleading

The most important thing to understand before you touch anything is this: the top-level error your application throws is almost never the real cause. It is a downstream symptom. Always scroll to the very bottom of the stack trace to find the true root cause.

For Sitecore with Solr, the index rebuild job logs show something like this:

Job started: Index_Update_IndexName=sitecore_jss_web_index
#Exception: System.Reflection.TargetInvocationException
---> SolrNet.Exceptions.SolrConnectionException:
  this IndexWriter is closed
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
Caused by: java.io.IOException: No space left on device

Here is the exact chain of events that causes this error. First, the Solr PVC fills to 100% capacity. Next, Lucene tries to merge index segments during the rebuild — a process that requires significant temporary extra disk space. Because no space is available, the write fails. As a result, the IndexWriter closes itself as a safety measure to protect data integrity. After that, Sitecore detects the closed IndexWriter and throws the AlreadyClosedException. Finally, the rebuild job fails — and keeps failing on every retry until the underlying disk problem is resolved.

⚠️ Until you fix the AKS PVC disk full condition, every single index rebuild will fail with the same error — no matter how many times you retry it from Sitecore.

Step-by-Step: AKS PVC Disk Full Fix

Step 1: Confirm the AKS PVC Disk Is Full

Before making any changes, confirm that disk exhaustion is actually the root cause. First, check your PersistentVolumeClaims in the relevant namespace:

kubectl get pvc -n solr

The output will show all PVCs in a Bound status — which looks completely healthy. However, Bound only means the PVC is attached to the pod. It tells you nothing about how much space is actually used inside it:

NAME              NAMESPACE  STATUS  CAPACITY  STORAGECLASS
solr-leader-disk  solr       Bound   10Gi      managed-premium

Next, exec directly into the pod to check actual disk usage:

kubectl exec -it <solr-pod-name> -n solr -- df -h

Look for the mount point where your application stores data — typically /var/solr for Solr, /var/lib/postgresql for Postgres, or /data/db for MongoDB. If you see Use% at 100%, you have confirmed the root cause. Furthermore, grep the pod logs directly for the IOException to be certain:

kubectl logs <solr-pod-name> -n solr | grep -i "no space"

Step 2: AKS PVC Disk Full Fix — Expand the PVC Online

Azure’s managed-premium storage class supports online volume expansion by default. Consequently, you can grow the PVC without stopping the pod, without losing data, and without any application downtime. Note that you can only ever expand a PVC — Kubernetes does not support shrinking.

You have two methods to expand. Choose whichever fits your workflow:

Method 1: kubectl patch (faster — one command)

kubectl patch pvc solr-leader-disk-2023081617 -n solr \
  -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'

Method 2: kubectl edit (visual — easier to verify)

kubectl edit pvc solr-leader-disk-2023081617 -n solr

In the editor, find the spec.resources.requests.storage field and change it from 10Gi to 20Gi:

spec:
  resources:
    requests:
      storage: 20Gi   # changed from 10Gi

After saving, watch the PVC status in real time. You will briefly see FileSystemResizePending — that is completely normal. Wait for it to return to Bound:

kubectl get pvc -n solr -w

The resize typically takes 5–10 minutes on Azure. The pod keeps running and all data remains accessible throughout the entire process.

💡 Pro tip: During a Solr index rebuild, Lucene needs up to 3× the current index size as temporary working space. If your current index is 8Gi, plan for at least 30Gi total — not just 20Gi. The same rule applies to Elasticsearch index merges and PostgreSQL VACUUM operations.

Step 3: Restart the Pod to Complete Recovery

This is the step most guides skip — and it is the one that trips people up most often. Even after the AKS PVC disk is expanded, the application process inside the pod is still in a broken state from the original crash. Simply having more disk space does not automatically fix this. You must restart the pod so the application remounts the expanded volume cleanly and resets its internal error state.

For Solr running as a StatefulSet (the most common AKS setup):

kubectl rollout restart statefulset/solr -n solr

Alternatively, if your workload runs as a Deployment:

kubectl rollout restart deployment/solr -n solr

For other workloads, simply replace solr with your StatefulSet or Deployment name and adjust the namespace accordingly. The restart causes a brief interruption of around 1–2 minutes. As a result of the restart, three things happen automatically — the broken application state is cleared, the expanded volume is remounted with the full new capacity, and the application performs its own internal recovery on startup.

Step 4: Verify the Pod and Disk Are Healthy

Before touching the application layer, confirm the pod is fully healthy at the infrastructure level. First, check that all pods are back in a Running state:

kubectl get pods -n solr

Next, check the pod logs for clean startup output with no errors:

kubectl logs <solr-pod-name> -n solr --tail=50

In addition, confirm the new disk size is visible and showing healthy usage inside the pod:

kubectl exec -it <solr-pod-name> -n solr -- df -h

The mount point should now show 20Gi total with plenty of free space. If it still shows 10Gi, the filesystem resize has not completed yet — wait a few more minutes and check again. If it still has not updated after 15 minutes, the pod restart itself will trigger the filesystem resize on remount.

Step 5: Application-Level Recovery (Per Workload)

Once the pod is healthy at the infrastructure level, perform the application-specific recovery step for your workload. Furthermore, this step varies depending on what was running in the pod:

  • Sitecore + Solr: Log in to Sitecore → Control Panel → Indexing Manager → select sitecore_jss_web_index → click Rebuild. The job should now complete successfully with no IndexWriter errors.
  • Elasticsearch: Check cluster health with GET /_cluster/health. If any indices are red or yellow, trigger a manual shard allocation using the Cluster Reroute API.
  • PostgreSQL: Run VACUUM ANALYZE on affected tables to clean up any incomplete transactions. Check for replication lag if running a replica set.
  • MongoDB: Check replica set status with rs.status(). Run db.repairDatabase() if any collections show corruption flags.
  • Redis: Verify persistence is working again with INFO persistence — confirm rdb_last_bgsave_status: ok.
  • Custom app pod: Trigger whatever write operation was failing before the disk was full. Check the application logs to confirm the error is gone.

How to Prevent AKS PVC Disk Full in the Future

The most frustrating thing about this problem is that it is completely preventable. Here is what we put in place after this incident — and what I recommend for every production AKS deployment running stateful workloads:

  • Set Azure Monitor alerts at 80% PVC usage. By the time you hit 100% it is already too late. An alert at 80% gives you comfortable time to expand before anything breaks. In Azure Portal, go to Monitor → Alerts and create a metric alert on Persistent Volume Used Bytes.
  • Use Prometheus and Grafana on AKS. The kubelet_volume_stats_used_bytes metric gives you real-time disk usage per PVC across every namespace. Pair it with a Grafana dashboard and a Slack alert — you will never be caught off guard again.
  • Start bigger for production. A 10Gi PVC is fine for development. In production, start at 50Gi or more for any write-heavy workload like Solr, Elasticsearch, or PostgreSQL. Storage is cheap — downtime is not.
  • Plan for 3× headroom during operations. Lucene index rebuilds, PostgreSQL VACUUM, and MongoDB compaction all need temporary space that can be 2–3× the current data size. Always leave enough headroom before triggering these operations.
  • Schedule regular maintenance operations. For Solr, run periodic OPTIMIZE commands to merge segments and reduce disk footprint. For Postgres, schedule regular VACUUM. These can reduce disk usage by 20–40% over time.
  • Verify allowVolumeExpansion on your storage class. Run kubectl describe storageclass managed-premium and confirm AllowVolumeExpansion: true is set. Azure managed-premium and managed-csi-premium both support it by default — but custom storage classes may not.

Conclusion : AKS PVC Disk Full Fix in Under 15 Minutes

An AKS PVC disk full condition is one of the most deceptive problems in Kubernetes operations. The pod stays Running, no obvious alert fires, and the error your application throws is almost never the one that points to disk space. In our case it was this IndexWriter is closed for Sitecore Solr — but it could just as easily be a Postgres write failure, a MongoDB corruption error, or a Redis persistence warning.

In summary, the fix is always the same three steps — confirm the disk is full with df -h inside the pod, expand the PVC online using kubectl patch or edit, and restart the pod to clear the broken application state. Furthermore, the entire process takes under 15 minutes and preserves all your data completely. Most importantly, it is 100% preventable with the right Azure Monitor alerts in place before the next incident hits.

Quick Reference: Commands for AKS PVC Disk Full Fix

  • Check PVCs: kubectl get pvc -n <namespace>
  • Check disk inside pod: kubectl exec -it <pod> -n <namespace> -- df -h
  • Grep for disk error: kubectl logs <pod> -n <namespace> | grep -i "no space"
  • Expand PVC: kubectl patch pvc <name> -n <namespace> -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'
  • Watch resize: kubectl get pvc -n <namespace> -w
  • Restart StatefulSet: kubectl rollout restart statefulset/<name> -n <namespace>
  • Restart Deployment: kubectl rollout restart deployment/<name> -n <namespace>
  • Verify pod health: kubectl get pods -n <namespace>

Frequently Asked Questions

Q: Will expanding the PVC delete my data?
No. PVC expansion on AKS is completely non-destructive. Azure grows the underlying managed disk and extends the filesystem. Your data — whether it is Solr index segments, Postgres tables, or MongoDB collections — is fully preserved throughout.

Q: Does the application go down during PVC expansion?
No. The disk expansion itself causes zero downtime. The only brief interruption — around 1–2 minutes — happens when you restart the pod in Step 3. The application itself stays up during the disk resize.

Q: Can I shrink the PVC back after fixing the issue?
No. Kubernetes does not support PVC shrinking. Once expanded, the size is permanent. Plan your initial PVC sizes carefully — especially for production write-heavy workloads.

Q: The df -h still shows the old size after PVC expansion. Why?
The Kubernetes PVC resize and the filesystem resize inside the pod are two separate operations. If the filesystem has not caught up, wait a few more minutes and recheck. If it still shows the old size after 15 minutes, the pod restart in Step 3 will trigger the filesystem resize on the next mount.

Q: My storage class does not support volume expansion. What do I do?
Run kubectl describe storageclass <name> and check for AllowVolumeExpansion: true. If it is not set, you will need to provision a new larger PVC and migrate the data manually using a tool like kubectl cp or a pod-to-pod rsync job.

Q: I have multiple Solr leader and follower PVCs. Do I need to expand all of them?
Yes. Expand every PVC that is full. If only the leader PVC is full, start there — but check the follower PVCs too, as they often fill at a similar rate.

Related Articles

Your Backend Isn’t Ready — But Your Frontend Doesn’t Have to Wait: Azure APIM Mock APIs Explained

Your backend team says they will be ready in two weeks. Meanwhile, your frontend developers are stuck. Your QA team cannot test. Your client demo is tomorrow. Sound familiar? If you have worked on any API-driven project, you have hit this wall before. Fortunately, Azure API Management (APIM) mock responses solve this exact problem — and most developers do not know they are already included in their APIM subscription at no extra cost.

In this guide, you will learn not just how to set up APIM mock APIs — but when to use them, what mistakes to avoid, and how to use dynamic mocking for real-world scenarios that the official Microsoft docs do not cover.

Introduction: Why Mock APIs Save Real Projects

On a recent project, our team needed to integrate a payment gateway API that was still three weeks away from being production-ready. Instead of blocking three frontend developers, we stood up an APIM mock in under 10 minutes. The frontend team shipped on time. QA ran a full regression suite against the mock. The client saw a working demo. When the real backend arrived, we swapped it in with zero frontend changes.

That is the real value of APIM mock responses — not just as a development convenience, but as a team productivity multiplier that eliminates blockers across the entire delivery pipeline.

What Can You Mock in Azure APIM?

Before diving into the steps, it helps to understand what APIM mocking actually supports. Most tutorials only show the basic 200 OK response. In reality, you can mock much more:

  • Static responses — a fixed JSON body returned every time. Good for happy-path testing.
  • Dynamic responses — responses that change based on query parameters, headers, or request body values. Essential for realistic testing.
  • Error simulation — return 404, 429, 500, or any HTTP status code to test how your frontend handles failures.
  • Conditional responses — return different data based on who is calling (e.g., admin vs. standard user).

💡 Key insight: APIM mock responses live in the inbound policy pipeline. This means they intercept the request before it ever reaches your backend — making them completely safe to use alongside live APIs in the same APIM instance.

When Should You Use APIM Mock Responses?

Here are the four most common real-world scenarios where mocking in APIM saves the day:

  • Backend not ready yet: Frontend and QA teams can start immediately. No more waiting on the backend team to hit a deadline before other work can begin.
  • Client demos and prototypes: Stand up a fully working API in minutes. Show the client a live demo without deploying a single line of backend code.
  • Error handling validation: Return 500, 503, or 429 on demand to confirm your frontend degrades gracefully. This is almost impossible to test reliably against a real backend.
  • Contract-first API design: Define the API contract up front. Let both backend and frontend teams build against the mock simultaneously — then replace the mock when the real service is ready.

Step-by-Step: Create an Azure APIM Mock API Response

Follow these steps to set up your first mock API in Azure APIM. All screenshots below are from the Azure portal.

Step 1: Create a Test API (No Backend Required)

First, create an HTTP API with no backend. This is the blank canvas you will attach your mock policy to.

  1. Sign in to the Azure portal and navigate to your API Management instance.
  2. Select APIs → + Add API → HTTP tile.
  3. In the Create an HTTP API window, select Full.
  4. Enter Test API for Display name.
  5. Select Unlimited for Products.
  6. Ensure that Managed is selected for Gateways.
  7. Click Create.

💡 Pro tip: Use a clear naming convention like “mock-[service-name]-api” so your team can instantly tell which APIs are mocks versus live. For example: mock-payment-api, mock-user-api.

Step 2: Add an Operation to the Test API

Next, add an operation. An operation is a single endpoint — for example, GET /users or POST /orders. In this step, you also define what the response should look like so APIM knows what to return when mocking is enabled.

  1. Select the API you just created and click + Add Operation.
  2. In the Frontend window, enter values for Display name, Name, and URL.
  3. Select the Responses tab, located under the URL and Description fields.
  4. Select + Add response and choose 200 OK from the list.
  5. Under Representations, select + Add representation. Enter application/json and in the Sample text box, enter your mock JSON body. For example:
    { "sampleField" : "test" }

  6. Click Save.

⚠️ Common mistake: Many developers enter a minimal sample like { “id”: 1 } and then wonder why their frontend tests miss edge cases. Instead, use a realistic payload that matches your actual API contract — including all fields the frontend will consume.

Step 3: Enable the Mock Response Policy

Now, enable mocking by attaching the mock-response policy to the operation. This is the step that tells APIM to intercept the request and return your sample — without ever calling a backend.

  1. Select the API and ensure the Design tab is selected.
  2. Select the test operation you added in Step 2.
  3. In the Inbound processing window, select + Add policy.
  4. Select Mock responses from the policy gallery.
  5. In the API Management response textbox, type 200 OK, application/json.
  6. Click Save. You will see a yellow “Mocking is enabled” banner appear at the top of the operation. That confirms the policy is active.

Step 4: Test the Mocked API

Finally, verify that your mock is working correctly. APIM has a built-in Test Console — no Postman needed.

  1. Select your API and click the Test tab.
  2. Select the Test call operation and click Send.
  3. The HTTP response appears at the bottom of the screen. You should see a 200 OK status and the JSON body you defined.
  4. Confirm the JSON response matches the sample you entered in Step 2.

💡 If your APIM instance has IP restrictions, test by calling the APIM gateway URL directly in a browser or Postman. Find your gateway URL in the APIM instance → Overview tab.

Beyond Basics: Dynamic Mock Responses in Azure APIM

The basic setup above returns the same JSON every time. However, real-world testing needs more. For instance, you may want to return different data based on a query parameter, or simulate a 404 when a resource is not found. Fortunately, APIM policies make this straightforward.

Here is an example that returns a different response based on a userId query parameter:

<policies>
  <inbound>
    <base />
    <choose>
      <when condition="@(context.Request.Url.Query.GetValueOrDefault("userId", "") == "999")">
        <return-response>
          <set-status code="404" reason="Not Found" />
          <set-header name="Content-Type" exists-action="override">
            <value>application/json</value>
          </set-header>
          <set-body>{ "error": "User not found", "userId": "999" }</set-body>
        </return-response>
      </when>
      <otherwise>
        <mock-response status-code="200" content-type="application/json" />
      </otherwise>
    </choose>
  </inbound>
</policies>

With this policy in place, calling /users?userId=999 returns a 404 error. Any other userId returns the standard 200 mock. As a result, your frontend team can test both success and failure paths without any backend at all.

Mistakes to Avoid When Using APIM Mock Responses

Based on real project experience, here are the most common mistakes teams make with APIM mocking — and how to avoid them:

  • Forgetting to remove the mock policy before go-live. The mock intercepts requests before they reach your backend. If you forget to remove it, your production API will keep returning fake data. Always add a task to your go-live checklist to verify the mock policy is removed.
  • Using unrealistic sample data. A sample body of { "id": 1 } tells your frontend almost nothing. Instead, use a full realistic payload with all fields — including optional and nullable ones.
  • Only mocking the happy path. Most bugs happen at the edges — 404s, 500s, timeouts, and empty arrays. Mock those too so your frontend handles them correctly from day one.
  • Not communicating which APIs are mocked. Without clear naming or documentation, developers may not realise they are calling a mock. Consequently, they report bugs that do not actually exist in the real backend. Use a naming convention and keep a shared list of active mocks.

Summary

Azure APIM mock responses are one of the most underused features in the entire APIM toolset. By using the mock-response policy, your team can eliminate development blockers, run realistic QA cycles, and deliver client demos — all without a live backend. Furthermore, dynamic mocking with the choose policy takes this further by simulating real error scenarios and conditional logic.

Key Takeaways

  • Use mock-response policy to return static JSON responses without a backend.
  • Combine choose and return-response policies for dynamic, conditional mock responses.
  • Always use realistic sample payloads — not minimal placeholders.
  • Remove mock policies before go-live and add it to your deployment checklist.
  • Validate responses using APIM’s built-in Test Console or Postman.

Frequently Asked Questions

Q: Does enabling mock responses affect other APIs in the same APIM instance?
No. Mock policies are scoped to a specific operation. They only affect the operation they are applied to — all other APIs and operations remain completely unaffected.

Q: Can I mock different HTTP status codes like 404 or 500?
Yes. Use the return-response policy with set-status to return any HTTP status code. This is particularly useful for testing how your frontend handles error states.

Q: Will APIM mock responses work in all pricing tiers?
Yes. The mock-response policy is available across all Azure APIM tiers, including the Consumption tier.

Q: How do I know if a mock policy is active on an operation?
A yellow “Mocking is enabled” banner appears at the top of the operation design view whenever a mock policy is active. In addition, always check the inbound policy XML before deploying to production.

Q: Can I use APIM mock responses for load testing?
Yes — and this is one of the best use cases. Because the mock bypasses your backend entirely, you can run load tests against the APIM gateway itself without stressing your backend services at all.

Related Articles

SNAT Port Exhaustion in AKS: The Silent Killer of Outbound Connectivity (Step-by-Step Fix)

For cloud engineers and DevOps professionals managing applications on Azure Kubernetes Service (AKS), one of the most common yet overlooked issues is SNAT (Source Network Address Translation) port exhaustion. This problem can severely impact outbound connectivity. In particular, it hits hard when applications create a high number of simultaneous connections to a small set of external destinations — such as databases or third-party APIs.

In this article, you will learn what SNAT port exhaustion is, how to detect it, and step-by-step methods to fix it using Azure CLI. You will also learn how to increase outbound ports and frontend IPs to keep your AKS cluster stable under heavy workloads.

Introduction

When SNAT ports run out, your applications can no longer open outbound connections. As a result, you get downtime, timeouts, and performance degradation — often at the worst possible moment. Fortunately, Azure gives you direct control over outbound rules, allocated ports, and frontend IPs on your AKS load balancer. In this guide, we walk through how to identify the problem and fix it step by step.

What Is SNAT Port Exhaustion?

To understand the problem, it helps to know how outbound traffic flows in AKS. Each outbound connection from a pod uses a combination of the pod’s IP address and a port number. SNAT then translates those internal addresses and ports to the load balancer’s public IP and a different port number.

However, the load balancer only has a limited number of ports available for this translation. When your app opens many connections to the same destination IP — for example, a database — it consumes those ports quickly. If all ports are used up, new connections fail. That failure is called SNAT port exhaustion.

💡 In simple terms: think of SNAT ports like phone lines. If all lines are busy, no new calls can get through — even if your app keeps trying.

Real-Life Scenario: SNAT Port Exhaustion in AKS

Imagine you have a high-traffic frontend application running on AKS. It connects to a database hosted on a public IP address. Over time, users start reporting intermittent connectivity errors. After investigating, you discover that the root cause is SNAT port exhaustion — the app has run out of outbound ports to open new connections to the database.

This is a very common pattern in production AKS clusters. Moreover, it is easy to miss because the app appears to work fine under normal load. Only under peak traffic does the exhaustion show up.

How to Identify SNAT Port Exhaustion

Before making any changes, first confirm that SNAT port exhaustion is the actual root cause. Specifically, look for these symptoms:

  • Connection failures: Applications fail to open new outbound connections, even though the destination is reachable.
  • Timeout errors: Connections time out because no SNAT port is available to complete the handshake.
  • Intermittent connectivity: Everything works at low traffic but fails under heavy load — a classic sign of port exhaustion.
  • Azure Monitor alerts: The metric SNATPortExhaustion appears in your Load Balancer metrics dashboard.

To confirm the issue, check your application logs for connection refused or timeout errors. In addition, open the Azure Portal → Load Balancer → Metrics and monitor the SNAT Connection Count and Used SNAT Ports metrics. A spike close to your allocated limit confirms exhaustion.

⚠️ Do not skip this step. Increasing ports without confirming the root cause wastes resources and may not solve the actual problem.

Once confirmed, you need to take two actions:

  • Identify the current outbound rule configuration on the AKS load balancer.
  • Increase the allocated outbound ports and frontend IPs to handle more simultaneous connections.

Using Azure CLI to Fix SNAT Port Exhaustion in AKS

The following steps use Azure CLI to inspect and fix the outbound configuration on your AKS cluster’s load balancer. Follow each step in order.

Step 1: Get the Node Resource Group

First, you need to find the node resource group. In AKS, the underlying infrastructure — including VMs and load balancers — is managed in a separate, auto-generated resource group. Run this command to get its name:

NODE_RG=$(az aks show --resource-group myResourceGroup --name myAKSCluster --query nodeResourceGroup -o tsv)

What each part does:

  • az aks show — retrieves details about your AKS cluster.
  • –resource-group myResourceGroup — the resource group where your AKS cluster lives.
  • –name myAKSCluster — the name of your AKS cluster.
  • –query nodeResourceGroup — extracts only the node resource group name from the response.
  • -o tsv — outputs the result as plain text, ready to use in the next command.

Step 2: List Current Outbound Rules

Next, inspect the current outbound rule configuration on your load balancer. This shows you exactly how many ports are currently allocated and how many frontend IPs are in use:

az network lb outbound-rule list --resource-group $NODE_RG --lb-name kubernetes -o table

What each part does:

  • az network lb outbound-rule list — lists all outbound rules for the specified load balancer.
  • –resource-group $NODE_RG — uses the node resource group from Step 1.
  • –lb-name kubernetes — targets the load balancer (always named kubernetes in AKS by default).
  • -o table — formats the output as a readable table so you can easily spot the current port allocation.

Review the output carefully. Look at the AllocatedOutboundPorts and FrontendIPConfigurations columns. If the allocated ports are low and your app is high-traffic, that is your problem confirmed.

Step 3: Increase Outbound Ports and Frontend IPs

Now that the problem is confirmed, fix it by increasing the allocated outbound ports and the number of frontend IPs. More frontend IPs means more total SNAT ports available across the load balancer. Run the command below — adjust the values to match your cluster name and resource group:

az aks update \
  --resource-group myResourceGroup \
  --name myAKSCluster \
  --load-balancer-managed-outbound-ip-count 7 \
  --load-balancer-outbound-ports 2000

What each part does:

  • az aks update — updates the configuration of an existing AKS cluster.
  • –load-balancer-managed-outbound-ip-count 7 — increases the number of managed outbound IPs to 7. Each IP adds 64,000 available SNAT ports.
  • –load-balancer-outbound-ports 2000 — sets the number of outbound ports allocated per node to 2,000. Adjust this based on your expected connection volume.

📊 Example calculation: 7 IPs × 64,000 ports per IP = 448,000 total SNAT ports available. That is a significant increase over the default configuration.

For a real production cluster, replace the placeholder values with your actual cluster details:

az aks update \
  --resource-group rg-dgtl-dot-prd-we-01 \
  --name aks-dgtl-ms-dot-we-01 \
  --load-balancer-managed-outbound-ip-count 7 \
  --load-balancer-outbound-ports 2000

Step 4: Verify the Fix

After applying the update, verify the new configuration by running the outbound rule list command again:

az network lb outbound-rule list --resource-group $NODE_RG --lb-name kubernetes -o table

Confirm that the AllocatedOutboundPorts and FrontendIPConfigurations columns now reflect your new values. In addition, monitor your Azure Load Balancer metrics over the next 30–60 minutes. Specifically, watch the Used SNAT Ports metric — it should stay well below your new limit.

How to Choose the Right Port Count

Choosing the right values depends on your cluster size and traffic pattern. Here is a simple way to calculate what you need:

  • Ports per node = Total SNAT ports ÷ Number of nodes. For example, 7 IPs × 64,000 = 448,000 ports ÷ 10 nodes = 44,800 ports per node.
  • Start conservative — set --load-balancer-outbound-ports to 1,000 or 2,000 per node and monitor the metrics.
  • Scale up gradually — if Used SNAT Ports still approaches the limit under peak load, increase the IP count or port count further.

⚠️ Important: Setting –load-balancer-outbound-ports too high reduces the number of ports available per backend instance. Always test in a staging environment before applying to production.

Conclusion

For cloud engineers and DevOps teams, managing SNAT port exhaustion in AKS is a critical part of keeping applications reliable under load. By using simple Azure CLI commands, you can identify your current outbound rule configuration, confirm whether exhaustion is occurring, and increase allocated ports and frontend IPs to fix it.

In summary, the fix involves three steps — get the node resource group, inspect the outbound rules, and update the load balancer configuration. Furthermore, adding more managed outbound IPs is the most scalable solution. With 7 IPs, for example, you unlock 448,000 SNAT ports — more than enough for most high-traffic workloads.

Most importantly, always monitor your Used SNAT Ports metric in Azure Monitor after applying changes. Consequently, you will catch any future exhaustion early — before it impacts your users.

Quick Reference: Commands Used in This Guide

  • Get node resource group: az aks show ... --query nodeResourceGroup
  • List outbound rules: az network lb outbound-rule list ... -o table
  • Increase outbound ports and IPs: az aks update ... --load-balancer-managed-outbound-ip-count 7 --load-balancer-outbound-ports 2000
  • Verify the fix: Re-run the outbound rule list and check Azure Monitor metrics.

How Azure API Management Fragments Prevent API Failures from Outdated Mobile Apps

Introduction

Azure API version handling is one of the most overlooked challenges in mobile app development. When older Android and iOS clients send outdated API version headers, backends built on newer versions break. As a result, you get cascading failures across downstream services. Moreover, these failures do not just affect one team — they hit backend engineers, frontend developers, and operations staff all at once.

When building modern APIs, supporting multiple app versions is very common. Over time, backend services evolve — and older mobile apps keep sending requests with outdated headers. In addition, missing metadata makes the problem worse. As a result, three practical challenges appear:

  • How do you enforce the correct API version without breaking older clients?
  • How do you ensure backward compatibility during a migration period?
  • How do you apply routing and transformation logic consistently across multiple APIs?

Fortunately, Azure API Management (APIM) policy fragments solve all three. A fragment lets you share reusable policy logic across APIs. In other words, you write the logic once and apply it everywhere — covering conditional logic, header checks, routing rules, and transformation.

In this article, we walk through a real-world example. Specifically, a fragment checks incoming request headers for platform and API version. It then upgrades outdated mobile clients to the current version automatically — so you get predictable behavior without emergency patches.

Prerequisites

  • An Azure API Management instance with at least one API configured.
  • Access to the APIM portal or ARM/Bicep templates to manage policy fragments.
  • Basic familiarity with APIM inbound policies and C# policy expressions.

Why Azure API Version Handling Fails Without a Gateway Layer

In real-world mobile environments, not all users upgrade their apps right away. For example, some apps still send x-api-version: 1 long after the backend has moved to version 2. Consequently, this gap causes serious problems.

Specifically, this becomes critical when:

  • The microservice backend no longer supports API version 1.
  • Newer frontend apps (web, kiosk, partner systems) already expect API version 2.
  • The backend team has already removed version 1 logic.
  • Older devices or users cannot — or will not — update their apps quickly.

⚠️ Without a protective layer, requests from old mobile apps cause backend failures, break dependent downstream services, and force frontend teams into costly emergency deployments.

Therefore, proper Azure API version handling at the gateway is essential. Instead of patching each service one by one, you apply a single policy fragment. It upgrades outdated requests silently and works the same way every time.

Step-by-Step Guide to Configure the APIM Policy Fragment

Step 1: How Azure API Version Handling Reads Incoming Headers

First, the fragment reads three headers from every incoming request. If any header is missing, safe defaults apply automatically. In other words, the policy never fails on missing input.

[table]

Step 2: Azure API Version Handling — Apply Conditional Logic

Next, the condition checks two things at once. It looks for x-api-version == "1" and also checks whether the platform is Android or iOS. Both must be true for the override to fire. As a result, web, kiosk, and partner systems are completely unaffected.

💡 Why target only Android and iOS? Web browsers and partner integrations are updated centrally and quickly. Only mobile apps — distributed via app stores with slow adoption curves — require this protective fallback.

Step 3: Set Override Headers When Condition Matches

When both conditions are true, the fragment takes two actions:

  • Rewrites x-api-version to "2" — so the backend always treats the request as version 2.
  • Adds is-wrong-api-version: true — a tracking header for logging pipelines, analytics dashboards, and push notification systems.

1️⃣ x-api-version — API version override

This header is rewritten from "1" to "2" transparently. Consequently, the backend never sees the old version at all.

🔹 exists-action=”override” replaces the header value if it already exists, rather than adding a duplicate.

2️⃣ is-wrong-api-version — Observability flag

This tracking header is useful in several ways. For instance, backend services can log a warning. Similarly, analytics pipelines can track how many outdated clients are still active. In addition, push notification systems can use it to prompt users to upgrade.

Step 4: Silent Failure Handling

Furthermore, the entire condition block sits inside a try-catch. If an exception occurs during header parsing, the catch block returns false. As a result, the request moves forward unchanged through the normal pipeline. In other words, the fragment fails open, not closed — so one bad request never causes a service outage.

⚠️ Never remove the try-catch. In production APIM policies, an unhandled exception in a condition expression will return HTTP 500 to the caller.

Step 5: The Complete Policy Fragment

To get started, open the APIM portal and go to APIs → Policy Fragments → Add Fragment. Next, paste the code below and save. Finally, reference it from any API’s inbound policy using <include-fragment fragment-id="api-version-handler" />.

<!--
  APIM Policy Fragment: API Version Handler
  Scope: Global / Per-API / Per-Operation
  Purpose: Upgrade mobile clients on x-api-version=1 to version 2
-->
<fragment>
  <choose>
    <when condition="@{
      var version  = context.Request.Headers
                     .GetValueOrDefault("x-api-version", "1");
      var platform = context.Request.Headers
                     .GetValueOrDefault("platform", "other");
      try {
        if (version == "1" &&
           (platform == "Android" || platform == "iOS")) {
          return true;
        }
        return false;
      } catch (Exception e) {
        return false;
      }
    }">
      <!-- Upgrade the API version transparently -->
      <set-header name="x-api-version" exists-action="override">
        <value>2</value>
      </set-header>
      <!-- Flag the request for analytics and monitoring -->
      <set-header name="is-wrong-api-version" exists-action="override">
        <value>true</value>
      </set-header>
    </when>
  </choose>
</fragment>

Step 6: Reference the Fragment in Your API Policy

Once the fragment is saved, add it to any API’s inbound policy block. As a result, Azure API version handling logic applies to that API instantly — with no duplicated code.

<policies>
  <inbound>
    <base />
    <include-fragment fragment-id="api-version-handler" />
  </inbound>
  <backend>
    <base />
  </backend>
  <outbound>
    <base />
  </outbound>
</policies>

💡 You can include the same fragment across multiple APIs — changes to the fragment automatically apply to all APIs referencing it, with no need to edit each policy individually.

Step 7: Validate the Azure API Version Handling Behaviour

To verify the setup, use the APIM Test Console or Postman. Simply send the headers below and confirm the backend gets the correct overridden values.

Test case 1 — Old mobile client (override should fire)

GET /api/flights HTTP/1.1
x-api-version: 1
platform: Android
version: 3.2.1

Expected result: Backend receives x-api-version: 2 and is-wrong-api-version: true.

Test case 2 — Web client on version 1 (no override)

GET /api/flights HTTP/1.1
x-api-version: 1
platform: web

Expected result: Backend gets the request unchanged — x-api-version: 1 stays as-is and no tracking header is added.

Test case 3 — Updated mobile client (no override)

GET /api/flights HTTP/1.1
x-api-version: 2
platform: iOS
version: 4.0.0

Expected result: Backend gets the request unchanged — version is already 2, so the condition does not match.

Conclusion: Reliable Azure API Version Handling at Scale

In summary, this APIM policy fragment gives you a practical way to enforce Azure API version handling rules without breaking existing clients. By checking requests at the gateway and upgrading outdated headers silently, the fragment delivers four key benefits:

  • Zero downtime — old clients keep working during the migration period without any changes on their end.
  • Consistent versioning — one fragment applies the same logic across all APIs, with no duplicated policy code.
  • Observability — the is-wrong-api-version header flags outdated clients for analytics and logging.
  • Safe failure handling — the try-catch ensures the fragment never breaks the pipeline if an exception occurs.

Without this setup, outdated client requests cascade into backend failures. Moreover, they break downstream services and force emergency responses across multiple teams. Fortunately, a single Azure API version handling fragment — applied once at the gateway — stops all of that before it starts.

Notes

  • Security: The is-wrong-api-version header is added server-side. As a result, clients cannot spoof it — making it reliable for audit trails.
  • Performance: Policy fragment checks add negligible latency. In addition, the try-catch stops any exception from affecting request throughput.
  • Extensibility: You can extend the condition to cover more platform values (e.g. "HuaweiOS") or version ranges without touching any individual API policy.
  • Monitoring: Pair the is-wrong-api-version header with Azure Monitor or Application Insights to build a real-time dashboard of outdated client traffic.

Related Articles

Top Kubectl Commands Every DevOps Engineer Needs for Kubernetes Troubleshooting

Introduction

In the fast-paced world of cloud-native applications, Azure Kubernetes Service (AKS) has become a go-to platform for DevOps teams across the world. However, managing and troubleshooting Kubernetes clusters can be challenging, especially when dealing with issues in pods, deployments, or containers. Thankfully, Kubernetes provides a powerful command-line tool called kubectl that allows you to interact with your AKS cluster. This article will walk you through the most essential kubectl commands to help DevOps engineers and cloud professionals effectively Kubernetes Troubleshooting  issues in their AKS clusters.

When to Use These Commands

You should use these kubectl commands when:

  • Pods are in a CrashLoopBackOff or Error state.
  • A deployment is not scaling as expected.
  • Containers fail to start or exhibit unexpected behavior.
  • Networking issues prevent services from communicating.
  • You need to inspect logs to understand what’s happening inside a container.
  • You want to check the status and events related to a pod or deployment.
  • You need to execute commands inside a running container for debugging purposes.

Essential Kubectl Commands for Troubleshooting in AKS

1. Get the Status of Pods, Deployments, and Services

This command lists all the pods, deployments, or services in a specific namespace. It helps you quickly identify if any pods are in a CrashLoopBackOff, Pending, or Error state.

kubectl get pods
kubectl get deployments
kubectl get services
#Example: 
kubectl get pods -n <namespace>

2. Describe a Pod or Deployment

The describe command provides detailed information about a specific pod or deployment, including events, configuration, and status. This is useful for understanding why a pod might be failing or why a deployment isn’t scaling correctly.

The kubectl describe command provides detailed information about various Kubernetes resources, such as pods, nodes, and deployments. By running kubectl describe <resource> <resource-name> -n <namespace>, you can access a wealth of data, including events, conditions, and configuration details, helping you pinpoint the root cause of problems.

kubectl describe pod 
kubectl describe deployment 
#Example: 
kubectl describe pod my-pod -n <namespace>
kubectl describe deployment customer-service -n customer

The output will contain detailed information about the specified pod, including its metadata, container information, conditions, and events. This information can be invaluable for troubleshooting issues with the pod, such as initialization problems, readiness issues, or events related to its lifecycle.

3. View Pod Logs

This command fetches the logs from a specific pod, which is crucial for debugging issues within the container. You can also use the -f flag to follow the logs in real-time.

When application-level issues arise, examining pod logs is crucial. Use kubectl logs <pod-name> -n <namespace> to view the logs of a specific pod in a given namespace. This command is invaluable for identifying errors, exceptions, or issues within your application code.

kubectl logs 
#Example: 
kubectl logs my-pod -n 

4. Execute Commands in a Running Container

This command allows you to open a shell inside a running container. It’s useful for running diagnostic commands or inspecting the file system directly within the container.

#Linux pod: 
kubectl exec -it  -- /bin/bash
#Window pod: 
kubectl exec -it  -- powershell
#Example: 
kubectl exec -it my-pod -n  -- /bin/bash

5. Check Event Logs for a Namespace

This command lists all events in the cluster, sorted by creation time. Events can provide insights into what’s happening in your cluster, such as why a pod was evicted or why a deployment failed.

kubectl get events --sort-by=.metadata.creationTimestamp
#Example: 
kubectl get events -n <namespace> --sort-by=.metadata.creationTimestamp

Real-World Scenario: Troubleshooting Pod Initialization

Suppose you encounter an issue where pods are not initializing correctly. You can use kubectl get events –all-namespaces to identify events related to pod initialization failures, helping you pinpoint the root cause.

6. View Node Resource Utilization

This command shows the CPU and memory usage of pods or nodes. It’s useful for identifying resource bottlenecks that might be causing issues in your AKS cluster.

kubectl top pod
kubectl top node
#Example: 
kubectl top pod -n <namespace>

Conclusion

Troubleshooting issues in Azure Kubernetes Service (AKS) can be challenging, but with the right kubectl commands, you can quickly identify and resolve problems in your pods, deployments, and containers. By using commands like kubectl get, kubectl describe, kubectl logs, and kubectl exec, you can gain deep insights into the state of your AKS cluster and take corrective actions. Whether you’re dealing with a crashing pod, a misbehaving deployment, or a container that’s not responding, these commands are essential tools in your Kubernetes troubleshooting toolkit.

By mastering these essential kubectl commands, you can minimize downtime, improve operational efficiency, and keep your AKS workloads running smoothly.

Troubleshooting DNS Failures in Azure Kubernetes Service (AKS) Clusters

Introduction

Monitoring DNS resolution inside Azure Kubernetes Service (AKS) is essential for maintaining reliable application connectivity. When DNS failures occur, services can experience intermittent connectivity or complete outages. One effective way to diagnose these issues is by using tcpdump with CoreDNS — the DNS server used by AKS.

In this guide, you’ll learn how to configure tcpdump within your CoreDNS deployment to capture and analyze DNS request/response traffic in real time

Prerequisites:

  • You need kubectl configured and connected to your AKS cluster.
  • Ensure you have permissions to manage deployments and pods in your Kubernetes cluster.

Step-by-Step Guide to Configure tcpdump to Find DNS failures

Step 1: Identify the CoreDNS Deployment

First, you need to find the CoreDNS deployment in your Kubernetes cluster: Look for a deployment named coredns in the kube-system namespace.

kubectl get deployments -n kube-system

Look for a deployment named coredns in the kube-system namespace.

Step 2 : Backup the coredns to Local

As a precautionary measure, create a local backup of your CoreDNS deployment YAML file before making any changes. You can achieve this using the following command:

kubectl get deployment coredns -n kube-system -o yaml > coredns.yaml

Step 3 : Add tcpdump Container to CoreDNS Deployment

Method 1: Edit the CoreDNS Deployment

Edit the coredns deployment and add the tcpdump container under the spec.template.spec.containers section of the CoreDNS deployment YAML. Here’s an example of how you can add it and save:

spec:
template:
spec:
containers:
- name: tcpdump
image: docker.io/corfr/tcpdump
args: ["-C", "100", "-W", "20", "-v", "-w", "/data/dump" ]

When you use tcpdump to capture packets, it writes them into a file (like dump00.pcap). If you keep capturing for a long time, that file can grow very large — potentially filling up the pod’s disk.

1️⃣ -C <file_size> — Limit the size of each capture file

This flag sets the maximum size (in megabytes) of a single capture file.

🔹 Example: -C 100
→ Each .pcap file will be limited to 100 MB.
Once the file reaches 100 MB, tcpdump automatically starts a new file.

2️⃣ -W <file_count> — Limit how many files to keep

This flag sets the number of rotated files tcpdump should keep.

🔹 Example: -W 20
→ tcpdump will keep 20 files maximum, rotating them like a circular buffer.
When it reaches the 21st file, it overwrites the oldest file.

Method 2: patch the deployment

Create patch.yaml file with above spec content and save to local and execute using below kubectl.

kubectl patch deployment coredns -n kube-system –patch-file patch.yaml


Post either followed Method 1 or 2, This changes will apply and trigger a rollout of the CoreDNS deployment.

Step 4 : Check the tcpdump container status

Once the rollout is complete, use the following command to check if the tcpdump container is running within a coredns pod: The output should display the tcpdump container listed among the containers running within the pod.

kubectl describe pod coredns -n kube-system

Step 5 : Verify the dump 

Access the core dns pod using the following command, replacing <coredns-pod-name> with the actual pod name:

kubectl exec -it <coredns-pod-name> -n kube-system -c tcpdump — sh

Once inside the pod, navigate to the /data directory to view the captured packets. You should find a file named “dump00” containing the captured network traffic data.

ls /data

There should be a dump00 file present.

Step 6: Downloading Logs from coredns 

At this point we wait for a few occurrences of the issue. Not sure how long this will take, that would depend on how often you see the error which you expect to collect in dumps. Once enough data is collected, you can exec into each of the pods and rename the file to apply the proper extension and then copy it to your local directory.

After enter in to the coredns pod, execute the below cmd ( you can able to see more dumps like dump00 , dump01, dump01….) to rename all the dump files as you required (XX replace with the your own name). Ensure you executing this command after moving to /data directory. Repeat this step, based on how much dump file present inside the coredns pod.

mv dump00 dumpXX.pcap

Finally exit the pod and in powershell/cmd, you can start to download from coredns to you local

kubectl cp kube-system/coredns-86c697cd8-6qtx9:/data/dump00.pcap -c tcpdump ./coredns-dump00

Step 7: Cleanup (Optional)

Remember to remove the tcpdump container from the CoreDNS deployment once you have completed your troubleshooting to avoid unnecessary resource usage and potential security risks. By restarting the coredns pods, the latest change will removed and back to its original state.

Conclusion

Configuring tcpdump on CoreDNS in AKS allows you to monitor DNS traffic effectively for troubleshooting and analysis purposes. By integrating tcpdump with CoreDNS, you can observe real DNS traffic patterns and isolate failures within your AKS cluster. This approach is invaluable for diagnosing name resolution issues that affect microservice communication or external dependencies. Use it in a controlled manner, and remove the tcpdump container once troubleshooting is complete. 

Notes

  • Security: Exercise caution with tcpdump as it can capture sensitive information. Ensure appropriate access controls and secure practices are in place.
  • Performance: Running tcpdump may impact pod performance and network throughput. Use it judiciously, especially in production environments.
  • Logging and Monitoring: Consider integrating tcpdump logs with your existing logging and monitoring solutions for better visibility and analysis.

Mastering Persistent Storage in Azure Kubernetes Service (AKS): A Step-by-Step Guide Using Azure Disks

Kubernetes is the leading managed container orchestration platform preferred by customers deploying microservices-based architectures in the cloud. Azure Kubernetes Service offers Kubernetes as a managed service, where the container orchestration platform is handled by Azure, enabling customers to focus on the developments of applications. Then how does AKS meet Kubernetes storage demands?

Containers are stateless, which means data is not stored locally—these containers depend on attached persistent volumes to handle the data lifecycle. This blog will walk through the steps required for provisioning persistent volumes and configuring them to be used by containers in Azure Kubernetes Service clusters.

What is Persistent Storage?

Persistent Storage is a mechanism to store data outside the lifecycle of a container or application, ensuring that critical information (like databases, user uploads, or configuration files) persists even if the container restarts, crashes, or is replaced. It’s essential for stateful applications (e.g., databases, CMS) that require data durability.

By default, the storage associated with pods is deleted when the pod lifecycle ends. For stateful applications, however, storage is expected to be persistent so that the data can remain available every time the pods gets recreated in the cluster. Container persistent storage in a Kubernetes cluster is provisioned using the PersistentVolume subsystem, which provides PersistentVolume and PersistentVolumeClaim API resources.

The PersistentVolumeClaim requests for a specific storage class, say Azure disks or Azure Files, and the underlying storage resource gets provisioned. This PersistentVolumeClaim is referenced in the pod definition so that the provisioned storage is mounted to the pod. In this way the PersistentVolume is linked to the PersistentVolumeClaim whenever the provisioned storage is mounted to the pod requesting the resource.

Note: The following steps will assume that the AKS cluster is already provisioned and that the administrator has access to execute the commands listed in this blog.

Provision Persistent Storage Using Azure Disks

There are four major steps in creating and attaching persistent storage using Azure disks in Azure Kubernetes Service.

1. Define/Create of storage class.
2. Configuration of persistent volume claim (PVC) that references of storage class.
3. Create the persistent volume claim and provision the volume.
4. Reference the PVC in the Pod specification to attach the Azure Disk ie., Attaching the provisioned volume to the pod by referencing the specific pod in the pod definition file.’

Let’s explore this process in detail with sample configuration files.

How AKS Uses Azure Disk Storage Classes

Azure Kubernetes Service (AKS) simplifies persistent storage management by offering two pre-configured storage classes for Azure Disks: default and managed-premium. These classes let teams provision storage tailored to workload requirements while abstracting backend complexity. Here’s how they work and where they shine—or fall short.

  • Built-in Storage Classes
  • default (Standard HDD)
  1. default (Standard HDD)
    • Backend: Relies on cost-effective Azure Standard HDD storage.
    • Use Case: Ideal for non-critical workloads like backups, logs, or dev/test environments where high throughput isn’t a priority.
  2. managed-premium (Premium SSD)
    • Backend: Leverages Azure Premium SSD for low latency and high IOPS.
    • Use Case: Suited for production workloads (e.g., databases, transactional apps) demanding consistent performance and faster data access.

Limitations of Built-in Classes

While convenient, the pre-built storage classes come with constraints:

No Post-Provisioning Resizing: Once a volume is created, you can’t expand its size—a hurdle for growing datasets or scaling applications.

Fixed Configuration: Default settings (e.g., performance tiers, redundancy) may not align with specialized needs.

Default: This storage class uses the standard Azure storage which leverages HDDs in the backend.

Creating Custom Storage Classes

In this section we’ll show you how users can create custom storage classes to suit their specific requirements.

1. Make Ready of StorageClass YAML

Connect to the AKS cluster from your management tool of choice (here i used powershell/CMD to connect the cluster using kubectl). For this demonstration we will be using Azure CL. From the Azure CLI, save the following manifest as storage-class.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: managed-standard-custom
provisioner: disk.csi.azure.com
reclaimPolicy: retain
allowVolumeExpansion: true
parameters:
  storageaccounttype: Premium_LRS
  kind: Managed

This file can be used to create a storage class that uses Premium_LRS managed disks. The reclaimPolicy is set to retain so that the persistent volume is not deleted even if the pod to which it is attached gets deleted. With the parameter allowVolumeExpansion set to true, the volume can now be resized even after provisioning.

 

ProvisionerStorage TypeRecommended UseNotes
disk.csi.azure.comAzure Managed Disks (Block Storage)High-performance workloads requiring dedicated block storage on a single nodeCSI-based; supports dynamic provisioning, volume expansion, and advanced features. Recommended for new deployments. :contentReference[oaicite:0]{index=0}
kubernetes.io/azure-disk (deprecated)Azure Managed Disks (Block Storage)Legacy setups using in-tree driversDeprecated in favor of the CSI-based disk.csi.azure.com provisioner.
file.csi.azure.comAzure Files (File Storage)Workloads needing shared storage (ReadWriteMany), such as content management or shared configuration filesCSI-based; supports SMB or NFS file shares with dynamic provisioning. :contentReference[oaicite:1]{index=1}
kubernetes.io/azure-file (deprecated)Azure Files (File Storage)Legacy deployments requiring shared file storageDeprecated in favor of the CSI-based file.csi.azure.com provisioner.
azureblob-csiAzure Blob Storage (Object Storage mounted as a filesystem)Workloads that use large unstructured datasets, such as logs or archival dataEnables mounting of Blob storage via CSI as a filesystem (using protocols like NFS or BlobFuse). Suitable for applications that do not require block storage.

 

2. Apply the yaml file using the following command

$ kubectl apply -f storage-class.yaml

Upon successful execution, you will get a message that the storage class has been created.

Create Persistent Volume Claim and Persistent Volume

The next step is to create a persistent volume claim (PVC), which uses the storage class defined in the above to provision an Azure disk as a persistent volume.

1. Create custom-pvc.yaml file in the Azure CLI window

Create the PVC creation YAML as shown below,

The PVC will request for an Azure disk of 10 GB with accessmode as ReadWriteOnce. That means only one node can mount the volume as read-write.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: custom-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: managed-standard-custom
  volumeMode: Filesystem
Access ModeDescription
ReadWriteOnceThe volume can be mounted as read-write by a single node. It can allow multiple pods to access it when running on the same node.
ReadOnlyManyThe volume can be mounted as read-only by many nodes.
ReadWriteManyThe volume can be mounted as read-write by many nodes.
ReadWriteOncePodFEATURE STATE: Kubernetes v1.29 [stable]
The volume can be mounted as read-write by a single Pod. This ensures that only one pod across the whole cluster can read or write to the PVC.

 

2. To create the volume itself, run the following command:

kubectl apply -f custom-pvc.yaml

On successful execution you will see a message that the persistent volume claim has been created.

After creation, the provisioned volume can also be seen from the Azure portal. Browse to the resource group where the AKS nodes are created to find the newly provisioned disk listed there. 

As highlighted in yellow, you can able to view  Persistent Volume has created with name of  “pvc-ae14b4f9-68d5-4aed-baf3-6addfe4e8b3d”

Once PVC is create successfully (status is bound) then the Persistent volume will create automatically and mapped to the PVC (refer above and below image)

 

And you can also verify disk with name of Persistent Volume by searching in the Azure portal as shown in below, where our mount data will be stored.

You can also run the following command to list the pvc status. It will be listed as “bound” to the created persistent volume:

You can also get all the PVC in your cluster, you can use kubectl get pvc

Attach a Persistent Volume to AKS

After creating the persistent volume claim and the Azure disk, it can be attached to a new pod by referencing the name of the persistent volume claim in the deployment or pod yaml.

1. Create file Nginx.yaml with the following content

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      restartPolicy: Always
      containers:
        - name:  nginx
          image: crdgtlshared02.azurecr.io/nginx:latest
          volumeMounts:
            - name: volume
              mountPath: /tmp/
      volumes:
        - name: volume
          persistentVolumeClaim:
            claimName: custom-pvc
      nodeSelector:
        kubernetes.azure.com/agentpool: userlnxpool
        kubernetes.azure.com/mode: user
        kubernetes.io/os: linux

The volume is mounted at /tmp/ as specified by the mountpath parameter.

Note that the image for the container is taken from my Azure container registry. Alternatively, it can also be downloaded from the Azure container registry.

2. Next, create the pod using the following command:

kubectl apply -f Nginx.yaml

When successfully executed, we can able to view the deployment is created successfully as shown below.

We also ensure in Azure portal by viewing pod will be created as shown below images under default namespace.

3. Verify that the persistent volume is attached to the pod 

kubectl describe pod nginx

The PVC will request for an Azure disk of 10 GB with access mode as ReadWriteOnce. That means only one node can mount the volume as read-write.

You can see that the volume is listed and and uses the claim name “custom-pv.”

Check Volume mount in your pod

To connect to the running Azure container instance, use the following command and run the df command to view the volumes:

kubectl exec -it <pod-name> -n <namespace> -- /bin/bash
kubectl exec -it nginx-7466c47dd6-h9tvz -n default -- /bin/bash

/dev/sdb : >> This indicates the device file associated with the storage. In this case, /dev/sdb refers to the second SCSI disk on the system.]

1K-blocks: 10218772 >>This shows the total size of the filesystem in 1-kilobyte blocks. Here, it amounts to approximately 10,218,772 KB, or roughly 9.75 GB.

Use%: 1% >> This shows the percentage of the filesystem’s capacity that is currently used. In this case, only 1% of the space is utilized.

Mounted on: /tmp >>This specifies the mount point, which is the directory where the filesystem is attached to the system. Here, the device /dev/sdb is mounted on /tmp, the standard directory for temporary files in Unix-like systems.

When AKS Won’t Scale Down: How We Fixed a Real Node Pool Autoscaling Failure in Production

Managing autoscaling in Azure Kubernetes Service (AKS) is not always straightforward. Recently, we ran into a problem where node pool scale down was not happening in both regions, and workloads were consuming resources beyond the defined cluster threshold. This article walks through the scenario, investigation, and resolution.

Scenario: Node Pool Not Scaling Down

The Context: A Planned Node Migration

Our production AKS cluster was running a node pool named mslnxpool02 (D-series VMs) on Kubernetes version 1.31.9. Following a performance review, Microsoft recommended migrating our workloads to E-series VMs for better resource utilization and cost-efficiency aligned with our application’s profile.

  1. To execute this migration, our team proceeded with a blue-green approach:
  2. Created a new node pool with E-series VMs.
  3. Used identical node labels on both the old (D-series) and new (E-series) node pools.

Relied on our Deployments, which used these labels as node Selectors (from starting we having this node selectors so only applied same label to new node pool), to automatically schedule new pods onto the E-series nodes as we cordoned and drained the D-series nodes.

The strategy was to incrementally scale up the E-series pool while simultaneously scaling down the D-series pool, allowing the cluster auto scaler to seamlessly re-schedule pods and reduce node count.

The Problem: Scale-Down Stalls, Incurring Cost

The initial phase of scaling up the new node pool and migrating pods worked as expected. However, we soon noticed a critical issue: the D-series node pool failed to scale down.

Despite workloads being successfully migrated and cluster resource usage dropping significantly, several nodes in the mslnxpool02 pool remained active. Even after waiting for an extended period to account for the cluster auto scaler’s cooldown delays, the node count remained stubbornly high. This resulted in unnecessary cloud costs and a cluster full of underutilized nodes.

The question was: Why was the scale-down stuck?

The following sections detail our investigation and the root cause we uncovered.

Step 1: Investigating the Node health

In my cluster, I had a node pool mslnxpool02 with multiple nodes running Kubernetes v1.31.9. Despite low workloads, the cluster auto scaler wasn’t scaling down the pool. Checking the nodes showed that they were all in a Ready state:

#check with powershell
kubectl get nodes | findstr lnxpool02

At first glance, the nodes looked healthy, but the auto scaler wasn’t freeing up any of them.

Step 2: Describe the Each Node for cause

To understand why the node couldn’t be scaled down, I began with a detailed inspection of one of the stuck nodes: aks-mslnxpool02-33931444-vmss000017.

kubectl describe node aks-lnxpool02-33931444-vmss000017

This command provides a comprehensive overview of the node’s status, capacity, and all the pods running on it. Here’s a breakdown of what I looked for and what I found:

  1. The node was Ready.
  2. Multiple workloads were running on it, including system pods (e.g., kube-proxy, csi-azure*, ama-logs, myapplication pods).
  3. One workload (my service) (customerfeed-service) had large CPU and memory requests as shown below:

CPU Requests: 6100m of 7820m (78%)
CPU Limits: 13 cores (166%)
Memory Requests: 2688Mi
Memory Limits: 10Gi

 

This explained part of the problem: auto scaler cannot evict pods that request a large percentage of node resources unless they can be rescheduled elsewhere.

Now we found the cause, the scaling down was not happened due to the high utilization of some application pods, which not allowing Node pool to scale down.

Step 3: Isolating the Node with Cordon Command

After identifying that the node was blocked from scaling down due to the customerfeed-service pod, the next step was to manually intervene and signal to the cluster that this node was a candidate for removal. This is where the kubectl cordon command comes in.

The primary goal of cordoning a node is to isolate it from receiving any new workloads. Think of it as putting up a “Do Not Enter” sign for the Kubernetes scheduler. This is a crucial, non-disruptive first step in any node maintenance or decommissioning process.

In simple , To prepare the node for scale-down, I manually cordoned it so no new pods would schedule on it:

kubectl cordon aks-mslnxpool02-33931444-vmss000017

Now the node appeared as Ready,SchedulingDisabled

kubectl get nodes | findstr mslnxpool02

aks-lnxpool02-33931444-vmss000017 Ready,SchedulingDisabled <none> 47d v1.31.9

This change told the cluster auto scaler that the node was un-schedulable.

In my case, i had done cordon for all the node pool and  you can confirm in the UI as well like below

Key Takeaways Highlights:

  • cordon is Non-Disruptive: It’s a safe first step that doesn’t affect running pods.

  • It’s a Signal: It tells both the Kubernetes scheduler and the Cluster Autoscaler to avoid this node.

  • It’s Not a Solution by Itself: Cordoning prepares the node but doesn’t solve the underlying resource fragmentation problem. It’s often the prelude to a drain operation.

Step 4: Allocation adjustment or restart the effected pod

After cordoning the node pool, we needed to address the affected pods that were preventing the autoscaler from acting. Simply cordoning ensures no new pods are scheduled on the node, but the existing high-utilization pod continues running and blocking scale down.

To resolve this, we took two possible actions:

Option A:  Restart the Affected Pod (Quick Fix)

If adjusting manifests immediately isn’t feasible (for example, in production during peak hours), a faster approach is to restart the affected pod so it can be rescheduled to a healthier node pool(as old node pool is restricted to schedule so it will map to new node pool):

1. Delete the Pod

 
kubectl delete pod <pod-name> -n <namespace>

2. Kubernetes Rescheduling

Kubernetes automatically recreates the pod on another available node pool.

This helps free up utilization on the cordoned pool, enabling the Cluster Auto scaler to scale it down.

⚠️ Note: This is a temporary workaround. Without fixing the resource requests in the deployment, the issue may reappear.

Option B:  Allocation Adjustment (Preferred)

1. Review Pod Resource Requests and Limits

Check the resources.requests and resources.limits configuration in the deployment manifest. In our case, the application pod had very high actual CPU utilization, but its configuration did not reflect realistic usage, keeping the node pinned.

In our case, the application pod had very high CPU requests, which forced the scheduler to keep nodes active.

Example snippet:

resources:
  requests:
    cpu: "2000m"   # 2 cores requested
    memory: "4Gi"
  limits:
    cpu: "3000m"
    memory: "6Gi"

2. Adjust Resource Requests

Increase the CPU or memory requests (based on observed utilization) to match actual needs. This ensures Kubernetes schedules pods more accurately and avoids overcommitting nodes.

Example snippet:

resources:
  requests:
    cpu: "6000m"   # increased to 4 core
    memory: "4Gi"
  limits:
    cpu: "4000m"
    memory: "6Gi"

This alignment gives the auto scaler a true view of resource usage, enabling it to correctly evaluate which nodes can safely scale down.

Conclusion

Node pool scale-down issues in AKS often come down to workload placement and resource requests. In this scenario, a single oversized pod was preventing the auto scaler from acting. By analyzing node workloads, cordoning, and draining, I was able to resolve the problem and allow auto scaler to scale down efficiently.