Dataverse API Execution Limits, Retry Policies, and Batch Optimisation

1. Conceptual Foundation

To build enterprise-grade integrations with Microsoft Dataverse, developers must thoroughly understand the platform's resource-governance mechanisms. Dataverse is a multi-tenant, cloud-scale data platform. To ensure high availability, consistent performance, and protection against noisy-neighbor scenarios, the platform enforces strict execution limits. These limits are categorized into Entitlement Limits and Service Protection Limits.

+---------------------------------------------------------------------------------+
|                                 Dataverse API                                   |
+---------------------------------------------------------------------------------+
                                         |
                                         v
                    +-----------------------------------------+
                    |        Service Protection Limits        |
                    |  (Evaluated per User per Web Server)    |
                    +-----------------------------------------+
                                         |
         +-------------------------------+-------------------------------+
         |                               |                               |
         v                               v                               v
+------------------+            +------------------+            +------------------+
| Request Count    |            | Execution Time   |            | Concurrency      |
| Limit: 6,000     |            | Limit: 1,200s    |            | Limit: 52        |
| Window: 300s     |            | Window: 300s     |            | Window: Instant  |
+------------------+            +------------------+            +------------------+

Service Protection Limits: The Three Facets

Service Protection Limits are designed to protect the physical web servers hosting a Dataverse environment from being overwhelmed by extraordinary API demands. Unlike Entitlement Limits, which are evaluated over a 24-hour period, Service Protection Limits are evaluated within a five-minute (300-second) sliding window and are enforced per user, per web server.

The platform monitors three distinct facets of API usage:

1. Number of Requests (Rate Throttling):

   • Default Limit: 6,000 requests per user, per web server, within a 300-second sliding window.
   • Mechanics: Every incoming external API request (Web API or Organization Service) increments a counter for the authenticated user. If this counter exceeds 6,000 within the preceding 300 seconds, subsequent requests are blocked, and the server returns an HTTP 429 Too Many Requests (Web API) or an OrganizationServiceFault with error code 0x80072322 (SDK).
   • Enterprise Gateway Variations: In highly customized enterprise architectures utilizing Azure API Management (APIM) or custom reverse proxies, a tighter 100-second window with a 52,000-request limit may be enforced at the gateway layer to protect downstream microservices. Resilient clients must be designed to parse and respect throttling headers regardless of whether they originate from the native Dataverse web server or an intermediary gateway.

2. Combined Execution Time (Resource Throttling):

   • Default Limit: 20 minutes (1,200 seconds or 1,200,000 milliseconds) of combined execution time per user, per web server, within a 300-second sliding window.
   • Mechanics: Dataverse tracks the actual server-side processing time (CPU and database execution) for all requests initiated by a user. This is critical because some operations (such as complex FetchXML queries, deep inserts, or requests triggering synchronous plug-ins) consume significantly more resources than simple CRUD operations. If a user's cumulative execution time exceeds 1,200 seconds within the 300-second window, subsequent requests fail with HTTP 429 or SDK error code 0x80072321.

3. Number of Concurrent Requests (Concurrency Throttling):

   • Default Limit: 52 concurrent requests per user, per web server.
   • Mechanics: This limit is evaluated instantly rather than over a sliding window. It prevents a single user from monopolizing the web server's thread pool by sending a massive burst of parallel requests. If a client application attempts to execute more than 52 requests simultaneously, the server immediately rejects the excess requests with HTTP 429 or SDK error code 0x80072326.

The Role of Server Affinity and Load Distribution

Dataverse environments are hosted across multiple web servers behind a load balancer. By default, client connections utilize Server Affinity (managed via the ARRAffinity cookie). This binds a client session to a specific web server to optimize caching.

However, because Service Protection Limits are enforced per web server, keeping server affinity enabled means a high-throughput integration will quickly exhaust the limits of its assigned server, even if other servers in the cluster are idle.

To maximize throughput, enterprise integrations must disable server affinity. By removing the affinity cookie, the load balancer distributes requests round-robin across all available web servers. This effectively multiplies the client's throttling threshold by the number of active web servers in the environment.

Entitlement Limits vs. Service Protection Limits

It is vital to distinguish between these two governance layers:

Feature	Service Protection Limits	Entitlement Limits (Power Platform Requests)
Primary Purpose	Protect server health and prevent service degradation.	Govern platform consumption based on licensing.
Evaluation Window	5-minute (300-second) sliding window (or 100-second gateway window).	24-hour static window (daily reset).
Enforcement Point	Enforced immediately at the web server or gateway layer.	Monitored via admin reports; overages managed via licensing add-ons.
Scope of Impact	Applies to all external web service requests.	Applies to both external API calls and internal operations (flows, plug-ins).
Error Response	Returns HTTP 429 or OrganizationServiceFault immediately.	Does not block interactive users; may throttle non-compliant flows.

Impact on Plug-ins and Custom Workflow Activities

Service Protection Limits do not apply to data operations originating internally from plug-ins or custom workflow activities, as these run within the isolated Sandbox Service and bypass public API endpoints.

However, any database execution time or API calls triggered by synchronous plug-ins are added to the execution time of the external request that initiated the transaction. If an external client calls an API that triggers a heavy synchronous plug-in, the plug-in's execution time directly contributes to the client's 1,200-second execution time limit, accelerating the onset of resource throttling.

---

2. Architecture & Decision Matrix

When designing high-throughput data integration pipelines, selecting the correct architectural pattern is critical to balancing performance, cost, and maintainability.

Implementation Approaches

1. Power Automate (Cloud Flows)

Best suited for low-to-medium volume, event-driven integrations. While easy to configure, Power Automate is subject to strict API connector limits and execution timeouts. It lacks native, granular control over HTTP headers and thread-level concurrency, making it inefficient for bulk migrations.

2. Azure Functions (.NET / C#)

The gold standard for high-volume, enterprise-grade integrations. Azure Functions provide serverless scalability, native integration with Azure Key Vault and Managed Identities, and full access to the Dataverse SDK for .NET (ServiceClient). Developers have complete control over thread pools, concurrency, and custom Polly retry policies.

3. Dataverse Plug-ins (C#)

Synchronous or asynchronous code executed directly within the Dataverse pipeline. Plug-ins are ideal for real-time validation and data enrichment but are subject to a hard 2-minute execution timeout. They cannot be used as the primary engine for bulk data loading due to sandbox isolation limits and the prohibition of parallel thread execution.

4. Power Apps Component Framework (PCF) / Client-side JS

Executed entirely within the user's browser context. Client-side code must use the Web API and is highly constrained by browser thread limits and network latency. It is unsuitable for backend integrations but must implement basic retry logic to handle transient errors during interactive operations.

Comparison Matrix

Criteria	Power Automate	Azure Functions	Dataverse Plug-ins	PCF / Client-side JS
Complexity	Low (Low-code)	Medium (Pro-code)	High (Pro-code)	Medium (Pro-code)
Scalability	Medium (Throttled by flow limits)	Extremely High (Elastic scale)	Low (Bound to transaction)	Low (Bound to browser)
Execution Context	Asynchronous / Event-driven	Asynchronous / Scheduled	Sync/Async Pipeline	Client-side Browser
Offline Support	No	No	Yes (via Mobile Offline)	Yes (via Client Cache)
Licensing	Power Automate / Power Apps	Azure Subscription (Pay-as-you-go)	Included in Dataverse	Included in Power Apps
PL-400 Relevance	High (Triggering & Actions)	High (Custom API integration)	Critical (Pipeline & Sandbox)	High (Web API usage)

Architectural Decision Flow

Is the integration real-time and triggered by a Dataverse event?
  ├── Yes: Is the execution time guaranteed to be under 2 seconds?
  │     ├── Yes: Implement a synchronous Dataverse Plug-in.
  │     └── No: Register an Asynchronous Plug-in or offload to Azure Service Bus.
  └── No: Is it a bulk data load or scheduled synchronization?
        ├── Is the data volume low (< 10,000 records daily) and logic simple?
        │     ├── Yes: Use Power Automate with the Dataverse Connector.
        │     └── No: Use Azure Functions with ServiceClient and ExecuteMultipleRequest.
        └── Is transactional integrity across multiple tables required?
              ├── Yes: Use ExecuteTransactionRequest (max 1,000 records).
              └── No: Use CreateMultipleRequest / UpdateMultipleRequest.

---

3. Step-by-Step Implementation Guide

This guide details how to build a high-throughput, resilient .NET 8 integration client that connects to Dataverse, disables server affinity, parses throttling headers, and optimizes throughput using bulk operations.

Step 1: Project Setup and Dependency Management

Create a new .NET 8 Console Application and install the required NuGet packages. The Microsoft.PowerPlatform.Dataverse.Client package is the modern, cross-platform SDK replacing the legacy CrmServiceClient.

# Create a new console application
dotnet new console -n Dataverse.ResilientIntegration -f net8.0
cd Dataverse.ResilientIntegration

# Install the modern Dataverse SDK
dotnet add package Microsoft.PowerPlatform.Dataverse.Client --version 1.2.10

# Install Polly for advanced resilience and transient fault handling
dotnet add package Polly --version 8.3.0
dotnet add package Polly.Core --version 8.3.0

# Install configuration and logging packages
dotnet add package Microsoft.Extensions.Configuration.Json --version 8.0.0
dotnet add package Microsoft.Extensions.Logging.Console --version 8.0.0

Step 2: Configure the ServiceClient with Native Retries

The ServiceClient class has built-in capabilities to handle Service Protection Limits. By default, it detects 429 errors and pauses execution. We must explicitly configure these properties to ensure optimal behavior.

Create an appsettings.json file in the root of your project:

{
  "Dataverse": {
    "ConnectionString": "AuthType=ClientSecret;Url=https://org8b1a2f.crm.dynamics.com;ClientId=00000000-0000-0000-0000-000000000000;ClientSecret=YourSecretHere;",
    "MaxRetryCount": 5,
    "RetryPauseTimeSeconds": 10
  }
}

Ensure the file is copied to the output directory by adding this to your .csproj:

<ItemGroup>
  <None Update="appsettings.json">
    <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
  </None>
</ItemGroup>

Step 3: Implement a Custom DelegatingHandler for Web API Requests

If your integration bypasses the SDK and uses the Web API directly via HttpClient, you must implement a custom DelegatingHandler to parse the Retry-After header and execute an exponential backoff strategy.

using System;
using System.Net;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.Logging;

public class DataverseServiceProtectionHandler : DelegatingHandler
{
    private readonly int _maxRetries;
    private readonly ILogger _logger;

    public DataverseServiceProtectionHandler(int maxRetries, ILogger logger)
    {
        _maxRetries = maxRetries;
        _logger = logger;
    }

    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        int attempt = 0;
        while (true)
        {
            attempt++;
            HttpResponseMessage response = await base.SendAsync(request, cancellationToken);

            if (response.StatusCode == (HttpStatusCode)429 && attempt <= _maxRetries)
            {
                double delaySeconds = 10; // Default fallback delay

                if (response.Headers.RetryAfter != null)
                {
                    if (response.Headers.RetryAfter.Delta.HasValue)
                    {
                        delaySeconds = response.Headers.RetryAfter.Delta.Value.TotalSeconds;
                    }
                    else if (response.Headers.RetryAfter.Date.HasValue)
                    {
                        delaySeconds = (response.Headers.RetryAfter.Date.Value - DateTimeOffset.UtcNow).TotalSeconds;
                    }
                }

                // Log rate limit headers for debugging
                if (response.Headers.TryGetValues("x-ms-ratelimit-burst-remaining-xrm-requests", out var burstRemaining))
                {
                    _logger.LogWarning("Burst requests remaining: {Remaining}", string.Join(",", burstRemaining));
                }

                _logger.LogWarning("Request throttled (429). Attempt {Attempt} of {MaxRetries}. Waiting {Delay} seconds...", 
                    attempt, _maxRetries, delaySeconds);

                await Task.Delay(TimeSpan.FromSeconds(Math.Max(delaySeconds, 1)), cancellationToken);
                continue;
            }

            return response;
        }
    }
}

Step 4: Disable Server Affinity

To ensure that requests are distributed across all available Dataverse web servers, we must disable cookies on the underlying handler.

var handler = new HttpClientHandler
{
    UseCookies = false // Disables ARRAffinity cookies to bypass single-server throttling
};

var resilientHandler = new DataverseServiceProtectionHandler(5, logger)
{
    InnerHandler = handler
};

var httpClient = new HttpClient(resilientHandler);

Step 5: Execute Optimized Batch Requests

When using the SDK, group operations using ExecuteMultipleRequest. If you are targeting standard tables that support it, use CreateMultipleRequest or UpdateMultipleRequest for maximum performance.

---

4. Complete Code Reference

C# Sample 1: Resilient Dataverse Batch Processor

This production-grade class manages connection initialization, disables server affinity, handles batch size faults dynamically, and processes records using ExecuteMultipleRequest with robust error handling.

// <summary>
// Production-grade batch processor for Microsoft Dataverse.
// Handles Service Protection Limits, dynamic batch resizing, and server affinity removal.
// </summary>

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.ServiceModel;
using System.Threading.Tasks;
using Microsoft.Extensions.Logging;
using Microsoft.PowerPlatform.Dataverse.Client;
using Microsoft.Xrm.Sdk;
using Microsoft.Xrm.Sdk.Messages;

namespace Dataverse.ResilientIntegration
{
    public class DataverseBatchProcessor
    {
        private readonly ServiceClient _serviceClient;
        private readonly ILogger<DataverseBatchProcessor> _logger;
        private int _currentMaxBatchSize = 1000; // Default platform limit

        public DataverseBatchProcessor(string connectionString, ILogger<DataverseBatchProcessor> logger)
        {
            _logger = logger ?? throw new ArgumentNullException(nameof(logger));

            // Initialize ServiceClient with disabled server affinity
            _serviceClient = new ServiceClient(connectionString)
            {
                MaxRetryCount = 10,
                RetryPauseTime = TimeSpan.FromSeconds(5),
                EnableHttpCookies = false // CRITICAL: Disables ARRAffinity cookies to distribute load
            };

            if (!_serviceClient.IsReady)
            {
                throw new InvalidOperationException($"Failed to connect to Dataverse: {_serviceClient.LastError}", _serviceClient.LastException);
            }

            _logger.LogInformation("Successfully connected to Dataverse organization: {OrgName}", _serviceClient.ConnectedOrgFriendlyName);
        }

        public async Task ProcessEntitiesInBatchesAsync(string entityLogicalName, List<Entity> entities, int targetBatchSize = 1000)
        {
            _currentMaxBatchSize = targetBatchSize;
            var queue = new Queue<Entity>(entities);
            int batchCounter = 0;

            while (queue.Count > 0)
            {
                batchCounter++;
                var currentBatchList = new List<Entity>();

                while (currentBatchList.Count < _currentMaxBatchSize && queue.Count > 0)
                {
                    currentBatchList.Add(queue.Dequeue());
                }

                _logger.LogInformation("Processing batch {BatchNumber} containing {Count} records...", batchCounter, currentBatchList.Count);

                var requestCollection = new OrganizationRequestCollection();
                foreach (var entity in currentBatchList)
                {
                    var createRequest = new CreateRequest { Target = entity };
                    requestCollection.Add(createRequest);
                }

                var executeMultipleRequest = new ExecuteMultipleRequest
                {
                    Requests = requestCollection,
                    Settings = new ExecuteMultipleSettings
                    {
                        ContinueOnError = true,
                        ReturnResponses = true
                    }
                };

                try
                {
                    var stopwatch = Stopwatch.StartNew();
                    var response = (ExecuteMultipleResponse)await Task.Run(() => _serviceClient.Execute(executeMultipleRequest));
                    stopwatch.Stop();

                    _logger.LogInformation("Batch {BatchNumber} executed in {Duration}ms.", batchCounter, stopwatch.ElapsedMilliseconds);
                    ProcessBatchResponses(response, currentBatchList, queue);
                }
                catch (FaultException<OrganizationServiceFault> ex)
                {
                    // Handle Batch Size Fault (Error Code: -2147220970 or similar when exceeding max batch size)
                    if (ex.Detail.ErrorDetails.Contains("MaxBatchSize"))
                    {
                        int platformLimit = (int)ex.Detail.ErrorDetails["MaxBatchSize"];
                        _logger.LogWarning("Batch size {CurrentSize} exceeded platform limit. Resizing to {PlatformLimit} and retrying...", 
                            _currentMaxBatchSize, platformLimit);
                        
                        _currentMaxBatchSize = platformLimit;

                        // Re-queue the failed batch items
                        foreach (var entity in currentBatchList)
                        {
                            queue.Enqueue(entity);
                        }
                    }
                    else
                    {
                        _logger.LogError(ex, "Fatal Organization Service Fault occurred during batch execution.");
                        throw;
                    }
                }
                catch (Exception ex)
                {
                    _logger.LogError(ex, "Unexpected error during batch execution.");
                    throw;
                }
            }
        }

        private void ProcessBatchResponses(ExecuteMultipleResponse response, List<Entity> originalBatch, Queue<Entity> mainQueue)
        {
            foreach (var responseItem in response.Responses)
            {
                if (responseItem.Fault != null)
                {
                    var failedEntity = originalBatch[responseItem.RequestIndex];
                    _logger.LogError("Request index {Index} failed. Error: {Message}", responseItem.RequestIndex, responseItem.Fault.Message);

                    // Check for Service Protection Limit within the batch response
                    if (responseItem.Fault.ErrorDetails.ContainsKey("Retry-After"))
                    {
                        var retryAfter = (TimeSpan)responseItem.Fault.ErrorDetails["Retry-After"];
                        _logger.LogWarning("Service Protection Limit hit inside batch. Retry-After: {Delay}s. Re-queuing record...", retryAfter.TotalSeconds);
                        
                        // Re-queue the specific failed record to the front of the queue
                        mainQueue.Enqueue(failedEntity);
                        
                        // Pause execution to respect the Retry-After header
                        Task.Delay(retryAfter).Wait();
                    }
                }
                else
                {
                    var createResponse = (CreateResponse)responseItem.Response;
                    _logger.LogDebug("Successfully created record with ID: {Id}", createResponse.id);
                }
            }
        }
    }
}

TypeScript Sample: Web API Multipart Batch Client

This TypeScript class constructs a raw OData $batch request using the multipart/mixed format, parses the response, and handles 429 errors with exponential backoff.

/**
 * Production-grade TypeScript client for Dataverse Web API batch operations.
 * Handles multipart/mixed payload generation, response parsing, and 429 Retry-After headers.
 */

export interface BatchOperation {
    method: "POST" | "PATCH" | "DELETE";
    entitySet: string;
    id?: string;
    data: any;
}

export class DataverseBatchClient {
    private clientUrl: string;
    private accessToken: string;
    private maxRetries: number = 5;

    constructor(clientUrl: string, accessToken: string) {
        this.clientUrl = clientUrl.replace(/\/$/, "");
        this.accessToken = accessToken;
    }

    public async executeBatch(operations: BatchOperation[]): Promise<any[]> {
        const boundary = `batch_`${this.generateUniqueIdentifier()}``;
        const changesetBoundary = `changeset_`${this.generateUniqueIdentifier()}``;
        
        let body = "";
        body += `--`${boundary}`\r\n`;
        body += `Content-Type: multipart/mixed; boundary=`${changesetBoundary}`\r\n\r\n`;

        operations.forEach((op, index) => {
            body += `--`${changesetBoundary}`\r\n`;
            body += `Content-Type: application/http\r\n`;
            body += `Content-Transfer-Encoding: binary\r\n`;
            body += `Content-ID: `${index + 1}`\r\n\r\n`;

            const url = op.id ? ``${op.entitySet}`(`${op.id}`)` : op.entitySet;
            body += ``${op.method}` `${this.clientUrl}`/api/data/v9.2/`${url}` HTTP/1.1\r\n`;
            body += `Content-Type: application/json;type=entry\r\n\r\n`;
            body += ``${JSON.stringify(op.data)}`\r\n`;
        });

        body += `--`${changesetBoundary}`--\r\n`;
        body += `--`${boundary}`--\r\n`;

        return this.sendRequestWithRetry(``${this.clientUrl}`/api/data/v9.2/$batch`, body, boundary, 1);
    }

    private async sendRequestWithRetry(url: string, body: string, boundary: string, attempt: number): Promise<any[]> {
        try {
            const response = await fetch(url, {
                method: "POST",
                headers: {
                    "Authorization": `Bearer `${this.accessToken}``,
                    "Content-Type": `multipart/mixed; boundary=`${boundary}``,
                    "OData-MaxVersion": "4.0",
                    "OData-Version": "4.0",
                    "Accept": "application/json"
                },
                body: body
            });

            if (response.status === 429) {
                if (attempt > this.maxRetries) {
                    throw new Error(`Max retries exceeded for batch operation. Status: 429`);
                }

                const retryAfterHeader = response.headers.get("Retry-After");
                let delaySeconds = Math.pow(2, attempt); // Exponential backoff fallback
                
                if (retryAfterHeader) {
                    delaySeconds = parseInt(retryAfterHeader, 10);
                }

                console.warn(`[429] Throttled by Dataverse. Waiting `${delaySeconds}` seconds before retry attempt `${attempt}`...`);
                await new Promise(resolve => setTimeout(resolve, delaySeconds * 1000));
                return this.sendRequestWithRetry(url, body, boundary, attempt + 1);
            }

            if (!response.ok) {
                const errorText = await response.text();
                throw new Error(`HTTP Error `${response.status}`: `${errorText}``);
            }

            const responseText = await response.text();
            return this.parseBatchResponse(responseText);
        } catch (error) {
            console.error("Error executing batch request:", error);
            throw error;
        }
    }

    private parseBatchResponse(responseText: string): any[] {
        const responses: any[] = [];
        const lines = responseText.split("\r\n");
        let currentStatus: number | null = null;

        lines.forEach(line => {
            if (line.startsWith("HTTP/1.1")) {
                const parts = line.split(" ");
                currentStatus = parseInt(parts[1], 10);
            }
            if (line.startsWith("{") && currentStatus !== null) {
                try {
                    const json = JSON.parse(line);
                    responses.push({ status: currentStatus, data: json });
                    currentStatus = null;
                } catch {
                    // Handle non-JSON or partial lines
                }
            }
        });

        return responses;
    }

    private generateUniqueIdentifier(): string {
        return Math.random().toString(36).substring(2, 15);
    }
}

---

5. Configuration & Environment Setup

Environment Variables & App Settings

To deploy the resilient integration securely, configuration parameters must be externalized. Below is the schema definition and configuration for production environments.

Environment Variable Schema Definition (envvariabledefinitions.json)

This schema defines the environment variables used within a Dataverse Solution to store the integration endpoint and configuration.

[
  {
    "schemaname": "contoso_DataverseEndpoint",
    "displayname": "Dataverse API Endpoint",
    "type": 100000000, 
    "description": "The base URL of the target Dataverse environment.",
    "defaultvalue": "https://org8b1a2f.crm.dynamics.com"
  },
  {
    "schemaname": "contoso_MaxRetryCount",
    "displayname": "Max Retry Count",
    "type": 100000001, 
    "description": "Maximum number of retries for throttled requests.",
    "defaultvalue": "5"
  }
]

Azure Resource Configuration (Bicep)

When hosting the integration in Azure (e.g., Azure Functions), use a System-Assigned Managed Identity to authenticate against Dataverse. This eliminates the need to manage client secrets.

param functionAppName string
param location string = resourceGroup().location
param keyVaultName string

resource keyVault 'Microsoft.KeyVault/vaults@2023-07-01' existing = {
  name: keyVaultName
}

resource functionApp 'Microsoft.Web/sites@2023-12-01' = {
  name: functionAppName
  location: location
  kind: 'functionapp'
  identity: {
    type: 'SystemAssigned' // Enables Managed Identity
  }
  properties: {
    httpsOnly: true
    siteConfig: {
      appSettings: [
        {
          name: 'DataverseEndpoint'
          value: '@Microsoft.KeyVault(SecretUri=`${keyVault.properties.vaultUri}`secrets/DataverseEndpoint/)'
        },
        {
          name: 'MaxRetryCount'
          value: '5'
        }
      ]
    }
  }
}

output principalId string = functionApp.identity.principalId

---

6. Security & Permission Matrix

To execute high-throughput API operations, the security principal (Application User or Interactive User) must be configured with the minimum required privileges to prevent security exceptions during batch execution.

Principal	Permission	Scope	Reason
Application User (Service Principal)	Read / Write / Create	Organization	Required to perform CRUD operations on target tables within the batch.
Application User (Service Principal)	prvShare / prvAssign	Organization	Required if the batch contains AssignRequest or GrantAccessRequest messages.
Managed Identity	Azure Active Directory Sign-In	Tenant	Required to acquire OAuth2 tokens from Microsoft Entra ID without client secrets.
Integration Security Role	Execute Multiple Privilege	Global	Allows the user to execute the ExecuteMultipleRequest message.
System Customizer / Admin	Read on EnvironmentVariableValue	Organization	Required to retrieve configuration values at runtime.

---

7. Pipeline Execution Internals

Understanding the internal execution pipeline of Dataverse is critical when optimizing batch operations.

External Request (ExecuteMultiple / ExecuteTransaction)
  │
  ├── Web Server Layer (Service Protection Limits Evaluated)
  │     └── If limits exceeded -> Return HTTP 429 / Fault
  │
  └── Sandbox Service (Sandbox Isolation Mode)
        │
        ├── Pre-Validation (Stage 10) - Out-of-transaction validation
        │
        ├── Pre-Operation (Stage 20) - Inside database transaction
        │
        ├── Main Operation (Stage 30) - Database write operation
        │
        └── Post-Operation (Stage 40) - Inside database transaction

Transaction Boundaries

• ExecuteMultipleRequest: Each request in the input collection is executed in a separate database transaction. If request #5 fails, requests #1 through #4 remain committed in the database. This is highly performant but requires the client to handle partial failures.
• ExecuteTransactionRequest: All requests are executed within a single, atomic database transaction. If any request fails, the entire batch is rolled back. This guarantees data integrity but introduces significant database locking and increases the risk of SQL timeouts.

Sandbox Isolation Mode & Callout Restrictions

All custom plug-ins and workflow activities run in a restricted Sandbox Isolation Mode.

• No Parallel Execution: Spawning parallel threads (e.g., Task.Run, Parallel.ForEach) within a plug-in is strictly prohibited and will cause a runtime security exception.
• Outbound Network Restrictions: Plug-ins can only make outbound web calls via HTTP/HTTPS (ports 80/443). IP addresses must be publicly resolvable (no local intranet access).
• 2-Minute Timeout: Any execution path exceeding 120 seconds is terminated immediately, rolling back the transaction.

Depth and Loop-Guard Limits

To prevent infinite recursive loops (e.g., a plug-in on update of Account updates the same Account, triggering itself), Dataverse enforces a Depth Limit.

• Mechanics: The platform tracks the Depth property in the IExecutionContext. Every nested execution increments this value.
• Default Limit: If the depth exceeds 8 within a 1-hour window, the platform aborts execution and throws a MaxDepthExceeded exception.

---

8. Error Handling & Retry Patterns

Common Failure Modes & Error Codes

Error Name	Hex Code	Decimal Code	Root Cause	Remediation
ThrottlingBurstRequestLimitExceededError	0x80072322	-2147015902	Exceeded 6,000 requests in 5 minutes.	Parse Retry-After header, pause execution, and switch to batching.
ThrottlingTimeExceededError	0x80072321	-2147015903	Exceeded 20 minutes of execution time.	Optimize queries, disable heavy plug-ins, or reduce batch size.
ThrottlingConcurrencyLimitExceededError	0x80072326	-2147015898	Exceeded 52 concurrent requests.	Reduce client-side thread count / degree of parallelism.
ExecuteMultiplePerOrgMaxConnectionsPerServer	0x80044150	-2147204784	Exceeded concurrent ExecuteMultiple limit.	Implement client-side throttling across integration instances.
MessageSizeExceeded	0x8004E007	-2147220970	Payload size exceeded 116.85 MB.	Reduce batch size or chunk binary files.

Parsing OrganizationServiceFault for Retry-After

When using the SDK, the Retry-After duration is returned as a TimeSpan inside the ErrorDetails collection of the OrganizationServiceFault.

try
{
    _serviceClient.Execute(request);
}
catch (FaultException<OrganizationServiceFault> ex)
{
    if (ex.Detail.ErrorDetails.ContainsKey("Retry-After"))
    {
        var delay = (TimeSpan)ex.Detail.ErrorDetails["Retry-After"];
        Console.WriteLine($"Throttled. Waiting {delay.TotalSeconds} seconds...");
        await Task.Delay(delay);
        // Retry operation here
    }
}

Exponential Backoff with Jitter Implementation

To prevent a "thundering herd" effect where multiple throttled clients retry simultaneously, implement exponential backoff with randomized jitter.

public static async Task<T> ExecuteWithJitterRetryAsync<T>(Func<Task<T>> operation, int maxRetries = 5)
{
    var random = new Random();
    for (int attempt = 1; attempt <= maxRetries; attempt++)
    {
        try
        {
            return await operation();
        }
        catch (Exception ex) when (IsTransientError(ex))
        {
            if (attempt == maxRetries) throw;

            // Exponential backoff: 2^attempt seconds
            double delay = Math.Pow(2, attempt);
            // Add jitter: +/- 20% of the delay
            double jitter = (random.NextDouble() * 0.4 - 0.2) * delay;
            var finalDelay = TimeSpan.FromSeconds(Math.Max(0.5, delay + jitter));

            Console.WriteLine($"Transient error encountered. Retrying in {finalDelay.TotalSeconds:F2}s (Attempt {attempt})...");
            await Task.Delay(finalDelay);
        }
    }
    throw new InvalidOperationException("Execution failed after maximum retries.");
}

private static bool IsTransientError(Exception ex)
{
    if (ex is FaultException<OrganizationServiceFault> faultEx)
    {
        var code = faultEx.Detail.ErrorCode;
        return code == -2147015902 || code == -2147015903 || code == -2147015898;
    }
    return false;
}

---

9. Performance Optimisation & Limits

Platform Limits Reference

• Plug-in Timeout: 2 minutes (120 seconds) hard limit.
• SQL Command Timeout: 120 seconds.
• Max URL Length (Web API): 8,192 characters (use $batch with FetchXML in the body to bypass).
• Max Batch Size (ExecuteMultiple): 1,000 requests.

Throughput Optimization Strategies

                                 Throughput Strategy
                                          │
         +--------------------------------+--------------------------------+
         |                                                                 |
         v                                                                 v
+----------------------------------+                             +----------------------------------+
|      Standard Tables             |                             |      Elastic Tables              |
+----------------------------------+                             +----------------------------------+
         │                                                                 │
         ├── Use CreateMultipleRequest                                     ├── Use CreateMultipleRequest
         ├── Use UpdateMultipleRequest                                     ├── Use UpdateMultipleRequest
         └── Max Batch Size: 100-200                                       └── Max Batch Size: 1,000

1. Batching: ExecuteMultiple vs. CreateMultiple / UpdateMultiple

• ExecuteMultipleRequest: A generic container that can mix different operations (Create, Update, Delete) and different tables. It executes sequentially on the server, which reduces network roundtrips but still triggers individual plug-in pipelines for each record.
• CreateMultipleRequest / UpdateMultipleRequest: Highly optimized bulk messages introduced for standard and elastic tables. They pass an EntityCollection of a single table type. The platform processes these records in a single, optimized pipeline execution, drastically reducing CPU overhead and database roundtrips. Always prefer these messages over ExecuteMultiple for single-table bulk operations.

2. Paging with QueryExpression

When retrieving large datasets, never retrieve all records in a single call. Use paging cookies to stream data efficiently.

var query = new QueryExpression("account")
{
    ColumnSet = new ColumnSet("name", "revenue"),
    PageInfo = new PagingInfo
    {
        Count = 5000,
        PageNumber = 1,
        PagingCookie = null
    }
};

while (true)
{
    EntityCollection results = _serviceClient.RetrieveMultiple(query);
    foreach (var entity in results.Entities)
    {
        // Process record
    }

    if (!results.MoreRecords) break;

    query.PageInfo.PageNumber++;
    query.PageInfo.PagingCookie = results.PagingCookie; // CRITICAL: Pass the cookie for performance
}

3. Caching Metadata

Querying table metadata (logical names, option set values) consumes API limits. Cache metadata locally using memory caches (e.g., IMemoryCache) with a sliding expiration of 24 hours.

---

10. ALM & Deployment Checklist

Moving resilient integration components across environments requires strict adherence to Application Lifecycle Management (ALM) best practices.

Step-by-Step Deployment Process

1. Solution Packaging: Package all custom plug-ins, workflows, and Environment Variable definitions into a Managed Solution for downstream environments (UAT, Production).
2. Connection Reference Mapping: Ensure that any cloud flows or custom connectors use Connection References rather than hardcoded connections.
3. Environment Variable Injection: Do not include values for Environment Variables in the exported solution. Instead, inject environment-specific values during the deployment pipeline.
4. Service Principal Configuration: Register the target Application User in the target environment and assign the custom Integration Security Role.

Azure DevOps Pipeline YAML Snippet

This pipeline snippet demonstrates how to automate the deployment of a Dataverse solution and inject environment variable values using the Power Platform Build Tools.

trigger:
- main

pool:
  vmImage: 'windows-latest'

variables:
  SolutionName: 'ContosoIntegration'

steps:
- task: PowerPlatformToolInstaller@2
  displayName: 'Install Power Platform Build Tools'

- task: PowerPlatformExportSolution@2
  displayName: 'Export Solution as Managed'
  inputs:
    authenticationType: 'PowerPlatformSPN'
    PowerPlatformSPN: 'DataverseDevConnection'
    SolutionName: '$(SolutionName)'
    Managed: true
    OutputPath: '$(Build.ArtifactStagingDirectory)/$(SolutionName)_managed.zip'

- task: PowerPlatformImportSolution@2
  displayName: 'Import Solution to Production'
  inputs:
    authenticationType: 'PowerPlatformSPN'
    PowerPlatformSPN: 'DataverseProdConnection'
    SolutionFile: '$(Build.ArtifactStagingDirectory)/$(SolutionName)_managed.zip'
    UseDeploymentSettingsFile: true
    DeploymentSettingsFile: '$(Build.SourcesDirectory)/deploymentSettings.json'

Sample Deployment Settings File (deploymentSettings.json)

This file maps environment variable values during import:

{
  "EnvironmentVariables": [
    {
      "SchemaName": "contoso_DataverseEndpoint",
      "Value": "https://contosoprod.crm.dynamics.com"
    },
    {
      "SchemaName": "contoso_MaxRetryCount",
      "Value": "10"
    }
  ]
}

---

11. Common Pitfalls & Troubleshooting Guide

1. Keeping Server Affinity Enabled

• Symptom: The integration client is throttled (HTTP 429) after only a few thousand requests, despite the environment having multiple web servers.
• Root Cause: The client is reusing the ARRAffinity cookie, routing all traffic to a single web server and exhausting its individual limits.
• Diagnostic Steps: Inspect the response headers of the HTTP requests. If the Set-Cookie header contains ARRAffinity, cookies are enabled.
• Fix: Set EnableHttpCookies = false on the ServiceClient or disable cookies on your HttpClientHandler.

2. Using ExecuteMultiple Inside Plug-ins

• Symptom: Plug-in execution fails with a timeout exception or a platform error.
• Root Cause: Executing batch requests (ExecuteMultipleRequest) inside a plug-in registered on a synchronous stage is an anti-pattern. It nests transactions, causes database locks, and easily exceeds the 2-minute timeout.
• Diagnostic Steps: Check the Plug-in Trace Log for InvalidOperationException referencing batch requests.
• Fix: Refactor the plug-in to perform individual CRUD operations or offload the batch processing to an asynchronous Azure Function.

3. Ignoring the Retry-After Header in Custom Clients

• Symptom: The client continues to bombard the server with requests after receiving a 429 error, leading to progressively longer blockages.
• Root Cause: The client's retry logic uses a static delay instead of parsing the server-provided Retry-After header.
• Diagnostic Steps: Monitor network traffic. If requests are sent immediately after a 429 response, the header is being ignored.
• Fix: Parse response.Headers.RetryAfter and pass its value to Task.Delay().

4. Exceeding the Max Batch Size

• Symptom: The entire batch fails immediately with a fault before any individual request is executed.
• Root Cause: The number of requests in the ExecuteMultipleRequest collection exceeds the platform limit (typically 1,000).
• Diagnostic Steps: Check the exception message for: "The input request collection contains too many requests."
• Fix: Implement dynamic batch resizing. Catch the fault, extract the MaxBatchSize from ErrorDetails, and chunk the collection accordingly.

5. Mixing Read and Write Operations in ExecuteTransaction

• Symptom: High database concurrency latency and frequent SQL timeouts.
• Root Cause: Including Retrieve requests inside an ExecuteTransactionRequest holds shared locks on the database for too long, blocking other transactions.
• Diagnostic Steps: Analyze SQL timeout errors in Application Insights.
• Fix: Remove Retrieve operations from the transaction. Perform reads before initiating the transaction, and only include write operations (Create, Update, Delete) in the transaction block.

6. Parallel Execution Inside Plug-ins

• Symptom: Plug-in fails with a security exception: "Asynchronous operations are not allowed in this context."
• Root Cause: Attempting to use multi-threading (e.g., Parallel.ForEach, Task.WhenAll) within the sandbox isolation environment.
• Diagnostic Steps: Inspect the stack trace for System.Security.SecurityException originating from custom plug-in code.
• Fix: Execute operations sequentially within the plug-in, or move the parallel processing logic to an external Azure Function.

7. Not Handling Partial Failures in ExecuteMultiple

• Symptom: Data is partially imported, but the integration client reports a success status, leading to silent data loss.
• Root Cause: The client calls ExecuteMultiple with ContinueOnError = true but does not inspect the Responses collection for individual faults.
• Diagnostic Steps: Verify that the database record count is lower than the source file count, despite no exceptions being thrown by the client.
• Fix: Iterate through ExecuteMultipleResponse.Responses and log/handle any item where Fault != null.

8. Exceeding the 2-Minute Plug-in Timeout

• Symptom: Long-running operations fail with: "Operation not allowed as plugin execution exceeded maximum allowed time."
• Root Cause: Synchronous plug-ins performing heavy calculations, external API calls, or processing large loops.
• Diagnostic Steps: Check the execution duration in the Plug-in Trace Log. If it is exactly 120,000ms, it is a timeout.
• Fix: Move long-running logic to an asynchronous plug-in step, or offload to an external worker via Azure Service Bus.

9. Hardcoding Connection Strings in Code

• Symptom: Deploying to Production fails because the integration attempts to connect to the Development environment.
• Root Cause: Hardcoded connection strings or API endpoints in the compiled code.
• Diagnostic Steps: Decompile the assembly or inspect the source code for environment-specific URLs.
• Fix: Use Dataverse Environment Variables and retrieve them at runtime using a query on the environmentvariablevalue table.

10. Missing OAuth Token Refresh Logic

• Symptom: Long-running migrations run successfully for exactly 60 minutes, then fail with Unauthorized (401) errors.
• Root Cause: The client acquires an access token at startup but does not refresh it before it expires (typically 1 hour).
• Diagnostic Steps: Check the timestamp of the first failure. If it is exactly 1 hour after startup, token expiration is the cause.
• Fix: Use the SDK's ServiceClient, which handles token refresh automatically, or implement an active token refresh loop in custom HTTP clients.

---

12. Exam Focus: Key Facts & Edge Cases

To prepare for the PL-400 exam, memorize these critical platform limits, behaviors, and API signatures:

• Service Protection Sliding Window: Exactly 300 seconds (5 minutes).
• Request Limit: 6,000 requests per user, per web server, per 300-second window.
• Execution Time Limit: 1,200 seconds (20 minutes) of server processing time per 300-second window.
• Concurrency Limit: 52 concurrent requests per user, per web server.
• Web API Throttling Status Code: Always returns HTTP 429 Too Many Requests.
• SDK Throttling Exception: Throws a FaultException<OrganizationServiceFault> with specific error codes:
  • 0x80072322 (Request Count Exceeded)
  • 0x80072321 (Execution Time Exceeded)
  • 0x80072326 (Concurrency Limit Exceeded)
• Retry-After Location (SDK): Found in OrganizationServiceFault.ErrorDetails with the key "Retry-After" as a TimeSpan.
• Retry-After Location (Web API): Found in the HTTP response headers with the key "Retry-After" as an integer representing seconds.
• Server Affinity Bypass: Disabling cookies (setting EnableHttpCookies = false on ServiceClient or UseCookies = false on HttpClientHandler) is the only way to distribute requests across multiple web servers to maximize throughput.
• ExecuteMultiple Recursion: Strictly prohibited. An ExecuteMultipleRequest cannot contain another ExecuteMultipleRequest.
• ExecuteTransaction Recursion: An ExecuteMultipleRequest can contain ExecuteTransactionRequest instances, but an ExecuteTransactionRequest cannot contain ExecuteMultipleRequest or ExecuteTransactionRequest instances.
• Max Batch Size: Default is 1,000 requests for ExecuteMultipleRequest. Exceeding this throws a fault before execution begins, returning the limit in the fault details.
• Plug-in Sandbox Timeout: Hard limit of 2 minutes (120 seconds). This cannot be changed or bypassed.
• Plug-in Depth Limit: Hard limit of 8 nested executions within a 1-hour window. Exceeding this aborts the transaction to prevent infinite loops.
• CreateMultiple vs ExecuteMultiple: CreateMultipleRequest and UpdateMultipleRequest are optimized bulk operations for a single table type. They are significantly faster than ExecuteMultipleRequest because they execute in a single pipeline execution. Use them whenever possible.