Retry Policies and Resilient Message Processing

Transient vs Permanent Failures

┌──────────────┐     ┌─────────────────┐     ┌──────────────────┐
│   Message    │     │   Processor     │     │  Downstream      │
│   Queue      │────▶│   (Peek-Lock)   │────▶│  Service         │
└──────────────┘     └────────┬────────┘     └──────────────────┘
                              │
                    ┌─────────┼─────────┐
                    │ Success │ Transient│  Permanent
                    ▼         ▼         ▼
              Complete()  Abandon()  DeadLetter()
                         (retry)    (no retry)

Prerequisites

• Azure Service Bus Standard or Premium namespace
• .NET 8 SDK
• Azure CLI installed

Step 1: Understand the Peek-Lock Mechanism

RESOURCE_GROUP="rg-servicebus-tutorials"
NAMESPACE="sb-tutorials-ns"
QUEUE_NAME="orders-resilient-queue"

# Create queue with specific retry settings
az servicebus queue create \
  --resource-group $RESOURCE_GROUP \
  --namespace-name $NAMESPACE \
  --name $QUEUE_NAME \
  --lock-duration PT1M \
  --max-delivery-count 5 \
  --dead-lettering-on-message-expiration true \
  --default-message-time-to-live P7D

Step 2: Implement Retry with Exponential Backoff

using Azure.Messaging.ServiceBus;

var connectionString = "<YOUR_CONNECTION_STRING>";
var queueName = "orders-resilient-queue";

await using var client = new ServiceBusClient(connectionString, new ServiceBusClientOptions
{
    RetryOptions = new ServiceBusRetryOptions
    {
        Mode = ServiceBusRetryMode.Exponential,
        MaxRetries = 3,
        Delay = TimeSpan.FromSeconds(1),
        MaxDelay = TimeSpan.FromSeconds(30),
        TryTimeout = TimeSpan.FromSeconds(60)
    }
});

await using var processor = client.CreateProcessor(queueName, new ServiceBusProcessorOptions
{
    MaxConcurrentCalls = 10,
    AutoCompleteMessages = false,
    PrefetchCount = 0
});

processor.ProcessMessageAsync += async args =>
{
    var body = args.Message.Body.ToString();
    int deliveryCount = args.Message.DeliveryCount;

    Console.WriteLine($"Processing (attempt {deliveryCount}): {body}");

    try
    {
        await ProcessOrderAsync(body);
        await args.CompleteMessageAsync(args.Message);
    }
    catch (TransientException ex)
    {
        Console.WriteLine($"Transient error: {ex.Message}. Abandoning for retry.");
        await args.AbandonMessageAsync(args.Message, new Dictionary<string, object>
        {
            { "LastError", ex.Message },
            { "RetryAt", DateTime.UtcNow.AddSeconds(Math.Pow(2, deliveryCount)) }
        });
    }
    catch (PermanentException ex)
    {
        Console.WriteLine($"Permanent error: {ex.Message}. Dead-lettering.");
        await args.DeadLetterMessageAsync(args.Message,
            deadLetterReason: "PermanentFailure",
            deadLetterErrorDescription: ex.Message);
    }
};

processor.ProcessErrorAsync += args =>
{
    Console.WriteLine($"Processor error: {args.Exception.Message}");
    return Task.CompletedTask;
};

await processor.StartProcessingAsync();

Step 3: Circuit Breaker Pattern

public class CircuitBreaker
{
    private int _failureCount;
    private DateTime _lastFailure;
    private readonly int _threshold;
    private readonly TimeSpan _resetTimeout;

    public CircuitBreakerState State { get; private set; } = CircuitBreakerState.Closed;

    public CircuitBreaker(int threshold = 5, int resetTimeoutSeconds = 60)
    {
        _threshold = threshold;
        _resetTimeout = TimeSpan.FromSeconds(resetTimeoutSeconds);
    }

    public bool AllowRequest()
    {
        if (State == CircuitBreakerState.Closed) return true;
        if (State == CircuitBreakerState.Open &&
            DateTime.UtcNow - _lastFailure > _resetTimeout)
        {
            State = CircuitBreakerState.HalfOpen;
            return true;
        }
        return State == CircuitBreakerState.HalfOpen;
    }

    public void RecordSuccess()
    {
        _failureCount = 0;
        State = CircuitBreakerState.Closed;
    }

    public void RecordFailure()
    {
        _failureCount++;
        _lastFailure = DateTime.UtcNow;
        if (_failureCount >= _threshold)
            State = CircuitBreakerState.Open;
    }
}

public enum CircuitBreakerState { Closed, Open, HalfOpen }

┌─────────────────────────────────────────────────┐
│            Circuit Breaker States                 │
│                                                   │
│  ┌────────┐  failures >= threshold  ┌────────┐  │
│  │ CLOSED │ ───────────────────────▶│  OPEN  │  │
│  │(normal)│                         │(reject)│  │
│  └────┬───┘                         └───┬────┘  │
│       ▲                                  │       │
│       │ success            timeout       │       │
│       │                    expires       ▼       │
│       │                          ┌───────────┐  │
│       └──────────────────────────│ HALF-OPEN │  │
│                                  │  (probe)  │  │
│                                  └───────────┘  │
└─────────────────────────────────────────────────┘

Step 4: Poison Message Handling

When a message exceeds MaxDeliveryCount, Service Bus automatically moves it to the dead-letter queue:

var dlqPath = $"{queueName}/$deadletterqueue";
await using var dlqReceiver = client.CreateReceiver(dlqPath);

while (true)
{
    var message = await dlqReceiver.ReceiveMessageAsync(TimeSpan.FromSeconds(5));
    if (message == null) break;

    Console.WriteLine($"Poison message: {message.Body}");
    Console.WriteLine($"  Reason: {message.DeadLetterReason}");
    Console.WriteLine($"  Error: {message.DeadLetterErrorDescription}");
    Console.WriteLine($"  Delivery count: {message.DeliveryCount}");

    // Log, alert team, or move to parking lot
    await dlqReceiver.CompleteMessageAsync(message);
}

Step 5: Auto-Forwarding to Retry Queue

Create a delayed retry pattern using a secondary queue:

# Create retry queue that auto-forwards expired messages back to main queue
az servicebus queue create \
  --resource-group $RESOURCE_GROUP \
  --namespace-name $NAMESPACE \
  --name "${QUEUE_NAME}-retry" \
  --default-message-time-to-live PT30S \
  --forward-dead-lettered-messages-to $QUEUE_NAME \
  --dead-lettering-on-message-expiration true

┌──────────────┐     ┌──────────────┐     ┌──────────────────┐
│  Main Queue  │     │  Retry Queue │     │  Dead Letter      │
│              │     │  (TTL: 30s)  │     │  Queue            │
└──────┬───────┘     └──────┬───────┘     └──────────────────┘
       │                     │                      ▲
       │  Send to retry      │  TTL expires         │ Max delivery
       │  on transient fail  │  → DLQ → forward     │ exceeded
       │─────────────────────▶  back to main        │
       │◀────────────────────────────────────────────│

// Forward to retry queue with scheduled delay
catch (TransientException ex) when (deliveryCount < 3)
{
    var retryMessage = new ServiceBusMessage(args.Message.Body)
    {
        ScheduledEnqueueTime = DateTimeOffset.UtcNow.AddSeconds(
            Math.Pow(2, deliveryCount) * 10)
    };
    foreach (var prop in args.Message.ApplicationProperties)
        retryMessage.ApplicationProperties.Add(prop);
    retryMessage.ApplicationProperties["RetryCount"] = deliveryCount;

    await sender.SendMessageAsync(retryMessage);
    await args.CompleteMessageAsync(args.Message);
}

Step 6: Monitor Failed Deliveries

# Check dead-letter queue message count
az servicebus queue show \
  --resource-group $RESOURCE_GROUP \
  --namespace-name $NAMESPACE \
  --name $QUEUE_NAME \
  --query "countDetails.deadLetterMessageCount" -o tsv

# Set up alert for DLQ messages
az monitor metrics alert create \
  --resource-group $RESOURCE_GROUP \
  --name "dlq-alert-${QUEUE_NAME}" \
  --scopes $(az servicebus namespace show \
    --resource-group $RESOURCE_GROUP \
    --name $NAMESPACE --query id -o tsv) \
  --condition "total DeadletteredMessages > 10" \
  --window-size PT5M \
  --evaluation-frequency PT1M \
  --description "Dead-letter messages exceeding threshold"

Retry Strategy Decision Table

Key Takeaways

• Distinguish transient from permanent failures — retry only what can succeed
• Use Abandon() for immediate retry, scheduled messages for delayed retry
• Circuit breakers prevent cascading failures to downstream services
• Monitor dead-letter queues and alert on thresholds
• Set MaxDeliveryCount based on your retry budget

← Previous: Managed Identity | Next: Premium Networking →