Service Bus Dead-Letter Queue (DLQ)

Handling Failed Messages and Implementing Retry Patterns

Introduction

The Dead-Letter Queue (DLQ) is a critical component of reliable message processing in Azure Service Bus. When messages fail to be processed successfully after multiple attempts, they are automatically moved to a dead-letter queue rather than being lost. This allows you to inspect failed messages, understand why they failed, and implement appropriate recovery strategies.

This guide covers:

DLQ fundamentals — How dead-lettering works
Configuration — Setting up DLQ for queues and subscriptions
Processing — Reading and handling dead-lettered messages
Analysis — Understanding why messages end up in DLQ
Recovery patterns — Fixing and reprocessing failed messages

How Messages Become Dead-Lettered

Triggers for Dead-Lettering

┌─────────────────────────────────────────────────────────────────────┐
│                    MESSAGE DEAD-LETTERING FLOW                      │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│   ┌──────────────┐                                                  │
│   │   Message    │                                                  │
│   │  Received    │                                                  │
│   └──────┬───────┘                                                  │
│          │                                                          │
│          ▼                                                          │
│   ┌──────────────────────────────────────┐                          │
│   │         Processing                   │                          │
│   │         attempts (1 to MaxDelivery)  │                          │
│   └──────────┬───────────────────────────┘                          │
│              │ Success?                                             │
│       ┌──────┴──────┐                                               │
│       │YES         NO│                                              │
│       ▼             ▼                                               │
│   ┌─────────┐   ┌─────────────────────────────────────────────┐     │
│   │Complete │   │ Delivery Attempts < MaxDeliveryCount?       │     │
│   │message  │   └───────────────┬─────────────────────────────┘     │
│   └─────────┘          │YES/NO                                      │
│                  ┌──────┴──────┐                                    │
│                  │             │                                    │
│                  ▼            ▼                                     │
│          ┌───────────┐   ┌──────────────┐                           │
│          │  Retry    │   │  Dead-Letter │                           │
│          │  message  │   │  (move to    │                           │
│          │           │   │   DLQ)       │                           │
│          └───────────┘   └──────────────┘                           │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

Three Ways Messages Enter DLQ

Exceeded MaxDeliveryCount — Message failed processing MaxDeliveryCount times
Message Expired (TTL) — Message Time-To-Live expired before processing
Explicit Dead-Lettering — Application explicitly calls DeadLetter()

Configure Dead-Letter Queue

Queue Configuration

// Create queue with DLQ settings
var queueOptions = new CreateQueueOptions("orders-queue")
{
    // Maximum delivery attempts before dead-lettering
    MaxDeliveryCount = 5,
    
    // Enable dead-lettering when message expires
    DeadLetterOnMessageExpiration = true,
    
    // Enable dead-lettering on max delivery exceeded
    DeadLetterOnMaxDeliveriesExceeded = true,
    
    // How long dead-lettered messages persist
    LockDuration = TimeSpan.FromMinutes(1),
    
    // Default message TTL
    DefaultMessageTimeToLive = TimeSpan.FromDays(1)
};

await namespaceClient.CreateQueueAsync(queueOptions);

Topic Subscription Configuration

// Configure DLQ for topic subscription
var subscriptionOptions = new CreateSubscriptionOptions
{
    TopicName = "orders-topic",
    SubscriptionName = "email-notifications",
    
    MaxDeliveryCount = 3,
    DeadLetterOnMessageExpiration = true,
    DeadLetterOnMaxDeliveriesExceeded = true
};

await topicClient.CreateSubscriptionAsync(subscriptionOptions);

Azure Portal Configuration

Queue Settings:
├── Enable dead-letter queue: ON
├── Max delivery attempts: 5
├── Message TTL: 1 day
├── Enable expiration: ON
└── Lock duration: 1 minute

CLI Configuration

# Create queue with DLQ
az servicebus queue create \
  --name orders-queue \
  --namespace-name mynamespace \
  --resource-group myrg \
  --enable-dead-lettering-on-message-expiration true \
  --enable-dead-lettering-on-max-deliveries true \
  --max-delivery-count 5 \
  --default-message-time-to-live "P1D"

Process Dead-Letter Queue

Read from DLQ

// Create processor for dead-letter queue
var deadLetterProcessor = client.CreateProcessor(
    "orders-queue", 
    "$DeadLetterQueue"
);

deadLetterProcessor.ProcessMessageAsync += async args =>
{
    var message = args.Message;
    
    // Get dead-letter reason
    var deadLetterReason = message.ApplicationProperties.TryGetValue(
        "DeadLetterReason", out var reason) ? reason : "Unknown";
    
    var deadLetterErrorDescription = message.ApplicationProperties.TryGetValue(
        "DeadLetterErrorDescription", out var errorDesc) ? errorDesc : "None";
    
    Console.WriteLine($"Dead-lettered message:");
    Console.WriteLine($"  Reason: {deadLetterReason}");
    Console.WriteLine($"  Description: {deadLetterErrorDescription}");
    Console.WriteLine($"  Body: {message.Body}");
    
    // Process the dead-lettered message
    await ProcessDeadLetterAsync(message);
    
    // Complete to remove from DLQ
    await args.CompleteMessageAsync(message);
};

await deadLetterProcessor.StartProcessingAsync();

Peek into DLQ (Without Locking)

// Peek messages without locking
var receiver = client.CreateReceiver("orders-queue", "$DeadLetterQueue");

// Peek first 10 messages
var messages = await receiver.PeekMessagesAsync(10);

foreach (var message in messages)
{
    Console.WriteLine($"Message ID: {message.MessageId}");
    Console.WriteLine($"Enqueued: {message.EnqueuedTime}");
    Console.WriteLine($"Body: {message.Body}");
}

Understanding Dead-Letter Reasons

System Properties

When a message is dead-lettered, Service Bus adds specific properties:

Property	Description
`DeadLetterReason`	System code or custom reason
`DeadLetterErrorDescription`	Detailed error information

Common Reasons

Reason	Description	Resolution
`MaxDeliveryCountExceeded`	Failed too many times	Check processing logic, increase max deliveries
`MessageExpired`	TTL expired before processing	Process messages faster, increase TTL
`MaxMessageSizeExceeded`	Message too large	Reduce message size, use external storage
`SenderDrivenDeadLetter`	Explicitly dead-lettered	Check application logic

Custom Dead-Lettering

// Explicitly dead-letter a message
await sender.DeadLetterAsync(message, "ValidationFailed", "Order data incomplete");

// Dead-letter with custom properties
var properties = new Dictionary<string, object>
{
    ["FailedField"] = "CustomerEmail",
    ["ExpectedFormat"] = "user@domain.com"
};
await sender.DeadLetterAsync(message, "InvalidFormat", "Email format invalid", properties);

Recovery and Reprocessing Patterns

Pattern 1: Retry After Fix

public async Task ReprocessFromDlqAsync(ServiceBusReceivedMessage deadLetter)
{
    // Analyze and fix the issue
    var originalBody = deadLetter.Body.ToString();
    var fixedBody = FixOrderData(originalBody);
    
    // Create new message
    var newMessage = new ServiceBusMessage(fixedBody)
    {
        MessageId = deadLetter.MessageId,
        ContentType = deadLetter.ContentType,
        Properties = deadLetter.ApplicationProperties
    };
    
    // Track retry
    var retryCount = (int)(deadLetter.ApplicationProperties.GetValueOrDefault("RetryCount", 0));
    newMessage.Properties["RetryCount"] = retryCount + 1;
    newMessage.Properties["OriginalDeadLetterReason"] = 
        deadLetter.ApplicationProperties.GetValueOrDefault("DeadLetterReason");
    
    // Send back to main queue
    var sender = client.CreateSender("orders-queue");
    await sender.SendMessageAsync(newMessage);
}

Pattern 2: Route to Error Queue

public async Task MoveToErrorQueueAsync(ServiceBusReceivedMessage deadLetter)
{
    // Move to a separate error queue for later analysis
    var errorQueueSender = client.CreateSender("orders-error-queue");
    
    var errorMessage = new ServiceBusMessage(deadLetter.Body)
    {
        ContentType = deadLetter.ContentType,
        MessageId = deadLetter.MessageId
    };
    
    // Add error context
    errorMessage.Properties["OriginalQueue"] = "orders-queue";
    errorMessage.Properties["DeadLetterReason"] = 
        deadLetter.ApplicationProperties.GetValueOrDefault("DeadLetterReason");
    errorMessage.Properties["DeadLetterTime"] = DateTime.UtcNow;
    errorMessage.Properties["OriginalProperties"] = 
        JsonSerializer.Serialize(deadLetter.ApplicationProperties);
    
    await errorQueueSender.SendMessageAsync(errorMessage);
    
    // Complete the DLQ message
    await deadLetter.CompleteAsync();
}

Pattern 3: Batch Reprocessing

public async Task ReprocessAllFailedOrdersAsync()
{
    var deadLetterReceiver = client.CreateReceiver("orders-queue", "$DeadLetterQueue");
    var mainQueueSender = client.CreateSender("orders-queue");
    
    // Peek all messages
    var failedMessages = new List<ServiceBusMessage>();
    ServiceBusReceivedMessage[] messages;
    
    do
    {
        messages = await deadLetterReceiver.PeekMessagesAsync(100);
        
        foreach (var msg in messages)
        {
            // Only reprocess certain types of failures
            var reason = msg.ApplicationProperties.GetValueOrDefault("DeadLetterReason", "");
            
            if (reason == "TransientError")
            {
                var reprocessed = new ServiceBusMessage(msg.Body)
                {
                    MessageId = msg.MessageId,
                    Properties =
                    {
                        ["Reprocessed"] = true,
                        ["OriginalFailure"] = reason
                    }
                };
                failedMessages.Add(reprocessed);
            }
        }
    } while (messages.Length > 0);
    
    // Send all to main queue
    await mainQueueSender.SendMessagesAsync(failedMessages);
}

Monitoring DLQ

Metrics to Track

# Get DLQ message count
az servicebus queue show \
  --name orders-queue \
  --namespace-name mynamespace \
  --resource-group myrg \
  --query "deadLetterMessageCount"

# Get subscription DLQ count
az servicebus topic subscription show \
  --name email-notifications \
  --topic-name orders-topic \
  --namespace-name mynamespace \
  --resource-group myrg \
  --query "deadLetterMessageCount"

Alert Configuration

{
  "alert": {
    "type": "metric",
    "condition": "deadLetterMessageCount > 10",
    "severity": "warning",
    "action": "Send email to team"
  }
}

Best Practices

Practice	Description
Monitor DLQ size	Alert when messages accumulate
Set appropriate max delivery	3-5 attempts is typical
Include context	Add custom properties when dead-lettering
Implement reprocessing	Fix issues and replay messages
Separate error queues	Use different queues for different failure types
Log failures	Track why messages fail for debugging
Regular cleanup	Process DLQ messages regularly

Processing Recommendations

// Recommended: Process DLQ with higher priority
var dlqOptions = new ProcessorOptions
{
    MaxConcurrentCalls = 1,  // Process one at a time
    PrefetchCount = 10
};

var processor = client.CreateProcessor(
    "orders-queue",
    "$DeadLetterQueue",
    dlqOptions);