.NET Microservices — Distributed Tracing with App Insights

Operation IDs, Dependency Tracking, Correlation

Introduction

In a microservices architecture, a single user request can flow through multiple services, making debugging and monitoring challenging. Distributed tracing provides end-to-end visibility by correlating requests across service boundaries.

Azure Application Insights provides:

• Automatic correlation — Track requests across services
• Dependency tracking — See external calls (SQL, HTTP, Service Bus)
• Application Map — Visual representation of service topology
• Performance analysis — Identify bottlenecks
• Failure diagnostics — Understand exception propagation

How Distributed Tracing Works

┌─────────────────────────────────────────────────────────────────────┐
│                    Request Flow with Tracing                        │
│                                                                     │
│   ┌────────────┐                                                    │
│   │  API GW    │── TraceId: abc-123 ── ParentId: -                  │
│   └──────┬─────┘                                                    │
│          │                                                          │
│          ▼                                                          │
│   ┌────────────┐    ┌────────────┐    ┌────────────┐                │
│   │ Order Svc  │──▶ │ Payment    │──▶ │ Inventory  │                │
│   │ abc-123.1  │    │ abc-123.2  │    │ abc-123.3  │                │
│   └────────────┘    └────────────┘    └────────────┘                │
│        │                  │                  │                      │
│        │                  │                  │                      │
│        ▼                  ▼                  ▼                      │
│   ┌───────────────────────────────────────────────────────────┐     │
│   │              Application Insights                         │     │
│   │  • Trace: abc-123                                         │     │
│   │  • Operation: GET /orders/123                             │     │
│   │  • Duration: 2.5s                                         │     │
│   │  • Success: true                                          │     │
│   │  • Dependencies: 3 calls                                  │     │
│   └───────────────────────────────────────────────────────────┘     │
└─────────────────────────────────────────────────────────────────────┘

Setup Application Insights

Install Package

dotnet add package Microsoft.ApplicationInsights.AspNetCore
dotnet add package Microsoft.ApplicationInsights.DependencyCollector

Program.cs Configuration

// Add services
builder.Services.AddApplicationInsightsTelemetry(options =>
{
    options.ConnectionString = builder.Configuration["APPINSIGHTS_CONNECTION_STRING"];
    options.EnableDependencyTrackingTelemetryModule = true;
    options.EnableServiceBusLifetimeTracking = true;
    options.EnableEventHubLifetimeTracking = true;
    options.EnableAzureSDKDiagnostics = true;
});

// Configure sampling
builder.Services.Configure<TelemetryConfiguration>(config =>
{
    config.DefaultSamplingSettings = new SamplingSettings
    {
        MaxTelemetryItemsPerSecond = 50,
        InitialSamplingPercentage = 100,
        SamplingPercentage = 100
    };
});

appsettings.json

{
  "ApplicationInsights": {
    "ConnectionString": "InstrumentationKey=xxx",
    "EnableAdaptiveSampling": false,
    "EnablePerformanceCounterCollectionModule": true,
    "EnableEventCounterCollectionModule": true
  }
}

Manual Tracing in Code

Using Activity (Recommended for .NET 5+)

public class OrderService
{
    private readonly ActivitySource _activitySource;
    private readonly ILogger<OrderService> _logger;
    
    public OrderService(ILogger<OrderService> logger)
    {
        _logger = logger;
        // Use Microsoft.Extensions.ActivitySource or create custom
        _activitySource = new ActivitySource("OrderService");
    }
    
    public async Task<Order> CreateOrderAsync(CreateOrderRequest request)
    {
        // Start activity
        using var activity = _activitySource.StartActivity(
            "OrderService.CreateOrder",
            ActivityKind.Server,
            parentContext: null,
            tags: new TagList
            {
                ["order.customerId"] = request.CustomerId,
                ["order.itemCount"] = request.Items.Count
            });
        
        try
        {
            _logger.LogInformation("Creating order for customer {CustomerId}", 
                request.CustomerId);
            
            activity?.SetTag("order.total", request.Items.Sum(i => i.Quantity * i.UnitPrice));
            
            // Call payment service
            using var paymentActivity = _activitySource.StartActivity(
                "PaymentService.ProcessPayment",
                ActivityKind.Client);
            
            var paymentResult = await _paymentClient.ProcessAsync(request.PaymentInfo);
            
            paymentActivity?.SetTag("payment.transactionId", paymentResult.TransactionId);
            
            if (!paymentResult.Success)
            {
                activity?.SetStatus(ActivityStatusCode.Error, "Payment failed");
                throw new PaymentException("Payment failed");
            }
            
            // Call inventory
            using var inventoryActivity = _activitySource.StartActivity(
                "InventoryService.Reserve",
                ActivityKind.Client);
            
            await _inventoryClient.ReserveItemsAsync(request.Items);
            
            // Create order
            var order = await _orderRepository.CreateAsync(request);
            
            activity?.SetTag("order.id", order.Id.ToString());
            
            _logger.LogInformation("Order {OrderId} created successfully", order.Id);
            
            return order;
        }
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            activity?.SetTag("error.message", ex.Message);
            
            throw;
        }
    }
}

Using TelemetryClient (Legacy)

public class OrderService
{
    private readonly TelemetryClient _telemetry;
    
    public OrderService(TelemetryClient telemetry)
    {
        _telemetry = telemetry;
    }
    
    public async Task<Order> CreateOrderAsync(CreateOrderRequest request)
    {
        // Track dependency
        using var operation = _telemetry.StartOperation<RequestTelemetry>("CreateOrder");
        
        operation.Telemetry.Properties["CustomerId"] = request.CustomerId;
        
        try
        {
            // Track event
            _telemetry.TrackEvent("OrderCreated", new Dictionary<string, string>
            {
                ["CustomerId"] = request.CustomerId,
                ["Total"] = request.Items.Sum(i => i.Quantity * i.UnitPrice).ToString()
            });
            
            // Process order
            var order = await ProcessOrderAsync(request);
            
            operation.Telemetry.ResponseCode = "200";
            operation.Telemetry.Success = true;
            
            return order;
        }
        catch (Exception ex)
        {
            operation.Telemetry.ResponseCode = "500";
            operation.Telemetry.Success = false;
            
            // Track exception
            _telemetry.TrackException(ex, new Dictionary<string, string>
            {
                ["CustomerId"] = request.CustomerId
            });
            
            throw;
        }
    }
}

Tracing Service Bus

Sending Messages

public class OrderPublisher
{
    private readonly ServiceBusClient _client;
    private readonly TelemetryClient _telemetry;
    
    public async Task PublishOrderCreatedAsync(Order order)
    {
        var startTime = DateTime.UtcNow;
        
        try
        {
            var sender = _client.CreateSender("orders");
            var message = new ServiceBusMessage(JsonSerializer.Serialize(order))
            {
                Subject = "OrderCreated",
                MessageId = order.Id.ToString()
            };
            
            await sender.SendMessageAsync(message);
            
            // Track successful dependency
            _telemetry.TrackDependency("ServiceBus", "orders", "Send", startTime, 
                DateTime.UtcNow - startTime, true);
        }
        catch (Exception ex)
        {
            _telemetry.TrackDependency("ServiceBus", "orders", "Send", startTime,
                DateTime.UtcNow - startTime, false);
            
            throw;
        }
    }
}

Processing Messages

public class OrderMessageProcessor
{
    [FunctionName("OrderProcessor")]
    public async Task Run(
        [ServiceBusTrigger("orders", Connection = "ServiceBusConnection")] 
        ServiceBusReceivedMessage message,
        ILogger log)
    {
        // Extract correlation ID from message
        var correlationId = message.CorrelationId ?? message.MessageId;
        
        using var operation = log.BeginScope(new Dictionary<string, object>
        {
            ["CorrelationId"] = correlationId,
            ["MessageId"] = message.MessageId
        });
        
        log.LogInformation("Processing order message");
        
        try
        {
            var order = JsonSerializer.Deserialize<Order>(Encoding.UTF8.GetString(message.Body));
            
            // Process
            await ProcessOrderAsync(order);
            
            log.LogInformation("Order processed successfully");
        }
        catch (Exception ex)
        {
            log.LogError(ex, "Failed to process order");
            throw;
        }
    }
}

Correlation Across Services

HTTP Correlation

public class ExternalApiClient
{
    private readonly HttpClient _httpClient;
    private readonly TelemetryClient _telemetry;
    
    public async Task<string> GetCustomerAsync(string customerId, string correlationId)
    {
        var request = new HttpRequestMessage(HttpMethod.Get, 
            $"https://api.customers.com/customers/{customerId}");
        
        // Add correlation headers
        request.Headers.Add("X-Correlation-Id", correlationId);
        request.Headers.Add("Request-Id", Guid.NewGuid().ToString());
        
        var startTime = DateTime.UtcNow;
        
        var response = await _httpClient.SendAsync(request);
        
        _telemetry.TrackDependency(
            "Http",
            "api.customers.com",
            "GET /customers/" + customerId,
            startTime,
            DateTime.UtcNow - startTime,
            response.IsSuccessStatusCode);
        
        response.EnsureSuccessStatusCode();
        
        return await response.Content.ReadAsStringAsync();
    }
}

KQL Queries for Analysis

Request Flow

// Trace a specific request
requests
| where operation_Id == "abc12345"
| order by timestamp asc
| extend durationMs = duration.TotalMilliseconds
| project timestamp, name, durationMs, success, cloud_RoleName

// Join with dependencies
requests
| where operation_Id == "abc12345"
| join kind=inner (dependencies) on operation_Id
| project timestamp, name, target, success, duration

Performance Analysis

// Slowest operations
requests
| where timestamp > ago(1h)
| summarize avg(duration), count() by name
| order by avg_duration desc
| take 10

Error Analysis

// Failed requests by service
requests
| where success == false
| summarize count() by cloud_RoleName
| order by count_ desc

Dependency Performance

// Dependency timing
dependencies
| where timestamp > ago(1h)
| summarize avg(duration), percentiles(duration, 50, 95) 
    by target, type
| order by avg_duration desc

Application Map

Configuration

The Application Map automatically shows:

• Service connections
• Number of requests between services
• Failure rates
• Performance data

Custom Roles

// Set cloud role name per service
builder.Services.AddApplicationInsightsTelemetry(options =>
{
    options.CloudRoleName = "order-service";
});

// Override in code
TelemetryConfiguration.Active
    .Context
    .Cloud
    .RoleName = "order-service";

Best Practices

Azure Integration Hub - Advanced Level