Convey.Tracing.Jaeger

Distributed tracing integration with Jaeger providing comprehensive request tracing, performance monitoring, and debugging capabilities for microservices architectures.

Installation

dotnet add package Convey.Tracing.Jaeger

Overview

Convey.Tracing.Jaeger provides:

• Distributed tracing - End-to-end request tracing across services
• Jaeger integration - Native integration with Jaeger tracing system
• OpenTelemetry support - Built on OpenTelemetry standards
• Automatic instrumentation - HTTP, database, and framework instrumentation
• Custom spans - Create custom spans for business operations
• Performance insights - Detailed timing and dependency analysis
• Error tracking - Trace error propagation across services

Configuration

Basic Setup

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddConvey()
    .AddJaeger(); // Enables Jaeger tracing

var app = builder.Build();

app.Run();

Advanced Configuration

Configure in appsettings.json:

{
  "jaeger": {
    "enabled": true,
    "serviceName": "my-service",
    "udpHost": "localhost",
    "udpPort": 6831,
    "maxPacketSize": 0,
    "sampler": "const",
    "maxTracesPerSecond": 5,
    "samplingRate": 1.0,
    "excludePaths": ["/health", "/metrics"],
    "tags": {
      "version": "1.0.0",
      "environment": "production"
    }
  }
}

Full Configuration

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddConvey()
    .AddJaeger(jaegerOptions =>
    {
        jaegerOptions.ServiceName = "order-service";
        jaegerOptions.UdpHost = "jaeger-agent";
        jaegerOptions.UdpPort = 6831;
        jaegerOptions.SamplingRate = 1.0; // Sample all traces in development
        jaegerOptions.ExcludePaths = new[] { "/health", "/metrics", "/swagger" };
    });

var app = builder.Build();

// Enable tracing middleware
app.UseTracing(); // Automatically traces HTTP requests

app.Run();

Key Features

1. Automatic HTTP Tracing

HTTP requests are automatically traced:

// HTTP requests are automatically instrumented when UseTracing() is configured
app.UseTracing(); // Tracks all HTTP requests with spans

// Automatic spans created for:
// - Incoming HTTP requests
// - Outgoing HTTP requests
// - Database operations
// - Message broker operations

2. Custom Spans

Create custom spans for business operations:

public class OrderService
{
    private readonly ITracer _tracer;
    private readonly IOrderRepository _repository;

    public OrderService(ITracer tracer, IOrderRepository repository)
    {
        _tracer = tracer;
        _repository = repository;
    }

    public async Task<OrderResult> ProcessOrderAsync(ProcessOrderCommand command)
    {
        using var span = _tracer.StartActiveSpan("order.process")
            .SetTag("order.id", command.OrderId.ToString())
            .SetTag("order.type", command.Type)
            .SetTag("customer.id", command.CustomerId.ToString());

        try
        {
            // Validate order
            using var validationSpan = _tracer.StartActiveSpan("order.validate");
            await ValidateOrderAsync(command);
            validationSpan.SetTag("validation.result", "success");

            // Process payment
            using var paymentSpan = _tracer.StartActiveSpan("order.payment")
                .SetTag("payment.amount", command.Amount)
                .SetTag("payment.method", command.PaymentMethod);

            var paymentResult = await ProcessPaymentAsync(command);
            paymentSpan.SetTag("payment.transaction_id", paymentResult.TransactionId);

            // Update inventory
            using var inventorySpan = _tracer.StartActiveSpan("order.inventory");
            await UpdateInventoryAsync(command);

            // Save order
            using var saveSpan = _tracer.StartActiveSpan("order.save");
            var order = await _repository.SaveAsync(command);

            span.SetTag("order.result", "success")
                .SetTag("order.total_amount", command.Amount);

            return new OrderResult { OrderId = order.Id, Success = true };
        }
        catch (ValidationException ex)
        {
            span.SetTag("error", true)
                .SetTag("error.type", "validation")
                .SetTag("error.message", ex.Message);
            throw;
        }
        catch (PaymentException ex)
        {
            span.SetTag("error", true)
                .SetTag("error.type", "payment")
                .SetTag("error.message", ex.Message);
            throw;
        }
        catch (Exception ex)
        {
            span.SetTag("error", true)
                .SetTag("error.type", "unknown")
                .SetTag("error.message", ex.Message);
            throw;
        }
    }
}

3. Tracing Decorators

Automatically trace handlers with decorators:

// Command handler tracing decorator
public class TracingCommandHandlerDecorator<TCommand> : ICommandHandler<TCommand>
    where TCommand : class, ICommand
{
    private readonly ICommandHandler<TCommand> _handler;
    private readonly ITracer _tracer;

    public TracingCommandHandlerDecorator(ICommandHandler<TCommand> handler, ITracer tracer)
    {
        _handler = handler;
        _tracer = tracer;
    }

    public async Task HandleAsync(TCommand command, CancellationToken cancellationToken = default)
    {
        var commandName = typeof(TCommand).Name;

        using var span = _tracer.StartActiveSpan($"command.{commandName}")
            .SetTag("command.type", commandName)
            .SetTag("component", "command-handler");

        // Add command-specific tags
        if (command is IIdentifiable identifiable)
        {
            span.SetTag("command.id", identifiable.Id.ToString());
        }

        try
        {
            await _handler.HandleAsync(command, cancellationToken);
            span.SetTag("command.result", "success");
        }
        catch (ValidationException ex)
        {
            span.SetTag("error", true)
                .SetTag("error.type", "validation")
                .SetTag("error.message", ex.Message)
                .SetTag("command.result", "validation_error");
            throw;
        }
        catch (Exception ex)
        {
            span.SetTag("error", true)
                .SetTag("error.type", ex.GetType().Name)
                .SetTag("error.message", ex.Message)
                .SetTag("command.result", "error");
            throw;
        }
    }
}

// Query handler tracing decorator
public class TracingQueryHandlerDecorator<TQuery, TResult> : IQueryHandler<TQuery, TResult>
    where TQuery : class, IQuery<TResult>
{
    private readonly IQueryHandler<TQuery, TResult> _handler;
    private readonly ITracer _tracer;

    public TracingQueryHandlerDecorator(IQueryHandler<TQuery, TResult> handler, ITracer tracer)
    {
        _handler = handler;
        _tracer = tracer;
    }

    public async Task<TResult> HandleAsync(TQuery query, CancellationToken cancellationToken = default)
    {
        var queryName = typeof(TQuery).Name;

        using var span = _tracer.StartActiveSpan($"query.{queryName}")
            .SetTag("query.type", queryName)
            .SetTag("component", "query-handler");

        // Add query-specific tags
        if (query is IPagedQuery pagedQuery)
        {
            span.SetTag("query.page", pagedQuery.Page)
                .SetTag("query.page_size", pagedQuery.PageSize);
        }

        try
        {
            var result = await _handler.HandleAsync(query, cancellationToken);

            span.SetTag("query.result", "success");

            // Add result information
            if (result is IPagedResult pagedResult)
            {
                span.SetTag("result.total_items", pagedResult.TotalItems)
                    .SetTag("result.total_pages", pagedResult.TotalPages);
            }

            return result;
        }
        catch (Exception ex)
        {
            span.SetTag("error", true)
                .SetTag("error.type", ex.GetType().Name)
                .SetTag("error.message", ex.Message)
                .SetTag("query.result", "error");
            throw;
        }
    }
}

// Registration
builder.Services.TryDecorate(typeof(ICommandHandler<>), typeof(TracingCommandHandlerDecorator<>));
builder.Services.TryDecorate(typeof(IQueryHandler<,>), typeof(TracingQueryHandlerDecorator<,>));

4. Database Operation Tracing

Trace database operations:

public class TracingUserRepository : IUserRepository
{
    private readonly IUserRepository _repository;
    private readonly ITracer _tracer;

    public TracingUserRepository(IUserRepository repository, ITracer tracer)
    {
        _repository = repository;
        _tracer = tracer;
    }

    public async Task<User> GetByIdAsync(Guid id)
    {
        using var span = _tracer.StartActiveSpan("db.query.users.get_by_id")
            .SetTag("db.type", "mongodb")
            .SetTag("db.collection", "users")
            .SetTag("db.operation", "find_one")
            .SetTag("user.id", id.ToString());

        try
        {
            var user = await _repository.GetByIdAsync(id);

            span.SetTag("db.result", user != null ? "found" : "not_found");

            return user;
        }
        catch (Exception ex)
        {
            span.SetTag("error", true)
                .SetTag("error.type", ex.GetType().Name)
                .SetTag("error.message", ex.Message);
            throw;
        }
    }

    public async Task<PagedResult<User>> BrowseAsync(BrowseUsersQuery query)
    {
        using var span = _tracer.StartActiveSpan("db.query.users.browse")
            .SetTag("db.type", "mongodb")
            .SetTag("db.collection", "users")
            .SetTag("db.operation", "find")
            .SetTag("query.page", query.Page)
            .SetTag("query.page_size", query.PageSize);

        try
        {
            var result = await _repository.BrowseAsync(query);

            span.SetTag("db.result_count", result.Items.Count())
                .SetTag("db.total_count", result.TotalItems);

            return result;
        }
        catch (Exception ex)
        {
            span.SetTag("error", true)
                .SetTag("error.type", ex.GetType().Name)
                .SetTag("error.message", ex.Message);
            throw;
        }
    }

    public async Task AddAsync(User user)
    {
        using var span = _tracer.StartActiveSpan("db.command.users.insert")
            .SetTag("db.type", "mongodb")
            .SetTag("db.collection", "users")
            .SetTag("db.operation", "insert_one")
            .SetTag("user.id", user.Id.ToString());

        try
        {
            await _repository.AddAsync(user);
            span.SetTag("db.result", "success");
        }
        catch (Exception ex)
        {
            span.SetTag("error", true)
                .SetTag("error.type", ex.GetType().Name)
                .SetTag("error.message", ex.Message);
            throw;
        }
    }
}

5. External Service Tracing

Trace calls to external services:

public class TracingPaymentService : IPaymentService
{
    private readonly IPaymentService _paymentService;
    private readonly ITracer _tracer;
    private readonly HttpClient _httpClient;

    public TracingPaymentService(IPaymentService paymentService, ITracer tracer, HttpClient httpClient)
    {
        _paymentService = paymentService;
        _tracer = tracer;
        _httpClient = httpClient;
    }

    public async Task<PaymentResult> ProcessPaymentAsync(PaymentRequest request)
    {
        using var span = _tracer.StartActiveSpan("external.payment.process")
            .SetTag("service.name", "payment-service")
            .SetTag("service.type", "external")
            .SetTag("payment.amount", request.Amount)
            .SetTag("payment.currency", request.Currency)
            .SetTag("payment.method", request.PaymentMethod);

        try
        {
            // Inject trace context into HTTP headers
            var headers = new Dictionary<string, string>();
            _tracer.Inject(span.Context, Format.HttpHeaders, new TextMapInjectAdapter(headers));

            foreach (var header in headers)
            {
                _httpClient.DefaultRequestHeaders.Add(header.Key, header.Value);
            }

            var result = await _paymentService.ProcessPaymentAsync(request);

            span.SetTag("payment.transaction_id", result.TransactionId)
                .SetTag("payment.status", result.Status)
                .SetTag("external.result", "success");

            return result;
        }
        catch (PaymentException ex)
        {
            span.SetTag("error", true)
                .SetTag("error.type", "payment_error")
                .SetTag("error.code", ex.ErrorCode)
                .SetTag("error.message", ex.Message);
            throw;
        }
        catch (HttpRequestException ex)
        {
            span.SetTag("error", true)
                .SetTag("error.type", "http_error")
                .SetTag("error.message", ex.Message);
            throw;
        }
        catch (Exception ex)
        {
            span.SetTag("error", true)
                .SetTag("error.type", ex.GetType().Name)
                .SetTag("error.message", ex.Message);
            throw;
        }
    }
}

// HTTP client tracing
public class TracingHttpMessageHandler : DelegatingHandler
{
    private readonly ITracer _tracer;

    public TracingHttpMessageHandler(ITracer tracer)
    {
        _tracer = tracer;
    }

    protected override async Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request, CancellationToken cancellationToken)
    {
        using var span = _tracer.StartActiveSpan("http.client.request")
            .SetTag("http.method", request.Method.ToString())
            .SetTag("http.url", request.RequestUri?.ToString())
            .SetTag("component", "http-client");

        // Inject trace context
        var headers = new Dictionary<string, string>();
        _tracer.Inject(span.Context, Format.HttpHeaders, new TextMapInjectAdapter(headers));

        foreach (var header in headers)
        {
            request.Headers.Add(header.Key, header.Value);
        }

        try
        {
            var response = await base.SendAsync(request, cancellationToken);

            span.SetTag("http.status_code", (int)response.StatusCode)
                .SetTag("http.response_size", response.Content.Headers.ContentLength ?? 0);

            if (!response.IsSuccessStatusCode)
            {
                span.SetTag("error", true);
            }

            return response;
        }
        catch (Exception ex)
        {
            span.SetTag("error", true)
                .SetTag("error.type", ex.GetType().Name)
                .SetTag("error.message", ex.Message);
            throw;
        }
    }
}

// Registration
builder.Services.AddHttpClient<IPaymentService, PaymentService>()
    .AddHttpMessageHandler<TracingHttpMessageHandler>();

Advanced Features

1. Correlation ID Propagation

Propagate correlation IDs across services:

public class CorrelationIdMiddleware
{
    private readonly RequestDelegate _next;
    private readonly ITracer _tracer;

    public CorrelationIdMiddleware(RequestDelegate next, ITracer tracer)
    {
        _next = next;
        _tracer = tracer;
    }

    public async Task InvokeAsync(HttpContext context)
    {
        var correlationId = context.Request.Headers["X-Correlation-ID"].FirstOrDefault()
                           ?? Guid.NewGuid().ToString();

        context.Items["CorrelationId"] = correlationId;
        context.Response.Headers["X-Correlation-ID"] = correlationId;

        // Add correlation ID to current span
        var activeSpan = _tracer.ActiveSpan;
        if (activeSpan != null)
        {
            activeSpan.SetTag("correlation.id", correlationId);
        }

        await _next(context);
    }
}

// Service that uses correlation ID
public class NotificationService
{
    private readonly ITracer _tracer;
    private readonly IHttpContextAccessor _httpContextAccessor;

    public async Task SendNotificationAsync(SendNotificationCommand command)
    {
        var correlationId = _httpContextAccessor.HttpContext?.Items["CorrelationId"]?.ToString();

        using var span = _tracer.StartActiveSpan("notification.send")
            .SetTag("correlation.id", correlationId)
            .SetTag("notification.type", command.Type)
            .SetTag("recipient.id", command.RecipientId);

        // Implementation...
    }
}

2. Baggage Propagation

Use baggage to propagate additional context:

public class BaggageMiddleware
{
    private readonly RequestDelegate _next;
    private readonly ITracer _tracer;

    public BaggageMiddleware(RequestDelegate next, ITracer tracer)
    {
        _next = next;
        _tracer = tracer;
    }

    public async Task InvokeAsync(HttpContext context)
    {
        var activeSpan = _tracer.ActiveSpan;
        if (activeSpan != null)
        {
            // Set baggage from headers or context
            var userId = context.User.FindFirst(ClaimTypes.NameIdentifier)?.Value;
            if (!string.IsNullOrEmpty(userId))
            {
                activeSpan.SetBaggageItem("user.id", userId);
            }

            var tenantId = context.Request.Headers["X-Tenant-ID"].FirstOrDefault();
            if (!string.IsNullOrEmpty(tenantId))
            {
                activeSpan.SetBaggageItem("tenant.id", tenantId);
            }
        }

        await _next(context);
    }
}

// Service that uses baggage
public class AuditService
{
    private readonly ITracer _tracer;

    public async Task LogActionAsync(string action, object data)
    {
        var activeSpan = _tracer.ActiveSpan;
        var userId = activeSpan?.GetBaggageItem("user.id");
        var tenantId = activeSpan?.GetBaggageItem("tenant.id");

        using var span = _tracer.StartActiveSpan("audit.log")
            .SetTag("audit.action", action)
            .SetTag("audit.user_id", userId)
            .SetTag("audit.tenant_id", tenantId);

        // Log audit event with context
        await LogAuditEvent(action, data, userId, tenantId);
    }
}

3. Sampling Strategies

Implement custom sampling strategies:

public class CustomSampler : ISampler
{
    private readonly double _defaultSamplingRate;
    private readonly Dictionary<string, double> _operationSamplingRates;

    public CustomSampler(double defaultSamplingRate, Dictionary<string, double> operationSamplingRates = null)
    {
        _defaultSamplingRate = defaultSamplingRate;
        _operationSamplingRates = operationSamplingRates ?? new Dictionary<string, double>();
    }

    public SamplingResult Sample(string operationName, TraceId traceId)
    {
        var samplingRate = _operationSamplingRates.GetValueOrDefault(operationName, _defaultSamplingRate);

        // Special sampling rules
        if (operationName.Contains("health") || operationName.Contains("metrics"))
        {
            return SamplingResult.Create(false); // Don't sample health/metrics endpoints
        }

        if (operationName.Contains("error") || operationName.Contains("exception"))
        {
            return SamplingResult.Create(true); // Always sample errors
        }

        // Use trace ID for deterministic sampling
        var traceIdBytes = traceId.ToByteArray();
        var hash = BitConverter.ToUInt64(traceIdBytes, 0);
        var shouldSample = (hash % 10000) < (samplingRate * 10000);

        return SamplingResult.Create(shouldSample);
    }
}

// Registration
builder.Services.AddJaeger(options =>
{
    options.Sampler = new CustomSampler(0.1, new Dictionary<string, double>
    {
        ["order.process"] = 1.0,    // Sample all order processing
        ["payment.process"] = 1.0,  // Sample all payment processing
        ["user.login"] = 0.5,       // Sample 50% of logins
        ["search.query"] = 0.01     // Sample 1% of searches
    });
});

Integration Patterns

1. Message Broker Integration

Trace message processing:

public class TracingEventHandler<TEvent> : IEventHandler<TEvent>
    where TEvent : class, IEvent
{
    private readonly IEventHandler<TEvent> _handler;
    private readonly ITracer _tracer;

    public TracingEventHandler(IEventHandler<TEvent> handler, ITracer tracer)
    {
        _handler = handler;
        _tracer = tracer;
    }

    public async Task HandleAsync(TEvent @event, CancellationToken cancellationToken = default)
    {
        var eventName = typeof(TEvent).Name;

        // Extract trace context from event metadata
        ISpanContext parentContext = null;
        if (@event is ITraceable traceable && traceable.TraceContext != null)
        {
            parentContext = ExtractTraceContext(traceable.TraceContext);
        }

        var spanBuilder = _tracer.BuildSpan($"event.{eventName}")
            .WithTag("event.type", eventName)
            .WithTag("component", "event-handler");

        if (parentContext != null)
        {
            spanBuilder = spanBuilder.AsChildOf(parentContext);
        }

        using var span = spanBuilder.Start();

        try
        {
            await _handler.HandleAsync(@event, cancellationToken);
            span.SetTag("event.result", "success");
        }
        catch (Exception ex)
        {
            span.SetTag("error", true)
                .SetTag("error.type", ex.GetType().Name)
                .SetTag("error.message", ex.Message);
            throw;
        }
    }

    private ISpanContext ExtractTraceContext(Dictionary<string, string> traceContext)
    {
        try
        {
            return _tracer.Extract(Format.TextMap, new TextMapExtractAdapter(traceContext));
        }
        catch
        {
            return null;
        }
    }
}

// Event with trace context
public interface ITraceable
{
    Dictionary<string, string> TraceContext { get; set; }
}

public class OrderCreatedEvent : IEvent, ITraceable
{
    public Guid OrderId { get; set; }
    public Guid CustomerId { get; set; }
    public decimal Amount { get; set; }
    public Dictionary<string, string> TraceContext { get; set; }
}

// Publish event with trace context
public class EventPublisher
{
    private readonly ITracer _tracer;
    private readonly IEventDispatcher _dispatcher;

    public async Task PublishAsync<TEvent>(TEvent @event) where TEvent : class, IEvent
    {
        if (@event is ITraceable traceable)
        {
            var traceContext = new Dictionary<string, string>();
            _tracer.Inject(_tracer.ActiveSpan.Context, Format.TextMap, new TextMapInjectAdapter(traceContext));
            traceable.TraceContext = traceContext;
        }

        await _dispatcher.PublishAsync(@event);
    }
}

Configuration Options

Jaeger Settings

public class JaegerOptions
{
    public bool Enabled { get; set; } = true;
    public string ServiceName { get; set; }
    public string UdpHost { get; set; } = "localhost";
    public int UdpPort { get; set; } = 6831;
    public int MaxPacketSize { get; set; } = 0;
    public string Sampler { get; set; } = "const";
    public double SamplingRate { get; set; } = 1.0;
    public int MaxTracesPerSecond { get; set; } = 5;
    public string[] ExcludePaths { get; set; } = Array.Empty<string>();
    public Dictionary<string, object> Tags { get; set; } = new();
}

API Reference

ITracer Interface

public interface ITracer
{
    ISpan ActiveSpan { get; }
    ISpanBuilder BuildSpan(string operationName);
    ISpan StartActiveSpan(string operationName);
    void Inject<TCarrier>(ISpanContext spanContext, IFormat<TCarrier> format, TCarrier carrier);
    ISpanContext Extract<TCarrier>(IFormat<TCarrier> format, TCarrier carrier);
}

ISpan Interface

public interface ISpan : IDisposable
{
    ISpanContext Context { get; }
    ISpan SetTag(string key, string value);
    ISpan SetTag(string key, bool value);
    ISpan SetTag(string key, int value);
    ISpan SetTag(string key, double value);
    ISpan SetBaggageItem(string key, string value);
    string GetBaggageItem(string key);
    ISpan Log(string message);
    ISpan Log(Dictionary<string, object> fields);
    void Finish();
}

Extension Methods

public static class ConveyExtensions
{
    public static IConveyBuilder AddJaeger(this IConveyBuilder builder, string sectionName = "jaeger");
    public static IConveyBuilder AddJaeger(this IConveyBuilder builder, Action<JaegerOptions> configure);
    public static IApplicationBuilder UseTracing(this IApplicationBuilder app);
}

Best Practices

1. Use meaningful span names - Use clear, hierarchical naming conventions
2. Add relevant tags - Include business context and technical details
3. Handle errors properly - Mark spans with error tags and log exceptions
4. Use appropriate sampling - Balance observability with performance
5. Propagate context - Ensure trace context flows across service boundaries
6. Monitor trace performance - Keep tracing overhead minimal
7. Use child spans - Break down complex operations into smaller spans
8. Include business metrics - Add business-relevant tags and logs

Troubleshooting

Common Issues

1. Traces not appearing in Jaeger
   • Check Jaeger agent connectivity
   • Verify service name configuration
   • Ensure sampling is not too restrictive
2. Missing trace context
   • Verify context propagation in HTTP headers
   • Check async/await patterns for context flow
   • Ensure proper decorator registration
3. Performance impact
   • Adjust sampling rates for high-volume endpoints
   • Use asynchronous reporting
   • Monitor memory usage of trace buffers
4. Incomplete traces
   • Check for exceptions that terminate spans early
   • Verify proper span disposal with using statements
   • Ensure child spans are properly linked to parents