Troubleshooting¶

Doc Version: 1.0.0 Last Updated: 2026-06-08 Git Commit: b9fa976b Author: Lincoln

This document provides diagnosis and solutions for common issues with the JAiRouter distributed tracing feature.

Tracing Data Issues¶

1. Missing Tracing Data¶

Symptoms: - No traceId and spanId in logs - No tracing data in monitoring panels - Exporter not receiving tracing information

Diagnosis Steps:

🔑 Admin credentials are required**: the /actuator/** calls below go through the normal authentication chain — only

/actuator/health (including the liveness / readiness sub-paths), /actuator/info and /actuator/prometheus are anonymous (for k8s probes and Prometheus scraping); everything else needs the ADMIN role. Without credentials you get 401 AUTH_MISSING. (Before the fix even a valid ADMIN JWT got 401: the auth chain was excluded while hasRole("ADMIN") was still required, so the two cancelled out — see issue #340.)

Fetch a JWT and call the endpoint (verified locally):

TOKEN=$(curl -s -X POST http://localhost:8080/api/auth/jwt/login \
  -H 'Content-Type: application/json' \
  -d '{"username":"admin","password":"<your admin password>"}' | jq -r .data.token)
curl -s -H "Jairouter_Token: $TOKEN" http://localhost:8080/actuator/metrics | jq

# 1. Check if tracing is enabled
curl http://localhost:8080/actuator/health/tracing

# 2. Check configuration
curl http://localhost:8080/actuator/configprops | jq '.jairouter.tracing'

# 3. Check sampling rate
curl http://localhost:8080/actuator/metrics/jairouter.tracing.sampling.rate

Common Causes and Solutions:

2. Partial Data Loss¶

Symptoms: - Only some requests have tracing data - Child Spans missing - Async operations have no tracing information

Solutions:

# Temporarily increase sampling rate for debugging
jairouter:
  tracing:
    sampling:
      ratio: 1.0

# Enable debug logs (checked in #159: tracing has no logging.level field, use Spring's own level)
logging:
  level:
    org.unreal.modelrouter.monitor.tracing: DEBUG

Performance Issues¶

1. Tracing Causing Performance Degradation¶

Symptoms: - Significantly increased response time - Rising CPU usage - High memory usage

Performance Analysis:

# View tracing-related metrics
curl -s http://localhost:8080/actuator/metrics | grep tracing

# Check GC situation
curl -s http://localhost:8080/actuator/metrics/jvm.gc.pause

# View thread usage
curl -s http://localhost:8080/actuator/metrics/jvm.threads.live

Optimization Measures:

jairouter:
  tracing:
    # Reduce sampling rate
    sampling:
      ratio: 0.1

    # Enable async processing
    # Checked in issues #159 / #224: there is no async section (the switch is
    # performance.async-processing, the pool lives under performance.thread-pool); the batch
    # processor knobs also live under performance, not under open-telemetry. Merged into one
    # performance mapping below.
    performance:
      async-processing: true
      thread-pool:
        core-size: 4
      # Optimize batch processing
      batch:
        size: 512
        timeout: 5s

2. Memory Leak¶

Symptoms: - Continuously growing heap memory - OutOfMemoryError occurs - Frequent GC but memory not released

Troubleshooting Steps:

# 1. Check Span count
curl http://localhost:8080/actuator/metrics/jairouter.tracing.spans.active

# 2. Check memory usage
jmap -histo <pid> | grep Span

# 3. Generate heap dump
jcmd <pid> GC.run_finalization
jcmd <pid> VM.gc

Solutions:

jairouter:
  tracing:
    # Checked in issue #159: the real path is performance.memory, and cleanup-interval /
    # span-ttl do not exist (memory pressure is driven by memory-limit-mb).
    performance:
      memory:
        max-spans-in-memory: 5000    # Limit Span count
        memory-limit-mb: 80          # Tighten the limit to evict sooner
        gc-interval: 15s             # Check more often

Configuration Issues¶

1. Configuration Not Taking Effect¶

Symptoms: - No changes after modifying configuration - Configuration validation fails - Configuration errors at startup

Check Configuration Syntax:

# Validate YAML syntax
python -c "import yaml; yaml.safe_load(open('application.yml'))"

# Check configuration binding
curl http://localhost:8080/actuator/configprops | jq '.jairouter.tracing'

Common Configuration Errors:

# ❌ Wrong configuration
jairouter:
  tracing:
    sampling:
      ratio: 1.5                   # Out of range [0.0, 1.0]
    exporter:
      otlp:
        endpoint: "localhost:4317"   # Missing protocol

The correct form of the same configuration (the two halves are split into separate blocks so each one passes YAML syntax validation):

# ✅ Correct configuration  
jairouter:
  tracing:
    sampling:
      ratio: 1.0
    exporter:
      otlp:
        endpoint: "http://localhost:4317"

2. Dynamic Configuration Update Failure¶

Diagnostic Methods:

# Check configuration service status
curl http://localhost:8080/actuator/health/config

# View configuration history
curl http://localhost:8080/api/admin/config/history

Exporter Issues¶

1. Jaeger Connection Failure¶

Error Message:

Failed to export spans to Jaeger: Connection refused

Resolution Steps:

# 1. Check Jaeger service status
curl http://localhost:14268/api/traces

# 2. Verify network connection
telnet localhost 14268

# 3. Check firewall settings
netstat -an | grep 14268

Configuration Adjustment:

jairouter:
  tracing:
    exporter:
      type: "jaeger"
      jaeger:
        endpoint: "http://jaeger:14268/api/traces"  # Use service name
        timeout: 30s                                # Increase timeout
        # Checked in issue #159: there is no retry-enabled knob
        # (JaegerConfig only has endpoint / timeout / headers).

2. OTLP Export Errors¶

Common Errors:

Context Propagation Issues¶

1. Context Loss in Reactive Streams¶

Issue Manifestation: - No traceId in async operations - Child Span creation fails - MDC information missing

Solutions:

// ✅ Correct reactive context propagation
return Mono.just(data)
    .flatMap(this::processAsync)
    .contextWrite(Context.of("tracing", TracingContextHolder.getCurrentContext()));

// ❌ Wrong usage - no context propagation
return Mono.just(data)
    .flatMap(this::processAsync);

2. Context Loss in Thread Pools¶

Configure Thread Pool Context Propagation:

@Bean
public TaskExecutor tracingTaskExecutor() {
    ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
    executor.setCorePoolSize(4);
    executor.setTaskDecorator(new TracingTaskDecorator());
    return executor;
}

Debugging Tips¶

1. Enable Debug Logs¶

logging:
  level:
    org.unreal.modelrouter.tracing: DEBUG
    io.opentelemetry: DEBUG

2. Use Debug Endpoints¶

# View current active Spans
curl http://localhost:8080/actuator/tracing/active-spans

# View tracing statistics
curl http://localhost:8080/actuator/tracing/stats

# Force export all Spans
curl -X POST http://localhost:8080/actuator/tracing/flush

3. Local Testing Tools¶

# Use curl to test tracing
curl -H "X-Trace-Debug: true" http://localhost:8080/api/v1/chat/completions

# Check tracing information in response headers
curl -I http://localhost:8080/health

Monitoring Alerts¶

1. Key Metrics Monitoring¶

# Prometheus alert rules
groups:
  - name: tracing_alerts
    rules:
      - alert: TracingExportFailure
        expr: rate(jairouter_tracing_export_errors_total[5m]) > 0.1
        labels:
          severity: warning

      - alert: TracingMemoryHigh
        expr: jairouter_tracing_memory_used_ratio > 0.8
        labels:
          severity: critical

2. Health Checks¶

# Set up tracing health check
curl http://localhost:8080/actuator/health/tracing

# Expected response
{
  "status": "UP",
  "details": {
    "exporter": "healthy",
    "sampling": "active",
    "memory": "normal"
  }
}

Common Error Codes¶

The tracing module emits no coded error identifiers: no TRACING_* code exists anywhere in the repository. Diagnose tracing failures with the logs and the /actuator endpoints described above; for client-visible error codes see the error code reference.

Getting Support¶

If you encounter unresolved issues:

1. View Logs: Enable DEBUG level logs for detailed information
2. Check Configuration: Use actuator endpoints to validate configuration
3. Performance Analysis: Use JVM tools to analyze performance issues
4. Community Support: Submit issue reports in GitHub Issues

Next Steps¶

• Performance Tuning - Optimize tracing system performance
• Operations Guide - Production environment operations best practices
• Configuration Reference - View detailed configuration options