Skip to content

Monitoring API

Doc Version: 1.0.2
Last Updated: 2026-05-21
Git Commit: 61384b4a
Author: Lincoln

JAiRouter provides comprehensive monitoring APIs for health checks, metrics collection, and system status monitoring.

Overview

The Monitoring API includes:

  • Health Checks - Service and instance health status
  • Metrics - Performance and usage statistics
  • System Status - Overall system health and configuration

All monitoring endpoints are available under the /actuator/* path and provide real-time insights into your JAiRouter deployment.

Health Check Endpoints

System Health

Get overall system health status:

GET /actuator/health

Response:

{
  "status": "UP",
  "components": {
    "diskSpace": {
      "status": "UP",
      "details": {
        "total": 499963174912,
        "free": 91943821312,
        "threshold": 10485760,
        "exists": true
      }
    },
    "ping": {
      "status": "UP"
    },
    "tracing": {
      "status": "UP"
    },
    "sharedState": {
      "status": "UP"
    },
    "rbacEndpointCoverage": {
      "status": "UP",
      "details": {
        "missing": 0
      }
    }
  }
}

⚠️ Component names are the registered health indicators: tracing (TracingHealthIndicator.java:28), sharedState (SharedStateHealthIndicator.java:31), rbacEndpointCoverage (RbacEndpointCoverageHealthIndicator.java:27) and jwtRedisHealthIndicator (JwtRedisHealthMonitor.java:21), plus Spring Boot's standard diskSpace etc. There is no modelRouter component; the activeInstances / totalInstances / circuitBreakerStatus structure does not exist in the code.

Detailed Health Information

Get detailed health information including all components:

GET /actuator/health

⚠️ There is no /actuator/health/detailed endpoint: only the liveness and readiness groups are defined (config/base/monitoring-base.yml:70-84). To see every component, call GET /actuator/health under the dev profile (show-details: always).

Response:

{
  "status": "UP",
  "components": {
    "diskSpace": {
      "status": "UP",
      "details": {
        "total": 499963174912,
        "free": 91943821312,
        "threshold": 10485760,
        "exists": true
      }
    }
  }
}

⚠️ The original document also listed a modelRouter component here (with services.*.instances[] detail) — no such component exists in the code, so it has been removed; see the note at the top of this section for the real health components.

Which Endpoint Should Each Probe Use

/actuator/health is the aggregate health including downstream components (db, jwtRedis, ...). It is suitable for humans and alerting, but a poor fit for orchestration probes: a dependency blip will take the replica out of rotation even though the replica could still serve traffic.

ConsumerEndpointSemantics
Process liveness/actuator/health/livenessContains only livenessState and ping; UP while the process is alive
Accepting traffic (readiness)/actuator/health/readinessContains only readinessState; excludes db/redis by default
Container self-check (Docker HEALTHCHECK)/actuator/health/livenessAll five Dockerfile definitions in this repo are configured this way
Alerting and troubleshooting/actuator/healthDOWN as soon as any component is DOWN — which is the signal you want
External load balancer (self-managed)/actuator/health/readinessIf you want dependency failures to drain traffic, use aggregate health instead and accept the false drain

The Kubernetes manifests (deploy/k8s/base/deployment.yaml) already do the right thing: startupProbe / livenessProbe hit /actuator/health/liveness, readinessProbe hits /actuator/health/readiness.

Why readiness does not include dependencies

readiness includes only readinessState (config/base/monitoring-base.yml). Spring Boot validates health group include entries strictly: naming a contributor that is not present in the current context makes the application fail to start. Whether db or redis is assembled depends on configuration (for example REDIS_ENABLED), so it cannot be listed statically in YAML.

The trade-off: when Redis or the database is unavailable, the replica stays in the ready set and keeps receiving traffic. For deployments where a dependency failure must drain traffic, aggregate it at the orchestration layer (an extra probe against aggregate health, accepting the false drain), or enable jairouter.health.readiness.include-dependencies (available from 3.8.7, defaults to false). When on, /actuator/health/readiness evaluates db and redis through the readinessDependencies aggregator, so a dependency failure leaves the ready set — at the cost of a dependency blip potentially keeping new replicas out of the ready set during a rolling update.

That aggregator is always assembled, so listing it in the readiness include does not trip the strict-existence check above. Internally it only evaluates db / redis that are actually assembled; absent dependencies are skipped rather than reported DOWN.

Health probe settings

Both switches are readable and editable on Platform Settings → Health Probe Settings (/system/health) or over the API. Both are hot-editable and need no restart.

KeyDefaultEffect
jairouter.health.readiness.include-dependenciesfalseWhether readiness aggregates db / redis
jairouter.security.jwt.redis-health.fail-healthtrueWhether jwtRedis turns aggregate health DOWN when Redis is unreachable
# Read current values
curl -s -H "Authorization: Bearer <token>" http://localhost:8080/api/config/health

# Change one of them (partial update)
curl -s -X PUT -H "Authorization: Bearer <token>" -H "Content-Type: application/json" \
  -d '{"jwtRedisFailHealth": false}' http://localhost:8080/api/config/health

Impact of the jwtRedis component in multi-replica setups

Once both jairouter.security.jwt.persistence.redis.enabled and jairouter.security.jwt.blacklist.redis.enabled are enabled (required for multi-replica), jwtRedisHealthIndicator performs real Redis round-trips; when Redis is unreachable it reports DOWN and aggregate /actuator/health returns 503.

From 3.8.7, jairouter.security.jwt.redis-health.fail-health=false downgrades it to UP (with the impact noted in the details: rate limiting and quota fail open, blacklist may be stale), so a Redis blip no longer drains healthy replicas.

Metrics Endpoints

Application Metrics

Get Prometheus-format metrics:

GET /actuator/prometheus

Response:

# HELP jvm_memory_used_bytes The amount of used memory
# TYPE jvm_memory_used_bytes gauge
jvm_memory_used_bytes{area="heap",id="PS Eden Space",} 2.38026752E8
jvm_memory_used_bytes{area="heap",id="PS Survivor Space",} 1048576.0
jvm_memory_used_bytes{area="heap",id="PS Old Gen",} 4.2991616E7

# HELP jairouter_requests_total Total number of requests
# TYPE jairouter_requests_total counter
jairouter_requests_total{service="chat",method="POST",status="200",} 1247.0
jairouter_requests_total{service="chat",method="POST",status="500",} 23.0
jairouter_requests_total{service="chat",method="POST",status="200",} 1156.0
jairouter_requests_total{service="chat",method="POST",status="500",} 18.0

# HELP jairouter_request_duration_seconds Request duration in seconds
# TYPE jairouter_request_duration_seconds histogram
jairouter_request_duration_seconds_bucket{service="chat",method="POST",le="0.1",} 234.0
jairouter_request_duration_seconds_bucket{service="chat",method="POST",le="0.5",} 892.0
jairouter_request_duration_seconds_bucket{service="chat",method="POST",le="1.0",} 1156.0
jairouter_request_duration_seconds_bucket{service="chat",method="POST",le="+Inf",} 1270.0

# HELP jairouter_circuit_breaker_state Circuit breaker state (0=CLOSED, 1=OPEN, 2=HALF_OPEN)
# TYPE jairouter_circuit_breaker_state gauge
jairouter_circuit_breaker_state{service="chat",method="POST",} 0.0
jairouter_circuit_breaker_state{service="chat",method="POST",} 0.0

# HELP jairouter_rate_limit_remaining Rate limit remaining requests
# TYPE jairouter_rate_limit_remaining gauge
jairouter_rate_limit_remaining{service="chat",client_ip="192.168.1.100",} 45.0
jairouter_rate_limit_remaining{service="chat",client_ip="192.168.1.101",} 38.0

Metrics Summary

Get human-readable metrics summary:

🔑 Admin credentials are required**: the /actuator/** calls below go through the normal authentication chain — only

/actuator/health (including the liveness / readiness sub-paths), /actuator/info and /actuator/prometheus are anonymous (for k8s probes and Prometheus scraping); everything else needs the ADMIN role. Without credentials you get 401 AUTH_MISSING. (Before the fix even a valid ADMIN JWT got 401: the auth chain was excluded while hasRole("ADMIN") was still required, so the two cancelled out — see issue #340.)

Fetch a JWT and call the endpoint (verified locally):

TOKEN=$(curl -s -X POST http://localhost:8080/api/auth/jwt/login \
  -H 'Content-Type: application/json' \
  -d '{"username":"admin","password":"<your admin password>"}' | jq -r .data.token)
curl -s -H "X-Jairouter-Token: $TOKEN" http://localhost:8080/actuator/metrics | jq
GET /actuator/metrics

Response:

{
  "names": [
    "jvm.memory.used",
    "jvm.memory.max", 
    "jvm.gc.pause",
    "http.server.requests",
    "jairouter_requests_total",
    "jairouter_request_duration_seconds",
    "jairouter_circuit_breaker_state",
    "jairouter_rate_limit_remaining",
    "jairouter_loadbalancer_selections_total"
  ]
}

Specific Metric Details

Get details for a specific metric:

GET /actuator/metrics/jairouter_requests_total

Response:

{
  "name": "jairouter_requests_total",
  "description": "Total number of requests processed by model router",
  "baseUnit": null,
  "measurements": [
    {
      "statistic": "COUNT",
      "value": 2647.0
    }
  ],
  "availableTags": [
    {
      "tag": "service",
      "values": ["chat", "embedding", "rerank", "tts", "stt", "imgGen", "imgEdit"]
    },
    {
      "tag": "method", 
      "values": ["POST", "GET"]
    },
    {
      "tag": "status",
      "values": ["200", "400", "500"]
    }
  ]
}

System Information

Application Info

Get application information:

GET /actuator/info

Response:

{
  "app": {
    "name": "JAiRouter",
    "version": "3.2.2",
    "description": "AI Model Service Router and Load Balancer"
  },
  "build": {
    "version": "3.2.2",
    "artifact": "model-router",
    "name": "model-router",
    "group": "org.unreal",
    "time": "2025-08-19T08:15:30.123Z"
  },
  "git": {
    "branch": "main",
    "commit": {
      "id": "3418d3f6",
      "time": "2025-08-19T08:00:00Z"
    }
  },
  "java": {
    "version": "17.0.8",
    "vendor": "Eclipse Adoptium"
  }
}

⚠️ The build / git sections require the build-info goal in pom.xml (not configured today), so the default /actuator/info does not return them; the output above is illustrative. The current project version is 3.2.2.

Environment Information

Get environment and configuration details:

GET /actuator/env

⚠️ env is not in the default exposure list (config/base/monitoring-base.yml:60 exposes only health / info / metrics / prometheus / jairouter-metrics / error-tracking; see application-prod.yml:40 for prod). Add env to management.endpoints.web.exposure.include first if you need it.

Response:

{
  "activeProfiles": ["default"],
  "propertySources": [
    {
      "name": "server.ports",
      "properties": {
        "local.server.port": {
          "value": 8080
        }
      }
    },
    {
      "name": "applicationConfig: [classpath:/application.yml]",
      "properties": {
        "model-router.load-balancer.default-strategy": {
          "value": "ROUND_ROBIN"
        },
        "model-router.rate-limit.default-algorithm": {
          "value": "TOKEN_BUCKET"
        },
        "model-router.circuit-breaker.failure-threshold": {
          "value": 5
        }
      }
    }
  ]
}

⚠️ The output above is illustrative: every configuration key in this project lives under the jairouter.* prefix — there are no model-router.* keys (no match anywhere under src/main/resources).

Custom Monitoring Endpoints

The gateway exposes the following custom monitoring endpoints:

System Health Status

Get the overall system health status:

GET /api/monitoring/health

Monitoring Configuration

Get the current monitoring configuration:

GET /api/monitoring/config

Hot-edit the basic configuration (writes only touch memory and fall back to the configuration files on restart; each successful change records one MONITORING_CONFIG_CHANGED audit event):

PUT /api/monitoring/config/enabled
PUT /api/monitoring/config/prefix
PUT /api/monitoring/config/collection-interval
PUT /api/monitoring/config/categories
PUT /api/monitoring/config/sampling
PUT /api/monitoring/config/performance

The six sampling rates take values in 0~1 (requests / backend / infrastructure / trace / trace processing / trace analysis); performance requires batchSize >= 1 and bufferSize >= batchSize, otherwise the endpoint returns 400. These endpoints require monitoring:config:write (granted to ADMIN and OPERATOR by default). They cannot be replaced by /actuator/metrics/*: that surface is ADMIN-only. Both paths now share the same value validation (see the next section), but the actuator path emits no configuration-change event and writes no audit record.

Preview the impact of an emergency action before running it (read-only, no audit entry):

GET /api/monitoring/emergency/preview?action=clear-cache

action accepts clear-cache / reset-errors / force-recovery / breaker-open / breaker-close. The impact object depends on the action: cache clearing returns the entry count and sample keys, error reset returns the key count, total errors and top keys, forced recovery returns the current degradation level, sampling rates and auto-mode flag, and breaker open/close returns state, failure rate, failure count and request count. Detail lists are capped at 20 rows while totals are reported separately. The endpoint requires monitoring:config:read (granted to ADMIN and OPERATOR by default) — seeing the page and being allowed to change configuration are separate, so writes still require monitoring:config:write.

Internal operations surface (/actuator/metrics/*)

MetricsManagementController exposes a second set of configuration and emergency endpoints under /actuator/metrics. The whole family is guarded by hasRole('ADMIN') in SecurityConfiguration and is a break-glass path for operators; the web console does not use it.

GET  /actuator/metrics/status
GET  /actuator/metrics/performance
GET  /actuator/metrics/memory
GET  /actuator/metrics/circuit-breaker
GET  /actuator/metrics/sampling
POST /actuator/metrics/sampling                     # marked @Deprecated
GET  /actuator/metrics/performance-config
POST /actuator/metrics/performance-config           # marked @Deprecated
POST /actuator/metrics/memory/cleanup
POST /actuator/metrics/memory/clear-cache
POST /actuator/metrics/metrics-pipeline/circuit-breaker/{action}
POST /actuator/metrics/circuit-breaker/{action}
GET  /actuator/metrics/error-handler
POST /actuator/metrics/error-handler/reset
GET  /actuator/metrics/degradation
POST /actuator/metrics/degradation/level
POST /actuator/metrics/degradation/auto-mode
POST /actuator/metrics/degradation/force-recovery

The two write endpoints (POST /sampling, POST /performance-config) overlap with PUT /api/monitoring/config/* and now share the same value validation (sampling rates in 0~1, batchSize >= 1 and bufferSize >= batchSize; out-of-range input returns 400). What differs is what they skip: they mutate the in-memory configuration object directly, emit no configuration-change event, and write no audit record. Use /api/monitoring/config/* for routine operations.

Prefix overlap: /actuator/metrics/sampling shares its prefix with the built-in /actuator/metrics/{metricName} endpoint. Production exposes only health,info,prometheus,threaddump by default (see application-prod.yml), so these custom write endpoints are normally available in dev only — do not treat them as a production hot-reload entry point.

Metrics Pipeline Circuit Breaker Statistics

Get statistics for the metrics pipeline circuit breaker (MetricsCircuitBreaker):

GET /api/monitoring/metrics-pipeline/circuit-breaker/stats

Scope: this circuit breaker protects the monitoring metrics processing pipeline (asynchronous metrics handling). It is not the per-instance routing circuit breaker (CircuitBreakerManager). Neither the statistics nor the force-open / force-close switches below change the routing circuit-breaker state of any instance.

The legacy path GET /api/monitoring/circuit-breaker/stats is a compatibility alias (marked @Deprecated in code) that shares the same implementation.

Degradation Status

Get the service degradation status:

GET /api/monitoring/degradation/status

Error Statistics

Get error statistics:

GET /api/monitoring/errors/stats

Cache Statistics

Get cache statistics:

GET /api/monitoring/cache/stats

Note: These are the custom monitoring endpoints JAiRouter (currently 3.2.2) actually provides; each maps to MonitoringController.java:65,226,240,268,309,390,426. See the Management API for more endpoints.

Monitoring Integration

Prometheus Integration

JAiRouter exposes metrics in Prometheus format at /actuator/prometheus. Configure Prometheus to scrape these metrics:

# prometheus.yml
scrape_configs:
  - job_name: 'jairouter'
    static_configs:
      - targets: ['localhost:8080']
    metrics_path: '/actuator/prometheus'
    scrape_interval: 15s

Grafana Dashboard

Import the JAiRouter Grafana dashboard for visualization:

  • Dashboard ID: Coming soon
  • Metrics: Request rate, response time, error rate, circuit breaker status
  • Alerts: High error rate, circuit breaker open, instance down

Health Check Integration

Configure external monitoring tools to check JAiRouter health:

# Simple health check (process liveness)
curl -f http://localhost:8080/actuator/health/liveness || exit 1

# Detailed health check with specific component
curl -f http://localhost:8080/actuator/health/tracing || exit 1

Monitoring Best Practices

Key Metrics to Monitor

  1. Request Metrics
  2. Request rate (requests/second)
  3. Response time (p50, p95, p99)
  4. Error rate (percentage)

  5. Instance Health

  6. Instance availability
  7. Health check response time
  8. Circuit breaker state

  9. Resource Usage

  10. JVM memory usage
  11. CPU utilization
  12. Disk space

  13. Rate Limiting

  14. Rate limit utilization
  15. Rejected requests
  16. Client-specific limits

Alerting Rules

Set up alerts for critical conditions:

# Prometheus alerting rules
groups:
  - name: jairouter
    rules:
      - alert: HighErrorRate
        expr: rate(jairouter_requests_total{status="500"}[5m]) / rate(jairouter_requests_total[5m]) > 0.1
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "High error rate detected"

      - alert: CircuitBreakerOpen
        expr: jairouter_circuit_breaker_state > 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Circuit breaker is open"

      - alert: InstanceDown
        expr: up{job="jairouter"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "JAiRouter instance is down"

Log Monitoring

Monitor application logs for:

  • Error patterns
  • Performance issues
  • Configuration changes
  • Security events
# Tail logs with filtering
tail -f logs/application.log | grep -E "(ERROR|WARN|Circuit|Rate)"

Troubleshooting

Common Issues

  1. High Response Time
  2. Check instance health
  3. Review load balancer distribution
  4. Monitor resource usage

  5. Circuit Breaker Open

  6. Check instance connectivity
  7. Review error logs
  8. Verify instance configuration

  9. Rate Limit Exceeded

  10. Review rate limit configuration
  11. Check client request patterns
  12. Consider increasing limits

Debug Endpoints

Enable debug logging for detailed monitoring:

# application.yml
logging:
  level:
    org.unreal.modelrouter: DEBUG
    org.springframework.web: DEBUG

Access debug information:

GET /actuator/loggers/org.unreal.modelrouter

⚠️ loggers is not in the default exposure list — add it to management.endpoints.web.exposure.include first.

Security Considerations

Monitoring Endpoint Security

Secure monitoring endpoints in production:

# application.yml
management:
  endpoints:
    web:
      exposure:
        include: health,info,prometheus
  endpoint:
    health:
      show-details: when-authorized
  security:
    enabled: true

Sensitive Information

Avoid exposing sensitive data in metrics:

  • API keys
  • Internal URLs
  • User information
  • Configuration secrets

Next Steps


Read-only monitoring surface: 37 management endpoints with no frontend entry and no docs

None of these endpoints is called by the console, and none was documented (the #454 reconciliation). By maintainer decision they are kept as a read-only monitoring / diagnostics surface and are documented here.

Where authorization comes from: these endpoints declare no method-level @PreAuthorize. Authorization is decided by PermissionAuthorizationManager over the /api/** matcher, using the URL rules in PermissionRuleRegistry. The codes below are the rules that are actually in effect, not an intended design.

Dynamic metric registration

MethodPathEffective permission code
DELETE/api/monitoring/metrics/batchmonitoring:metrics:write
GET/api/monitoring/metrics/category/{}monitoring:metrics:read
POST/api/monitoring/metrics/cleanupmonitoring:metrics:write
GET/api/monitoring/metrics/exists/{}monitoring:metrics:read
GET/api/monitoring/metrics/listmonitoring:metrics:read
GET/api/monitoring/metrics/metadatamonitoring:metrics:read
GET/api/monitoring/metrics/metadata/{}monitoring:metrics:read
PUT/api/monitoring/metrics/metadata/{}monitoring:metrics:write
POST/api/monitoring/metrics/registermonitoring:metrics:write
POST/api/monitoring/metrics/register/batchmonitoring:metrics:write
GET/api/monitoring/metrics/searchmonitoring:metrics:read
GET/api/monitoring/metrics/statisticsmonitoring:metrics:read
POST/api/monitoring/metrics/validatemonitoring:metrics:write
DELETE/api/monitoring/metrics/{}monitoring:metrics:write
PUT/api/monitoring/metrics/{}/configurationmonitoring:metrics:write

Call history

MethodPathEffective permission code
DELETE/api/call-history/cleanupno write rule ⇒ denied by default (RbacUnmatchedPolicy DENY_ALL, ADMIN only)
GET/api/call-history/countcallhistory:read
GET/api/call-history/errorscallhistory:read
GET/api/call-history/recentcallhistory:read
POST/api/call-history/record**ingestion exemption** (RbacExemptEndpoints, #454)
GET/api/call-history/trace/{}callhistory:read

Responses from GET /api/call-history/** are redacted per caller: with callhistory:full-content the detail endpoint decrypts FULL-level bodies and records one FULL_CONTENT_ACCESS audit event; otherwise (including an operator that only holds callhistory:read) the encrypted fields are null and only the summary columns stay visible.

Routing monitor

MethodPathEffective permission code
PUT/api/v1/routing-monitor/configlb:config:write
DELETE/api/v1/routing-monitor/history/{}lb:config:write
GET/api/v1/routing-monitor/history/{}monitoring:routing:read
GET/api/v1/routing-monitor/stats/{}monitoring:routing:read

Circuit breaker history

MethodPathEffective permission code
GET/api/config/circuit-breaker/history/instanceconfig:circuitbreaker:read
GET/api/config/circuit-breaker/history/service-typeconfig:circuitbreaker:read
GET/api/config/circuit-breaker/history/time-rangeconfig:circuitbreaker:read

Adapter templates

MethodPathEffective permission code
GET/api/config/adapter/templates/{}config:adapters:read
POST/api/config/adapter/templates/{}/createconfig:adapters:write

Exception management

MethodPathEffective permission code
GET/api/exceptions/dashboardmonitoring:exceptions:read
GET/api/exceptions/recent/{}monitoring:exceptions:read

Slow query analysis

MethodPathEffective permission code
GET/api/monitoring/slow-queries/stats/{}monitoring:slowquery:read
GET/api/monitoring/slow-queries/trendsmonitoring:slowquery:read
DELETE/api/monitoring/slow-queries/statsmonitoring:slowquery:write (added by #468; before that the write had no rule and was only caught by the default DENY_ALL)

Health status SSE

MethodPathEffective permission code
GET/api/health-status/streamAuthenticated only

This SSE channel pushes an instance-health snapshot every 5 seconds. Since #866 the snapshot is computed once per tick and shared by all connections, so opening more console tabs no longer multiplies the database queries. The interval is configurable through jairouter.health-status.stream-interval-ms (default 5000, minimum 100).

Rate limit monitor

MethodPathEffective permission code
GET/api/rate-limiter/statusrl:monitoring:read

State persistence

History: POST /api/state-persistence/test/save-circuit-breaker/{} was a test-fixture endpoint (it wrote hard-coded state=OPEN / failureCount=5 straight into the persistence tier). #483 first isolated it behind @Profile("!prod"), then it was removed — production code should not ship a data-fabrication endpoint; the same chain (real state → persistence → startup recovery) is now covered by CircuitBreakerStatePersistenceAdapterTest in the test tree.

Two caveats

  1. Writes under /api/monitoring/metrics/** now have an explicit rule — reads use monitoring:metrics:read, writes use monitoring:metrics:write (added by #454). Previously writes had no rule and were protected only by the RbacUnmatchedPolicy default DENY_ALL ("secure by accident"); the OPERATOR template now receives the write code automatically.
  2. /api/call-history/** registers a GET rule only (callhistory:read); its write surface relies on deny-by-default and the exemption list — POST /record is on the RbacExemptEndpoints ingestion exemption (authentication only), while DELETE /cleanup has no write rule and therefore falls into the DENY_ALL default posture (ADMIN only) rather than being allowed by callhistory:read. #514 fixed the defect where callhistory:view authorized reads, writes and deletes alike; the current shape is "one GET rule + deny-by-default writes + ingestion exemption".