Monitoring API¶
Doc Version: 1.0.2
Last Updated: 2026-05-21
Git Commit: 61384b4a
Author: Lincoln
JAiRouter provides comprehensive monitoring APIs for health checks, metrics collection, and system status monitoring.
Overview¶
The Monitoring API includes:
- Health Checks - Service and instance health status
- Metrics - Performance and usage statistics
- System Status - Overall system health and configuration
All monitoring endpoints are available under the /actuator/* path and provide real-time insights into your JAiRouter deployment.
Health Check Endpoints¶
System Health¶
Get overall system health status:
Response:
{
"status": "UP",
"components": {
"diskSpace": {
"status": "UP",
"details": {
"total": 499963174912,
"free": 91943821312,
"threshold": 10485760,
"exists": true
}
},
"ping": {
"status": "UP"
},
"tracing": {
"status": "UP"
},
"sharedState": {
"status": "UP"
},
"rbacEndpointCoverage": {
"status": "UP",
"details": {
"missing": 0
}
}
}
}
⚠️ Component names are the registered health indicators:
tracing(TracingHealthIndicator.java:28),sharedState(SharedStateHealthIndicator.java:31),rbacEndpointCoverage(RbacEndpointCoverageHealthIndicator.java:27) andjwtRedisHealthIndicator(JwtRedisHealthMonitor.java:21), plus Spring Boot's standarddiskSpaceetc. There is nomodelRoutercomponent; theactiveInstances/totalInstances/circuitBreakerStatusstructure does not exist in the code.
Detailed Health Information¶
Get detailed health information including all components:
⚠️ There is no
/actuator/health/detailedendpoint: only thelivenessandreadinessgroups are defined (config/base/monitoring-base.yml:70-84). To see every component, callGET /actuator/healthunder the dev profile (show-details: always).
Response:
{
"status": "UP",
"components": {
"diskSpace": {
"status": "UP",
"details": {
"total": 499963174912,
"free": 91943821312,
"threshold": 10485760,
"exists": true
}
}
}
}
⚠️ The original document also listed a
modelRoutercomponent here (withservices.*.instances[]detail) — no such component exists in the code, so it has been removed; see the note at the top of this section for the real health components.
Which Endpoint Should Each Probe Use¶
/actuator/health is the aggregate health including downstream components (db, jwtRedis, ...). It is suitable for humans and alerting, but a poor fit for orchestration probes: a dependency blip will take the replica out of rotation even though the replica could still serve traffic.
| Consumer | Endpoint | Semantics |
|---|---|---|
| Process liveness | /actuator/health/liveness | Contains only livenessState and ping; UP while the process is alive |
| Accepting traffic (readiness) | /actuator/health/readiness | Contains only readinessState; excludes db/redis by default |
Container self-check (Docker HEALTHCHECK) | /actuator/health/liveness | All five Dockerfile definitions in this repo are configured this way |
| Alerting and troubleshooting | /actuator/health | DOWN as soon as any component is DOWN — which is the signal you want |
| External load balancer (self-managed) | /actuator/health/readiness | If you want dependency failures to drain traffic, use aggregate health instead and accept the false drain |
The Kubernetes manifests (
deploy/k8s/base/deployment.yaml) already do the right thing:startupProbe/livenessProbehit/actuator/health/liveness,readinessProbehits/actuator/health/readiness.
Why readiness does not include dependencies¶
readiness includes only readinessState (config/base/monitoring-base.yml). Spring Boot validates health group include entries strictly: naming a contributor that is not present in the current context makes the application fail to start. Whether db or redis is assembled depends on configuration (for example REDIS_ENABLED), so it cannot be listed statically in YAML.
The trade-off: when Redis or the database is unavailable, the replica stays in the ready set and keeps receiving traffic. For deployments where a dependency failure must drain traffic, aggregate it at the orchestration layer (an extra probe against aggregate health, accepting the false drain), or enable jairouter.health.readiness.include-dependencies (available from 3.8.7, defaults to false). When on, /actuator/health/readiness evaluates db and redis through the readinessDependencies aggregator, so a dependency failure leaves the ready set — at the cost of a dependency blip potentially keeping new replicas out of the ready set during a rolling update.
That aggregator is always assembled, so listing it in the readiness
includedoes not trip the strict-existence check above. Internally it only evaluatesdb/redisthat are actually assembled; absent dependencies are skipped rather than reported DOWN.
Health probe settings¶
Both switches are readable and editable on Platform Settings → Health Probe Settings (/system/health) or over the API. Both are hot-editable and need no restart.
| Key | Default | Effect |
|---|---|---|
jairouter.health.readiness.include-dependencies | false | Whether readiness aggregates db / redis |
jairouter.security.jwt.redis-health.fail-health | true | Whether jwtRedis turns aggregate health DOWN when Redis is unreachable |
# Read current values
curl -s -H "Authorization: Bearer <token>" http://localhost:8080/api/config/health
# Change one of them (partial update)
curl -s -X PUT -H "Authorization: Bearer <token>" -H "Content-Type: application/json" \
-d '{"jwtRedisFailHealth": false}' http://localhost:8080/api/config/health
Impact of the jwtRedis component in multi-replica setups¶
Once both jairouter.security.jwt.persistence.redis.enabled and jairouter.security.jwt.blacklist.redis.enabled are enabled (required for multi-replica), jwtRedisHealthIndicator performs real Redis round-trips; when Redis is unreachable it reports DOWN and aggregate /actuator/health returns 503.
From 3.8.7, jairouter.security.jwt.redis-health.fail-health=false downgrades it to UP (with the impact noted in the details: rate limiting and quota fail open, blacklist may be stale), so a Redis blip no longer drains healthy replicas.
Metrics Endpoints¶
Application Metrics¶
Get Prometheus-format metrics:
Response:
# HELP jvm_memory_used_bytes The amount of used memory
# TYPE jvm_memory_used_bytes gauge
jvm_memory_used_bytes{area="heap",id="PS Eden Space",} 2.38026752E8
jvm_memory_used_bytes{area="heap",id="PS Survivor Space",} 1048576.0
jvm_memory_used_bytes{area="heap",id="PS Old Gen",} 4.2991616E7
# HELP jairouter_requests_total Total number of requests
# TYPE jairouter_requests_total counter
jairouter_requests_total{service="chat",method="POST",status="200",} 1247.0
jairouter_requests_total{service="chat",method="POST",status="500",} 23.0
jairouter_requests_total{service="chat",method="POST",status="200",} 1156.0
jairouter_requests_total{service="chat",method="POST",status="500",} 18.0
# HELP jairouter_request_duration_seconds Request duration in seconds
# TYPE jairouter_request_duration_seconds histogram
jairouter_request_duration_seconds_bucket{service="chat",method="POST",le="0.1",} 234.0
jairouter_request_duration_seconds_bucket{service="chat",method="POST",le="0.5",} 892.0
jairouter_request_duration_seconds_bucket{service="chat",method="POST",le="1.0",} 1156.0
jairouter_request_duration_seconds_bucket{service="chat",method="POST",le="+Inf",} 1270.0
# HELP jairouter_circuit_breaker_state Circuit breaker state (0=CLOSED, 1=OPEN, 2=HALF_OPEN)
# TYPE jairouter_circuit_breaker_state gauge
jairouter_circuit_breaker_state{service="chat",method="POST",} 0.0
jairouter_circuit_breaker_state{service="chat",method="POST",} 0.0
# HELP jairouter_rate_limit_remaining Rate limit remaining requests
# TYPE jairouter_rate_limit_remaining gauge
jairouter_rate_limit_remaining{service="chat",client_ip="192.168.1.100",} 45.0
jairouter_rate_limit_remaining{service="chat",client_ip="192.168.1.101",} 38.0
Metrics Summary¶
Get human-readable metrics summary:
🔑 Admin credentials are required**: the /actuator/** calls below go through the normal authentication chain — only
/actuator/health(including theliveness/readinesssub-paths),/actuator/infoand/actuator/prometheusare anonymous (for k8s probes and Prometheus scraping); everything else needs the ADMIN role. Without credentials you get401 AUTH_MISSING. (Before the fix even a valid ADMIN JWT got 401: the auth chain was excluded whilehasRole("ADMIN")was still required, so the two cancelled out — see issue #340.)Fetch a JWT and call the endpoint (verified locally):
Response:
{
"names": [
"jvm.memory.used",
"jvm.memory.max",
"jvm.gc.pause",
"http.server.requests",
"jairouter_requests_total",
"jairouter_request_duration_seconds",
"jairouter_circuit_breaker_state",
"jairouter_rate_limit_remaining",
"jairouter_loadbalancer_selections_total"
]
}
Specific Metric Details¶
Get details for a specific metric:
Response:
{
"name": "jairouter_requests_total",
"description": "Total number of requests processed by model router",
"baseUnit": null,
"measurements": [
{
"statistic": "COUNT",
"value": 2647.0
}
],
"availableTags": [
{
"tag": "service",
"values": ["chat", "embedding", "rerank", "tts", "stt", "imgGen", "imgEdit"]
},
{
"tag": "method",
"values": ["POST", "GET"]
},
{
"tag": "status",
"values": ["200", "400", "500"]
}
]
}
System Information¶
Application Info¶
Get application information:
Response:
{
"app": {
"name": "JAiRouter",
"version": "3.2.2",
"description": "AI Model Service Router and Load Balancer"
},
"build": {
"version": "3.2.2",
"artifact": "model-router",
"name": "model-router",
"group": "org.unreal",
"time": "2025-08-19T08:15:30.123Z"
},
"git": {
"branch": "main",
"commit": {
"id": "3418d3f6",
"time": "2025-08-19T08:00:00Z"
}
},
"java": {
"version": "17.0.8",
"vendor": "Eclipse Adoptium"
}
}
⚠️ The
build/gitsections require thebuild-infogoal inpom.xml(not configured today), so the default/actuator/infodoes not return them; the output above is illustrative. The current project version is 3.2.2.
Environment Information¶
Get environment and configuration details:
⚠️
envis not in the default exposure list (config/base/monitoring-base.yml:60exposes only health / info / metrics / prometheus / jairouter-metrics / error-tracking; seeapplication-prod.yml:40for prod). Addenvtomanagement.endpoints.web.exposure.includefirst if you need it.
Response:
{
"activeProfiles": ["default"],
"propertySources": [
{
"name": "server.ports",
"properties": {
"local.server.port": {
"value": 8080
}
}
},
{
"name": "applicationConfig: [classpath:/application.yml]",
"properties": {
"model-router.load-balancer.default-strategy": {
"value": "ROUND_ROBIN"
},
"model-router.rate-limit.default-algorithm": {
"value": "TOKEN_BUCKET"
},
"model-router.circuit-breaker.failure-threshold": {
"value": 5
}
}
}
]
}
⚠️ The output above is illustrative: every configuration key in this project lives under the
jairouter.*prefix — there are nomodel-router.*keys (no match anywhere undersrc/main/resources).
Custom Monitoring Endpoints¶
The gateway exposes the following custom monitoring endpoints:
System Health Status¶
Get the overall system health status:
Monitoring Configuration¶
Get the current monitoring configuration:
Hot-edit the basic configuration (writes only touch memory and fall back to the configuration files on restart; each successful change records one MONITORING_CONFIG_CHANGED audit event):
PUT /api/monitoring/config/enabled
PUT /api/monitoring/config/prefix
PUT /api/monitoring/config/collection-interval
PUT /api/monitoring/config/categories
PUT /api/monitoring/config/sampling
PUT /api/monitoring/config/performance
The six
samplingrates take values in 0~1 (requests / backend / infrastructure / trace / trace processing / trace analysis);performancerequiresbatchSize >= 1andbufferSize >= batchSize, otherwise the endpoint returns 400. These endpoints requiremonitoring:config:write(granted to ADMIN and OPERATOR by default). They cannot be replaced by/actuator/metrics/*: that surface is ADMIN-only. Both paths now share the same value validation (see the next section), but the actuator path emits no configuration-change event and writes no audit record.
Preview the impact of an emergency action before running it (read-only, no audit entry):
action accepts clear-cache / reset-errors / force-recovery / breaker-open / breaker-close. The impact object depends on the action: cache clearing returns the entry count and sample keys, error reset returns the key count, total errors and top keys, forced recovery returns the current degradation level, sampling rates and auto-mode flag, and breaker open/close returns state, failure rate, failure count and request count. Detail lists are capped at 20 rows while totals are reported separately. The endpoint requires monitoring:config:read (granted to ADMIN and OPERATOR by default) — seeing the page and being allowed to change configuration are separate, so writes still require monitoring:config:write.
Internal operations surface (/actuator/metrics/*)¶
MetricsManagementController exposes a second set of configuration and emergency endpoints under /actuator/metrics. The whole family is guarded by hasRole('ADMIN') in SecurityConfiguration and is a break-glass path for operators; the web console does not use it.
GET /actuator/metrics/status
GET /actuator/metrics/performance
GET /actuator/metrics/memory
GET /actuator/metrics/circuit-breaker
GET /actuator/metrics/sampling
POST /actuator/metrics/sampling # marked @Deprecated
GET /actuator/metrics/performance-config
POST /actuator/metrics/performance-config # marked @Deprecated
POST /actuator/metrics/memory/cleanup
POST /actuator/metrics/memory/clear-cache
POST /actuator/metrics/metrics-pipeline/circuit-breaker/{action}
POST /actuator/metrics/circuit-breaker/{action}
GET /actuator/metrics/error-handler
POST /actuator/metrics/error-handler/reset
GET /actuator/metrics/degradation
POST /actuator/metrics/degradation/level
POST /actuator/metrics/degradation/auto-mode
POST /actuator/metrics/degradation/force-recovery
The two write endpoints (POST /sampling, POST /performance-config) overlap with PUT /api/monitoring/config/* and now share the same value validation (sampling rates in 0~1, batchSize >= 1 and bufferSize >= batchSize; out-of-range input returns 400). What differs is what they skip: they mutate the in-memory configuration object directly, emit no configuration-change event, and write no audit record. Use /api/monitoring/config/* for routine operations.
Prefix overlap:
/actuator/metrics/samplingshares its prefix with the built-in/actuator/metrics/{metricName}endpoint. Production exposes onlyhealth,info,prometheus,threaddumpby default (seeapplication-prod.yml), so these custom write endpoints are normally available in dev only — do not treat them as a production hot-reload entry point.
Metrics Pipeline Circuit Breaker Statistics¶
Get statistics for the metrics pipeline circuit breaker (MetricsCircuitBreaker):
Scope: this circuit breaker protects the monitoring metrics processing pipeline (asynchronous metrics handling). It is not the per-instance routing circuit breaker (
CircuitBreakerManager). Neither the statistics nor the force-open / force-close switches below change the routing circuit-breaker state of any instance.The legacy path
GET /api/monitoring/circuit-breaker/statsis a compatibility alias (marked@Deprecatedin code) that shares the same implementation.
Degradation Status¶
Get the service degradation status:
Error Statistics¶
Get error statistics:
Cache Statistics¶
Get cache statistics:
Note: These are the custom monitoring endpoints JAiRouter (currently 3.2.2) actually provides; each maps to
MonitoringController.java:65,226,240,268,309,390,426. See the Management API for more endpoints.
Monitoring Integration¶
Prometheus Integration¶
JAiRouter exposes metrics in Prometheus format at /actuator/prometheus. Configure Prometheus to scrape these metrics:
# prometheus.yml
scrape_configs:
- job_name: 'jairouter'
static_configs:
- targets: ['localhost:8080']
metrics_path: '/actuator/prometheus'
scrape_interval: 15s
Grafana Dashboard¶
Import the JAiRouter Grafana dashboard for visualization:
- Dashboard ID: Coming soon
- Metrics: Request rate, response time, error rate, circuit breaker status
- Alerts: High error rate, circuit breaker open, instance down
Health Check Integration¶
Configure external monitoring tools to check JAiRouter health:
# Simple health check (process liveness)
curl -f http://localhost:8080/actuator/health/liveness || exit 1
# Detailed health check with specific component
curl -f http://localhost:8080/actuator/health/tracing || exit 1
Monitoring Best Practices¶
Key Metrics to Monitor¶
- Request Metrics
- Request rate (requests/second)
- Response time (p50, p95, p99)
Error rate (percentage)
Instance Health
- Instance availability
- Health check response time
Circuit breaker state
Resource Usage
- JVM memory usage
- CPU utilization
Disk space
Rate Limiting
- Rate limit utilization
- Rejected requests
- Client-specific limits
Alerting Rules¶
Set up alerts for critical conditions:
# Prometheus alerting rules
groups:
- name: jairouter
rules:
- alert: HighErrorRate
expr: rate(jairouter_requests_total{status="500"}[5m]) / rate(jairouter_requests_total[5m]) > 0.1
for: 2m
labels:
severity: warning
annotations:
summary: "High error rate detected"
- alert: CircuitBreakerOpen
expr: jairouter_circuit_breaker_state > 0
for: 1m
labels:
severity: critical
annotations:
summary: "Circuit breaker is open"
- alert: InstanceDown
expr: up{job="jairouter"} == 0
for: 1m
labels:
severity: critical
annotations:
summary: "JAiRouter instance is down"
Log Monitoring¶
Monitor application logs for:
- Error patterns
- Performance issues
- Configuration changes
- Security events
Troubleshooting¶
Common Issues¶
- High Response Time
- Check instance health
- Review load balancer distribution
Monitor resource usage
Circuit Breaker Open
- Check instance connectivity
- Review error logs
Verify instance configuration
Rate Limit Exceeded
- Review rate limit configuration
- Check client request patterns
- Consider increasing limits
Debug Endpoints¶
Enable debug logging for detailed monitoring:
Access debug information:
⚠️
loggersis not in the default exposure list — add it tomanagement.endpoints.web.exposure.includefirst.
Security Considerations¶
Monitoring Endpoint Security¶
Secure monitoring endpoints in production:
# application.yml
management:
endpoints:
web:
exposure:
include: health,info,prometheus
endpoint:
health:
show-details: when-authorized
security:
enabled: true
Sensitive Information¶
Avoid exposing sensitive data in metrics:
- API keys
- Internal URLs
- User information
- Configuration secrets
Next Steps¶
- Management API - Configuration management
- Universal API - OpenAI-compatible endpoints
- OpenAPI Specification - Interactive API documentation
Read-only monitoring surface: 37 management endpoints with no frontend entry and no docs¶
None of these endpoints is called by the console, and none was documented (the #454 reconciliation). By maintainer decision they are kept as a read-only monitoring / diagnostics surface and are documented here.
Where authorization comes from: these endpoints declare no method-level
@PreAuthorize. Authorization is decided byPermissionAuthorizationManagerover the/api/**matcher, using the URL rules inPermissionRuleRegistry. The codes below are the rules that are actually in effect, not an intended design.
Dynamic metric registration¶
| Method | Path | Effective permission code |
|---|---|---|
DELETE | /api/monitoring/metrics/batch | monitoring:metrics:write |
GET | /api/monitoring/metrics/category/{} | monitoring:metrics:read |
POST | /api/monitoring/metrics/cleanup | monitoring:metrics:write |
GET | /api/monitoring/metrics/exists/{} | monitoring:metrics:read |
GET | /api/monitoring/metrics/list | monitoring:metrics:read |
GET | /api/monitoring/metrics/metadata | monitoring:metrics:read |
GET | /api/monitoring/metrics/metadata/{} | monitoring:metrics:read |
PUT | /api/monitoring/metrics/metadata/{} | monitoring:metrics:write |
POST | /api/monitoring/metrics/register | monitoring:metrics:write |
POST | /api/monitoring/metrics/register/batch | monitoring:metrics:write |
GET | /api/monitoring/metrics/search | monitoring:metrics:read |
GET | /api/monitoring/metrics/statistics | monitoring:metrics:read |
POST | /api/monitoring/metrics/validate | monitoring:metrics:write |
DELETE | /api/monitoring/metrics/{} | monitoring:metrics:write |
PUT | /api/monitoring/metrics/{}/configuration | monitoring:metrics:write |
Call history¶
| Method | Path | Effective permission code |
|---|---|---|
DELETE | /api/call-history/cleanup | no write rule ⇒ denied by default (RbacUnmatchedPolicy DENY_ALL, ADMIN only) |
GET | /api/call-history/count | callhistory:read |
GET | /api/call-history/errors | callhistory:read |
GET | /api/call-history/recent | callhistory:read |
POST | /api/call-history/record | **ingestion exemption** (RbacExemptEndpoints, #454) |
GET | /api/call-history/trace/{} | callhistory:read |
Responses from GET /api/call-history/** are redacted per caller: with callhistory:full-content the detail endpoint decrypts FULL-level bodies and records one FULL_CONTENT_ACCESS audit event; otherwise (including an operator that only holds callhistory:read) the encrypted fields are null and only the summary columns stay visible.
Routing monitor¶
| Method | Path | Effective permission code |
|---|---|---|
PUT | /api/v1/routing-monitor/config | lb:config:write |
DELETE | /api/v1/routing-monitor/history/{} | lb:config:write |
GET | /api/v1/routing-monitor/history/{} | monitoring:routing:read |
GET | /api/v1/routing-monitor/stats/{} | monitoring:routing:read |
Circuit breaker history¶
| Method | Path | Effective permission code |
|---|---|---|
GET | /api/config/circuit-breaker/history/instance | config:circuitbreaker:read |
GET | /api/config/circuit-breaker/history/service-type | config:circuitbreaker:read |
GET | /api/config/circuit-breaker/history/time-range | config:circuitbreaker:read |
Adapter templates¶
| Method | Path | Effective permission code |
|---|---|---|
GET | /api/config/adapter/templates/{} | config:adapters:read |
POST | /api/config/adapter/templates/{}/create | config:adapters:write |
Exception management¶
| Method | Path | Effective permission code |
|---|---|---|
GET | /api/exceptions/dashboard | monitoring:exceptions:read |
GET | /api/exceptions/recent/{} | monitoring:exceptions:read |
Slow query analysis¶
| Method | Path | Effective permission code |
|---|---|---|
GET | /api/monitoring/slow-queries/stats/{} | monitoring:slowquery:read |
GET | /api/monitoring/slow-queries/trends | monitoring:slowquery:read |
DELETE | /api/monitoring/slow-queries/stats | monitoring:slowquery:write (added by #468; before that the write had no rule and was only caught by the default DENY_ALL) |
Health status SSE¶
| Method | Path | Effective permission code |
|---|---|---|
GET | /api/health-status/stream | Authenticated only |
This SSE channel pushes an instance-health snapshot every 5 seconds. Since
#866the snapshot is computed once per tick and shared by all connections, so opening more console tabs no longer multiplies the database queries. The interval is configurable throughjairouter.health-status.stream-interval-ms(default5000, minimum100).
Rate limit monitor¶
| Method | Path | Effective permission code |
|---|---|---|
GET | /api/rate-limiter/status | rl:monitoring:read |
State persistence¶
History:
POST /api/state-persistence/test/save-circuit-breaker/{}was a test-fixture endpoint (it wrote hard-codedstate=OPEN/failureCount=5straight into the persistence tier).#483first isolated it behind@Profile("!prod"), then it was removed — production code should not ship a data-fabrication endpoint; the same chain (real state → persistence → startup recovery) is now covered byCircuitBreakerStatePersistenceAdapterTestin the test tree.
Two caveats¶
- Writes under
/api/monitoring/metrics/**now have an explicit rule — reads usemonitoring:metrics:read, writes usemonitoring:metrics:write(added by#454). Previously writes had no rule and were protected only by theRbacUnmatchedPolicydefaultDENY_ALL("secure by accident"); theOPERATORtemplate now receives the write code automatically. /api/call-history/**registers a GET rule only (callhistory:read); its write surface relies on deny-by-default and the exemption list —POST /recordis on theRbacExemptEndpointsingestion exemption (authentication only), whileDELETE /cleanuphas no write rule and therefore falls into theDENY_ALLdefault posture (ADMIN only) rather than being allowed bycallhistory:read.#514fixed the defect wherecallhistory:viewauthorized reads, writes and deletes alike; the current shape is "one GET rule + deny-by-default writes + ingestion exemption".