Monitoring API¶
Doc Version: 1.0.2
Last Updated: 2026-05-21
Git Commit: 61384b4a
Author: Lincoln
JAiRouter provides comprehensive monitoring APIs for health checks, metrics collection, and system status monitoring.
Overview¶
The Monitoring API includes:
- Health Checks - Service and instance health status
- Metrics - Performance and usage statistics
- System Status - Overall system health and configuration
All monitoring endpoints are available under the /actuator/* path and provide real-time insights into your JAiRouter deployment.
Health Check Endpoints¶
System Health¶
Get overall system health status:
Response:
{
"status": "UP",
"components": {
"diskSpace": {
"status": "UP",
"details": {
"total": 499963174912,
"free": 91943821312,
"threshold": 10485760,
"exists": true
}
},
"ping": {
"status": "UP"
},
"tracing": {
"status": "UP"
},
"sharedState": {
"status": "UP"
},
"rbacEndpointCoverage": {
"status": "UP",
"details": {
"missing": 0
}
}
}
}
⚠️ Component names are the registered health indicators:
tracing(TracingHealthIndicator.java:28),sharedState(SharedStateHealthIndicator.java:31),rbacEndpointCoverage(RbacEndpointCoverageHealthIndicator.java:27) andjwtRedisHealthIndicator(JwtRedisHealthMonitor.java:21), plus Spring Boot's standarddiskSpaceetc. There is nomodelRoutercomponent; theactiveInstances/totalInstances/circuitBreakerStatusstructure does not exist in the code.
Detailed Health Information¶
Get detailed health information including all components:
⚠️ There is no
/actuator/health/detailedendpoint: only thelivenessandreadinessgroups are defined (config/base/monitoring-base.yml:70-84). To see every component, callGET /actuator/healthunder the dev profile (show-details: always).
Response:
{
"status": "UP",
"components": {
"diskSpace": {
"status": "UP",
"details": {
"total": 499963174912,
"free": 91943821312,
"threshold": 10485760,
"exists": true
}
}
}
}
⚠️ The original document also listed a
modelRoutercomponent here (withservices.*.instances[]detail) — no such component exists in the code, so it has been removed; see the note at the top of this section for the real health components.
Metrics Endpoints¶
Application Metrics¶
Get Prometheus-format metrics:
Response:
# HELP jvm_memory_used_bytes The amount of used memory
# TYPE jvm_memory_used_bytes gauge
jvm_memory_used_bytes{area="heap",id="PS Eden Space",} 2.38026752E8
jvm_memory_used_bytes{area="heap",id="PS Survivor Space",} 1048576.0
jvm_memory_used_bytes{area="heap",id="PS Old Gen",} 4.2991616E7
# HELP jairouter_requests_total Total number of requests
# TYPE jairouter_requests_total counter
jairouter_requests_total{service="chat",method="POST",status="200",} 1247.0
jairouter_requests_total{service="chat",method="POST",status="500",} 23.0
jairouter_requests_total{service="chat",method="POST",status="200",} 1156.0
jairouter_requests_total{service="chat",method="POST",status="500",} 18.0
# HELP jairouter_request_duration_seconds Request duration in seconds
# TYPE jairouter_request_duration_seconds histogram
jairouter_request_duration_seconds_bucket{service="chat",method="POST",le="0.1",} 234.0
jairouter_request_duration_seconds_bucket{service="chat",method="POST",le="0.5",} 892.0
jairouter_request_duration_seconds_bucket{service="chat",method="POST",le="1.0",} 1156.0
jairouter_request_duration_seconds_bucket{service="chat",method="POST",le="+Inf",} 1270.0
# HELP jairouter_circuit_breaker_state Circuit breaker state (0=CLOSED, 1=OPEN, 2=HALF_OPEN)
# TYPE jairouter_circuit_breaker_state gauge
jairouter_circuit_breaker_state{service="chat",method="POST",} 0.0
jairouter_circuit_breaker_state{service="chat",method="POST",} 0.0
# HELP jairouter_rate_limit_remaining Rate limit remaining requests
# TYPE jairouter_rate_limit_remaining gauge
jairouter_rate_limit_remaining{service="chat",client_ip="192.168.1.100",} 45.0
jairouter_rate_limit_remaining{service="chat",client_ip="192.168.1.101",} 38.0
Metrics Summary¶
Get human-readable metrics summary:
🔑 Admin credentials are required**: the /actuator/** calls below go through the normal authentication chain — only
/actuator/health(including theliveness/readinesssub-paths),/actuator/infoand/actuator/prometheusare anonymous (for k8s probes and Prometheus scraping); everything else needs the ADMIN role. Without credentials you get401 AUTH_MISSING. (Before the fix even a valid ADMIN JWT got 401: the auth chain was excluded whilehasRole("ADMIN")was still required, so the two cancelled out — see issue #340.)Fetch a JWT and call the endpoint (verified locally):
Response:
{
"names": [
"jvm.memory.used",
"jvm.memory.max",
"jvm.gc.pause",
"http.server.requests",
"jairouter_requests_total",
"jairouter_request_duration_seconds",
"jairouter_circuit_breaker_state",
"jairouter_rate_limit_remaining",
"jairouter_loadbalancer_selections_total"
]
}
Specific Metric Details¶
Get details for a specific metric:
Response:
{
"name": "jairouter_requests_total",
"description": "Total number of requests processed by model router",
"baseUnit": null,
"measurements": [
{
"statistic": "COUNT",
"value": 2647.0
}
],
"availableTags": [
{
"tag": "service",
"values": ["chat", "embedding", "rerank", "tts", "stt", "imgGen", "imgEdit"]
},
{
"tag": "method",
"values": ["POST", "GET"]
},
{
"tag": "status",
"values": ["200", "400", "500"]
}
]
}
System Information¶
Application Info¶
Get application information:
Response:
{
"app": {
"name": "JAiRouter",
"version": "3.2.2",
"description": "AI Model Service Router and Load Balancer"
},
"build": {
"version": "3.2.2",
"artifact": "model-router",
"name": "model-router",
"group": "org.unreal",
"time": "2025-08-19T08:15:30.123Z"
},
"git": {
"branch": "main",
"commit": {
"id": "3418d3f6",
"time": "2025-08-19T08:00:00Z"
}
},
"java": {
"version": "17.0.8",
"vendor": "Eclipse Adoptium"
}
}
⚠️ The
build/gitsections require thebuild-infogoal inpom.xml(not configured today), so the default/actuator/infodoes not return them; the output above is illustrative. The current project version is 3.2.2.
Environment Information¶
Get environment and configuration details:
⚠️
envis not in the default exposure list (config/base/monitoring-base.yml:60exposes only health / info / metrics / prometheus / jairouter-metrics / error-tracking; seeapplication-prod.yml:40for prod). Addenvtomanagement.endpoints.web.exposure.includefirst if you need it.
Response:
{
"activeProfiles": ["default"],
"propertySources": [
{
"name": "server.ports",
"properties": {
"local.server.port": {
"value": 8080
}
}
},
{
"name": "applicationConfig: [classpath:/application.yml]",
"properties": {
"model-router.load-balancer.default-strategy": {
"value": "ROUND_ROBIN"
},
"model-router.rate-limit.default-algorithm": {
"value": "TOKEN_BUCKET"
},
"model-router.circuit-breaker.failure-threshold": {
"value": 5
}
}
}
]
}
⚠️ The output above is illustrative: every configuration key in this project lives under the
jairouter.*prefix — there are nomodel-router.*keys (no match anywhere undersrc/main/resources).
Custom Monitoring Endpoints¶
The gateway exposes the following custom monitoring endpoints:
System Health Status¶
Get the overall system health status:
Monitoring Configuration¶
Get the current monitoring configuration:
Metrics Pipeline Circuit Breaker Statistics¶
Get statistics for the metrics pipeline circuit breaker (MetricsCircuitBreaker):
Scope: this circuit breaker protects the monitoring metrics processing pipeline (asynchronous metrics handling). It is not the per-instance routing circuit breaker (
CircuitBreakerManager). Neither the statistics nor the force-open / force-close switches below change the routing circuit-breaker state of any instance.The legacy path
GET /api/monitoring/circuit-breaker/statsis a compatibility alias (marked@Deprecatedin code) that shares the same implementation.
Degradation Status¶
Get the service degradation status:
Error Statistics¶
Get error statistics:
Cache Statistics¶
Get cache statistics:
Note: These are the custom monitoring endpoints JAiRouter (currently 3.2.2) actually provides; each maps to
MonitoringController.java:65,226,240,268,309,390,426. See the Management API for more endpoints.
Monitoring Integration¶
Prometheus Integration¶
JAiRouter exposes metrics in Prometheus format at /actuator/prometheus. Configure Prometheus to scrape these metrics:
# prometheus.yml
scrape_configs:
- job_name: 'jairouter'
static_configs:
- targets: ['localhost:8080']
metrics_path: '/actuator/prometheus'
scrape_interval: 15s
Grafana Dashboard¶
Import the JAiRouter Grafana dashboard for visualization:
- Dashboard ID: Coming soon
- Metrics: Request rate, response time, error rate, circuit breaker status
- Alerts: High error rate, circuit breaker open, instance down
Health Check Integration¶
Configure external monitoring tools to check JAiRouter health:
# Simple health check
curl -f http://localhost:8080/actuator/health || exit 1
# Detailed health check with specific component
curl -f http://localhost:8080/actuator/health/tracing || exit 1
Monitoring Best Practices¶
Key Metrics to Monitor¶
- Request Metrics
- Request rate (requests/second)
- Response time (p50, p95, p99)
Error rate (percentage)
Instance Health
- Instance availability
- Health check response time
Circuit breaker state
Resource Usage
- JVM memory usage
- CPU utilization
Disk space
Rate Limiting
- Rate limit utilization
- Rejected requests
- Client-specific limits
Alerting Rules¶
Set up alerts for critical conditions:
# Prometheus alerting rules
groups:
- name: jairouter
rules:
- alert: HighErrorRate
expr: rate(jairouter_requests_total{status="500"}[5m]) / rate(jairouter_requests_total[5m]) > 0.1
for: 2m
labels:
severity: warning
annotations:
summary: "High error rate detected"
- alert: CircuitBreakerOpen
expr: jairouter_circuit_breaker_state > 0
for: 1m
labels:
severity: critical
annotations:
summary: "Circuit breaker is open"
- alert: InstanceDown
expr: up{job="jairouter"} == 0
for: 1m
labels:
severity: critical
annotations:
summary: "JAiRouter instance is down"
Log Monitoring¶
Monitor application logs for:
- Error patterns
- Performance issues
- Configuration changes
- Security events
Troubleshooting¶
Common Issues¶
- High Response Time
- Check instance health
- Review load balancer distribution
Monitor resource usage
Circuit Breaker Open
- Check instance connectivity
- Review error logs
Verify instance configuration
Rate Limit Exceeded
- Review rate limit configuration
- Check client request patterns
- Consider increasing limits
Debug Endpoints¶
Enable debug logging for detailed monitoring:
Access debug information:
⚠️
loggersis not in the default exposure list — add it tomanagement.endpoints.web.exposure.includefirst.
Security Considerations¶
Monitoring Endpoint Security¶
Secure monitoring endpoints in production:
# application.yml
management:
endpoints:
web:
exposure:
include: health,info,prometheus
endpoint:
health:
show-details: when-authorized
security:
enabled: true
Sensitive Information¶
Avoid exposing sensitive data in metrics:
- API keys
- Internal URLs
- User information
- Configuration secrets
Next Steps¶
- Management API - Configuration management
- Universal API - OpenAI-compatible endpoints
- OpenAPI Specification - Interactive API documentation
Read-only monitoring surface: 37 management endpoints with no frontend entry and no docs¶
None of these endpoints is called by the console, and none was documented (the #454 reconciliation). By maintainer decision they are kept as a read-only monitoring / diagnostics surface and are documented here.
Where authorization comes from: these endpoints declare no method-level
@PreAuthorize. Authorization is decided byPermissionAuthorizationManagerover the/api/**matcher, using the URL rules inPermissionRuleRegistry. The codes below are the rules that are actually in effect, not an intended design.
Dynamic metric registration¶
| Method | Path | Effective permission code |
|---|---|---|
DELETE | /api/monitoring/metrics/batch | monitoring:metrics:write |
GET | /api/monitoring/metrics/category/{} | monitoring:metrics:read |
POST | /api/monitoring/metrics/cleanup | monitoring:metrics:write |
GET | /api/monitoring/metrics/exists/{} | monitoring:metrics:read |
GET | /api/monitoring/metrics/list | monitoring:metrics:read |
GET | /api/monitoring/metrics/metadata | monitoring:metrics:read |
GET | /api/monitoring/metrics/metadata/{} | monitoring:metrics:read |
PUT | /api/monitoring/metrics/metadata/{} | monitoring:metrics:write |
POST | /api/monitoring/metrics/register | monitoring:metrics:write |
POST | /api/monitoring/metrics/register/batch | monitoring:metrics:write |
GET | /api/monitoring/metrics/search | monitoring:metrics:read |
GET | /api/monitoring/metrics/statistics | monitoring:metrics:read |
POST | /api/monitoring/metrics/validate | monitoring:metrics:write |
DELETE | /api/monitoring/metrics/{} | monitoring:metrics:write |
PUT | /api/monitoring/metrics/{}/configuration | monitoring:metrics:write |
Call history¶
| Method | Path | Effective permission code |
|---|---|---|
DELETE | /api/call-history/cleanup | callhistory:view(同码,见 #514) |
GET | /api/call-history/count | callhistory:view |
GET | /api/call-history/errors | callhistory:view |
GET | /api/call-history/recent | callhistory:view |
POST | /api/call-history/record | callhistory:view(同码,见 #514) |
GET | /api/call-history/trace/{} | callhistory:view |
Routing monitor¶
| Method | Path | Effective permission code |
|---|---|---|
PUT | /api/v1/routing-monitor/config | lb:config:write |
DELETE | /api/v1/routing-monitor/history/{} | lb:config:write |
GET | /api/v1/routing-monitor/history/{} | monitoring:routing:read |
GET | /api/v1/routing-monitor/stats/{} | monitoring:routing:read |
Circuit breaker history¶
| Method | Path | Effective permission code |
|---|---|---|
GET | /api/config/circuit-breaker/history/instance | config:circuitbreaker:read |
GET | /api/config/circuit-breaker/history/service-type | config:circuitbreaker:read |
GET | /api/config/circuit-breaker/history/time-range | config:circuitbreaker:read |
Adapter templates¶
| Method | Path | Effective permission code |
|---|---|---|
GET | /api/config/adapter/templates/{} | config:adapters:read |
POST | /api/config/adapter/templates/{}/create | config:adapters:write |
Exception management¶
| Method | Path | Effective permission code |
|---|---|---|
GET | /api/exceptions/dashboard | monitoring:exceptions:read |
GET | /api/exceptions/recent/{} | monitoring:exceptions:read |
Slow query analysis¶
| Method | Path | Effective permission code |
|---|---|---|
GET | /api/monitoring/slow-queries/stats/{} | monitoring:slowquery:read |
GET | /api/monitoring/slow-queries/trends | monitoring:slowquery:read |
DELETE | /api/monitoring/slow-queries/stats | monitoring:slowquery:write (added by #468; before that the write had no rule and was only caught by the default DENY_ALL) |
Health status SSE¶
| Method | Path | Effective permission code |
|---|---|---|
GET | /api/health-status/stream | Authenticated only |
This SSE channel pushes an instance-health snapshot every 5 seconds. Since
#866the snapshot is computed once per tick and shared by all connections, so opening more console tabs no longer multiplies the database queries. The interval is configurable throughjairouter.health-status.stream-interval-ms(default5000, minimum100).
Rate limit monitor¶
| Method | Path | Effective permission code |
|---|---|---|
GET | /api/rate-limiter/status | rl:monitoring:read |
State persistence¶
History:
POST /api/state-persistence/test/save-circuit-breaker/{}was a test-fixture endpoint (it wrote hard-codedstate=OPEN/failureCount=5straight into the persistence tier).#483first isolated it behind@Profile("!prod"), then it was removed — production code should not ship a data-fabrication endpoint; the same chain (real state → persistence → startup recovery) is now covered byCircuitBreakerStatePersistenceAdapterTestin the test tree.
Two caveats¶
- Writes under
/api/monitoring/metrics/**now have an explicit rule — reads usemonitoring:metrics:read, writes usemonitoring:metrics:write(added by#454). Previously writes had no rule and were protected only by theRbacUnmatchedPolicydefaultDENY_ALL("secure by accident"); theOPERATORtemplate now receives the write code automatically. /api/call-history/**sharescallhistory:viewacross read, write and delete —POST /recordandDELETE /cleanupare both allowed by the "view" code; see#514.