Skip to content

Capability Reference

This page answers three questions: what it does, what it does not do, and where the boundaries are. Every configuration key below maps to real code in the repository — there are no "planned" entries.

Describes the current release. Default values are in the "Default" column.

Request endpoints

CapabilityStatusDefaultConfigurationBoundary
OpenAI-compatible Chat CompletionsAvailableOnmodel.services.chat (config/router/services.yml)Streaming and non-streaming
OpenAI-compatible EmbeddingsAvailableOnmodel.services.embedding—
RerankAvailableOnmodel.services.rerank—
Speech synthesis / recognitionAvailableOnmodel.services.tts / model.services.sttServed under /v1/audio/*
Anthropic Messages protocolAvailableOnDetected from the request pathRuns alongside the OpenAI surface; clients use their own protocol
Native passthrough (/v1/**)AvailableOn—Returns native JSON, not wrapped in RouterResponse
Console surface (/api/**, /admin/**)AvailableOn—Returns the unified RouterResponse envelope

Routing and fault tolerance

CapabilityStatusDefaultConfigurationBoundary
Multi-instance load balancingAvailableOnmodel.services.<type>.instancesEight strategies: round-robin / random / least-connections / consistent-hash / IP-hash / latency-aware / sticky-session
Same-name instances form a candidate setAvailableOnInstance nameThe request's model field matches instance names; identical names mean "replicas of the same model"
Circuit breakingAvailableOnmodel.circuit-breaker.*State is persisted in the database and survives restarts; half-open recovery is implemented
RetryAvailableOnmodel.services.<type>.retry—
FailoverAvailableOnmodel.fallback.*—
Health checkingAvailableOnmodel.health-check.*A port-only probe can report "healthy but unusable" — see Known Limits

Rate limiting and quota

CapabilityStatusDefaultConfigurationBoundary
Global rate limitAvailableOnmodel.rate-limitOutermost of the three layers
Service-level rate limitAvailableOnmodel.services.<type>.rate-limit—
Instance-level rate limitAvailableOnmodel.services.<type>.instances[i].rate-limitCannot be overridden per item via environment variables (it replaces the whole list; see configuration/index.md)
API key quotaAvailableOnjairouter.quota.*Daily / weekly / monthly windows plus token dimension
Over-quota responseAvailable——Returns 429, not 401, so clients can tell "out of quota" from "bad credentials"

Caching

CapabilityStatusDefaultConfigurationBoundary
Response cache (exact)AvailablePer configjairouter.cache.response.*Isolated per tenant
Response cache (semantic)AvailableOffjairouter.semantic-cache.*SimHash near-match reuse; L2 per-tenant and L3 public pool, the latter gated by seven admission rules
Streaming response cacheAvailableFollows response cacheSame as aboveReplayed as SSE frames on a hit
PII-bearing responses are never cachedAvailableOn—Skipped when any of three signals trips (request replacements / response restorations / an independent detector)

Data protection

CapabilityStatusDefaultConfigurationBoundary
Response sanitization (one-way)AvailableOn in prodsanitization.response.*Rules are configurable; redaction is not reversible
Reversible sanitization (PII vault)AvailableOffjairouter.pii.reversible.*Placeholders on the request side, restored on the response side; streaming restores chunk by chunk, so a split placeholder stays split
Field-level sanitization scopeAvailableWhole textjairouter.pii.reversible.json-pathsSupports $.a.b and $.a[*].b; a misconfigured path falls back to whole-text matching (never under-redacts)
Sanitization access auditAvailableOn with reversible sanitization—Metadata only, never the original values
Streaming guardAvailableOffjairouter.guard.*Regex hits can REDACT or ABORT; no cross-chunk joining, so a split sensitive string is not detected
SSRF protectionAvailableOn—Outbound address validation rejects internal and reserved ranges

Observability

CapabilityStatusDefaultConfigurationBoundary
Prometheus metricsAvailableOnmanagement.endpoints.web.exposure.includeProduction exposes only health,info,prometheus
Call historyAvailableOnjairouter.monitoring.call-history.*Cleanup runs at 02:00
Token usage recordsAvailableOnjairouter.monitoring.token-usage.*Cleanup runs at 03:00 (90-day default)
Audit logAvailableOnjairouter.security.audit.*Event types are an enum; unknown names are rejected
Distributed tracingAvailablePer configjairouter.tracing.*OTLP / Zipkin / log exporters
Capacity metricsAvailableOn—SSE connections, outbound in-flight requests, cache entries, event-loop delay, WebSocket sessions
ConsoleAvailableOn—Vue 3; overview / routing / cache / monitoring / configuration

Deployment and operations

CapabilityStatusDefaultConfigurationBoundary
Single-jar startupAvailable——java -jar model-router.jar
Docker imagesAvailable—Dockerfile / Dockerfile.optimized / Dockerfile.distroless / Dockerfile.jlinkThe three variants use jarmode=tools layering
KubernetesAvailable—docs/*/deployment/kubernetes.mdDeployment / Service / ConfigMap examples
DatabaseAvailableH2spring.datasource.*H2 and PostgreSQL; R2DBC was removed in v3.3.6
Shared state for multiple replicasAvailableOffjairouter.security.*.redis.enabled (three switches)Startup warns when replicas are detected without them; the semantic cache pool is still per-process (see Known Limits)
Configuration hot updatePartial——Quota and monitoring change memory only (lost on restart); sanitization and circuit-breaker settings are persisted; instance config is persisted but a sync deletes instances absent from YAML

What this gateway does not do

Not doneWhyWhose job
Model inferenceThe gateway forwards; it does not computevLLM / Ollama / cloud APIs
Billing and settlementBilling involves accounts, tax, reconciliation — a separate systemYour billing platform
Prompt management and versioningAn application-layer concern; putting it here turns the gateway into a business systemApplication / prompt platform
RAG and vector storageThe gateway forwards embedding requests; it stores no vectorsVector databases
Training and fine-tuningUnrelated to gateway dutiesTraining platforms
End-user application scaffoldingThe console is an admin UI, not an end-user productBusiness applications

Enterprise capabilities: are they in the open-source build?

This section lays out the current facts so a commercial split can be decided with eyes open.

CapabilityIn the open-source buildNotes
Multi-tenancy (API key scope)YesQuota, cache, and audit are all keyed by API key
Sub-tenant isolation (user field)YesUsed by semantic cache L2; user is client-reported and narrows scope only, never authenticates
Audit trailYesSecurity audit + sanitization access audit + cross-tenant cache hit audit
Compliant sanitization (reversible)YesPII vault plus field-level policy
Quota and rate limitingYesThree-layer limiting plus multi-window quota
ObservabilityYesPrometheus + tracing + console
Shared state across replicasYes (switch required)Three Redis switches; the semantic cache pool is still per-process
Fine-grained RBACYes53 permission codes plus URL rules
SSO / OIDCNoUsername/password plus JWT only
High-availability orchestrationNoK8s examples only, no operational orchestration
SLA and paid supportNoAn open-source project makes no SLA commitment

The conclusion is blunt: apart from SSO and orchestration, every enterprise capability is currently in the open-source build. If there is to be a commercial path, the line has to be drawn here — that is a product decision, not a technical one.

Known limits

These are deliberate trade-offs, not defects:

  1. A port-only health check does not mean "usable" — check_type: socket_connect proves the port is open; an upstream returning errors may still be reported healthy.
  2. The semantic cache L2/L3 pool is per-process — with multiple replicas the hit rate is divided by the replica count, and nothing signals why (issue #1002).
  3. The streaming guard does not join across chunks — a sensitive string split between two chunks is not detected.
  4. Reversible sanitization restores chunk by chunk in streaming — a split placeholder is not restored, so the client sees the placeholder text (the real value never left).
  5. The streaming path does not run the PII detector — non-streaming responses containing PII are not cached; the streaming path does not yet apply this check.
  6. Quota hot edits are not persisted — a console change reverts to the configuration file on restart (only QuotaConfigController surfaces this: restart-level fields return 400 with RESTART_REQUIRED).