Skip to content

JAiRouter¶

Doc Version: 1.0.0 Last Updated: 2026-08-15 Git Commit: b1088a7e Author: Lincoln

AI Model Service Unified Gateway
Connect to all models with a single line of code

🎯 Core Value¶

Feature	Description
🔌 Unified API	OpenAI-compatible interface, seamlessly connect to Ollama, vLLM, GPUStack, etc.
⚖️ Smart Load Balancing	5 strategies for automatic request distribution, health checks to remove failed nodes
🛡️ Traffic Control	Rate limiting, circuit breaking, degradation for service stability
🔐 Dual Authentication	JWT + API Key with audit logs and blacklist support
📊 Full-chain Tracing	OpenTelemetry integration, performance bottlenecks at a glance
💾 State Persistence	Redis / file storage, supports high-availability cluster deployment
🎫 Token Quota & Multi-dimensional Limiting	Ledger across dimensions (tenant / API Key / user / service / model) × four windows (minute / hour / day / month), with Redis distributed counting and failover; disabled by default
🔗 Multi-protocol Entry	OpenAI-native `/v1` and Anthropic `/v1/messages` served directly, so Claude Code can connect as-is (including tool calls)
⚡ Response Cache	Exact-key and streaming SSE caching; cache hits do not consume downstream quota and can be invalidated on demand

Management Console

🏗️ Architecture Overview¶

┌─────────────────┐     ┌─────────────────────┐     ┌─────────────────┐
│   OpenAI SDK    │────▶│     JAiRouter       │────▶│    Ollama       │
│   Langchain     │     │  • Load Balancing   │     │    vLLM         │
│   HTTP Client   │     │  • Rate Limiting    │     │    GPUStack     │
└─────────────────┘     │  • Authentication   │     │    Xinference   │
                        └─────────────────────┘     └─────────────────┘

⚡ 1-Minute Quick Start¶

# Start the service (dev mode, zero config)
docker run -d --name jairouter -p 8080:8080 -e SPRING_PROFILES_ACTIVE=dev sodlinken/jairouter:latest

# For production, also provide JWT_SECRET and INITIAL_ADMIN_PASSWORD (at least 32 characters / a strong password), otherwise the process exits at startup

# Access the console
open http://localhost:8080

I want to...	Recommended Docs
Get started quickly	Quick Start Guide
Deploy in production	Deployment Guide
Configure services	Configuration Guide
Integrate via API	API Reference
Set up monitoring	Monitoring Guide
Troubleshoot issues	Troubleshooting

📈 Project Status¶

Current Release: v3.3.5 (2026-10-17)

🔧 Supported Backends¶

Backend	Description
GPUStack	GPUStack private model service
Ollama	Local LLM runtime
vLLM	High-performance LLM inference engine
Xinference	Distributed model inference framework
LocalAI	OpenAI-compatible local inference
OpenAI	Official OpenAI API

🤝 Community Support¶

GitHub Issues - Bug reports
GitHub Discussions - Discussions
Contributing Guide - Join development

If JAiRouter helps you, please give it a ⭐️ Star!