Routing Rules Configuration¶

Doc Version: 1.0.1 Last Updated: 2026-08-25 Git Commit: f8a2eebe Author: Lincoln

Overview¶

JAiRouter provides a rule engine (v2.8.5) that enables conditional routing rules configured via the Web console or YAML. Route requests flexibly by model name, request headers, client IP, and more — without writing code.

The rule engine evaluates on every request: rules are matched by priority (highest first) and the first hit takes effect. When no rule matches, the original routing logic runs unchanged — behavior is identical to a deployment without the rule engine.

Use Cases¶

Matching Semantics¶

Condition Combination¶

• AND within a rule: all conditions must match
• OR across rules: matched by priority descending, first match wins

Rule Fields¶

Condition Types (type)¶

Operators (operator)¶

WEIGHT condition: stable hash of (clientIp + "|" + modelName), hit when hash % 100 < weight. The same (IP, model) request always yields the same result, suitable for percentage-based splits. weight comes from the condition's weight field, falling back to value, then 50.

Action Types (action.type)¶

LB_STRATEGY values: random / round-robin / least-connections / ip-hash / consistent-hash. Unknown strategies fall back to the configured one.

TARGET_TAGS action (v2.9.7): selects instances by tags (AND semantics); tags are set via the action's tags map. See "Tag Routing (v2.9.7+)" at the end of this page.

RATE_LIMIT action (v2.8.8): Once a rule with this action matches, requests are rate-limited per rule ID (independent of service/instance-level limits). capacity (token bucket capacity) and rate (refill per second) are required and must be > 0; algorithm defaults to token-bucket, scope defaults to rule. Requests over the limit get 429 Too Many Requests. Deleting a rule or changing its action type automatically cleans up the corresponding limiter. GET /api/services/{serviceType}/ratelimit exposes limiter status including the rule level.

Web Console Configuration¶

1. Log in to the JAiRouter admin console
2. Click Configuration → Routing Rules in the left menu

Creating a Rule¶

Click 「New Rule」 and fill in the form:

The form has a 「Simulate Test」 button — validate rule matching before saving.

Alternatively click 「Create from Template」: pick a built-in scenario template (canary release / tenant isolation / model rewrite / weight split / adapter switch / VIP pin), enter a name, and a rule draft is generated and pre-filled into the form for editing.

Managing Rules¶

• Enable/Disable: the table switch takes effect immediately
• Priority drag-and-drop: drag the row handle to reorder by row position (top = highest priority) and commit in batch; YAML rules show a YAML badge and cannot be dragged (their priority is unaffected)
• Hit statistics: the "hits" column shows accumulated match counts per rule (Prometheus metric jairouter_rule_hits_total); the refresh button reloads list and stats
• Edit/Delete: via the action column

Rule changes take effect immediately — no restart required.

YAML Configuration¶

Edit src/main/resources/config/router/rules.yml:

model:
  rules:
    - id: route-vllm-header
      name: Route to vLLM adapter by header
      enabled: true
      priority: 100
      conditions:
        - type: HEADER
          field: x-routing
          operator: EQUALS
          value: vllm
      action:
        type: TARGET_ADAPTER
        adapter-name: vllm

    - id: route-internal-ip
      name: Pin internal IPs to dedicated instance
      enabled: true
      priority: 90
      conditions:
        - type: CLIENT_IP
          operator: CIDR_MATCH
          value: 10.0.0.0/8
      action:
        type: TARGET_INSTANCE
        instance-id: internal-gpu-1

    - id: route-model-rewrite
      name: Rewrite gpt-4 to claude-3
      enabled: true
      priority: 80
      conditions:
        - type: MODEL_NAME
          operator: EQUALS
          value: gpt-4
      action:
        type: TARGET_MODEL
        model-name: claude-3

Default is an empty list (model.rules: []), i.e. no rules enabled. YAML rules merge with Web-created rules: for the same id, the persisted (Web) rule overrides YAML.

API Reference¶

Base path: /api/config/rules

Create Rule Example¶

curl -X POST http://localhost:8080/api/config/rules \
  -H "Content-Type: application/json" \
  -H "Jairouter_Token: your-admin-token" \
  -d '{
    "name": "Route to vLLM by header",
    "priority": 100,
    "enabled": true,
    "conditions": [
      {"type": "HEADER", "field": "x-routing", "operator": "EQUALS", "value": "vllm"}
    ],
    "action": {"type": "TARGET_ADAPTER", "adapterName": "vllm"}
  }'

Batch Priority Example¶

curl -X PUT http://localhost:8080/api/config/rules/priority \
  -H "Content-Type: application/json" \
  -H "Jairouter_Token: your-admin-token" \
  -d '[{"id": "rule-id-1", "priority": 200}, {"id": "rule-id-2", "priority": 100}]'

Rule Simulation Test (dry-run)¶

Before saving a rule, verify it matches as expected using a sample request — read-only, no state change:

curl -X POST http://localhost:8080/api/config/rules/validate \
  -H "Content-Type: application/json" \
  -H "Jairouter_Token: your-admin-token" \
  -d '{
    "serviceType": "chat",
    "modelName": "gpt-4",
    "clientIp": "127.0.0.1",
    "headers": {"x-routing": "vllm"}
  }'

On match:

{
  "success": true,
  "data": {
    "matched": true,
    "ruleId": "xxx",
    "ruleName": "Route to vLLM by header",
    "priority": 100,
    "action": {"type": "TARGET_ADAPTER", "target": "vllm"},
    "message": "Matched rule: Route to vLLM by header"
  }
}

When nothing matches, matched=false. The rule form in the Web console also has a built-in 「Simulate Test」 panel.

Verification¶

1. Use AI Playground → Chat Test to send requests and observe routing
2. Check backend logs for Selected adapter / routing selection info
3. Or call /v1/* APIs with/without the conditional header and compare routing

Notes¶

1. Hot reload: rule changes take effect immediately, no restart needed
2. Persistence: Web-created rules are stored in StoreManager (key=rule_definitions) and restored on restart
3. Priority: rules with the same priority keep insertion order; semantics are "first match wins" — avoid overlapping rules
4. Performance: keep the rule count under ~100; per-request evaluation cost is negligible
5. TARGET_ADAPTER: if the specified adapter is not registered, it logs a warning and falls back to the instance-level adapter
6. HEADER conditions: only apply to /v1/* request paths (headers are used for matching only, not outbound forwarding)

Resource Pools & auto-model (v2.8.9)¶

A resource pool bundles a named set of instances of one service type. When a request's model is a pool name (convention: uto-model), the gateway picks a healthy pool member to serve.

Configuration¶

1. YAML (config/router/pools.yml):

yaml model: pools: - pool-name: auto-model name: Default auto-routing pool service-type: chat enabled: true strategy: weighted-random members: - instance-id: inst-gpt weight: 9 - instance-id: inst-claude weight: 1

1. Web console: Configuration → Resource Pools (CRUD with hot reload; persisted to StoreManager key=pool_definitions)

Behavior¶

• Pool name = virtual model name: requesting model=auto-model (or any configured pool name) triggers pool routing; with no pool configured, uto-model falls back to all healthy instances of the service type
• Selection flow: pool members (referenced by instance ID) → status/health/circuit-breaker filter → pool weights → pool strategy; deleted instances are skipped
• Strategies: weighted-random (default) / round-robin / least-connections / ip-hash / consistent-hash (note: consistent hash ignores weights)
• Rule synergy: a rule's TARGET_MODEL target can be a pool name — the matched request then routes through the pool
• Response echo: after pool routing, the response and downstream request model field reflect the actual serving instance model (non-streaming; streaming outbound requests are rewritten too)
• Rate limiting & circuit breaker: the pool-selected instance continues through the existing service/instance rate-limit and circuit-breaker chain

Tag Routing (v2.9.7+)¶

Tag routing narrows the candidate instances by instance tags: when a TARGET_TAGS rule matches or the request carries a tag header, only instances whose tags contain all the specified key-value pairs (AND semantics) remain eligible.

Instance Tags Configuration¶

Instances (ModelInstance) gain a tags field (Map<String, String>), configured in YAML at the same level as instance-id:

model:
  services:
    chat:
      instances:
        - name: "qwen2.5-72b"
          instance-id: "inst-gpu-a100-1"
          base-url: "http://10.0.0.1:8000"
          tags:
            gpu_type: a100
            region: cn-north
            tier: premium

Tags are arbitrary key-value pairs, commonly gpu_type (GPU model), region (region), tier (service tier). Instances without tags are excluded when tag selection is active.

TARGET_TAGS Action¶

Set the action type to TARGET_TAGS and list the selection tags in the action's tags map:

model:
  rules:
    - id: route-premium-a100
      name: Pin premium traffic to A100
      enabled: true
      priority: 100
      conditions:
        - type: HEADER
          field: X-User-Tier
          operator: EQUALS
          value: premium
      action:
        type: TARGET_TAGS
        tags:
          gpu_type: a100

On match, instances are selected by tags (an instance must contain every key-value pair to qualify).

The Web console supports this too: pick the 「Tag routing」 action type and add 「tag name / value」 rows under Target Tags.

Request-Level Header Selection¶

Without a rule, select instances directly via the request header:

X-JAiRouter-Tags: gpu_type=a100,region=cn-north

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "X-JAiRouter-Tags: gpu_type=a100,region=cn-north" \
  -d '{"model": "gpt-4", "messages": [{"role": "user", "content": "hi"}]}'

• Comma-separated key-value pairs; multiple keys are AND semantics
• Spaces and empty items are tolerated (e.g. gpu_type=a100, region=cn-north ,,)
• A missing, blank, or empty-after-parse header skips selection and keeps the original routing logic

Priority and Empty-Candidate Semantics¶

• TARGET_TAGS takes priority over the request header: when a TARGET_TAGS rule matches, its tags win and the header is ignored; X-JAiRouter-Tags is only parsed when no rule provides tags
• TARGET_INSTANCE lock > tag filter: instance locking narrows candidates by instanceId/name first; the tag filter narrows further on top (the lock action itself imposes no tag requirement)
• Empty candidates → 404: 404 Not Found when no instance matches the tags after filtering (same as TARGET_INSTANCE with no target instance)
• With no tag selection configured, behavior is identical to not enabling the feature

Relationship with Resource Pools¶

Pool routing (request model = pool name) is not affected by tag selection: pool members are resolved from the pool configuration, not through the tag filter.