Rate Limiting Configuration¶

Doc Version: 1.0.2
Last Updated: 2026-05-21 Git Commit: 61384b4a
Author: Lincoln

JAiRouter provides multiple rate limiting algorithms, supporting global, service-level, and instance-level rate limiting configurations, as well as independent rate limiting based on client IP. This document details the configuration and usage of various rate limiting strategies.

Modular Configuration Overview¶

Starting from v1.0.0, JAiRouter adopts a modular configuration structure with rate limiting related configurations moved to separate configuration files:

Main configuration file: application.yml
Model services base configuration: config/base/model-services-base.yml

You can find all rate limiting related configurations in the config/base/model-services-base.yml file, including global configurations, service type configurations, and instance configurations.

Rate Limiting Overview¶

Supported Algorithms¶

Algorithm	Characteristics	Use Cases	Burst Handling
Token Bucket	Allows burst traffic	Steady traffic + occasional bursts	Excellent
Leaky Bucket	Smooths output traffic	Requires steady output	Moderate
Sliding Window	Precise time window control	Precise traffic control	Moderate
Warm Up	Cold start protection	Service startup scenarios	Gradual

Rate Limiting Hierarchy¶

graph TB
    A[Global Rate Limiting Configuration] --> B[Service-Level Rate Limiting Configuration]
    B --> C[Instance-Level Rate Limiting Configuration]

    A --> D[Default Algorithm and Parameters]
    B --> E[Service-Specific Rate Limiting]
    C --> F[Instance-Independent Rate Limiting]

    G[Client IP Rate Limiting] --> B
    G --> C

    style A fill:#e1f5fe
    style B fill:#f3e5f5
    style C fill:#e8f5e8
    style G fill:#fff3e0

Layer order (v2.8.8): Requests pass through service-level → instance-level checks (global / client-IP limits apply when enabled). Exceeding the service limit returns 429 Too Many Requests; exceeding an instance limit skips to the next instance, and 503 is returned only when all instances are limited. The rule engine's RATE_LIMIT action additionally provides per-rule-ID limiting (see routing-rules.md).

Service-Level Dynamic Rate Limiting Configuration (v2.8.8)¶

Service-level limits can be configured dynamically and hot-applied from the Web console (no restart):

API: PUT /api/services/{serviceType}/ratelimit (persists + applies immediately), GET /api/services/{serviceType}/ratelimit (reads current config)
Config format (canonical):

{
  "enabled": true,
  "algorithm": "token-bucket",
  "capacity": 100,
  "rate": 10,
  "scope": "service",
  "key": ""
}

enabled=false removes the service limiter; when enabled, capacity/rate must be > 0 or 400 is returned
Config is persisted to the config store (model-router-config) and re-applied automatically on restart (loaded from the merged config at boot); services without a YAML rate-limit config are untouched
The service management edit dialog is wired to this endpoint (saving the rate-limit form hot-applies it)

Global Rate Limiting Configuration¶

Basic Configuration¶

Configure global rate limiting in the config/base/model-services-base.yml file:

# config/base/model-services-base.yml
model:
  # Global rate limiting configuration
  rate-limit:
    enabled: true
    algorithm: "token-bucket"
    capacity: 1000
    rate: 100
    scope: "service"
    client-ip-enable: true  # Enable client IP rate limiting

  services:
    # Chat service configuration
    chat:
      load-balance:
        type: least-connections
      adapter: gpustack # Use GPUStack adapter
      # Service-level rate limiting configuration
      rate-limit:
        enabled: true
        algorithm: "token-bucket"
        capacity: 100
        rate: 10
        scope: "service"
        client-ip-enable: true

Advanced Configuration¶

model:
  rate-limit:
    enabled: true
    algorithm: "token-bucket"
    capacity: 1000
    rate: 100
    scope: "service"
    client-ip-enable: true  # Enable client IP rate limiting

Detailed Rate Limiting Algorithms¶

1. Token Bucket Algorithm¶

Algorithm Principle¶

The token bucket algorithm maintains a fixed-capacity bucket and adds tokens at a fixed rate. Requests consume tokens when they arrive, and are rejected if no tokens are available.

graph LR
    A[Token Generator] -->|Fixed Rate| B[Token Bucket]
    C[Request] --> D{Tokens in Bucket?}
    D -->|Yes| E[Consume Token, Allow Request]
    D -->|No| F[Reject Request]
    B --> D

Configuration Example¶

Configure the token bucket algorithm in the config/base/model-services-base.yml file:

model:
  services:
    chat:
      rate-limit:
        enabled: true
        algorithm: "token-bucket"
        capacity: 100            # Bucket capacity (maximum tokens)
        rate: 10                # Token refill rate (per second)
        scope: "service"        # Rate limiting scope
        client-ip-enable: true  # Enable client IP rate limiting

JSON Configuration¶

JAiRouter also supports updating rate limiting configurations via the dynamic configuration API:

{
  "services": {
    "chat": {
      "rateLimit": {
        "algorithm": "token-bucket",
        "capacity": 100,
        "rate": 10,
        "clientIpEnable": true
      }
    }
  }
}

Parameter Description¶

Parameter	Type	Description	Default Value
`capacity`	Integer	Bucket capacity, maximum number of tokens	100
`rate`	Integer	Token refill rate (per second)	10
`clientIpEnable`	Boolean	Whether to enable client IP rate limiting	false

Use Cases¶

Steady Traffic + Occasional Bursts: Daily traffic is steady with occasional burst requests
API Gateway: Need to allow short-term bursts but control long-term average rate
User Request Limiting: Allow users to send multiple requests in a short time

Configuration Recommendations¶

# High concurrency scenario
rate-limit:
  algorithm: "token-bucket"
  capacity: 1000              # Large capacity to support bursts
  rate: 100                  # Moderate refill rate

# Backend protection scenario
rate-limit:
  algorithm: "token-bucket"
  capacity: 50               # Small capacity to limit bursts
  rate: 5                    # Low rate to protect backend

2. Leaky Bucket Algorithm¶

Algorithm Principle¶

The leaky bucket algorithm maintains a fixed-capacity bucket. Requests enter the bucket to queue, and are processed at a fixed rate from the bottom of the bucket.

graph TB
    A[Request Inflow] --> B{Is Bucket Full?}
    B -->|No| C[Request Enters Bucket]
    B -->|Yes| D[Reject Request]
    C --> E[Queue in Bucket]
    E -->|Fixed Rate| F[Process Request]

Configuration Example¶

Configure the leaky bucket algorithm in the config/base/model-services-base.yml file:

model:
  services:
    embedding:
      rate-limit:
        enabled: true
        algorithm: "leaky-bucket"
        capacity: 50            # Bucket capacity
        rate: 5                # Leak rate (per second)
        scope: "service"
        client-ip-enable: true

JSON Configuration¶

JAiRouter also supports updating rate limiting configurations via the dynamic configuration API:

{
  "services": {
    "embedding": {
      "rateLimit": {
        "algorithm": "leaky-bucket",
        "capacity": 50,
        "rate": 5,
        "clientIpEnable": true
      }
    }
  }
}

Parameter Description¶

Parameter	Type	Description	Default Value
`capacity`	Integer	Bucket capacity, maximum queued requests	50
`rate`	Integer	Leak rate (per second)	5

Use Cases¶

Steady Output Required: Backend needs to receive requests at a steady rate
Message Queue: Need to control message processing rate
Database Protection: Protect database from burst traffic impact

3. Sliding Window Algorithm¶

Algorithm Principle¶

The sliding window algorithm counts requests within a fixed time window and rejects requests when the threshold is exceeded.

graph LR
    A[Time Window] --> B[Count Requests]
    C[New Request] --> D{Requests in Window < Threshold?}
    D -->|Yes| E[Allow Request]
    D -->|No| F[Reject Request]
    B --> D

Configuration Example¶

Configure the sliding window algorithm in the config/base/model-services-base.yml file:

model:
  services:
    tts:
      rate-limit:
        enabled: true
        algorithm: "sliding-window"
        capacity: 100          # Bucket capacity
        rate: 100              # Maximum requests per 1-second window
        scope: "service"
        client-ip-enable: true

JSON Configuration¶

JAiRouter also supports updating rate limiting configurations via the dynamic configuration API:

{
  "services": {
    "tts": {
      "rateLimit": {
        "algorithm": "sliding-window",
        "capacity": 100,
        "rate": 100,
        "clientIpEnable": true
      }
    }
  }
}

Parameter Description¶

Parameter	Type	Description	Default Value
`capacity`	Integer	Bucket capacity	100
`rate`	Integer	Maximum requests per 1-second window	10

Note: The sliding window uses a fixed 1-second window. rate controls the maximum number of requests allowed within that window.

Use Cases¶

Precise Traffic Control: Need precise control of requests within time window
API Quota Management: Allocate API call quotas by time period
Interface Protection: Prevent large number of duplicate requests in short time

4. Warm Up Algorithm¶

Algorithm Principle¶

The warm-up algorithm gradually increases the allowed request rate during service startup to avoid performance issues during cold start.

graph LR
    A[Service Startup] --> B[Low Rate]
    B --> C[Gradually Increase]
    C --> D[Reach Target Rate]

    E[Warm-up Time] --> F[Current Allowed Rate]
    F --> G[= Target Rate * (Current Time / Warm-up Time)]

Configuration Example¶

Configure the warm-up algorithm in the config/base/model-services-base.yml file:

model:
  services:
    chat:
      rate-limit:
        enabled: true
        algorithm: "warm-up"
        capacity: 100           # Final capacity
        rate: 10               # Final rate
        warm-up-period: 300    # Warm-up time (seconds)
        scope: "service"

JSON Configuration¶

JAiRouter also supports updating rate limiting configurations via the dynamic configuration API:

{
  "services": {
    "chat": {
      "rateLimit": {
        "algorithm": "warm-up",
        "capacity": 100,
        "rate": 10,
        "warmUpPeriod": 300,
        "clientIpEnable": true
      }
    }
  }
}

Parameter Description¶

Parameter	Type	Description	Default Value
`capacity`	Integer	Final capacity	100
`rate`	Integer	Final rate (per second)	10
`warmUpPeriod`	Integer	Warm-up time (seconds)	600

Use Cases¶

Service Cold Start: Service needs warm-up when starting
Cache Warm-up: Time needed to build cache
Connection Pool Warm-up: Database connection pool needs time to establish

Client IP Rate Limiting¶

Basic Configuration¶

Enable client IP rate limiting via client-ip-enable in config/base/model-services-base.yml:

model:
  rate-limit:
    enabled: true
    algorithm: "token-bucket"
    capacity: 1000
    rate: 100
    scope: "service"
    client-ip-enable: true      # Enable client IP rate limiting

When enabled, each client IP gets its own rate limiter instance using the same algorithm, capacity, and rate as the parent configuration. The cache holds up to 10,000 client entries with 30-minute idle expiry.

Service-Level IP Rate Limiting¶

model:
  services:
    chat:
      rate-limit:
        enabled: true
        algorithm: "token-bucket"
        capacity: 1000          # Service-level total capacity
        rate: 100              # Service-level total rate
        client-ip-enable: true  # Enable IP rate limiting

Instance-Level IP Rate Limiting¶

model:
  services:
    chat:
      instances:
        - name: "model-1"
          base-url: "http://server-1:8080"
          rate-limit:
            enabled: true
            algorithm: "token-bucket"
            capacity: 100
            rate: 10
            scope: "instance"
            client-ip-enable: true

IP Rate Limiting Monitoring¶

# View rate limiter status (includes per-IP limiters)
curl "http://localhost:8080/api/rate-limiter/status"

# View rate limiter metrics (remaining capacity, usage ratio)
curl "http://localhost:8080/api/rate-limiter/metrics"

Multi-Layer Rate Limiting Configuration¶

Layered Rate Limiting Strategy¶

Configure multi-layer rate limiting in the config/base/model-services-base.yml file:

model:
  # Global rate limiting: Protect the entire system
  rate-limit:
    enabled: true
    algorithm: "token-bucket"
    capacity: 10000
    rate: 1000
    client-ip-enable: true

  services:
    chat:
      # Service-level rate limiting: Protect specific service
      rate-limit:
        enabled: true
        algorithm: "token-bucket"
        capacity: 1000
        rate: 100
        client-ip-enable: true

      instances:
        - name: "high-perf-model"
          base-url: "http://gpu-server:8080"
          # Instance-level rate limiting: Protect specific instance
          rate-limit:
            enabled: true
            algorithm: "token-bucket"
            capacity: 500
            rate: 50
            scope: "instance"

Rate Limiting Priority¶

Instance-Level Rate Limiting: Highest priority, directly protects instances
Service-Level Rate Limiting: Medium priority, protects service types
Global Rate Limiting: Lowest priority, protects the entire system

Dynamic Rate Limiting Configuration¶

Dynamic Adjustment via Configuration Files¶

JAiRouter also supports updating rate limiting configurations via the dynamic configuration API:

{
  "services": {
    "chat": {
      "rateLimit": {
        "algorithm": "token-bucket",
        "capacity": 200,
        "rate": 20,
        "clientIpEnable": true
      },
      "instances": [
        {
          "name": "model-1",
          "baseUrl": "http://server-1:8080",
          "rateLimit": {
            "algorithm": "token-bucket",
            "capacity": 100,
            "rate": 10
          }
        }
      ]
    }
  }
}

Rate Limiting Parameter Tuning¶

# Monitor current rate limiting effectiveness
curl "http://localhost:8080/api/rate-limiter/summary"

# Adjust configuration based on monitoring results
# If rejection rate is too high, increase capacity or rate
# If backend pressure is high, reduce capacity or rate

Rate Limiting Monitoring and Alerting¶

Monitoring Metrics¶

# Rate limiting events (allowed / denied)
curl "http://localhost:8080/actuator/metrics/jairouter_rate_limit_events_total"

# Remaining capacity per limiter
curl "http://localhost:8080/actuator/metrics/jairouter_rate_limit_remaining"

# Usage ratio per limiter
curl "http://localhost:8080/actuator/metrics/jairouter_rate_limit_usage_ratio"

Prometheus Metrics¶

# Rate limiting request rate
rate(jairouter_rate_limit_events_total[5m])

# Rate limiting rejection rate
rate(jairouter_rate_limit_events_total{result="denied"}[5m]) / rate(jairouter_rate_limit_events_total[5m])

# Remaining capacity per limiter
jairouter_rate_limit_remaining

# Usage ratio per limiter
jairouter_rate_limit_usage_ratio

Alert Rules¶

# Prometheus alert rules
groups:
  - name: jairouter_ratelimit
    rules:
      - alert: HighRateLimitRejection
        expr: rate(jairouter_rate_limit_events_total{result="denied"}[5m]) / rate(jairouter_rate_limit_events_total[5m]) > 0.1
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "High rate limiting rejection rate"
          description: "Service {{ $labels.service }} rejection rate exceeds 10%"

Performance Optimization¶

1. Memory Optimization¶

The client IP rate limiter cache uses Caffeine with automatic cleanup:

Maximum 10,000 entries
30-minute idle expiry (access-based)
Automatic eviction with statistics tracking

2. Algorithm Selection Optimization¶

# High concurrency scenario: Choose the best performing algorithm
model:
  services:
    high-traffic:
      rate-limit:
        algorithm: "token-bucket"  # Best performance

# Precise control scenario: Choose the most accurate algorithm
model:
  services:
    precise-control:
      rate-limit:
        algorithm: "sliding-window"  # Highest accuracy

Practical Use Cases¶

Case 1: API Gateway Rate Limiting¶

# Multi-layer rate limiting protection
model:
  # Global rate limiting: Protect the entire gateway
  rate-limit:
    enabled: true
    algorithm: "token-bucket"
    capacity: 10000
    rate: 1000
    client-ip-enable: true

  services:
    # Chat service: High frequency usage
    chat:
      rate-limit:
        algorithm: "token-bucket"
        capacity: 5000
        rate: 500
        client-ip-enable: true

    # Image generation: Resource intensive
    image-generation:
      rate-limit:
        algorithm: "leaky-bucket"
        capacity: 100
        rate: 10
        client-ip-enable: true

Case 2: Anti-Scraping Protection¶

# Prevent malicious interface scraping
model:
  services:
    sensitive-api:
      rate-limit:
        enabled: true
        algorithm: "sliding-window"
        capacity: 50           # Bucket capacity
        rate: 50               # Maximum 50 requests per second
        client-ip-enable: true

Case 3: Service Warm-up¶

# New service online warm-up
model:
  services:
    new-service:
      rate-limit:
        enabled: true
        algorithm: "warm-up"
        capacity: 1000
        rate: 100
        warm-up-period: 600     # 10-minute warm-up

Troubleshooting¶

Common Issues¶

Rate Limiting Too Strict

# Check rejection rate
curl "http://localhost:8080/api/rate-limiter/summary"

# Solution: Increase capacity or rate

High Memory Usage

# Check limiter count
curl "http://localhost:8080/api/rate-limiter/status"

# Solution: Adjust cleanup strategy

Rate Limiting Not Working

# Check if configuration is loaded correctly
curl "http://localhost:8080/api/config/instance/type/chat"

# Check if rate limiting is enabled

Debugging Commands¶

# View rate limiting configuration
curl "http://localhost:8080/actuator/configprops" | jq '.model.rate-limit'

# View rate limiting metrics
curl "http://localhost:8080/actuator/metrics" | grep ratelimit

# Test rate limiting effect
for i in {1..20}; do
  curl -w "%{http_code}\n" "http://localhost:8080/v1/chat/completions" \
    -H "Content-Type: application/json" \
    -d '{"model": "test", "messages": [{"role": "user", "content": "test"}]}'
done

Next Steps¶

After completing rate limiting configuration, you can continue to learn about:

Circuit Breaker Configuration - Configure fault protection mechanisms
Monitoring Guide - Set up rate limiting monitoring and alerts
Troubleshooting - Diagnose and resolve rate limiting issues
Performance Tuning - Optimize rate limiting performance