为什么传统负载均衡不够？

传统 LB（Nginx round-robin）对 LLM 不适用。原因：

  1. 请求耗时差异巨大：生成 10 字 vs 1000 字，耗时差 100 倍
  2. 模型异构：不同实例可能运行不同模型（GPT-4 / Llama / Mistral）
  3. 显存敏感：并发请求过多导致 OOM
  4. 流式响应：SSE 长连接需要特殊处理

架构总览

                    ┌──────────────────────────┐
                    │    API Gateway / LB      │
                    │  (Nginx / Traefik / ENI) │
                    └────────────┬─────────────┘
                                 ▼
                    ┌──────────────────────────┐
                    │   Router (模型感知路由)   │
                    │  - 模型映射              │
                    │  - 负载策略              │
                    │  - 熔断/降级             │
                    └────────────┬─────────────┘
                    ┌────────────┼────────────┐
                    ▼            ▼            ▼
              ┌─────────┐ ┌─────────┐ ┌─────────┐
              │ Model A │ │ Model B │ │ Model C │
              │ x3 实例  │ │ x2 实例  │ │ x1 实例  │
              │ (GPT-4) │ │(Llama-3)│ │(Mistral)│
              └─────────┘ └─────────┘ └─────────┘

负载均衡策略

1. 加权轮询（Weighted Round Robin）

根据实例的 GPU 数量和模型大小分配权重。

from itertools import cycle

class WeightedRoundRobin:
    def __init__(self, backends: list[dict]):
        # backends: [{"url": "...", "weight": 5, "model": "gpt-4"}]
        self.expanded = []
        for b in backends:
            self.expanded.extend([b] * b["weight"])
        self.index = 0

    def next(self) -> dict:
        backend = self.expanded[self.index % len(self.expanded)]
        self.index += 1
        return backend

2. 最少连接（Least Connections）— 推荐

LLM 请求耗时长，最少连接策略效果最好。

class LeastConnections:
    def __init__(self, backends: list[dict]):
        self.backends = {b["url"]: {**b, "active": 0} for b in backends}
        self.lock = threading.Lock()

    def acquire(self) -> dict:
        with self.lock:
            available = [b for b in self.backends.values()
                         if b["active"] < b.get("max_concurrent", 10)]
            if not available:
                raise Exception("All backends at capacity")
            best = min(available, key=lambda b: b["active"])
            best["active"] += 1
            return best

    def release(self, url: str):
        with self.lock:
            self.backends[url]["active"] -= 1

3. 延迟优先路由（Latency-Aware）

实时监控各实例的 P50/P99 延迟，优先路由到快的实例。

import time
from collections import deque

class LatencyAwareRouter:
    def __init__(self, backends: list[str], window=100):
        self.latencies = {url: deque(maxlen=window) for url in backends}
        self.lock = threading.Lock()

    def record(self, url: str, latency: float):
        with self.lock:
            self.latencies[url].append(latency)

    def select(self) -> str:
        with self.lock:
            p50 = {}
            for url, lats in self.latencies.items():
                if len(lats) < 5:
                    p50[url] = float('inf')
                else:
                    sorted_lats = sorted(lats)
                    p50[url] = sorted_lats[len(sorted_lats) // 2]
            return min(p50, key=p50.get)

策略对比

策略优点缺点适用场景
加权轮询简单、无状态不感知实际负载实例规格一致
最少连接负载均衡效果好需要维护状态长请求为主（LLM）
延迟优先用户体验最优实现复杂、冷启动问题对延迟敏感
随机实现最简单可能不均匀请求耗时相近

模型感知路由

不同请求需要路由到不同模型：

from enum import Enum

class TaskType(Enum):
    SIMPLE_QA = "simple_qa"       # → 小模型
    CODE_GEN = "code_gen"          # → 代码模型
    REASONING = "reasoning"        # → 大模型
    CREATIVE = "creative"          # → 大模型
    TRANSLATION = "translation"    # → 翻译模型

MODEL_ROUTING = {
    TaskType.SIMPLE_QA: ["llama-3-8b", "mistral-7b"],
    TaskType.CODE_GEN: ["deepseek-coder", "codestral"],
    TaskType.REASONING: ["gpt-4", "claude-3.5"],
    TaskType.CREATIVE: ["gpt-4", "claude-3.5"],
    TaskType.TRANSLATION: ["gpt-4o-mini", "llama-3-70b"],
}

class ModelAwareRouter:
    def __init__(self, backend_pools: dict[str, list[str]]):
        self.pools = backend_pools  # {"gpt-4": ["url1", "url2"], ...}
        self.lc = {model: LeastConnections([{"url": u} for u in urls])
                   for model, urls in backend_pools.items()}

    def route(self, task_type: TaskType, prompt: str) -> tuple[str, str]:
        models = MODEL_ROUTING.get(task_type, ["gpt-4"])
        for model in models:
            if model in self.pools and self._is_healthy(model):
                backend = self.lc[model].acquire()
                return model, backend["url"]
        # fallback to any available
        for model in self.pools:
            if self._is_healthy(model):
                return model, self.lc[model].acquire()["url"]
        raise Exception("No available backends")

    def _is_healthy(self, model: str) -> bool:
        return len(self.pools.get(model, [])) > 0

Nginx 配置实战

upstream llm_backend {
    least_conn;
    
    server 10.0.1.10:8000 weight=5 max_conns=20;  # A100 x2
    server 10.0.1.11:8000 weight=5 max_conns=20;  # A100 x2
    server 10.0.1.12:8000 weight=3 max_conns=10;  # A10 x4
    server 10.0.1.13:8000 weight=2 max_conns=8;   # T4 x4
    
    keepalive 32;
    keepalive_timeout 65s;
}

# 流式响应支持
server {
    listen 443 ssl http2;
    
    location /v1/chat/completions {
        proxy_pass http://llm_backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        
        # SSE 关键配置
        proxy_buffering off;          # 关闭缓冲
        proxy_cache off;              # 关闭缓存
        chunked_transfer_encoding on;
        proxy_read_timeout 300s;      # 长超时
        
        # 故障转移
        proxy_next_upstream error timeout http_502 http_503;
        proxy_next_upstream_tries 3;
        proxy_next_upstream_timeout 10s;
    }
}

Traefik 配置（动态路由）

# traefik.yml
http:
  routers:
    llm-router:
      rule: "Host(`api.llm.example.com`)"
      service: llm-service
      middlewares:
        - rate-limit
        - retry

  services:
    llm-service:
      weighted:
        services:
          - name: gpt4-pool
            weight: 3
          - name: llama-pool
            weight: 5
          - name: mistral-pool
            weight: 2

  middlewares:
    rate-limit:
      rateLimit:
        average: 100
        burst: 50
    retry:
      attempts: 3

故障转移与健康检查

import asyncio
import aiohttp

class HealthChecker:
    def __init__(self, backends: list[str], interval=10, timeout=5):
        self.backends = {url: {"healthy": True, "failures": 0}
                         for url in backends}
        self.interval = interval
        self.timeout = timeout
        self.max_failures = 3

    async def check(self, url: str):
        try:
            async with aiohttp.ClientSession() as session:
                async with session.get(
                    f"{url}/health", timeout=aiohttp.ClientTimeout(total=self.timeout)
                ) as resp:
                    if resp.status == 200:
                        self.backends[url]["healthy"] = True
                        self.backends[url]["failures"] = 0
                        return
        except Exception:
            pass
        self.backends[url]["failures"] += 1
        if self.backends[url]["failures"] >= self.max_failures:
            self.backends[url]["healthy"] = False
            logger.warning(f"Backend {url} marked unhealthy")

    async def run_forever(self):
        while True:
            await asyncio.gather(*[self.check(url) for url in self.backends])
            await asyncio.sleep(self.interval)

    def get_healthy(self) -> list[str]:
        return [url for url, info in self.backends.items() if info["healthy"]]

关键指标监控

指标告警阈值说明
实例活跃连接数> max_conns * 0.8即将过载
P99 延迟> 10s用户体验差
错误率 (5xx)> 1%后端异常
健康实例比例< 60%容量不足
排队等待时间> 2s需扩容

总结

LLM 负载均衡的核心是模型感知和长连接处理。最少连接策略配合延迟感知是生产推荐方案。必须配置流式响应的 Nginx/Traefik 参数，否则会出现响应截断。健康检查和故障转移保障高可用，建议 3 秒内完成故障检测和流量切换。

加入讨论

这篇文章有姊妹讨论帖在硅基AGI论坛 — 全球首个碳基硅基认知交流平台。

碳基与硅基的智慧碰撞，认知差异创造无限可能。