Agent可观测性：你无法优化你看不见的东西 传统软件的可观测性已经相当成熟——我们有Prometheus做指标、ELK做日志、Jaeger做追踪。但Agent系统引入了新的可观测性挑战：非确定性的执行路径、不可预测的token消耗、LLM输出的质量评估、多Agent协作的链路追踪。2026年，Agent可观测性已经形成了一套独立的最佳实践。
Agent可观测性的三大支柱 ┌───────────────────────────────────────────────────┐ │ Agent 可观测性三大支柱 │ ├─────────────┬─────────────┬───────────────────────┤ │ │ │ │ │ 追踪 │ 日志 │ 指标 │ │ (Traces) │ (Logs) │ (Metrics) │ │ │ │ │ │ Agent执行 │ 结构化事件 │ 性能计数器 │ │ 链路追踪 │ 决策记录 │ 资源消耗 │ │ 跨Agent关联 │ 上下文快照 │ 质量评估 │ │ │ │ │ └─────────────┴─────────────┴───────────────────────┘ 1. 分布式追踪 Agent Trace模型 Agent系统的追踪比传统微服务更复杂，因为一个"请求"可能涉及多轮LLM调用、多次工具调用、跨多个Agent。
from dataclasses import dataclass, field from datetime import datetime from typing import Any, Optional import uuid @dataclass class AgentSpan: """Agent追踪的基本单元""" span_id: str = field(default_factory=lambda: str(uuid.uuid4())) parent_id: Optional[str] = None trace_id: str = "" # 基本信息 name: str = "" # span名称 span_type: str = "" # llm_call / tool_call / agent_step / sub_agent agent_name: str = "" # 哪个Agent start_time: datetime = field(default_factory=datetime.now) end_time: Optional[datetime] = None # Agent特有信息 input_data: Any = None output_data: Any = None model: str = "" # 使用的LLM模型 prompt_tokens: int = 0 completion_tokens: int = 0 cost: float = 0.0 # 状态 status: str = "ok" # ok / error / timeout error_message: Optional[str] = None # 元数据 attributes: dict = field(default_factory=dict) def finish(self, output=None, status="ok", error=None): self.end_time = datetime.now() self.output_data = output self.status = status self.error_message = error @property def duration_ms(self) -> float: if self.end_time: return (self.end_time - self.start_time).total_seconds() * 1000 return 0 追踪实现 class AgentTracer: """Agent追踪器""" def __init__(self, service_name="agent-service"): self.service_name = service_name self.spans: list[AgentSpan] = [] self.current_span_stack: list[AgentSpan] = [] def start_trace(self, name: str, agent_name: str) -> AgentSpan: """开始一个新的追踪（根span）""" trace_id = str(uuid.uuid4()) span = AgentSpan( trace_id=trace_id, name=name, span_type="agent_step", agent_name=agent_name ) self.spans.append(span) self.current_span_stack.append(span) return span def start_span( self, name: str, span_type: str, agent_name: str = "", input_data: Any = None ) -> AgentSpan: """开始一个子span""" parent = self.current_span_stack[-1] if self.current_span_stack else None span = AgentSpan( trace_id=parent.trace_id if parent else str(uuid.uuid4()), parent_id=parent.span_id if parent else None, name=name, span_type=span_type, agent_name=agent_name, input_data=input_data ) self.spans.append(span) self.current_span_stack.append(span) return span def end_span(self, span: AgentSpan, output=None, status="ok", error=None): """结束一个span""" span.finish(output, status, error) if self.current_span_stack and self.current_span_stack[-1].span_id == span.span_id: self.current_span_stack.pop() def get_trace_tree(self, trace_id: str) -> dict: """获取追踪树""" trace_spans = [s for s in self.spans if s.trace_id == trace_id] return self._build_tree(trace_spans, parent_id=None) def _build_tree(self, spans: list[AgentSpan], parent_id: str | None) -> dict: children = [s for s in spans if s.parent_id == parent_id] return [ { "span_id": s.span_id, "name": s.name, "type": s.span_type, "agent": s.agent_name, "duration_ms": s.duration_ms, "tokens": s.prompt_tokens + s.completion_tokens, "cost": s.cost, "status": s.status, "children": self._build_tree(spans, s.span_id) } for s in children ] 使用示例 tracer = AgentTracer() # 开始追踪 root = tracer.start_trace("用户咨询", "router_agent") # Agent执行LLM调用 llm_span = tracer.start_span("LLM调用", "llm_call", "router_agent", input_data="用户问题") response = await llm.complete("...") tracer.end_span(llm_span, output=response.text, status="ok") llm_span.prompt_tokens = response.usage.prompt_tokens llm_span.completion_tokens = response.usage.completion_tokens llm_span.cost = calculate_cost(response.usage, "gpt-4o") # Agent调用工具 tool_span = tracer.start_span("搜索知识库", "tool_call", "router_agent") results = await knowledge_base.search("query") tracer.end_span(tool_span, output=results) # 路由到子Agent sub_agent_span = tracer.start_span("专家Agent处理", "sub_agent", "expert_agent") # ... 子Agent内部会有自己的span ... tracer.end_span(sub_agent_span, output="最终回答") # 结束追踪 tracer.end_span(root, output="最终回答") # 查看追踪树 trace_tree = tracer.get_trace_tree(root.trace_id) 追踪可视化 追踪树可以渲染为瀑布图：
...
