Prompt as Code：理念 代码有 Git，有 CI/CD，有 code review，有单元测试。Prompt 呢？大多数团队的 Prompt 管理方式相当于把代码写在记事本里，用文件名标记版本。
Prompt as Code 的核心主张：
Prompt 是代码，不是配置 Prompt 变更需要 review 和审批 Prompt 变更需要测试和验证 Prompt 需要版本回退能力 Prompt 需要线上监控和告警 Git 管理 Prompt 仓库结构 prompt-repo/ ├── prompts/ │ ├── customer-service/ │ │ ├── intent-classification.yaml │ │ ├── response-generation.yaml │ │ └── escalation.yaml │ ├── data-analysis/ │ │ ├── sql-generation.yaml │ │ └── insight-summary.yaml │ └── _shared/ │ ├── system-prompts.yaml │ └── safety-rules.yaml ├── tests/ │ ├── golden-sets/ │ │ ├── customer-service-golden.jsonl │ │ └── data-analysis-golden.jsonl │ └── regression/ │ └── test_regression.py ├── eval/ │ ├── evaluators.py │ └── metrics.py ├── .promptlab.yaml # 工具配置 └── CHANGELOG.md Prompt 文件规范 # prompts/customer-service/intent-classification.yaml id: cs-intent-classification name: "客服意图分类" version: "2.3.1" author: "team-cs" status: production # draft | staging | production | archived variables: - name: user_message type: string required: true - name: context type: string required: false default: "" model: provider: openai name: gpt-4o temperature: 0.1 max_tokens: 256 template: | 系统：你是客服意图分类器。将用户消息分类为以下意图之一： [退款, 咨询, 投诉, 修改订单, 技术支持, 其他] {% if context %}上下文：{{context}}{% endif %} 用户消息：{{user_message}} 只输出意图类别，不要输出其他内容。 test_cases: - input: {user_message: "我要退货"} expected: "退款" - input: {user_message: "怎么使用优惠券"} expected: "咨询" metrics: - accuracy >= 0.95 - latency_p95 < 500ms - token_usage < 100 Git 工作流 # 创建 Prompt 变更分支 git checkout -b prompt/cs-intent-v2.4 # 修改 Prompt 后提交 git add prompts/customer-service/intent-classification.yaml git commit -m "feat(cs): 优化意图分类 Prompt，增加技术支持子类 - 新增 3 个 few-shot 示例覆盖技术支持场景 - 调整 temperature 0.2 → 0.1 减少随机性 - 黄金集准确率 92.3% → 96.1% - Closes #142" # CI 自动跑回归测试 git push origin prompt/cs-intent-v2.4 A/B 测试框架 架构 class PromptABTest: def __init__(self, config): self.control = config['control'] # 当前生产版本 self.treatment = config['treatment'] # 候选版本 self.split_ratio = config.get('split', 0.1) # 10% 流量到 treatment self.metrics = config['metrics'] def route(self, request_id, user_id): """决定使用哪个 Prompt 版本""" bucket = hash(f"{user_id}:{self.experiment_id}") % 100 if bucket < self.split_ratio * 100: return self.treatment return self.control def evaluate(self): """评估 A/B 测试结果""" control_results = collect_metrics(self.control) treatment_results = collect_metrics(self.treatment) return { 'control': control_results, 'treatment': treatment_results, 'significance': t_test( control_results['scores'], treatment_results['scores'] ), 'recommendation': self._recommend( control_results, treatment_results ) } def _recommend(self, control, treatment): if treatment['accuracy'] - control['accuracy'] < 0.02: return "no_significant_improvement" if treatment['cost_per_call'] > control['cost_per_call'] * 1.2: return "improvement_but_cost_prohibitive" if treatment['latency_p95'] > 2000: return "improvement_but_latency_too_high" return "promote_to_production" 流量分配 ┌──────────────────┐ │ 用户请求进入 │ └────────┬─────────┘ │ ┌────────▼─────────┐ │ Hash(user_id) │ │ % 100 │ └────────┬─────────┘ ┌─────┴─────┐ │ │ 90% ▼ 10% ▼ ┌──────────┐ ┌──────────┐ │ Control │ │ Treatment│ │ v2.3.1 │ │ v2.4.0 │ └────┬─────┘ └────┬─────┘ │ │ ┌────▼───────────────▼────┐ │ 指标收集 & 对比分析 │ └─────────────────────────┘ 回归测试 黄金集构建 def build_golden_set(production_logs, n=200): """从生产日志中采样构建黄金集""" # 1. 采样 samples = stratified_sample(production_logs, n) # 2. 人工标注/确认 golden = [] for sample in samples: golden.append({ 'input': sample.input, 'expected_output': sample.human_verified_output, 'min_quality_score': 0.85, 'category': sample.category }) return golden 回归测试执行 class PromptRegressionTest: def __init__(self, prompt_template, golden_set, evaluator): self.template = prompt_template self.golden_set = golden_set self.evaluator = evaluator def run(self, model_config): results = [] for case in self.golden_set: prompt = self.template.render(**case['input']) output = llm_call(prompt, **model_config) score = self.evaluator(output, case['expected_output']) results.append({ 'case_id': case.get('id'), 'score': score, 'passed': score >= case['min_quality_score'], 'output': output, 'expected': case['expected_output'] }) passed = sum(r['passed'] for r in results) total = len(results) return { 'pass_rate': passed / total, 'avg_score': sum(r['score'] for r in results) / total, 'failures': [r for r in results if not r['passed']], 'details': results } CI/CD 集成 # .github/workflows/prompt-ci.yml name: Prompt CI on: pull_request: paths: ['prompts/**'] jobs: regression-test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Install dependencies run: pip install promptfoo langsmith - name: Run regression tests run: | promptfoo eval \ --prompts prompts/customer-service/ \ --tests tests/golden-sets/customer-service-golden.jsonl \ --threshold 0.95 \ --output results.json - name: Check for regressions run: | python scripts/check_regression.py results.json # 如果准确率下降超过 2%，CI 失败 - name: Upload results if: always() uses: actions/upload-artifact@v4 with: name: prompt-test-results path: results.json 线上监控 监控指标 指标 类型 告警阈值 准确率 质量 < 基线 5% 延迟 P95 性能 > 2000ms Token 使用量 成本 > 预算 120% 安全拦截率 安全 > 1% 用户反馈率 满意度 差评 > 10% 空响应率 异常 > 0.5% class PromptMonitor: def __init__(self, prompt_id, version): self.prompt_id = prompt_id self.version = version self.baselines = load_baselines(prompt_id, version) def check(self, metrics): alerts = [] if metrics['accuracy'] < self.baselines['accuracy'] - 0.05: alerts.append({ 'level': 'critical', 'metric': 'accuracy', 'value': metrics['accuracy'], 'baseline': self.baselines['accuracy'], 'action': '考虑回退到上一版本' }) if metrics['latency_p95'] > 2000: alerts.append({ 'level': 'warning', 'metric': 'latency_p95', 'value': metrics['latency_p95'], 'action': '检查模型负载或简化 Prompt' }) if metrics['token_usage_avg'] > self.baselines['token_usage'] * 1.2: alerts.append({ 'level': 'warning', 'metric': 'cost', 'value': metrics['token_usage_avg'], 'action': '优化 Prompt 长度' }) return alerts 工具链：PromptHub 与 LangSmith PromptHub 功能矩阵： ├── Prompt 仓库（版本化存储） ├── 权限管理（RBAC：编辑/审批/部署） ├── 审批工作流（draft → review → staging → production） ├── 在线编辑器（实时预览 + 变量注入测试） ├── A/B 测试管理（实验配置 + 流量分配） └── 审计日志（谁在什么时候改了什么） LangSmith 集成 from langsmith import Client client = Client() # 创建 Prompt 版本 client.create_prompt( name="cs-intent-classification", prompt=template_body, metadata={ "version": "2.4.0", "author": "team-cs", "change_type": "minor" } ) # 线上追踪 @client.trace def classify_intent(user_message): prompt = load_prompt("cs-intent-classification", "2.4.0") response = llm_call(prompt.render(user_message=user_message)) client.record_evaluation( run_id=run.id, key="intent_correct", score=1 if response in VALID_INTENTS else 0 ) return response 实践路线图 阶段一（1-2 周）： Prompt 文件化 + Git 管理
...
