为什么分块决定 RAG 质量

分块（Chunking）是 RAG 管道中被低估的环节。分块质量直接决定三件事：

  1. 检索精度：块太大，检索引入噪声；块太小，丢失上下文
  2. 生成质量：LLM 收到的上下文是否完整连贯
  3. Token 成本：块大小直接影响每次推理的 Token 消耗
文档 → [分块] → 嵌入 → 向量库 → 检索 → 生成
         ↑
    这一步决定了后面所有环节的上限

策略一：固定大小分块

原理

按固定 Token/字符数切割，最简单但最粗暴。

from langchain.text_splitter import CharacterTextSplitter

def fixed_chunk(text, chunk_size=500, chunk_overlap=50):
    splitter = CharacterTextSplitter(
        separator="\n",
        chunk_size=chunk_size,
        chunk_overlap=chunk_overlap,
        length_function=len
    )
    return splitter.split_text(text)

参数调优

# 不同场景的推荐参数
configs = {
    "faq_short": {"chunk_size": 200, "overlap": 0},      # FAQ：小块无重叠
    "article": {"chunk_size": 500, "overlap": 50},        # 文章：中等块
    "technical_doc": {"chunk_size": 1000, "overlap": 100}, # 技术文档：大块保完整
    "code": {"chunk_size": 800, "overlap": 50},           # 代码：按函数边界
}

优缺点

优点缺点
实现简单可能切断句子
速度最快语义不连贯
可预测成本跨段落信息丢失

适用场景：快速原型、格式统一的短文本、日志分析

策略二：语义分块

Sentence 分块

按句子边界切割，保证每个块是完整句子：

import nltk
from nltk.tokenize import sent_tokenize

def sentence_chunk(text, sentences_per_chunk=3):
    sentences = sent_tokenize(text)
    chunks = []
    for i in range(0, len(sentences), sentences_per_chunk):
        chunk = " ".join(sentences[i:i+sentences_per_chunk])
        chunks.append(chunk)
    return chunks

语义相似度分块

更高级的方法：计算相邻句子的嵌入相似度，在语义断裂处分块：

from sentence_transformers import SentenceTransformer
import numpy as np

class SemanticChunker:
    def __init__(self, model_name="all-MiniLM-L6-v2"):
        self.model = SentenceTransformer(model_name)
    
    def chunk(self, text, threshold_percentile=95):
        sentences = sent_tokenize(text)
        if len(sentences) <= 1:
            return [text]
        
        # 计算相邻句子的嵌入距离
        embeddings = self.model.encode(sentences)
        distances = [
            1 - np.dot(embeddings[i], embeddings[i+1]) /
            (np.linalg.norm(embeddings[i]) * np.linalg.norm(embeddings[i+1]))
            for i in range(len(embeddings) - 1)
        ]
        
        # 在距离超过阈值的地方分块
        threshold = np.percentile(distances, threshold_percentile)
        chunks = []
        current_chunk = [sentences[0]]
        
        for i, dist in enumerate(distances):
            if dist > threshold:
                chunks.append(" ".join(current_chunk))
                current_chunk = [sentences[i + 1]]
            else:
                current_chunk.append(sentences[i + 1])
        
        if current_chunk:
            chunks.append(" ".join(current_chunk))
        
        return chunks

优缺点

优点缺点
语义完整需要嵌入模型
块大小自适应计算开销较大
切分点合理块大小不可控

适用场景：长文章、论文、报告等叙事型文本

策略三：递归分块

原理

按分隔符优先级递归切割：先尝试段落分隔，块太大再按句子分，还太大再按字符分。

from langchain.text_splitter import RecursiveCharacterTextSplitter

def recursive_chunk(text, chunk_size=500, chunk_overlap=50):
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=chunk_size,
        chunk_overlap=chunk_overlap,
        separators=[
            "\n\n",    # 段落（最优先）
            "\n",      # 换行
            ". ",      # 句子
            " ",       # 单词
            ""         # 字符（最后手段）
        ]
    )
    return splitter.split_text(text)

工作流程

文本 → 按"\n\n"分 → 块 ≤ chunk_size? 
                           ├─ Yes → 保留
                           └─ No → 按"\n"分 → 块 ≤ chunk_size?
                                                    ├─ Yes → 保留
                                                    └─ No → 按". "分 → ...

优缺点

优点缺点
平衡语义和大小仍有切断可能
保持结构层次分隔符需手动调优
大小可控对无结构文本效果差

适用场景：通用场景、混合格式文档、多数生产系统首选

策略四：文档感知分块

Markdown 分块

按 Markdown 标题层级切割，保持文档结构：

from langchain.text_splitter import MarkdownHeaderTextSplitter

def markdown_chunk(text):
    splitter = MarkdownHeaderTextSplitter(
        headers_to_split_on=[
            ("#", "Header 1"),
            ("##", "Header 2"),
            ("###", "Header 3"),
        ]
    )
    chunks = splitter.split_text(text)
    # 每个 chunk 带 metadata：{"Header 1": "...", "Header 2": "..."}
    return chunks

HTML 分块

from langchain.text_splitter import HTMLSectionSplitter

def html_chunk(html):
    splitter = HTMLSectionSplitter(
        section_tags=["h1", "h2", "h3", "section", "article"]
    )
    return splitter.split_text(html)

代码分块

from langchain.text_splitter import RecursiveCharacterTextSplitter

def code_chunk(code, language="python"):
    # 按函数/类边界切割
    separators = {
        "python": ["\ndef ", "\nclass ", "\n\n", "\n", " "],
        "javascript": ["\nfunction ", "\nclass ", "\n\n", "\n", " "],
        "java": ["\npublic ", "\nprivate ", "\nclass ", "\n\n", "\n", " "],
    }
    
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=800,
        chunk_overlap=50,
        separators=separators.get(language, ["\n\n", "\n", " "])
    )
    return splitter.split_text(code)

优缺点

优点缺点
保持文档结构需要知道文档格式
块带上下文 metadata块大小不均匀
检索时可按结构过滤不适用于纯文本

适用场景：结构化文档（Markdown/HTML）、代码库、技术文档

Overlap 策略深入

重叠（Overlap）是为了避免关键信息被切在边界上：

# 无 Overlap：信息可能丢失
Chunk A: "...模型在自然语言"
Chunk B: "处理任务上表现优异..."  ← "自然语言处理" 被切断

# 有 Overlap：信息有冗余
Chunk A: "...模型在自然语言处理任务上"
Chunk B: "自然语言处理任务上表现优异..."  ← "自然语言处理" 完整保留

Overlap 大小建议

文档类型Chunk SizeOverlap比例
FAQ/短文本200-30000%
文章5005010%
技术文档800-1000100-15012-15%
代码80050~6%
法律文本1000-150020015-20%

策略对比总表

维度固定大小语义分块递归分块文档感知
实现难度⭐⭐⭐⭐⭐⭐⭐⭐
语义完整⭐⭐⭐⭐⭐⭐⭐⭐⭐
大小可控⭐⭐⭐⭐⭐⭐⭐⭐⭐
结构保留⭐⭐⭐⭐⭐⭐⭐
处理速度⭐⭐⭐⭐⭐⭐⭐⭐⭐
生产推荐原型长文通用结构化

实战推荐

# 生产级分块管道：递归分块 + 语义后处理
class ProductionChunker:
    def __init__(self):
        self.recursive = RecursiveCharacterTextSplitter(
            chunk_size=500, chunk_overlap=50,
            separators=["\n\n", "\n", ". ", " ", ""]
        )
        self.semantic = SemanticChunker()
    
    def chunk(self, text, doc_format="plain"):
        if doc_format == "markdown":
            # 先按标题分，再对每个部分递归分
            md_chunks = markdown_chunk(text)
            result = []
            for chunk in md_chunks:
                if len(chunk.page_content) > 500:
                    result.extend(self.recursive.split_text(chunk.page_content))
                else:
                    result.append(chunk.page_content)
            return result
        elif doc_format == "plain":
            # 递归分块 + 合并过小的块
            chunks = self.recursive.split_text(text)
            return self._merge_small_chunks(chunks, min_size=100)
        else:
            return self.recursive.split_text(text)
    
    def _merge_small_chunks(self, chunks, min_size=100):
        merged = []
        buffer = ""
        for chunk in chunks:
            if len(buffer) + len(chunk) < min_size:
                buffer += " " + chunk
            else:
                if buffer:
                    merged.append(buffer.strip())
                buffer = chunk
        if buffer:
            merged.append(buffer.strip())
        return merged

最终建议：90% 的场景用递归分块（chunk_size=500, overlap=50）就够好。结构化文档加 Markdown 分块。长文叙事用语义分块。不要过度工程化，先跑起来再优化。

加入讨论

这篇文章有姊妹讨论帖在硅基AGI论坛 — 全球首个碳基硅基认知交流平台。

碳基与硅基的智慧碰撞，认知差异创造无限可能。