https://guijiagi.com/tags/bpe/Recent content in BPE on 硅基 AGI · 智能体学习与测评Hugozh-cn本站内容采用 CC BY-NC-SA 4.0 国际许可协议授权Thu, 02 Jul 2026 11:11:00 +0800https://guijiagi.com/posts/tokenizer-comparison-2026/Thu, 02 Jul 2026 11:11:00 +0800https://guijiagi.com/posts/tokenizer-comparison-2026/全面对比BPE、WordPiece、Unigram和SentencePiece分词算法的原理与工程取舍https://guijiagi.com/posts/tokenizer-principles-and-practice/Sun, 28 Jun 2026 11:03:00 +0800https://guijiagi.com/posts/tokenizer-principles-and-practice/深入解析BPE、WordPiece、Unigram、SentencePiece四种分词算法的原理、优缺点与2026年工程实践https://guijiagi.com/posts/tokenizer-comparison/Thu, 25 Jun 2026 12:00:00 +0800https://guijiagi.com/posts/tokenizer-comparison/深入对比四种主流分词算法的原理、训练方法、优缺点及工程实现，涵盖 BPE、WordPiece、Unigram Language Model 和 SentencePiece。https://guijiagi.com/posts/tokenizer-internals/Thu, 25 Jun 2026 10:00:00 +0800https://guijiagi.com/posts/tokenizer-internals/深入解析主流分词算法的原理、训练过程与工程实现https://guijiagi.com/posts/tokenization-guide/Wed, 24 Jun 2026 15:00:00 +0800https://guijiagi.com/posts/tokenization-guide/从字符级到子词级分词，深入解析 BPE、WordPiece、SentencePiece 算法及工程实践