https://guijiagi.com/tags/tokenizer/Recent content in Tokenizer on 硅基 AGI · 智能体学习与测评Hugozh-cn本站内容采用 CC BY-NC-SA 4.0 国际许可协议授权Mon, 13 Jul 2026 00:40:00 +0800https://guijiagi.com/posts/article-47/Mon, 13 Jul 2026 00:40:00 +0800https://guijiagi.com/posts/article-47/从数据不平衡到跨语言迁移，多语言大模型的_core技术挑战与前沿解法https://guijiagi.com/posts/b2-42dd7e26/Mon, 13 Jul 2026 00:40:00 +0800https://guijiagi.com/posts/b2-42dd7e26/从数据不平衡到跨语言迁移，多语言大模型的_core技术挑战与前沿解法https://guijiagi.com/posts/tokenizer-comparison-2026/Thu, 02 Jul 2026 11:11:00 +0800https://guijiagi.com/posts/tokenizer-comparison-2026/全面对比BPE、WordPiece、Unigram和SentencePiece分词算法的原理与工程取舍https://guijiagi.com/posts/tokenizer-comparison/Thu, 25 Jun 2026 12:00:00 +0800https://guijiagi.com/posts/tokenizer-comparison/深入对比四种主流分词算法的原理、训练方法、优缺点及工程实现，涵盖 BPE、WordPiece、Unigram Language Model 和 SentencePiece。https://guijiagi.com/posts/tokenizer-special-tokens/Thu, 25 Jun 2026 12:00:00 +0800https://guijiagi.com/posts/tokenizer-special-tokens/梳理大语言模型特殊标记（Special Tokens）的设计演进，从 BERT 到 ChatML，再到现代模型的统一 Prompt 格式。https://guijiagi.com/posts/tokenizer-internals/Thu, 25 Jun 2026 10:00:00 +0800https://guijiagi.com/posts/tokenizer-internals/深入解析主流分词算法的原理、训练过程与工程实现