Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.
Install
npx skills add davila7/claude-code-templates --skill sentencepiecenpx skills add davila7/claude-code-templates --skill sentencepiece -a claude-code -gnpx skills add davila7/claude-code-templates --skill sentencepiece -a codex -ggemini skills install https://github.com/davila7/claude-code-templates.git --path cli-tool/components/skills/ai-research/tokenization-sentencepiecegh skill install davila7/claude-code-templates sentencepiecegit clone --depth 1 https://github.com/davila7/claude-code-templates /tmp/davila7__claude-code-templates
mkdir -p ~/.claude/skills ~/.agents/skills
cp -R /tmp/davila7__claude-code-templates/cli-tool/components/skills/ai-research/tokenization-sentencepiece ~/.claude/skills/sentencepiece # Claude Code
cp -R /tmp/davila7__claude-code-templates/cli-tool/components/skills/ai-research/tokenization-sentencepiece ~/.agents/skills/sentencepiece # Codex, Cursor, Gemini CLI, Copilot, OpenCode