Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.
安装
npx skills add Orchestra-Research/AI-Research-SKILLs --skill huggingface-tokenizersnpx skills add Orchestra-Research/AI-Research-SKILLs --skill huggingface-tokenizers -a claude-code -gnpx skills add Orchestra-Research/AI-Research-SKILLs --skill huggingface-tokenizers -a codex -ggemini skills install https://github.com/Orchestra-Research/AI-Research-SKILLs.git --path 02-tokenization/huggingface-tokenizersgh skill install Orchestra-Research/AI-Research-SKILLs huggingface-tokenizersgit clone --depth 1 https://github.com/Orchestra-Research/AI-Research-SKILLs /tmp/Orchestra-Research__AI-Research-SKILLs
mkdir -p ~/.claude/skills ~/.agents/skills
cp -R /tmp/Orchestra-Research__AI-Research-SKILLs/02-tokenization/huggingface-tokenizers ~/.claude/skills/huggingface-tokenizers # Claude Code
cp -R /tmp/Orchestra-Research__AI-Research-SKILLs/02-tokenization/huggingface-tokenizers ~/.agents/skills/huggingface-tokenizers # Codex, Cursor, Gemini CLI, Copilot, OpenCode