Tag: tokenization
Yi
2026-06-08 model-intro text tokenization vision upscaling Overview of the Yi open-weights language models from 01.AI, a Chinese startup which seems to have been inactive since 2024. Access: Free account
Huffman to Byte Pair
2025-11-24 basics text tokenization Introduction to two data compression concepts, one of which is commonly used for LLM input: Huffman and byte pair encoding. Access: Free account
Rotary Position Encoding
2025-08-18 basics text AIAYN tokenization I review position encoding - why it's needed, and how classic Transformers do it - and then go in detail into the Rotary Positioning Embedding (RoPE) enhancement to position encoding. RoPE is widely used in recent large language models. Access: $ Basic
Better (than) tokenization with BLTs
2025-08-01 theory text LLaMA tokenization Using "patches" of input bytes, instead of a fixed token list, allows better scalability and improves performance on some tasks that are hard for token-based LLMs. Access: $ Basic