Tag: AIAYN
Mamba #1
2026-04-27 Mamba AIAYN audio model-intro text State-space models represent a thread of statistical modelling other than attention, often used for continuous domains like audio. This paper introduces Mamba, a model architecture where attention is replaced by state-space layers in a model aimed at language. Access: $ Basic
Linear attention
2026-04-20 AIAYN attention image math theory The simplest kind of attention mechanism for transformers consumes space and computation quadratic in the context window, and that limits how large the context window can be. This paper looks at modifying the attention mechanism to reduce that bound to linear, making much larger windows reasonable. Access: $ Basic
Vision Transformers
2025-12-01 AIAYN BERT attention basics image What if we applied the "attention is all you need" architecture to images instead of language? That's the question considered in this paper from 2021, which laid the groundwork for today's multi-modal models. Access: $ Basic
Rotary Position Encoding
2025-08-18 basics text AIAYN tokenization I review position encoding - why it's needed, and how classic Transformers do it - and then go in detail into the Rotary Positioning Embedding (RoPE) enhancement to position encoding. RoPE is widely used in recent large language models. Access: $ Basic
Grammar is all you get
2025-08-01 model-intro basics text AIAYN attention An overview of the classic "Attention is all you need" paper, with focus on the attention mechanism and its resemblance to dependency grammar. Access: $ Basic