Tag: math
Cross-entropy
2026-01-12 basics math theory training Entropy is the negative logarithm of probability, averaged over all outcomes. Cross-entropy is a similar calculation, involving logs of probabilities from one distribution averaged over a different distribution. These concepts form an excuse for reading Claude Shannon's classic paper A Mathematical Theory of Communication; and cross-entropy in particular is the most popular loss function for language model training. Access: $$$ Pro
Optimization with Adam
2025-12-15 basics math theory training Training consists of finding the parameters for a model that will give the lowest possible value of the loss function. How do we actually do that, and do it efficiently? The Adam algorithm, from 2015, is one way, and still popular today. Access: $$$ Pro
Eigenvectors and Eigenfaces
2025-12-08 applications basics image math theory video Introduction to eigenvectors, which abstract the concept of an "axis" along or around which one might scale or rotate things. Illustrated by a 1991 paper on "eigenfaces," which applies this concept to recognizing faces in images. Access: $ Basic
Quick look: Injective LLMs
2025-11-05 math prompting sampling text theory meta Brief thoughts on the "Injective and invertible LLMs" paper that is making the rounds. My general view on it is negative. Access: Free account (logged in)
Automatic differentiation
2025-11-03 basics math theory training Training a machine learning model is one case of the larger class of "optimization" problems; to solve it, you need to calculate how the output (i.e. the loss) changes in relation to inputs (such as weights). I introduce the calculus topic of the derivative, and discuss how to calculate the derivative of a piece of software by augmenting the compiler or interpreter to do it during execution. Access: $ Basic
Linear algebra intro
2025-10-06 basics theory math Introduction to basic concepts that are useful in reading papers: the meaning and purpose of mathematics; vectors; dot products; and matrices. Access: $ Basic