GPT-2 (2019) Paper Notes
Paper notes on GPT-2 covering its core ideas: decoder-only Transformer scaling, WebText, next-token prediction, zero-shot task transfer, and the staged release controversy.
Paper-reading
BERT (2018) Paper Notes
Paper notes covering the core ideas of BERT: the bidirectional Transformer encoder, masked language model, next sentence prediction, and the fine-tuning paradigm.
LLM Architecture Basics: Encoder-only vs. Decoder-only
A comparison of encoder-only and decoder-only architectures that distinguish the BERT and GPT families.
LLM Basics: RNNs and Sequential Processing
Explains how RNNs — the dominant architecture before Transformers — process sequences token by token, and the fundamental limitations that motivated moving beyond them.
LLM Learning Basics: Masked Language Model
Explains BERT's core training objective — the Masked Language Model — with formulas, commentary, and examples.
LLM Learning Basics: Pre-training and Fine-tuning
Explains large-scale pre-training and task-specific fine-tuning through the lens of the BERT workflow.
Transformer Basics: Encoder and Decoder
A clear explanation of what the Transformer encoder and decoder each do, grounded in the original architecture and illustrated with simple examples.
Attention Is All You Need (2017) — Paper Notes
A reading note on the Transformer paper — the core ideas, why it mattered, and what to read next.
LLM Learning Basics: cross-entropy and perplexity
Explains cross-entropy and perplexity — the metrics used to measure how wrong a model is — with formulas, commentary, and examples.
LLM Math Basics 2: Softmax and Probability Interpretation
A walkthrough of how softmax converts raw scores into probability-like values, with formulas, explanations, and examples.
LLM Math Basics 1: Vectors and the Dot Product
A walkthrough of vectors and the dot product — with notation, explanations, and examples — covering what you need to know before reading LLM papers.
LLM Paper Evolution Map (MOC)
A Map of Content page for reading core LLM papers in order, starting from the Transformer.
Transformer Basics: Intuition Behind Q, K, and V
Explains the role of Q, K, and V through the attention formula, a plain-language walkthrough, and a concrete example.
Transformer Basics: Residual, LayerNorm, FFN
Why Residual, LayerNorm, and FFN are necessary in a Transformer block — explained with equations, commentary, and examples.