Writing log70 posts277 tagslatest Jul 24, 2026
Transformer.
Latest posts
Llama 3 (2024) Paper Notes
Paper notes on The Llama 3 Herd of Models: 15T tokens, a 405B flagship, 128K context, and a design philosophy of 'managing complexity' (dense over MoE, SFT → rejection sampling → DPO instead of PPO) — the paper where an open-weight model first reaches GPT-4-class performance. It completes Llama 2's skeleton through scale, data, and simplicity.
Llama 2 (2023) Paper Notes
Paper notes on Llama 2 covering its commercial license, a base model scaled to 2T tokens with 4k context and GQA, and the Llama 2-Chat alignment pipeline (SFT → two reward models → iterative RLHF with rejection sampling + PPO → GAtt). It fuses LLaMA's open base lineage with InstructGPT's RLHF at production quality.
LLaMA (2023) Paper Notes
Paper notes on LLaMA covering its core strategy — train smaller models for longer on publicly available data, optimized for inference — the result where a 13B model beats 175B GPT-3, the RMSNorm/SwiGLU/RoPE architecture changes, the 2048-A100 training scale, and how it kicked off the open LLM ecosystem.
InstructGPT (2022) Paper Notes
Paper notes on InstructGPT covering its core method — the three-step RLHF recipe (SFT → reward model → PPO) — the alignment result where a 1.3B model beats 175B GPT-3, gains in truthfulness and toxicity, the alignment tax and PPO-ptx, and the limitations.
GPT-3 (2020) Paper Notes
Paper notes on GPT-3 covering its core ideas: 175B scaling, in-context learning (zero/one/few-shot), weighted-sampling training data, headline benchmark numbers, and the data-contamination and bias limitations.
GPT-2 (2019) Paper Notes
Paper notes on GPT-2 covering its core ideas: decoder-only Transformer scaling, WebText, next-token prediction, zero-shot task transfer, and the staged release controversy.
BERT (2018) Paper Notes
Paper notes covering the core ideas of BERT: the bidirectional Transformer encoder, masked language model, next sentence prediction, and the fine-tuning paradigm.
LLM Architecture Basics: Encoder-only vs. Decoder-only
A comparison of encoder-only and decoder-only architectures that distinguish the BERT and GPT families.
Transformer Basics: Encoder and Decoder
A clear explanation of what the Transformer encoder and decoder each do, grounded in the original architecture and illustrated with simple examples.
Attention Is All You Need (2017) — Paper Notes
A reading note on the Transformer paper — the core ideas, why it mattered, and what to read next.
Transformer Basics: Intuition Behind Q, K, and V
Explains the role of Q, K, and V through the attention formula, a plain-language walkthrough, and a concrete example.
Transformer Basics: Residual, LayerNorm, FFN
Why Residual, LayerNorm, and FFN are necessary in a Transformer block — explained with equations, commentary, and examples.