GitHub - llm-internals: Learn LLM internals step by step

In this blog, we will learn about BPE (Byte Pair Encoding) - the tokenization algorithm used by most modern Large Language Models (LLMs) to break text into smaller pieces before processing it.

We will understand what BPE is, why it is needed, and how it works step by step with a simple example.

We will cover the following:

• What is Tokenization?
• The Problem: How to Break Text into Tokens?
• What is BPE (Byte Pair Encoding)?
• How BPE Works: Step by Step
• How BPE Tokenizes New Text
• Why BPE is Used in Modern LLMs

Let's get started: Byte Pair Encoding in LLMs 

We will cover the following:

• The Attention Formula
• Setting Up: From Words to Vectors
• Creating Q, K, and V Matrices
• Computing Attention Scores (Q x K^T)
• Scaling the Scores
• Applying Softmax
• Computing the Final Output (Attention Weights x V)
• Putting It All Together

Let's get started: Math behind Attention - Q, K, and V