Machine Learning I Building Large Language Models LLMs
Life and a bit more
youtube· 104:21standard
1.Introduction to Large Language Models (LLMs)
0:00 / 0:55This chapter introduces Large Language Models (LLMs) as the chatbots we encounter today, such as ChatGPT, Claude, Gemini, and Llama. It outlines the key components involved in training LLMs, including architecture, training loss, data, evaluation, and system components.
- LLMs
- Training Components
What's inside this course
- 0:00
1. Introduction to Large Language Models (LLMs)
This chapter introduces Large Language Models (LLMs) as the chatbots we encounter today, such as ChatGPT, Claude, Gemini, and Llama. It outlines the key components involved in training LLMs, including architecture, training loss, data, evaluation, and system components.
- 0:55
2. Overview of LLM Training Paradigms
This section differentiates between pre-training and post-training paradigms for LLMs. Pre-training involves modeling the internet, while post-training focuses on making LLMs into AI assistants, a more recent development since ChatGPT.
- 2:55
3. Understanding Language Models and Probability
Language models are defined as models of probability distribution over sequences of tokens or words. The chapter explains how they assign probabilities to sentences, reflecting grammatical correctness and semantic plausibility.
- 4:54
4. Auto-Regressive Language Models
This segment delves into auto-regressive language models, which decompose the probability of a sequence into a product of conditional probabilities. It highlights the sequential generation process and its implications for generation speed.
- 6:43
5. Predicting the Next Word: Training and Inference
The core task of auto-regressive language models is predicting the next word. The process involves tokenization, passing through a model, obtaining a probability distribution, and then sampling. The chapter distinguishes between training, where the model learns to predict, and inference, where new text is generated.
- 8:29
6. Neural Language Model Architecture and Loss
This section explains how auto-regressive neural language models work. It covers embedding words, passing them through a neural network (like a transformer), and using a linear layer and softmax to get a probability distribution. The cross-entropy loss is introduced as the mechanism for training, aiming to maximize the likelihood of correct next tokens.
- 10:36
7. The Importance of Tokenizers
Tokenizers are crucial for LLMs, offering more generality than words and managing sequence length. They address issues like typos and non-Latin languages by breaking text into sub-word units, typically 3-4 letters long, to balance vocabulary size and sequence length.
- 12:39
8. Byte Pair Encoding (BPE) Tokenization
Byte Pair Encoding (BPE) is explained as a common tokenization algorithm. It starts with individual characters as tokens and iteratively merges the most frequent pairs of tokens into new, larger tokens, effectively creating a vocabulary of common sub-sequences.
- 15:13
9. Tokenizer Challenges and Future Directions
This section discusses the challenges with tokenizers, such as handling spaces, punctuation, and the fixed vocabulary size. It highlights how tokenization can impact a model's understanding of concepts like numbers and code, suggesting a potential future shift towards character-by-character tokenization if architectural limitations are overcome.
- 18:20
10. Evaluating LLMs: Perplexity
Perplexity is introduced as a key evaluation metric for LLMs, essentially a normalized validation loss. It quantifies how well a probability model predicts a sample, with lower perplexity indicating better performance. The chapter notes its decline in academic benchmarking but its continued importance in LLM development.
- 20:55
11. Academic Benchmarks for LLMs
Beyond perplexity, LLMs are evaluated using aggregated classical NLP benchmarks like HELM and Hugging Face Open LLM Leaderboard. These benchmarks, such as MMLU, involve tasks like question answering where models are assessed on their ability to select correct answers from multiple choices, often by comparing the likelihood of generating each option.
- 23:22
12. Evaluation Challenges: Inconsistencies and Contamination
Evaluating LLMs presents challenges, including inconsistencies in evaluation methods across different organizations, leading to varied results for the same model. Another significant issue is 'chain test contamination,' where test data might inadvertently be included in the training set, skewing performance metrics.
- 26:39
13. Data Collection for LLMs: The Internet's Dirtiness
Training LLMs on 'all of the internet' is a complex process due to the internet's 'dirty' and unrepresentative nature. The initial step involves using web crawlers like Common Crawl to gather billions of web pages, resulting in petabytes of raw, unstructured data.
- 29:57
14. Data Cleaning and Filtering Pipeline
A multi-step pipeline is essential for cleaning raw internet data. This involves extracting text from HTML, filtering undesirable content (e.g., NSFW, PII), deduplicating redundant information, and applying heuristic rules to remove low-quality documents. These steps are crucial for preparing data suitable for LLM training.
- 33:27
15. Advanced Data Curation: Model-Based Filtering and Domain Weighting
Advanced data curation techniques include model-based filtering, where classifiers are trained to identify high-quality data sources (e.g., Wikipedia references). Data is also classified into domains (e.g., code, books, entertainment) to allow for up- or down-weighting certain types of content, optimizing for specific model capabilities like reasoning.
- 37:00
16. Data Challenges and Scale in LLM Training
Despite extensive cleaning, data collection for LLMs remains a significant challenge, with ongoing research into efficient processing, data balancing, and synthetic data generation. The sheer scale of data has grown from 150 billion tokens to 15 trillion tokens for state-of-the-art models, highlighting the continuous need for massive datasets.
- 40:02
17. Scaling Laws: Predicting LLM Performance
Scaling laws describe the empirical observation that increasing data and model size consistently improves LLM performance, defying traditional overfitting concerns. These laws allow for predicting how much performance will improve with increased compute, data, or parameters, often showing a linear relationship on a log-log scale.
- 43:23
18. Optimizing Training with Scaling Laws
Scaling laws revolutionize LLM training by enabling a new pipeline: instead of tuning hyperparameters on large models, one can find a scaling recipe, tune on smaller models, extrapolate performance, and then train the final large model efficiently. This approach helps in deciding between different architectures (e.g., Transformers vs. LSTMs) by predicting their performance at scale.
- 48:00
19. Resource Allocation and the Chinchilla Paper
Scaling laws help optimize resource allocation, guiding decisions on whether to train larger models or use more data. The Chinchilla paper famously showed that for optimal training, a ratio of 20 tokens per parameter is ideal. This balance is crucial for achieving the lowest loss given a fixed computational budget, though inference costs might shift this ratio.
- 52:50
20. The Bitter Lesson and Cost of LLM Training
The 'Bitter Lesson' emphasizes that computational scale and efficient architectures that leverage it are more critical than minor architectural innovations. This chapter provides a back-of-the-envelope calculation for training a model like Llama 3 400B, estimating costs around $75 million, highlighting the immense financial and computational investment.
- 55:49
21. Post-Training: Aligning LLMs as AI Assistants
Post-training is crucial for aligning LLMs to act as AI assistants, moving beyond pure language modeling. The goal is to make LLMs follow user instructions and designers' desires, such as avoiding toxic content. This process transforms models like GPT-3 into conversational agents like ChatGPT.
- 59:59
22. Supervised Fine-Tuning (SFT) for Alignment
Supervised Fine-Tuning (SFT) is the first step in post-training, where a pre-trained LLM is fine-tuned on human-collected, desired question-answer pairs. This process teaches the model to format its answers according to human preferences, effectively specializing it for a particular user interaction style without necessarily teaching new knowledge.
- 63:49
23. Reinforcement Learning from Human Feedback (RLHF)
RLHF addresses SFT's limitations, such as being bound by human abilities and potentially causing hallucinations. Instead of merely cloning human behavior, RLHF aims to maximize human preference. It involves generating multiple answers, having humans rank them, and then using reinforcement learning to train the model to produce more preferred responses.
- 67:39
24. Reward Models and PPO in RLHF
RLHF utilizes a 'reward model' to provide a continuous reward signal, overcoming the sparsity of binary human feedback. This reward model is trained to predict human preferences. Proximal Policy Optimization (PPO), a reinforcement learning algorithm, then fine-tunes the LLM to maximize this reward, with regularization to prevent over-optimization.
- 72:09
25. Direct Preference Optimization (DPO)
Direct Preference Optimization (DPO) simplifies RLHF by directly optimizing the probability of preferred responses and minimizing dispreferred ones, without needing a separate reward model or complex reinforcement learning. DPO achieves similar performance to PPO but with significantly less complexity, making it a standard in the open-source community.
- 76:09
26. Challenges with Human Data in RLHF
Collecting human data for RLHF is slow, expensive, and prone to biases. Humans struggle to consistently label for correctness and often prioritize superficial aspects like length, leading to models generating overly verbose responses. The ethics of crowdsourcing and annotator distribution shifts also pose significant challenges.
- 80:19
27. LLM-Based Evaluation of Post-Training
Evaluating post-trained LLMs is challenging due to the open-ended nature of their responses and the inability to use traditional metrics like perplexity. A common approach is to use LLMs themselves as evaluators, comparing outputs from different models and having a 'judge' LLM determine which is better. This method, exemplified by Alpaca Eval, shows high correlation with human judgments and is cost-effective.
- 84:29
28. GPU Optimization for LLMs: Low Precision
GPU optimization is critical for LLMs due to the high computational demands. One key technique is using low-precision (e.g., 16-bit) floats for computations, which reduces memory consumption and speeds up communication between GPU components. While weights are often stored in 32-bit, computations are performed in lower precision to maximize throughput.
- 87:29
29. GPU Optimization: Operator Fusion
Operator fusion is another crucial GPU optimization technique that addresses the bottleneck of communication between GPU memory and processing units. Instead of repeatedly moving data for each operation, fusion combines multiple operations into a single kernel, reducing data transfers and significantly speeding up computations, often achieved with tools like `torch.compile`.
Every chapter ends with a checkpoint (quiz, flashcards, retell, diagram, or prediction) and the course closes with a final boss-fight. More courses →
Full transcript of “Machine Learning I Building Large Language Models LLMs”
— 3065 segments
0:06So, let's get started. Uh so, I'll be
0:07talking about building LLMs today. Um
0:10so, I think a lot of you have heard of
0:12LLMs before.
0:14Uh but just as a quick recap,
0:16uh LLMs, standing for large language
0:18models, are basically all the chatbots
0:21uh that you've been hearing about
0:22recently. So,
0:24uh ChatGPT from OpenAI, uh Claude uh
0:27from Anthropic, Gemini, and and Llama,
0:29and other type of models like this. And
0:32today we'll be talking about how do they
0:34actually work? So, it's going to be an
0:35overview cuz it's only one lecture, and
0:37it's hard to compress everything, but
0:39hopefully I'll touch a little bit about
0:40all the components that are needed to
0:42train uh some of these LLMs.
0:44Uh also, if you have questions, please
0:46interrupt me and ask. Uh if you have a
0:48question, most likely other people in
0:50the room or on Zoom have other have the
0:53same question. So, please ask.
0:56Um great. So, what matters when training
0:59LLMs?
1:00Um so, there's a few key components that
1:02matter.
1:03Uh one is the architecture. So, as you
1:05probably will know, LLMs are neural
1:07networks. And when you think about
1:09neural networks, you have to think about
1:10what architecture you're using. Uh
1:12another component, which is really
1:13important, uh is the training loss and
1:15the training algorithm. Um so, how you
1:18actually train these models. Then it's
1:20the data. So, uh what do you train these
1:23models on? Um the evaluation, which is
1:26how do you know whether you're actually
1:27making progress towards the goal of uh
1:31LLMs. And then the system component. So,
1:33that is like how do you actually make
1:35these models run on uh modern hardware,
1:38which is really important cuz these
1:40models are really large. Um so, now more
1:42than ever, systems actually really an
1:44important topic um
1:46for LLMs.
1:47So, those five components, um you
1:50probably all know that LLMs, and if you
1:52don't know, LLMs are all based on
1:54transformers, or at least some version
1:56of transformers.
1:57Uh I'm actually not going to talk about
2:00the architecture today. One because I
2:02gave a seal lecture on
2:04Transformers a few weeks ago and two
2:07because you can find so much information
2:09online on Transformers. But I think you
2:11can it's there's much less information
2:13about the other four topics. I really
2:15want to talk about those.
2:17Another thing to say is that most of
2:19academia actually focuses on
2:21architecture and training algorithm and
2:23losses.
2:25As academics and I've done that for a
2:27lot
2:27big part of my career is simply we like
2:30thinking that this is like we make new
2:33architectures, new models and it seems
2:36like it's very important. But in
2:37reality, honestly, what matters in
2:39practice is mostly the three other
2:41topics. So data, evaluation and systems,
2:44which is what of most of industry
2:46actually focuses on. So that's also one
2:49of the reason why I don't want to talk
2:50too much about the architecture cuz
2:52really the rest is super important.
2:55Great. So overview of the lecture. I'll
2:57be talking about pre-training. So
2:59pre-training, you probably heard that
3:00word. This is the general word. This is
3:02kind of the classical language modeling
3:05paradigm where you basically train your
3:07language model to essentially model all
3:09of internet.
3:11And then there's a post-training, which
3:12is a more recent paradigm, which is
3:14taking these large language models and
3:16making them essentially AI assistants.
3:18So this is more of a recent trend since
3:21ChatGPT.
3:22So if you ever heard of GPT-3 or GPT-2,
3:25that's really pre-training land.
3:27If you heard of ChatGPT, which you
3:29probably have, this is really
3:30post-training land.
3:32So I'll be talking about both, but I'll
3:34start with pre-training.
3:35And specifically I'll talk about what is
3:37the task of pre-training LLMs and what
3:40is the loss that people are actually
3:41use.
3:43So language modeling, this is a quick
3:46recap.
3:47Language models at a high level are
3:49simply models of probability
3:51distribution over sequences of tokens or
3:53words. So, it's basically some model of
3:56P of X1 to XL where X1 is basically word
4:00one and XL is the last word in the
4:02sequence or in the sentence.
4:04Um so, very concretely, if you have a
4:06sentence like the mouse ate the cheese,
4:08what the language model gives you is
4:10simply a a probability of this sentence
4:13being uttered by a human or being found
4:15on on online.
4:17Uh so, if you have another sentence like
4:19the the mouse ate cheese, uh here
4:22there's grammatical mistakes. So, the
4:24model should know that this uh should
4:26have some syntactic knowledge. So, it
4:28should know that this has less
4:29likelihood of appearing online.
4:32Uh if you have another sentence like the
4:35cheese ate the mouse, uh then the model
4:37should hopefully know about the fact
4:39that usually cheese don't eat mouse. Um
4:42so, there's some semantic knowledge and
4:44this is less likely than the first
4:45sentence. So, this is basically at a
4:46high-level what language models are.
4:49Um
4:50One word that you've probably have been
4:52hearing a lot in the news are generative
4:53models. Uh so, this is just something
4:55that can generate models that can
4:57generate sentences or can generate some
4:59data. Uh the reason why we say language
5:01models are generative models is that
5:02once you have a model of a distribution,
5:04you can simply sample from this model
5:06and now we can generate data.
5:08Uh so, you can generate sentences
5:10uh using a language model.
5:13So, the type of models that uh people
5:15are all currently using are what we call
5:16auto-regressive language models. And the
5:19key idea of auto-regressive language
5:22models is that you take this
5:23distribution over words and you
5:26basically decompose it into the dish
5:28into the distribution of the first word
5:31multiplied by the distribution of or the
5:33likelihood of the distribution of the
5:35second word given the first word uh
5:37multiplied by P of the third word given
5:39the first two words.
5:41Um so, there's no approximation here.
5:43This is just the chain rule of
5:44probability, which you hopefully you all
5:45know about. Uh really no approximation.
5:47This just one way of modeling a
5:49distribution.
5:51Uh so slightly more concisely you can
5:52write it as a product of uh
5:54of P's of the next word given everything
5:57which happened in the past, so of the
5:58context. Uh so this this is what we call
6:01auto-regressive language models. Again,
6:03this is really not the only way of
6:05modeling distribution, this is just one
6:07way. Uh it has some benefits and some
6:10downsides. One downside of
6:11auto-regressive language models is that
6:13when you actually sample from this
6:15auto-regressive language model, you
6:16basically have a for loop which
6:18generates the next word, then conditions
6:20on that next word, and then regenerates
6:22another one. So basically, if you have a
6:24longer sentence that you want to
6:25generate, you it takes more time to
6:27generate it. Uh so there are some
6:29downsides of this current paradigm, but
6:31that's what we currently have, so I'm
6:33going to talk about this one.
6:36Uh great. So auto-regressive language
6:38models. At a high level, um what the
6:40task of auto-regressive language model
6:42is is simply predicting the next word as
6:44I just said. So if you have a sentence
6:45like she likely prefers, uh one
6:47potential next word might be dogs.
6:51And the what the way we do it is that we
6:53first tokenize. So you take these words
6:56or sub-words, you tokenize them, um and
6:59then you give an ID for each token. So
7:01here you have 1 2 3.
7:03Uh then you pass it through this black
7:05box as I already said, we're not going
7:06to talk about the architecture. You just
7:08pass it through pass it through a model,
7:10and you then get a distribution, a
7:12probability distribution over the next
7:15word or the next token.
7:17And then you sample uh from this
7:19distribution, you get a new token, and
7:22then you de-tokenize. So you get a new
7:23ID, you then de-tokenize, and that's how
7:26you basically sample from a language
7:27model. Uh one thing which is important
7:29to note is that the last two tips uh two
7:31steps are actually only need needed
7:33during inference.
7:34Uh when you do training, you just need
7:36to predict uh the most likely token, and
7:38you can just compare to the real token
7:40which happened in practice, and then you
7:42basically change the weights of your
7:44model to increase the probability of
7:46generating that token.
7:49Um, great. So, auto-regressive neural
7:51language models, so to be slightly more
7:53specific still without talking about the
7:55architecture, uh, the first thing we do
7:57is that we have all of these Oh, sorry.
7:59Yes. On the previous slide, you're
8:02predicting the probability of the next
8:03token. Does this mean that your final,
8:04like, output vector has to be the same
8:07dimensionality as the number of tokens
8:09that you have? Yes.
8:10How do you deal with, like, if you have
8:12more tokens, like, if you're adding more
8:14tokens to your corpus or something?
8:16Yeah. So, we're going to talk about
8:18tokenization actually later. Uh, so you
8:20will get some sense of this. You
8:22basically can't deal with adding new
8:24tokens. I am I'm kind of exaggerating.
8:26There are methods for doing it, but
8:28essentially people don't do it. Um, so
8:31it's really important to think about how
8:33you tokenize your text, and that's why
8:34we'll talk about that later. Uh, but
8:36it's a very good point to notice that
8:37you basically the vocabulary size, so
8:39the number of tokens that you have is
8:40essentially the output of your, uh,
8:42language model. So, it's actually pretty
8:44pretty large.
8:46Okay, so auto-regressive neural language
8:48models.
8:49First thing you do is that you take
8:50every word or every token. You embed
8:53them, so you get a um,
8:55some vector representation for each of
8:57these tokens.
8:58Um, you pass them through some neural
9:00network. As we said, it's a transformer.
9:01Then you get a representation for all
9:04the word and all the words in the
9:06context. So, it's basically a
9:07representation of the entire sentence.
9:10Uh, you pass it through a linear layer,
9:12as you just said, to basically map it to
9:14the number so that the output the number
9:17of outputs is the number of tokens.
9:19Uh, you then pass it through some
9:21softmax, and you basically get a
9:23probability distribution over the next
9:26words given every word in the context.
9:31And the loss that you use is basically
9:33it's essentially a task of classifying
9:35the next token, so it's a very simple
9:36kind of machine learning task. So, you
9:38use a cross entropy loss where you
9:40basically you look at the actual target
9:44that happened, which is a target
9:45distribution, which is a one-hot
9:46encoding, which here in this in this
9:48case says, "I saw uh the real word that
9:51happened is cat." So, that's a one-hot
9:53um
9:54distribution over cat. And here this is
9:56the actual
9:58uh Do you see my mouse? Oh, yeah. This
9:59is the distribution that you generated,
10:01and basically you do cross entropy,
10:02which really just increases the
10:03probability of generating cat and
10:05decreases all the the probability of
10:06generating all the other tokens. One
10:08thing to notice is that as you all know
10:11again, uh this is just equivalent to
10:14maximizing the text log like the text
10:16log likelihood cuz you can just rewrite
10:18the the max over the probability of um
10:22this autoregressive language modeling
10:24task as just being this minimum over I
10:27just added the log here and minus, which
10:29is just the minimum of the loss, which
10:31is the cross entropy loss. So, basically
10:32minimizing the loss is the same thing as
10:34maximizing the likelihood of your text.
10:37Any question? Questions?
10:43Okay.
10:45Tokenizer.
10:47Um so, this is one thing that people
10:49usually don't talk that much about.
10:51Tokenizers are extremely important. Uh
10:54so, it's really important that you kind
10:55of understand at least uh what they do
10:57at a high level.
10:58So, why do we need tokenizers in the
11:00first place?
11:01Uh first, it's more general than words.
11:03So, one simple thing that you might
11:05think is, "Oh, we're just going to take
11:06every word that we all have. You just
11:08say every word is a new is a token in
11:10its own."
11:11Um but then what happens is if there's a
11:13typo in your word,
11:15then you might not have any token
11:17associated with this this word with a
11:19typo, and then you don't know how to
11:21actually pass this word with a typo into
11:23a large language model. So, what do you
11:25do next? And also, even if you think
11:27about words, words is a very like words
11:30are fine with like Latin-based
11:31languages, uh but if you think about the
11:34language like Thai, you won't have a
11:36simple way of tokenizing by spaces cuz
11:38there are no spaces between words.
11:40So really tokens are much more general
11:42than words.
11:44First thing. Second thing that you might
11:45think is that you might tokenize every
11:48sentence character by character. You
11:50might say A is one token, B is another
11:52token.
11:53That would actually work and probably
11:54very well. The issue is that then your
11:56sequence becomes super long. And as you
11:59probably remember from the lecture on on
12:01transformers,
12:02the complexity
12:04grows quadratically with the length of
12:06sequences. So you really don't want to
12:08have a super long sequence.
12:10So tokenizers basically try to deal with
12:13those two problems and give common
12:17subsequences
12:18a certain token. And usually how you
12:20should be thinking about it is around
12:23an average or every token is around
12:24three, four letters.
12:28And there are many algorithms for
12:30tokenization. I'll just talk about one
12:31of them to give you a high level, which
12:33is what we call byte pair encoding,
12:34which is actually pretty common, one of
12:36the two most common tokenizers.
12:38And the way that you train a tokenizer
12:40is that first you start with a very
12:42large corpus of text. And here I'm
12:43really not talking about training a
12:45large language model yet. This is purely
12:46for the tokenization step. So this is my
12:49large corpus of text with these five
12:51words.
12:52Then you associate every character in
12:56this corpus of text different token.
12:59So here I just split it up every
13:00character with its different token
13:02and I just color coded all of those
13:04tokens.
13:06And then what you do is that you go
13:07through your text and every time you see
13:10pairs of tokens that are very common,
13:12the most common pair of token, you just
13:14merge them. So here you see three times
13:17the
13:18the tokens T and O next to each other.
13:21So you're just going to say this is a
13:22new token.
13:23And then you continue. You repeat that.
13:24So now you have T O K, talk, which
13:27happens three times. Talk with an e that
13:30happens uh sorry, two times. And And
13:33token, which happens twice. And then e
13:35x, which also happened twice. So, this
13:38is that if you were to train a tokenizer
13:40on this corpus of text, which is very
13:42small, that's how you would uh finish
13:44with a token with a pre like a trained
13:46tokenizer.
13:47Uh in reality, you do it on on much
13:50larger corpuses of text. Um and this is
13:52the real tokenizer of uh actually, I
13:55think this is GPT-3 or ChatGPT. Uh and
13:58here you see how it would actually
13:59separate these words. So, basically, you
14:01see the same thing as what we gave in
14:03the previous example, token becomes its
14:05own token. So, tokenizer is actually
14:08split up into two tokens, token and
14:11izer. Um
14:13So, yeah. That's all about tokenizers.
14:15Any question on that? Yeah. How do you
14:17deal with spaces and how do you deal
14:18with punctuation?
14:20Yeah. So, actually, there's a a step
14:23before tokenizers, which is what we call
14:24pre-tokenizers, which is exactly what
14:27you just said. Uh so, this is mostly In
14:30theory, there's no reason to deal with
14:31spaces and punctuation separately. You
14:34could just say every space gets its own
14:36token, every
14:39uh punctuation get its own token, and
14:41you could just do all the merging. The
14:42problem is that so, there's an
14:44efficiency question. Actually, training
14:46these tokenizers takes a long time. Uh
14:48so, you're better off because you have
14:49to consider every pair of token. So,
14:52what you end up doing is saying if
14:53there's a space, this is very like
14:55pre-tokenizers are very English
14:56specific. You say if there's a space,
14:58we're not going to start looking at the
15:00the token that came before and the token
15:02that came afterwards. So, you're not
15:04merging in between spaces. But, this is
15:06just like a optimiz like a computational
15:09optimization. You could theoretically
15:11just deal with it um the same way as you
15:13deal with any other character.
15:16Yeah. When you merge tokens, do you
15:18delete the tokens that you merged away
15:20or do you keep the the smaller token in
15:22the vocabulary. Um, you actually keep
15:24the smaller tokens. I mean, in reality,
15:26it doesn't matter much because, um,
15:29usually on large corpus of text, you
15:31will have actually everything.
15:33Uh, but you usually keep the small ones.
15:35And the reason why you want to do that
15:36is because if, in case there's a, as we
15:38said before, you have some, um, some
15:40grammatical mistakes or some typos, you
15:42still want to be able to represent these
15:44words by character.
15:45Um,
15:47so, yeah.
15:48Yes. Are the tokens
15:50unique? So, I mean, say in this case, t
15:53o k e n, is there only one occurrence or
15:56could do you need to
15:58leave multiple occurrences so they could
16:00have take on different meanings or
16:01something? Oh, oh, I see what you would
16:03say. No, it's every token has its own,
16:06uh, unique ID.
16:08Um, so, a usual, this is a great
16:11question. For example, if you think
16:12about, uh, bank, which could be bank for
16:14like money or bank like water, um,
16:16it will have the same token, but the
16:18model will learn, the transformer will
16:20learn that based on the words that are
16:22around it, it will should associate
16:24that, I'm saying, I'm being very
16:26hand-wavy here, but associate that with
16:28the with a with a representation that is
16:31either more like the the bank money side
16:33or the bank, uh, water side. Um, but
16:35that's the transformer that does that.
16:36It's not a tokenizer.
16:38Yes.
16:39Yes, so you mentioned during
16:40tokenization keep the smaller tokens to
16:42start with.
16:43Like, if you start with a t, you keep
16:46the t and then you tell your tokenizer
16:48to just tokenize amount of
16:50So, let's say, um, maybe you didn't
16:51train on token by like in your data you
16:53are trying to encode token. So, how does
16:56the tokenizer know to encode it with
16:58token or to leave t? Yeah,
17:00that's a great question. You basically,
17:01when you, so, when you tokenize, so,
17:03that's after training of the tokenizer,
17:05when you actually apply the tokenizer,
17:07you basically always choose the largest,
17:09uh, token that you can apply.
17:12Uh, so, if you can do token, you will
17:13never do t. You will always do token.
17:15Um, but there's actually, so, people
17:18don't really talk that much about
17:19tokenizers, but
17:20there's a lot of of computational
17:22benefits
17:24or computational tricks that you can do
17:25for making these things faster.
17:27So I really don't think we and honestly,
17:29I think a lot of people think that we
17:30should just get away from tokenizers
17:33and just kind of tokenize character by
17:35character or bytes by bytes.
17:37But as I said right now there's this
17:38issue of like length. But maybe one day
17:41like in 5 or 10 years we'll have
17:43different architectures that don't scale
17:44quadratically with the length of the
17:45sequence and maybe we'll
17:48um, yeah, move away from tokenizers. So
17:51can you share with us the drawback? Why
17:53why do people want to move away from the
17:55tokenizer? Oh, um,
17:58yeah, so think
18:00one good example is
18:03math.
18:04If you think about math, actually
18:05numbers right now are not tokenized. So
18:07for example, 327 might have its own
18:10token, which means that models when they
18:12see numbers, they don't see them the
18:14same way as we do. And this is really
18:16annoying because what I mean, the reason
18:18why we can kind of generalize with math
18:20is because we can deal with every every
18:22letter separately and we can then do
18:23composition. Well, you know that
18:25basically if you are add stuff, it's the
18:26same thing as adding every one
18:28separately plus like whatever the unit
18:30that you add. So they can't do that. Um,
18:33so then you have to do like special
18:34tokenization. And like one of the big
18:37changes that GPT-4 did is changing the
18:41way that they tokenize code. So for
18:43example, if you have code, you know, you
18:45have like often in Python these four
18:47spaces at the beginning, those were
18:49dealt with kind of strangely before.
18:52And as a result like the model couldn't
18:54really understand
18:55how to deal with code.
18:57So so tokenizers actually matter a lot.
19:00Um,
19:01okay. So I'll move on right now, but we
19:03can come back later on tokenizers.
19:06Great. So we talked about the task the
19:08last the tokenizer. Let's talk a little
19:09bit about the evaluation.
19:11So the way that LLMs are usually
19:13evaluated is what we call is using what
19:15we call perplexity. Um, at a high level,
19:18it's basically just your validation
19:19loss. Uh, the slight difference with
19:21perplexity is that we use something that
19:23is slightly more interpretable, which is
19:25that we use the average per token loss,
19:28and then you exponentiate it. And the
19:30reason why you exponentiate it is
19:31because you want I mean the loss uh has
19:34a log inside and you like one humans are
19:36actually pretty bad at thinking in log
19:38space, but two logs depend on the base
19:40of the log. Uh, while what when you
19:42exponentiate you basically have
19:44everything in the uh kind of the
19:46vocabulary size uh unit. Um,
19:49and the average per token is just so
19:50that your your perplexity is independent
19:52of the length of your sequence. Um, so
19:54perplexity is just 2 to the power
19:57uh average of the loss of the sequence.
20:00Um, so perplexity is between one and the
20:03length of the vocabulary of your
20:05tokenizer. Uh, one is simply well, if
20:07you predict perfectly the thing which uh
20:10every word, then every word will have
20:12basically product of ones. Uh, so the
20:15best perplexity you can have is one. If
20:17you really have no idea, you basically
20:19predict with one divided by uh size of
20:21vocabulary, um and then you do simple
20:23math and you basically get perplexity of
20:25size of vocabulary. Uh, so then tuition
20:28of perplexity is that it's basically the
20:29number of tokens that your model is kind
20:31of hesitating between.
20:33Uh, so if you if your model is perfect,
20:35it doesn't hesitate, it know exactly the
20:36word. If it really has no idea, then it
20:39hesitates between uh
20:41all of the vocabulary.
20:43Uh, so perplexity really improved.
20:46That's perplexity on a standard data set
20:49between 2017 and 2023. It it went from a
20:52kind of 70 tokens to less than 10 tokens
20:55over these 5-6 years. So that means that
20:57the models were previously hesitating
20:59between 70 words every time it was
21:01generating a word, and now it's
21:03hesitating between like less than 10
21:04words. Uh, so that's much better.
21:07Uh, perplexity is actually not used
21:08anymore in academic benchmarking. Mostly
21:11cuz it depends on the tokenizers that
21:12you use. Uh it depends on the actual
21:14data that people are evaluating on, but
21:16it's still very important for
21:17development of LLMs. So, when you when
21:20you actually train your own LLM, people
21:22will still really look at the
21:23perplexity.
21:25Uh
21:26one common other way, and now more
21:29common in academia, of evaluating these
21:31LLMs is just by taking all the classical
21:34NLP benchmarks, and I'll give you a few
21:36examples later, and just kind of
21:38aggregating everything. Um so, collect
21:40as many automatically evaluatable
21:43benchmarks, and just evaluate across all
21:45of them. Um
21:47So, one such if or actually two such uh
21:50benchmarks are what we call uh HELM,
21:53which is from Stanford, and another one
21:55is the Hugging Face Open LLM
21:56Leaderboard, which are the probably the
21:57two two most common ones right now. Um
22:00so, just to give you an idea, in HELM
22:03there are all of these type of tasks,
22:05which are mostly things that can be
22:07easily evaluated, uh like question
22:09answering. So, think about many
22:11different question answering uh tasks.
22:13Um and the benefit with question
22:15answering is that you usually know what
22:17is the real answer. Um so, you can the
22:19way that you evaluate these models, and
22:21I'll give you a concrete example in 1
22:22second, um is that you can just look at
22:25how likely the language model is to
22:27generate the real answer compared to
22:29some other answers. And that's
22:31essentially at a high level how you
22:32evaluate these models. Um so, to give
22:34you a specific example, MMLU is probably
22:37the most common um
22:39academic benchmark for LLMs.
22:42Uh and this is just a collection of many
22:45question and answers in all of those
22:47domains. For example, college medicine,
22:49college physics, astronomy, and these
22:52type of topics. And the questions are
22:53things like, so this is in astronomy,
22:55what is true for type 1A supernova? Then
22:58you give uh four different potential
23:01answers, and you just ask the model
23:04which one is more likely. So, there are
23:05many different ways of doing it. Either
23:07you can look at the likelihood of
23:09generating all these answers, or you can
23:11ask the model which one is the most
23:12likely. So, there are different ways
23:14that you can prompt the model, but at a
23:15high level, you know which one is
23:17correct, and the other three are the
23:18mistakes.
23:20Yes.
23:22Creating these like constraints of text
23:24as an output. Yeah. How do you evaluate
23:27the model if
23:28it gives something that's, you know,
23:30semantically
23:31completely identical,
23:32but is not the exact token that you
23:35expected? Yeah, so that's a great
23:37question. I'll talk more about that
23:38later. Here in this case, we don't do
23:40unconstrained. So, the way you would
23:42evaluate MMLU is basically either you
23:45you ask the first question, and then you
23:47look at the likelihood of the model
23:50generating A, the likelihood of the
23:51model generating B, C, and D, and you
23:54look at which one is the most likely.
23:55Or, you can ask the model, "Out of A, B,
23:58C, D, which one is the most likely?" And
24:00you look at whether the the most likely
24:02next token is A, B, C, or D. So, you
24:05constrain the model to say it can only
24:07answer these four things.
24:09When you say you constrain the model,
24:11Yeah.
24:11you constrain it at the prompt, or do
24:13you mean the model's whole probability
24:15distribution of the outputs? You only
24:17are comparing the outputs of like you're
24:19only comparing the right token. Yeah.
24:21So, in the second case I gave you, you
24:23would do exactly the I mean, actually
24:25you would do both. You would prompt the
24:26model saying A, B, C, or D, plus you
24:28would constrain to only look at these
24:30two these four tokens. In the first
24:33case, you don't even need to generate
24:34anything. So, in the first case, you
24:35literally just look, given that it's a
24:37language model, it can give a a
24:39distribution over sentences, you just
24:40look at what is the the likelihood of
24:43generating all of these words.
24:45What is the likelihood of generating the
24:47second choice? And you just look at
24:49whether the most likely sentence
24:52is actually the real answer.
24:54So, you don't actually sample from it,
24:56you really just use P of X1 to XL.
25:00Does that make sense?
25:01Uh that being said, evaluation of
25:03open-ended questions is something we're
25:06going to talk about later and is
25:07actually really important and really
25:08challenging.
25:10Yes. Earlier you mentioned that um like
25:13um metrics like perplexity are not
25:15common and are not like usually used
25:17because it depends on like how you do
25:19your tokenization, some design choices.
25:21I was wondering if you could speak more
25:22to that.
25:24Oh, um yeah. So, think about perplexity.
25:27I told you perplexity is between 1 and
25:29vocabulary size. So, now imagine that
25:31ChatGPT uses a tokenizer that has like
25:3410,000 tokens, but uh Gemini from Google
25:37uses a tokenizer that had 100,000 uh
25:40potential tokens. Then actually the
25:42Gemini one will will have like the upper
25:45bound of the the perplexity that you can
25:47get is actually worse for Gemini than
25:50for ChatGPT. Does that make sense?
25:52So, that's just an idea. It's actually a
25:55little bit more complicated than that,
25:56but that's just like one uh first order
25:58bit of where you can see that the the
25:59tokenizer actually matters.
26:02Um
26:03great.
26:06Okay. So, evaluation challenges, there
26:08are many. I'll just talk about two
26:10really briefly. Uh one, as I told you,
26:12there are two ways of doing evaluation
26:14for these MMLUs. Actually there are many
26:15more than two, but I'll give you two
26:17examples. Um and it happens that for a
26:20long time, even though that was a very
26:21classical benchmark that everyone used,
26:24uh actually different uh
26:26different companies and different um
26:28diff- uh
26:30uh
26:31different organization were actually
26:32using different ways of evaluating MMLU.
26:35And as a result, you could you get
26:37completely different results. For
26:38example, Llama
26:3965B,
26:41uh which was the first model of of Meta
26:43in the Llama series, uh had on Helm 63.7
26:47accuracy, but on this other um
26:50benchmark had like 48.8.
26:53Um so, really the way that that you
26:55evaluate, and this is not even talking
26:56about prompting, this is really just
26:58kind of the the way that you evaluate
27:00the
27:01the models. Prompting is another issue.
27:03So really there are a lot of
27:04inconsistencies. It's not as easy as as
27:06it looks. Uh first thing. Yeah, sorry.
27:09How can we make sure that all these
27:10models aren't trained on the benchmark?
27:12Okay.
27:13Second thing. This is a great question.
27:15Uh chain test contamination. Uh this is
27:18something which I would say is really
27:21important in academia.
27:23In uh given that the talk is mostly
27:25about training large language models, uh
27:28for companies it's maybe not that
27:29important cuz they know what they
27:31trained on. Uh
27:33for us, we have no idea. So for us it's
27:36a real problem. Uh so there are many
27:38different ways of trying to test whether
27:41uh the test set or sorry, whether the
27:43test set was actually in the training
27:45set.
27:46Uh one kind of cute trick um
27:49that people uh
27:51in in the lab on Tezzo's lab have found
27:53is that what you can do is that given
27:55that most of the data set online are not
27:57randomized,
27:59you can just look at and in that
28:01language models what they do is just
28:02predict the next word. Um you can just
28:04look at the entire test set. Uh what if
28:07you generate all the examples in order
28:11versus all the examples in a different
28:13order? And if it's more likely to
28:15generate the thing in order given that
28:17there's no real order there, then it
28:20means that it probably was in the
28:21training set. Does that make sense?
28:23Um so there are many that's like one of
28:24them. There are many other ways of doing
28:26it. Chain test contamination, again, not
28:28that important for development, really
28:30important for academic benchmarking.
28:33Great. So there are many other
28:34challenges, but uh
28:36I'll move on for now.
28:38Great. Data.
28:39Um
28:41So data is a another really big topic.
28:43Um at a high level, people just say,
28:45"Oh, you basically train large language
28:47models on all of internet." What does
28:49that even mean? Um so, or people
28:52sometimes say all of clean internet,
28:53which is even less defined. Um so,
28:56internet is very dirty and really not
28:59representative of what we want in
29:00practice. If I download a random website
29:03right now, uh you would be shocked at
29:05what is in there. It's definitely not
29:07your Wikipedia.
29:08Um so,
29:10I'll go really briefly on like what
29:13people do. Um
29:14I can answer some questions, but I mean,
29:16data is on its own is a huge topic. Uh
29:19basically, first what you do is download
29:22all of internet. What that means is that
29:24you use uh
29:25web crawlers that will go on every
29:27webpage on internet or every webpage
29:29that is
29:30um on Google, uh and that is around 250
29:34billion pages right now. Um
29:36And that's around 1 petabyte of of data.
29:39So, this is actually a common Common
29:41Crawl is one web crawler. So, people
29:43don't usually write their own web
29:44crawlers. What they do is that they use
29:46standard web crawlers, and web Common
29:48Crawl is one of them uh that basically
29:51every month adds all the new websites
29:53that were added on uh internet that are
29:56found by by Google, and they put it in a
29:58big uh basically a big data set. Um so,
30:01that's on Common Crawl you have around
30:03250 billion pages right now. So, 1 E6 GB
30:07of data.
30:08Once you have this, uh so, this is a
30:10random webpage, like literally random,
30:13uh from this Common Crawl. And what you
30:15see is that one it really doesn't look
30:16at type of things that you would usually
30:18see, but actually So, this is an HTML
30:21page. Uh it's hard to see, but if you
30:24look through, you will see some content.
30:26For example,
30:28here, uh Test King World is your
30:31ultimate source for the System X
30:33high-performance server, and then you
30:35have three dots. So, you don't even the
30:36sentence is not even finished. That's
30:38how uh random internet looks like.
30:41Uh so, of course, it's not that useful
30:43if you just train a large language model
30:44to generate things like this.
30:46So what are some of the steps that are
30:47needed?
30:48First one, you extract the text from the
30:51HTML. So that's what I just tried to do
30:52by looking at uh basically the correct
30:54tags.
30:56Uh there are a lot of challenges by
30:57through this. For example, extracting
30:58math is actually very complicated but
31:01pretty important for training large
31:02language models.
31:04Um or for example, boiler plates. A lot
31:06of your forums will have the same type
31:08of headers, the same type of footers. Uh
31:10you don't want to repeat all of this in
31:12your data.
31:13Um then you will filter undesirable
31:16content. Uh so not safe for work,
31:19harmful content, PII. Uh so usually
31:21every company has basically a uh
31:24blacklist of websites that they don't
31:26want to train their models on. That
31:28blacklist is very long. And you
31:30basically say if it comes from there, we
31:31don't train on this. There are other
31:33ways of doing these things is that you
31:34can train a small model for classifying
31:37what is PII, removing these things. Um
31:40it's hard. Every point here that I'm
31:42going to show you is like a hard amount
31:45of work. Uh but I would just going to go
31:47go quickly through it. So filter
31:49undesirable content. Second or fourth is
31:52deduplication.
31:54As I said, um you might have things like
31:57headers and footers in forums that are
31:59always the same. You want to remove
32:00that. Another thing that you might have
32:02is a lot of URLs that are different but
32:05actually show the same website. Um and
32:09you might also have a lot of like uh
32:12paragraphs that come from like common
32:13books that are basically deduplicated a
32:16thousand times or ten thousand times on
32:18internet. So you need have to
32:19deduplicate. Also very challenging.
32:22Uh because you have to do that at scale.
32:24Once you do deduplication, you will do
32:26some heuristic filtering. You will try
32:28to remove low quality documents. Uh the
32:32way you do that are things like
32:33rule-based um filtering. For example, if
32:36you see that there are some outlier
32:37tokens, if the distribution of tokens in
32:39the website is very different than the
32:41usual distribution of tokens, then it's
32:42probably some outlier. If you see that
32:44the length of the words in this website
32:46is super long, there's something strange
32:48going on on that website. If you see
32:50that the the website has only three
32:52words, yeah, maybe is it worth training
32:54on it, maybe not. If it has like 10
32:56million words, maybe there's something
32:58also wrong going on that page.
33:00Um so, a lot of rules like this. Yes.
33:02What do we filter out undesirable
33:04content from our pages? So, that's kind
33:06of kind of like
33:08putting it in is like a supervised loss.
33:10Right? Like can we not just say like,
33:12you know, here's this like hate speech
33:14website, let's actively try to
33:17let's actively penalize them for getting
33:19in.
33:20We'll do exactly that, but not at this
33:22step. That's where the post training
33:24will come from.
33:25Uh pre-training, um
33:28the idea is just to say
33:30I want to model kind of how humans speak
33:33essentially.
33:34Um and I want to remove all these like
33:36headers, footers, and and menus, and
33:38things like this. Uh but it's a very
33:39good
33:40uh like idea that you just had, and
33:42that's exactly what we'll do later.
33:45Next step, model-based filtering. So,
33:47once you filter a lot of data, what you
33:49will do, uh so, that's actually a very
33:51cute trick. Uh you will take all of
33:53Wikipedia, and you will look at all the
33:55links that are linked through Wikipedia
33:58pages. Cuz probably if something is
34:00referenced by Wikipedia, it's probably
34:02some high-quality website. And we will
34:04train a classifier to predict whether
34:06something comes from whether a document
34:09comes from one of these references
34:12uh from Wikipedia, or whether it's from
34:14the random web. And you will try to
34:16basically say, I want more of the things
34:19that come from Wikipedia references.
34:22Does that make sense?
34:24So, yeah. So, you will train a a machine
34:25learning uh model. Usually also very
34:28simple models, cuz you need to do that
34:29really at scale. I mean, just think
34:31about the 250 billion pages.
34:34Uh next one, you will try to classify
34:37your data into different different um
34:40domains. You will say, "Okay, this is
34:42entertainment, this is books, this is
34:44code, this is like these type of
34:46domains." And then you will try to
34:48either um up or downweight some of the
34:52domains. Uh for example, you might say
34:54uh you might see that actually if you
34:56train more on code, then actually your
34:58model becomes better on reasoning. So,
34:59that's something that people uh usually
35:01say in a very hand-wavy way. train your
35:03model more on code, actually it helps
35:05reasoning. So, you want to upweight the
35:07coding uh distribution because that
35:09helps for general language modeling
35:11skills. Uh books is usually also another
35:13one that people usually um
35:15upweight. Entertainment, they usually
35:17downweight. Uh so, things like this. Of
35:20course, you want to do it So, people
35:22used to do it maybe
35:23uh
35:24kind of heuristically. Now, there's
35:26entire pipelines that we'll talk about
35:28of how to do these things uh slightly
35:30more um automatically.
35:34And then at the end of training, uh
35:36usually train um after training on all
35:39of this data that we saw, usually train
35:41on very high-quality data at the end of
35:44training a large language model where
35:46you decrease your learning rate. Uh and
35:48that basically means that you're kind of
35:50overfitting your model on a very
35:52high-quality data. So, usually what you
35:54do there is like Wikipedia. You
35:56basically overfit on Wikipedia and you
35:58overfit on like human uh
36:02data that was collected.
36:04Um the other things like continual
36:06pre-training for getting longer context.
36:08I'm I'm going to skip over all of these
36:09things. Uh but I just to give you a
36:11sense of how hard it is when people just
36:13say, "Oh, I'm going to train on
36:14internet." That's a lot of work.
36:17Um and really we haven't figured it out
36:19yet.
36:20So, collecting well data is a huge part
36:23of practical large language model. Uh
36:25some might say that it's actually the
36:26key. Yes.
36:28Now about the data, so based on
36:30questions, so usually when you start
36:31with like the total petabytes of data,
36:34after I go through all that steps,
36:35what's the typical amount of data you
36:37have in there?
36:38And then like how how large a team does
36:40it typically take to go through all the
36:42different steps that you talked about?
36:44So like how large is the question how
36:45large is the data after you filter?
36:47Yeah, after you filter and then to go
36:48through all the steps, how large a team
36:50do you need to go through like the
36:52filtering the other filtration steps you
36:53mentioned? Uh, how slow is it or How
36:57like how large how many people would you
36:59need Oh.
37:00to be able to do this, you know? Uh,
37:03okay, that's a great question. I'm going
37:04to somewhat answer about the data, uh,
37:07how large is the data set uh, at the end
37:09of this slide. Uh, for
37:12number of people that work on it,
37:14uh, that's a good question. I'm actually
37:16not quite sure, but I would say
37:19Yeah, I actually don't quite know, but I
37:23would say it's probably even bigger than
37:24number of people that work on kind of
37:26the tuning of the pre-training of the
37:29model. Uh, so the data is bigger than
37:32kind of the modeling aspect. Um,
37:35yeah, I I don't think I have a good
37:38sense. I would say probably in Llama's
37:39team, which have like 70-ish people, I
37:41would say maybe 15 work on data.
37:44Uh, I think yeah.
37:47All these things you don't need that
37:48many people, you need a lot of computer
37:49also cuz for data you need a lot of
37:51CPUs.
37:52Um,
37:53so yeah, and I'll answer the second
37:54question at the end of this slide.
37:57So, as I just kind of alluded to, really
38:00we haven't solved data at all for
38:02pre-training. So, there's a lot of
38:03research that that has to be done.
38:04First, how do you process these things
38:06super efficiently? Uh, second, how do
38:08you balance the kind of like all of
38:09these different domains? Uh, can you do
38:11synthetic data generation? That's
38:13actually a big one right now. Uh, and
38:15because we don't have uh, we'll talk
38:17about that later, but we don't have
38:18enough data on the internet. Um, can you
38:21use multi-modal data instead of just
38:23text data? And how does that improve
38:25even your text performance? Um, there's
38:29a lot of secrecy cuz really this is the
38:30key of most of the pre-trained
38:33pre-trained large language models. Uh,
38:34so for competitive dynamics, uh, usually
38:37these these, um,
38:38these companies don't talk about how
38:40they do the data collection. And also
38:42there's a copyright liability issue.
38:44They definitely don't want to tell you
38:45that they've trained on books even
38:46though they did. Um, because if not you
38:48can uh, sue them.
38:50Uh, common academic benchmarks, uh, so
38:52that will kind of answer what you asked.
38:54Um, it started, so those are the smaller
38:57ones. Uh, it's the names are not that
38:58important, but it started from a around
39:00150 billion tokens, which around 800 GB
39:04of data. Now it's around 15 trillion of
39:0715 trillion tokens, which is also uh,
39:10the size of the models that are right
39:12now the best models are probably trained
39:13on that amount of data. So 15 trillion
39:15tokens, uh, which is probably, I guess,
39:19two order of magnitude bigger than that.
39:20So 80 uh, E3 GB. So that would be around
39:26100 to 1,000 times, uh, filtering of the
39:30common crawl, if I'm not mistaken.
39:32Um, so yeah. One very One very, uh,
39:35famous one is the Pile. So this is an
39:38academic benchmark, the Pile, and we can
39:40just look at what distribution of data
39:42they have. It's things like, um,
39:44archive, PubMed Central, uh, which is
39:47all the the biology stuff. Uh,
39:50here it's Wikipedia, you see Stack
39:53Exchange, um,
39:55some GitHub, and some books and things
39:57like this. Um, again this is on the
39:59smaller side. So this is, if we look at
40:01here, this is on 280 B. So in reality
40:03it's like 100 times bigger, so you
40:05cannot have that much of GitHub and and
40:07and of Wikipedia.
40:09Um, in terms of closed-source models,
40:11uh, just to give you an idea, uh, Llama
40:132, um, it was trained on two two
40:16trillion tokens. Lama 3, 15 trillion
40:18tokens, which is currently the best
40:20model that we know on how much it was
40:22trained on, which is the same thing as
40:24this the the the best academic or the
40:27biggest academic benchmark, which is 15
40:28trillion tokens. GPT-4 we don't really
40:30know, but it's probably in the same
40:32order of magnitude. Or it's probably
40:33around that actually. It's probably
40:34around 13. Um
40:36from leaks, if the leaks are true.
40:38Um
40:40Great.
40:41So, scaling laws. Um any other questions
40:44on data before you go to the scaling
40:45laws?
40:49Sorry, I know I'm giving you a lot of
40:50information, but uh
40:52there's a lot into training at large
40:53language models.
40:55Great. Scaling laws.
40:57So, the idea is that what people saw um
41:00around 2020, or at least for a long
41:02time, but they've been able to kind of
41:04uh theoretically show it or empirically
41:07show it since 2020, is that the more
41:09data you train your models on and the
41:10larger the models, the better the
41:12performance.
41:13This is actually pretty different than
41:14what you've seen in this class. In this
41:16class we teach you about overfitting.
41:18Overfitting doesn't happen with large
41:19language models. Uh larger models,
41:22better performance.
41:23Um it's something that really took a
41:25long time for the community who took
41:27this type of class to realize. Um but
41:30for the exam, overfitting exists.
41:33And so
41:35Okay. The idea of scaling laws is that
41:37if given that you know that more data
41:39and larger models will always give you
41:41better performance, can we predict how
41:45much better your performance will be if
41:47you increase the amount of data and the
41:48size of your model?
41:50And surprisingly, it works.
41:52Uh so here you see three plots from a
41:54very famous paper called scaling laws
41:56from OpenAI. Um
41:58here you see on the x-axis compute. So,
42:00how much did you train like how much
42:02compute did you did you spend for
42:03training? And here you see test loss.
42:05So, this is essentially I mean it's not
42:07perplexity, but it's your validation
42:09loss. Um so it's the log of the
42:11perplexity. And if you put these two on
42:14a log scale,
42:15then you see that the the performance or
42:18like the the the
42:20Sorry, the
42:21the scaling law is linear.
42:23That means that if you increase your
42:24compute by a certain amount, you can you
42:26can say by how much your test loss will
42:29actually decrease.
42:30Same thing with data and same thing for
42:32parameters. If you increase the data set
42:35size, your loss will will decrease by an
42:38amount that is somewhat predictable. If
42:40you increase the number of parameters,
42:42it will the loss will decrease by an
42:44amount which is somewhat predictable.
42:46This is really amazing. Um very
42:48surprising. I mean, it looks innocuous
42:51when you look at these type of plots,
42:53but that's crazy cuz it means that you
42:54can predict uh how well we're going to
42:56perform in two, three years depending on
42:59how much compute we will add assuming
43:00that these things will hold. There's
43:02nothing theoretical about it.
43:03Um
43:04Yes.
43:06Two things. One, what is the loss that
43:08you're using here? Is this perplexity?
43:09Uh so, it's it's You know, I said
43:11perplexity was like two to the power of
43:13the log. So, this is the the the power
43:16of the perplexity. And then the second
43:18thing is
43:19when you like increase the number of
43:20parameters or you increase the total
43:22data set size, you're going multiple
43:25times. Doesn't that just
43:26inherently increase your compute? Like
43:28is all of this work not just
43:30Oh, yes.
43:32No, this is a great question. So, the
43:34compute here is actually a factor of two
43:36things, the data and the parameter. What
43:38I'm showing here is that you can Um
43:40well, actually we're going to talk about
43:41that in details, but basically, if you
43:43increase the number of parameters, you
43:45should increase the number of data that
43:46you have.
43:47Um
43:48So, you actually don't go multiple times
43:50through the same data set. No one does
43:52epochs in
43:54large at least not yet.
43:57Uh because we haven't still kind of
43:59enough data. Um so, yeah, this is all
44:01the same trend which is increased
44:02compute, decreased loss.
44:05Yes.
44:06Have we seen the numbers for the last
44:08two years? Are
44:10they still holding? It is still holding.
44:13I
44:13I don't have like good numbers to show
44:16you,
44:17uh but it is still holding,
44:18surprisingly.
44:21Yes. Is there no evidence like empirical
44:23evidence that it will ever plateau?
44:25But intuitively we would expect it to
44:27plateau, right?
44:29No empirical evidence of plateauing
44:31anytime soon.
44:32Um
44:33why?
44:34We don't know. Um will it happen?
44:38Probably. I mean, it doesn't need to
44:39because it's actually in log scale.
44:41So, it's not like as if it had to go it
44:44had to plateau like mathematically. It
44:46could continue decreasing like this. I
44:48mean, most people think that it will
44:49probably plateau at some point. We don't
44:51know when.
44:53Um
44:54Okay. So, that's I'll talk more about
44:56scaling laws now.
44:57So, why are scaling laws really cool?
45:00Imagine that I give you um you're very
45:03fortunate. I give you 10,000 GPUs for
45:05this month.
45:06What model will you train? How do you
45:08even go about answering that question?
45:10And I mean, this is a a hypothetical,
45:12but that's exactly what these companies
45:14are faced with.
45:16Uh the old pipeline
45:18um which was basically you tune hyper
45:20parameters on the big models. So, let's
45:22say I have 30 days, I will train 30
45:25models for 1 day each. I will pick the
45:27best one. Uh and that will be the final
45:30model that I will use in production. Um
45:32that means that the model that I
45:33actually used was only trained for 1
45:35day.
45:37The new pipeline is that you first find
45:39a scaling recipe. So, you find something
45:41that tells you, for example, oh, like
45:44one common thing is that if you increase
45:45the size of your model, you should
45:46decrease your learning rate. So, you
45:48find a scaling recipe such that you
45:49know, if I increase the num the the the
45:52size of my model, here's what I should
45:53do with some hyper parameters.
45:55Then you tune your hyper parameters
45:58on smaller models of different sizes.
46:01Let's say I will save for 3 days of my
46:0330 days. I will train many different
46:05models and I will do hyper parameter
46:07tuning on these small models, each of
46:08different sizes. Then I will fit a
46:10scaling law and try to extrapolate
46:13from these smaller models which one will
46:16be the best if I if I train it for much
46:19longer.
46:20Oh, sorry. If I train it for a larger
46:22model.
46:23And then I will train the final huge
46:25model for 27 days instead of just one
46:27day.
46:28Um so, the new pipeline is not train
46:31things or do hyper parameter tuning on
46:33the real scale of the model that you're
46:34going to use in practice, but do things
46:36on smaller ones
46:38at different scales, try to predict how
46:41well they will perform once you make
46:42them bigger. I will give the I will give
46:44you a very concrete example right now.
46:46Uh let's say transformers versus LSTMs.
46:50Uh let's say you're you have these
46:5110,000 GPUs, you're not sure which one
46:53you should be using. Should I be using
46:54transformer-based model or LSTM-based
46:56model? What I will do is I will train
46:58transformers at different scales. So,
47:00here you see different parameters on the
47:02x-axis, y-axis is my test loss. I will
47:05then train different different LSTMs at
47:07different scales.
47:09Once I have these points, I will see,
47:11oh, it kind of fits a scaling law. I
47:13will fit my scaling law and then I will
47:14be able to predict
47:16oh, if I had
47:1810 times more compute, here's how well I
47:19would perform. For the LSTM, it's
47:22actually slightly less linear for the
47:23LSTM, but like you could probably try to
47:25predict where you would end up. And
47:27clearly from this plot, you would see
47:29that transformers are better.
47:30Um one thing to notice when you read
47:32these type of scaling laws is that there
47:34are two things that are important. Uh
47:36one is really your scaling rate,
47:40uh which is kind of the
47:42uh the slope of the uh the slope of the
47:45scaling law. The other thing is your
47:48um your intercept.
47:50Like you could start worse, but actually
47:52become better over time. It just happens
47:54that LSTMs are worse for both. Uh but I
47:56could show you another one where things
47:59you can predict that actually after
48:01certain scale you're better off using
48:03that type of model than others. Uh so
48:05that's why scaling laws are actually
48:07really useful.
48:09Any questions on that?
48:12Yeah. So, these are all kind of very How
48:16how sensitive are these to like small
48:18differences in the architecture?
48:20One one like transformer architecture
48:22versus another transformer architecture.
48:23But you basically have to like fit your
48:26own curve and basically say like oh
48:27scaling laws told me that it should be
48:29some like logarithmic function. Like let
48:32me Yeah. extrapolate that for my own
48:34specific context.
48:35Yeah, so uh usually for example if
48:38you're in academic and you want to now
48:39at least that's like pretty recent and
48:41you want to propose a new like
48:42activation,
48:44uh that's exactly what you will do. You
48:45will fit a scaling law, show another
48:47scaling law with the standard like I
48:49don't know GELU, and you will say that
48:50it's better. In reality once you start
48:52thinking about it in scaling laws terms,
48:54you really realize that actually all the
48:56architecture differences that we can
48:57make like the small minor ones, all they
48:59do is maybe change a little bit the the
49:02intercept.
49:03But really that doesn't matter.
49:05Uh cuz just train it for 10 hours longer
49:07or like wait for the next for the next
49:09computer GPUs and these things are
49:11really secondary, which is exactly why I
49:13was telling you originally people spend
49:15too much time on the architecture and
49:16losses. Um in reality these things don't
49:18matter as much. Data though, if you use
49:20good data, you will have much better
49:22scaling laws than if you use bad data.
49:24So that really matters.
49:27Uh another really cool thing you can do
49:29with scaling laws is that you can ask
49:30yourself
49:31uh
49:32how to optimally allocate training
49:34resources. Should I train larger models?
49:37Cuz we saw that it's better when you
49:39train larger models, but we saw that
49:40it's also better when you use more data.
49:42So which one should I do? Should I just
49:44train on more data, a smaller model, or
49:46should I train a larger model on less
49:48data?
49:49Um
49:50so
49:51Chinchilla is a very famous paper that
49:52first showed this. Uh the way they did
49:55it, I want to give you a little bit of a
49:57sense of what these flops are. Uh here
49:59you see training loss again. On the
50:00x-axis, you see parameter parameter
50:02differences. Uh sorry, parameter size
50:04number of parameters, so the size of the
50:05model. And here all these curves are
50:08what we call isoflops, which is that all
50:11the models on this curve
50:14have been trained with the same amount
50:15of compute.
50:17Um the way that you do that is that you
50:19train you change Sorry, you vary the
50:21number of tokens that were trained on
50:23and the size of the models. But you vary
50:24in such a way that the total compute is
50:26constant, okay? So all these curves that
50:29you see with different colors have
50:30different amount of computes that they
50:32were trained on. Then you take the best
50:34one for each of those curves.
50:36Once you have the best one for each of
50:37those curves,
50:38um you can
50:40ask you can plot um how much flops it
50:44was and on which curve were you on and
50:46how much parameters did you actually use
50:49for training that specific point. You
50:51put that on the on the log log uh scale
50:54again and now you fit a scaling law
50:56again. So now I have something which
50:59tells me if I want to train a model of
51:0110 to the power 23 flops, here's exactly
51:04the number of parameters that I should
51:05be using, 100 100 B.
51:08And you can do the same thing with flops
51:10and tokens.
51:12So now you can predict
51:13if if I tell you exactly I have 1 month
51:15of compute,
51:17what size of model should I be training?
51:19Figure scaling law and I tell you.
51:21Um of course that all looks beautiful.
51:24In reality like there's like there's a
51:25lot of like small things of like should
51:27you be counting like embedding
51:28parameters? Like there's there's a lot
51:30of complexities, but if you do things
51:32well, these things actually do hold.
51:35Um so the optimal number of parameters
51:38that that Chinchilla paper have found is
51:40to use 20 tokens for every parameter
51:43that you train.
51:44Uh so if you add one more parameter, you
51:46should add you should train your thing
51:47on uh your model on 20 more tokens.
51:50So, one caveat here is that this is
51:52optimal training resources. So, that is
51:54telling me if you have 10 to the power
51:5623 flops, or if you have like 100 I
51:58don't know how much that is, 100 million
52:00dollars, or 10 No, that was much less
52:02actually. Let's say I have 5 million
52:04dollars to to train my best model that
52:06gets the lowest loss. How what would I
52:09train on? In reality, these companies
52:11need to think about inference also. If
52:13you have a smaller model, it they will
52:16spend less over time. Um so, actually if
52:19you consider the inference cost, you
52:20have other papers that try to show that
52:22um it's around 150
52:25uh parameters per Sorry, tokens per
52:28parameters. Cuz you prefer having a
52:29smaller model cuz over time you're going
52:32to you're going to actually um
52:35spend less money on inference of these
52:37models. So, 150 to 1, that's around what
52:40the best models are trained on right
52:43now, at least the ones that are
52:45that are used um
52:47in practice for in production.
52:50Great.
52:52Any question on Chinchilla?
52:56Great.
52:57Oh, sorry. In practice, how expensive is
53:00inference for these models relative to
53:02training?
53:03Actually very expensive. Uh I will not
53:06talk about inference cuz that would be
53:07another entire uh lecture, but just
53:10think about ChatGPT, where they have I
53:12don't know how much they it is in now,
53:14like 600 million people that use it. Um
53:18like
53:20that's a lot.
53:21Um
53:23yeah. So, it's actually very expensive.
53:25There's a lot of optimization you can do
53:26for inference though. Um and that's an
53:28entire other lecture, so I'm going to
53:29skip that uh this time.
53:32But it's very interesting.
53:34Okay, tuning. Um as I said, there are
53:36many things that you can uh answer with
53:38scaling laws. I just tried to give you
53:40two examples, uh but really there are
53:42many things. What data do you use? What
53:44mixture what data mixing waiting you use
53:47data mixtures? That's what we talked
53:48about before.
53:49What architecture you use, whether you
53:51should make your models wider or deeper.
53:54Should you be paying for more GPUs or
53:56actually collecting more data?
53:59All these things are things you can try
54:00to answer with scaling loss.
54:03One thing I want to say is the bitter
54:05lesson. If you ever heard
54:07Richard Sutton very famous blog post in
54:102019.
54:11What he realized
54:13which
54:15I think not enough people realize. I
54:17didn't definitely did not realize at
54:18that time.
54:21Once you see these type of scaling loss,
54:22you know that the more compute you have,
54:24the better models you will get. So with
54:27scale you will get better model. And you
54:28also know by Moore's law or these type
54:31of variants of Moore's law that you will
54:33always have better compute. Then the
54:34only thing that matters is just to have
54:37architectures that can leverage
54:39competition. So what matters is
54:41basically systems, data, and less so the
54:44architecture like the small architecture
54:46differences like your your your
54:48activation and thing like this.
54:50So I think that's like one of the
54:51reasons why most of research focuses on
54:54um
54:54some things that for industry matters
54:56less. And I was one of those researchers
54:58for a
54:59large part of my my career.
55:02Um so don't spend time overcomplicating.
55:05Do the simple things. Do it well. Scale
55:07them. That's really what OpenAI taught
55:10us with
55:11with ChatGPT and with all the GPTs
55:13before.
55:15Okay, I want to give you some back of
55:17the envelope computations. So I might be
55:19off by a few factors here, but I just
55:21want to give you a sense of how costly
55:23it is to train some of these models.
55:25I'll give as an example Llama 3 400B
55:28which is currently the best open source
55:30model that you can get.
55:31It was trained on 15.6 tokens. It has
55:36405 billion parameters. So just now that
55:39you know what is like this uh optimal
55:41tokens per parameter, that's around 40.
55:43So, that's uh a little bit more than
55:45Chinchilla, but less than this like
55:47inference uh optimal um
55:50model. So, they went for training
55:52optimality.
55:53Uh flops for this model, so one simple
55:57uh way to compute flops is six
55:59uh times the number of parameters times
56:01the number of data you train on. Uh so,
56:03if you do the simple calculation here,
56:05it's 3.8 e25 flops. The reason why this
56:08is important is that if you follow a
56:10little bit the news, there's an
56:11executive order from Biden that
56:13basically says that once you have
56:15uh 1 e26 parameters, uh sorry, flops, uh
56:19then you have special scrutiny on your
56:21models. So, they went 2x less than that.
56:24So, they really went right below this to
56:26not have special scrutiny. So, 3.8 uh I
56:29might be off by a little bit, but it's
56:30definitely under the 1 e26.
56:35Oh, um so, parameter P is parameters, N
56:40is data, number of tokens. This is uh uh
56:44this is just an approximation.
56:46We Yeah.
56:48Okay. Uh compute and we know that they
56:51trained on 16,000 H100s.
56:54Um
56:55And we know the throughput. They they
56:57said it, too.
56:58Uh so, if you do the computation, it
57:00takes around 70 days. Um or 26 million
57:04GPU hours. At least that's what my uh
57:07back-of-the-envelope computation. They
57:09actually said that they used 30 million
57:11instead of 26 million GPU hours. Um
57:14so, maybe they had like some
57:16uh some challenges. I don't really know.
57:18But, if you follow the simple
57:20computation, it's around 70 days.
57:22Um cost uh I mean, this it's hard to to
57:26approximate, but I'm just going to say
57:28it's kind of the rent like what if I
57:30were to rent H100s, that many H100s for
57:34that many days, how much will I pay?
57:37H100 a lower bound on the on the renting
57:40cost of H100 is around 2 hours $2 per
57:43hour. So, if you multiply this by 26
57:46million
57:47hours,
57:48you get 52 million dollars. So, they
57:51probably pay less than that, but not
57:54actually much less because all these um
57:57all these services that actually rent
57:59GPUs, they don't make that much money.
58:01So, it's it's probably slightly less,
58:02but not that much less.
58:04Um now salary,
58:06I said 50 employees,
58:08500k per year.
58:11Yeah, it's probably the right ballpark.
58:1225 million. So, if you put all together,
58:15around 75 million
58:17dollars for training
58:19uh this llama model.
58:21I'm probably off by like 10 million, but
58:23but that's kind of right
58:25ballpark.
58:28Carbon emitted,
58:29a lot of people might ask like also the
58:32cost is not the only thing that is
58:33important. So, I did a computation.
58:36It's around
58:374.4 4,000
58:40tons of CO2 equivalent. That is actually
58:44only 2,000 return tickets from JFK to
58:47London. So, right now
58:49carbon emitted is actually not
58:52I mean, it's huge, but it's not like um
58:55meaningful
58:56yet. I think in maybe GPT-6, GPT-7, once
59:01you multiply this by 100, that might
59:03become a real issue. Right now, it's
59:05still not I think
59:07an issue in the grand scheme of things.
59:10Next model, the way you should be
59:11thinking about these models is that
59:13every new generation, the number of
59:15flops essentially multiplies 10x. Well,
59:18at least that's what they try
59:19if they have enough energy and if they
59:21can buy enough GPUs.
59:23Great. Any question on these back of the
59:25envelope math.
59:30No. Okay.
59:32So, now we talked about pre-training. I
59:35wanted to also chat about systems cuz
59:37now we know computers really important.
59:39So, there's a question of how do you
59:40optimize the
59:42how do you optimize the computer? I will
59:43leave that for the end cuz I'm not sure
59:45how much time we will have. I think it's
59:47important, but hopefully I'll I'll be
59:48able to to talk about it later. It's
59:51slightly different than what we've been
59:53talking about right now. So, I'll move
59:55on to post-training for now.
59:57So, the task of post-training uh
59:59the reason why we need to do
60:00post-training is as I told you before
60:03um
60:04it's to make AI assistants. So, language
60:07modeling is not
60:09uh really the thing that you want when
60:11you have an AI assistant. Uh for
60:13example, if you ask to GPT-3, which is a
60:15purely language model, a pure language
60:17model, not a um not a aligned one, if
60:20you ask a question like explain the moon
60:22landing to a 6-year-old,
60:24the completion that you would get is
60:27something like explain the theory of
60:28gravity to a 6-year-old. Cuz what it
60:30learned is that on on on internet, if
60:32you have one question, you usually have
60:34maybe another bullet point of other
60:36similar questions. You don't usually
60:37have question and then answer later. Uh
60:39this is not what you want from an AI
60:42assistant.
60:43So, how do we uh do this alignment,
60:46which is this post-training and making
60:47these models assistants?
60:49Um so, the goal of this alignment is to
60:52basically get LLMs follow the
60:54instructions that are given um by users.
60:58And and maybe some designers' kind of
61:01desires. Um so, think about moderation.
61:04You don't want the models like OpenAI
61:06definitely doesn't want the model to say
61:07stuff that is very toxic.
61:10Um so, here you see on the left-hand
61:11side uh that when you ask a question, it
61:14actually provides a a real answer. So,
61:16it's not like uh before the LLM. And on
61:18the right-hand side, you see that it
61:20would uh if you ask to write a tweet
61:22describing how a certain p-
61:25part of the population are evil, it will
61:27say that it cannot do that.
61:29Um
61:30so that's kind of this alignment.
61:33Uh the background here is that
61:35uh
61:37basically the data that you want for
61:39training some of these models um
61:41is like we know what we want, which is
61:43just asking humans, this is a question,
61:45this is the answer that you want. Uh but
61:47the thing is that it's very expensive to
61:48collect that data and it's hard to find
61:50it online. Uh in contrast, pre-training
61:53data is not what you want, but there's a
61:55lot of it. Um so what what we will do or
61:58the main idea is simply take a
62:00pre-trained large language model,
62:01pre-train it on all of internet, and
62:03then you just fine-tune, so you just
62:04change a little bit the weights on the
62:06type of data that you actually want. And
62:08hopefully, given that you already
62:09pre-trained it on all of internet, it
62:10basically learns uh or knows how to
62:12speak in in English and and knows uh
62:15standard um
62:17language syntax, uh then you can really
62:20fine-tune it with very little data.
62:23Okay, SFT. So supervised fine-tuning is
62:26really exactly what I just said, which
62:28is the idea of fine-tuning the large
62:29language model on uh basically the
62:32desired answers that are collected from
62:34humans.
62:35Um so why is it called supervised
62:37fine-tuning? Because you basically want
62:39to do la- language modeling on the real
62:41answers, so language modeling is this
62:43like next word prediction, um and that's
62:45the fine-tuning part, and then you want
62:46to do it on desired answers given by
62:48humans, so that's why we call it
62:49supervised.
62:51So how do we collect this data? Well, we
62:53I just said it, you just ask humans uh
62:56to tell you this is the this is a
62:57question, this is the answer that you uh
62:59you would want from some of these
63:00models. So this is an example, um sorry,
63:03I can't read very well on computer, but
63:06uh my kid uh needs to do a science um
63:09now let's read this one. Can you write a
63:11short introduction about the relevance
63:13of the term monopsony? And then it says
63:14monopsony refers to a market structure,
63:16blah blah blah, and that's a human
63:17network that
63:19Um so actually this is Open Assistant,
63:21which was a a way to collect um
63:25uh data online by humans.
63:28So this type of supervised fine-tuning
63:31or alignment is really the key of
63:33ChatGPT. This is what made uh the big
63:36jump from GPT-3, which was mostly
63:38something that was known by AI
63:39researchers, to ChatGPT, which became
63:42known by basically everyone.
63:45Um
63:47So the problem with
63:51human data is that it's
63:53uh very slow to collect and very
63:55expensive.
63:56Um so one percent possible simple idea
64:00is to use LLMs to scale data collection.
64:03Uh so that's exactly what we did with
64:05Alpaca uh 1 year ago. What we did is
64:07that we asked uh humans or we used a
64:10data set of human uh question answers.
64:12So there were 175
64:14uh question answers here, and we asked
64:16the best model at the time, so
64:17text-davinci-003,
64:18to basically generate many more of these
64:21question and answers. So what we did is
64:23like this is what humans would write,
64:25now write similar answers and similar
64:27questions. And we collected uh 52,000
64:30LLM-generated question answers. And then
64:33what we did is simply we took Llama 7B,
64:35which was the best pre-trained model at
64:36the time, and we just fine-tuned this
64:38with supervised fine-tuning as I told
64:39you, and that's how we got um the Alpaca
64:427B model.
64:44Uh and this is the type of data that we
64:46collected. So things like what does
64:49algorithm mean? An algorithm is a
64:50step-by- a step-by-step uh
64:53set of instructions used to solve a
64:54problem or achieve a goal, blah blah
64:56blah blah. So the data is not actually
64:58it's actually pretty good, given it was
64:59LLM-generated by LLMs from essentially
65:02two generations ago.
65:04Um
65:05So that really started, at least for us,
65:07kind of as an academic replication of
65:09ChatGPT. Uh now it really there's a big
65:13field of like synthetic data generation
65:15of how to use LLMs to basically make
65:18development of LLMs faster.
65:21Um,
65:21and by basically by decreasing the
65:23amount of of human hours that you need.
65:27Quantity of data.
65:29So, we talked about what type of data
65:30and how we collect it. Um, one thing
65:32which is surprising with SFT is that you
65:34don't need that much data. Uh,
65:36so what this paper showed, this is
65:38called LIMA, is that if you have if you
65:41scale the amount of data that you use
65:43from uh, supervised fine-tuning from
65:452,000 to 32,000, it really doesn't help
65:47much. So, here scaling laws definitely
65:49don't help. Um,
65:51so the the intuition here is that all
65:53you learn um,
65:55is you learn how to format your desired
65:58answers. Another way of saying it is
66:00that your pre-trained models, they
66:02essentially model the distribution of
66:04every user on internet. One that might
66:06write bullet points, another one that
66:08might answer question answer question
66:10with an answer. So, all you tell your
66:12model is like, "Wait, you should
66:14actually be optimizing more for this
66:16type of user than another one." So,
66:17you're not actually teaching it you're
66:19not teaching anything through this um,
66:22SFT. Uh, so supervised fine-tuning, all
66:25you do is you tell the model to kind of
66:27optimize for one type of user that it
66:29saw already in a pre-trained data set.
66:31So, the knowledge is already in the
66:32pre-trained LLM
66:34uh, and you basically just specialize to
66:35one type of user.
66:38Great. Any question on SFT?
66:41Yes. So,
66:43I know it's a big issue with synthetic
66:45data where uh,
66:47if you keep generating data from the
66:49same distribution, eventually you're not
66:50learning a new distribution. You're
66:52essentially playing with the next
66:53bootstrap that that you
66:55Surely you can't scale that forever,
66:57right? You can't keep going on and
66:59generating from the same distribution
67:00and hope to learn something new. Yeah.
67:02Uh, so are there Is there an active area
67:04of research? Yeah. Any thoughts that you
67:06have around how people are maybe
67:08thinking around this and
67:10better ways to bootstrap or to give up
67:12on this idea and and realize that the
67:15church shows you don't need that many so
67:16just get humans to generate 2000 rating
67:18but
67:19Yeah,
67:20so that's a very good question. So for
67:22the data stuff, so I'm saying it's not
67:24that important for SFT but there will be
67:25another thing we'll talk about right
67:27after where actually data does matter.
67:30My intuition based on not that much
67:33empirical results
67:34is that you can still get
67:37even though you use your LLMs, if you
67:39use purely LLM generated text and you do
67:41that for like three four generations of
67:43LLMs, I agree with you that probably you
67:44won't improve much.
67:46But for me what is important is how do
67:47you use like human in the loop with
67:49LLMs? Not purely LLMs, not purely
67:53humans but maybe what you can do is just
67:54have the model regenerate some new text
67:57and just
67:58humans write a few edits. Edits are much
68:00faster than writing the entire text and
68:02I think that if you have that type of
68:03collaboration, then from like kind of an
68:06information theoretical point of view,
68:07you still get additional information but
68:09you still much faster than if you use
68:11humans. And I think that as a field
68:13we'll probably move towards these type
68:14of things which is really just finding
68:18the examples that are important and and
68:20asking humans it's kind of active
68:22learning. Just asking humans exactly
68:23when you need to to get the the inputs.
68:28Yes. Do we train with like the same loss
68:30function and the same like general
68:32training algorithm for the supervised
68:34fine tuning bit as we do for the
68:35pre-training, right? Because like
68:37the examples you showed, I think the the
68:40important thing of
68:42good examples is that they're like super
68:44factually accurate and first these more
68:46complex things
68:47Yeah, so don't just like train something
68:49Same loss. So that's why here I yeah, I
68:51didn't maybe didn't emphasize enough.
68:53This is just language modeling. Fine
68:54tune the LLM with language model on the
68:56desired answers. So this is literally
68:57the same loss.
68:59It will be different in two seconds. But
69:02the first step of SFT is literally the
69:04same loss, where you just say, "Okay, I
69:06want to actually specialize on that type
69:08of data." So, there's even a question of
69:09like, what is pre-training, what is
69:11post-training? Cuz in reality, it's just
69:12like a different data that you use. The
69:14reason why we usually call it
69:15post-training is that the way we collect
69:16that data is very different.
69:19Great great questions.
69:21Uh yes.
69:22Maybe it's the same question, but why
69:24would these 2,000 examples have such a
69:27overweighted influence? Uh
69:29Yeah, in fine-tuning. So, that's why we
69:31uh
69:32also that's another reason why we call
69:33it post-training, is that we use
69:34different type of hyper parameters. So,
69:36you know I told you basically at the end
69:37of pre-training, you essentially end up
69:39with a learning rate of zero. Uh here,
69:41you're going to increase your learning
69:42rate to like 1e-5, 1e-minus Yeah. And
69:44and so, um the weight that you give to
69:47them is actually different.
69:50Um
69:53Okay.
69:54Uh second step or second part of this uh
69:57post-training um is what we call
69:59reinforcement learning from human
70:01feedback or RLHF. Uh some of you might
70:03have heard of that. Um
70:06the idea is that SFT has a problem,
70:08namely that uh you do behavioral
70:10cloning, which means that you just try
70:12to clone what the humans would say. And
70:15that had many that has many issues. One
70:17of them is that you're bound by human
70:18abilities. So, if
70:21um
70:22like humans actually
70:24humans won't generate the things that
70:27they think is actually the best thing to
70:28generate. So, if you ask me to write a
70:30book, I mean, I can definitely enjoy a
70:32book. I can probably say one book is
70:33better than another, but I'm definitely
70:35not going to be as good as writing the
70:36book that I want to read. Uh so, you're
70:38going to be bound by the human ability
70:40to generate things, even though the
70:41humans might be better at distinguishing
70:43between things. That's one issue. Issue
70:45number two,
70:46uh I find that actually pretty
70:47interesting, is that it might if you've
70:49ever heard of the word hallucination, so
70:51this is LLMs generating uh fake like
70:54false information.
70:56Hallucination might uh these people have
70:58um hypothesized that that can come from
71:01the supervised fine-tuning even if you
71:03do supervised fine-tuning on data that
71:05is correct. And the reason why that is
71:08is that if
71:10given I told you that basically SFT is
71:12with very little data and it's with data
71:15that doesn't model doesn't learn
71:16anything new. So what if the human gives
71:19an answer
71:20that the model didn't know was true?
71:24From the model perspective you the human
71:26basically is telling the the model
71:28generate this thing that seems plausible
71:31but I actually have no idea if it's true
71:33or not. Um
71:34so just to give you a very concrete
71:36example, if we go back to this monopsomy
71:38example, can you write blah blah blah
71:40about monopsomy?
71:42Imagine that the human wrote a reference
71:44on this type of book. Um and that book
71:47might exist. That might be a correct
71:49reference. But what if the LLM never saw
71:51this reference during pre-training? Then
71:53it doesn't know that it's a correct
71:54reference. So really what you tell the
71:56model is to generate or make up some
71:58plausibly sounding reference.
72:00Um rather than actually tell the real
72:03reference that it saw during
72:04pre-training. Uh so hallucination might
72:08be um
72:10a like might be caused by this SFT.
72:12That's problem number two. Does that all
72:15make sense?
72:16Great.
72:17Problem number three, price. Generating
72:19the ideal answers is very pricey. And
72:22that comes back to your question
72:24of like humans writing an entire answer
72:26is actually pretty expensive.
72:28Um so that's where RLHF comes in. The
72:30idea is that instead of cloning the
72:32behaviors of humans, we're going to
72:35maximize human preference. Um and the
72:38way we're going to do that through the
72:39pipeline is that for a certain for every
72:42instruction, you're going to ask a model
72:43to generate two answers.
72:46Um and usually use a pretty good model,
72:48so you usually don't use an LLM here.
72:50You use a
72:51SFT fine-tuned use a fine-tuned LLM
72:54already to give like pretty good
72:56answers.
72:57And then you ask labelers which of these
73:00two answers was better. So, select the
73:02preferred one.
73:03And then with different type of
73:05algorithms, we're going to talk about
73:06the algorithms, um, you just fine-tune
73:08the model to generate more of the green
73:10thing than the red thing. So, more of
73:11the good stuff.
73:12Uh, so now the question is how? And
73:14we're going to talk about that right
73:15now.
73:17So, there are two ways that we're going
73:19to talk about and two that are mainly
73:21used in the community. Um, the first one
73:24is simply the idea of using
73:25reinforcement learning. So, hopefully
73:27you all know what reinforcement learning
73:28is now. Um, so when you think about
73:32using reinforcement learning, one
73:34important question is like what is the
73:35reward that we're optimizing? Uh, so
73:37then this case there are really two
73:38options that I could think about. The
73:40first one you could just say, "I'm going
73:42to compare the output generated by some
73:43baseline, the output generated by my
73:45model,
73:46uh, and I'm just going to ask the human
73:48to say which one is better, and I'm
73:50going to use this as a reward. So, if
73:52I'm better than the baseline, this is a
73:54plus one, if not, it's a minus one. Uh,
73:56so now it's binary reward. The problem
73:58with binary reward is that it's very
73:59sparse, and you don't get much
74:01information out of it. Uh, like maybe
74:03your answer was slightly better, maybe
74:05it was like way better, and you don't
74:07really know from this, um,
74:09how much better it was.
74:11So, option two is that you can train
74:13what we call the reward model, which is
74:15simply a classifier.
74:17Uh, so you use machine learning to to
74:19classify how much better, uh, two
74:22outputs are from the preference from the
74:25perspective of the human. Um, so this is
74:28a little bit meta, but what you
74:29basically do is that you train, uh, you
74:31take a, um,
74:33re- a reward model R, which is a, uh,
74:35just a large la- also a large, um,
74:38a large classifier. And you basically
74:40ask this reward model, you give it the
74:42input and the actual output that you
74:44have, one of the two outputs. Uh, and
74:46you just, um, exponential that, so
74:48that's the softmax loss that you all
74:50know about. And now you divide by
74:53um the the exponentiated reward uh on
74:57the first example, uh sorry, on the
74:59first output, and this is on the second
75:01output. And you basically train So, the
75:03reason why you do that is that you train
75:05your your model, you train this reward
75:07model to be able to classify
75:09um
75:10how much better one output is to another
75:13one. So, another uh slightly less
75:16convoluted way of saying it is that your
75:17reward model will output some reward
75:20that will be used as the logits of your
75:22softmax. So, now if you have high logits
75:25in your softmax, it means that you
75:27highly likely this um
75:30output is better.
75:32Uh so, that's what we call Bradley-Terry
75:34model.
75:35Yes. Is this reward model going over the
75:37entire output, or is it going token by
75:39token?
75:40Um so, this takes the entire
75:45uh
75:45Yeah, this takes the entire output at
75:47once. So, it takes all the input and all
75:48the output, and it gives one number.
75:51Yes.
75:52So, um let's say both of these are
75:53reward model. Where would the human be
75:55then?
75:55Sorry? With the reward model, where
75:57would the human be like
75:59Oh, I see. Uh sh- Okay, sorry. I I maybe
76:01I wasn't clear. Um you train this reward
76:04model to fit this green and and red
76:09preference from humans. So, basically
76:11you train a classifier to say whether
76:13the humans prefer red or green.
76:16Uh but instead of using the binary
76:18reward, which is what the human would
76:19tell you, you basically use the logits
76:22of the softmax. And the thing with the
76:24logits is that logits are continuous.
76:27So, now you know that if your reward
76:28model said it has high logits, then in
76:31some ways the human highly prefer this
76:33answer to some other answer.
76:37Great. Um
76:39So, like I said, continuous information
76:41is that's better. So, that's what people
76:42uh use in practice, or at least used to
76:45use in practice. I'll tell you about the
76:47other algorithm later.
76:48So, what you do at the end is that you
76:50basically try to just use reinforcement
76:53learning that you know about. Now we
76:54know where we we have our reward. What
76:56you sample through is the generation
76:58from your large language model. Um and
77:00then you just use some regularization
77:02terms. So, the reason why we do this
77:03regularization term is for avoiding what
77:05we call over-optimization. So, this
77:07reward model might not be really
77:09represent like might not perfectly model
77:11human preferences. So, you don't want to
77:13maximize this thing to essentially
77:15infinity. Um
77:17and you do it using PPO, which is a
77:21common
77:22reinforcement learning algorithm.
77:24Um one thing to note here, cuz it will
77:26be important for later, is that when we
77:29use maximum likelihood, um
77:33Oh, sorry. Now the large language models
77:35are actually a policy for your
77:37reinforcement learning. It's not
77:39maximizing maximum likelihood anymore,
77:41which means that you're not modeling any
77:43distribution anymore. And the reason why
77:45this is important is that models that
77:46went through this type of PPO actually
77:49don't give you likelihoods of text that
77:52are meaningful. Cuz what you optimize
77:54them to do is basically just optimize
77:56for generating the most likely thing.
77:58Not optimized for modeling like all the
78:01answers that humans might say. Another
78:03way of saying that is that there's
78:04nothing that incentivizes here the model
78:07to not give a like a
78:10a single possible generation. Nothing
78:12here says it's good if you have some
78:15distribution with some entropy.
78:17Um all right, if you haven't followed,
78:19it's not that important, but just good
78:21to know.
78:22Great.
78:23So, PPO is exactly what ChatGPT did
78:27originally. So, here's the on their blog
78:29post um
78:30what they have is step one, do
78:32supervised fine-tuning, which now you
78:34all know about. Step two, train a reward
78:36model on human preferences. Step three,
78:39do PPO multiple steps, which is where
78:41you see this this blue arrow. So, you
78:43continue to you train a model once with
78:45the PPO, you collect new data, you
78:46continue.
78:48Uh and that's why and that's exactly
78:49what ChatGPT did.
78:51Uh and that was a big breakthrough
78:52between GPT-3 and ChatGPT.
78:55One thing to note is that uh PPO has
78:58many challenges. Reinforcement learning
79:00is something that's super nice
79:02theoretically. In practice, anyone who
79:03ever worked with reinforcement learning
79:05knows it's such a mess. Uh there's a lot
79:07of things like the rollouts, outer
79:09loops, clipping, there's so many
79:10complications.
79:11Um
79:12so, it's messy. This is the idealized
79:14PPO used for LM settings. So, that's
79:16already much more complicated than this
79:18expectation we saw before. And in
79:19practice, it's actually much more
79:21complicated. So, we have one
79:22implementation of it that we had to do
79:24and I'm got not going to go through it,
79:25but basically you have like so much
79:27stuff that you have to think about when
79:29you implement that type of of uh PPO
79:31algorithm. So, you have clipping
79:32everywhere, you have a lot of
79:34complexities, and uh things are not well
79:36documented.
79:38All this to say um that we're going to
79:41there was a new method that was proposed
79:43uh also from Stanford 1 year ago called
79:45DPO, which is essentially a
79:47simplification of PPO. Um
79:50and the way uh
79:52what they did or the idea that they have
79:54is that instead of using reinforcement
79:56learning, you can just maximize the
79:57probability of generating the stuff that
79:59you like and minimizing the probability
80:01of the stuff that you don't like. Uh so,
80:03if you think about the human preference,
80:04the red and green, maximize
80:06uh green, minimize red. Um so, the loss
80:10is actually this one uh where what you
80:12see, this is simply um
80:14some log of the model. So, this is the
80:17likelihood of the model generating the
80:19things that the human preferred given
80:21the the inputs. Um
80:23and what you try to do is basically
80:26maximize uh
80:28the likelihood of generating the things
80:30that you like and minimize the
80:31likelihood of the things that you don't
80:32like.
80:33Um
80:34All the rest of the terms, here it's not
80:36too important. It's actually really not
80:38that complicated to understand. But at a
80:40high level, it's really just maximizing
80:42the things you like, minimizing the the
80:44rest.
80:45Um
80:46and one thing to note, uh which I was
80:49going to say just here, is that actually
80:50all the rest is chosen such that um the
80:53global minima of PPO and the global
80:56minima of like this DPO under some
80:59assumptions are essentially equivalent.
81:01So, this is the right thing to do uh
81:04mathematically. I'm not going to go
81:05through the derivations, but that's the
81:07right thing to do.
81:08Uh it's pretty different with PPO in the
81:10sense that now uh with PPO, what you had
81:12to do is collect the human preferences,
81:14then train your uh reward model with
81:15maximum likelihood, then use
81:16reinforcement learning. Now, all you do
81:18is basically maximum likelihood. Much
81:20simpler. Yes. I mean, yeah, so it seems
81:22like this is A much simpler, B like what
81:25you just intuitively do. But why didn't
81:28they start with this reward model? Like,
81:30what what let them do that?
81:32I think it's a great question. Uh I
81:33don't really know. What I can tell you
81:35is that at OpenAI, the people who did
81:38the
81:39um
81:40uh who did basically this PP uh sorry,
81:42who did ChatGPT initially are the ones
81:45who actually wrote PPO. And I think they
81:48were just like they were a lot of
81:49reinforcement learning people, and I
81:51think that for them it was very
81:52intuitive.
81:54Um
81:55so, there's also some additional like
81:57potential benefits. For example, I don't
82:00want to
82:01Yeah, for example, if you use the reward
82:02model, uh the cool thing here with
82:04reinforcement learning is that you can
82:05use unlabeled data
82:07with the reward model. So, here you can
82:09only use the labeled data for doing DPO.
82:12Um for PPO for PPO, you first train your
82:14reward model, and then you can use
82:16unlabeled data uh
82:18where the reward model will basically
82:20label this unlabeled data. So, there's
82:22there's additional kind of potential
82:25uh
82:25there could be potential improvements.
82:27In practice, this happens at OpenAI, and
82:30I think it's just that a lot of people
82:32in this team were reinforcement learning
82:34experts, including the main author of
82:36PPO, John Schulman.
82:39Um so, much simpler than PPO, and it
82:41basically performs as well. So, now this
82:44is the standard thing that people use,
82:46at least in the open-source community. I
82:48believe it's actually the standard also
82:50in
82:50in industry.
82:52So, that's called DPO.
82:54Gains. Um so, those are older papers on
82:57the left. Here, this is on a
82:58summarization task. You see All I want
83:01to show you is that basically the
83:02pre-trained models
83:04were okay, and they improve with scale.
83:06If you do supervised fine-tuning, you
83:07improve them a little bit more. If you
83:09do PPO or something with RLHF, with
83:11human feedback, you get performance that
83:13are as oftentimes, depending on the
83:16benchmark, even better than humans. So,
83:19this is the human reference summaries.
83:22Same thing. This is on a
83:23on a paper that we have, Alpaca Farm,
83:25where we see
83:26the evaluation here is not too
83:27important, but basically you see
83:28pre-trained model, you jump to SFT, and
83:31then you jump to PPO DPO, and PPO DPO
83:34have the exact same performance.
83:37So, basically RLHF helps. That's kind of
83:39the conclusion, and DPO is simple.
83:42Uh data. The way that you collect that
83:45type of data
83:46um
83:47First idea is just use humans, as we
83:49already talked about.
83:51Guidelines are very complicated for what
83:53humans should be labeling, and it's
83:55really not that easy. And actually, if
83:56you ever do some of the labeling, you
83:58will see that it's extremely
84:01complicated. Like, if I zoom in to this,
84:04here I have a question, "Tell me Tell me
84:06about self-driving cars." And you read
84:08both. "Self-driving cars are vehicles
84:10that are capable of detecting their
84:11surroundings, blah blah blah.
84:12Self-driving cars are cars that are
84:13equipped with sensors, blah blah blah,
84:15to navigate without the need for a
84:16driver." I mean, both seem okay. Like,
84:18which one is better? It's actually hard
84:20to say at a glance.
84:22And as a result, uh, the problem with
84:24humans is that
84:26you will start optimizing a lot of like
84:28high-level features. For example, the
84:29second one is longer. I can guarantee
84:31you that most humans will choose the
84:32second one, even though I mean maybe the
84:35first one is better, I don't know.
84:37I haven't read it carefully.
84:39So, challenges with humans. First, slow
84:41and expensive.
84:42Uh, second, as I just mentioned, it's
84:45hard to focus on things that matter,
84:47like correctness. And people, uh,
84:49usually look at things that don't matter
84:51as much, like the form, like length.
84:53Uh, and as a result, so what I show here
84:55is that, uh, when you do RLHF, the more
84:57you do of RLHF, the longer the output of
84:59the of the models become. So, if you've
85:02ever been annoyed at ChatGPT answering
85:04you super long sentences, this is
85:06because of RLHF.
85:08Um, annotator distribution shift. Uh,
85:12like the distribution of annotators that
85:13you use matters a lot. And you have to
85:15think like, what is what is even the
85:17humans that we want to represent in
85:19these models?
85:20Uh, now the question is like
85:21crowdsourcing ethics. Uh,
85:23like usually these basically a lot of
85:25the the labeling that is done,
85:27um,
85:28like the people who do it them are not
85:30paid well, and they have to go through a
85:31lot of toxic data. Uh, because you
85:33basically want the model to avoid saying
85:35the toxic data. Um, so crowdsourcing
85:38ethics, too.
85:40So, many challenges with human data. Um,
85:43so what we did also last year is again
85:46the same thing as Alpaca, just the idea
85:48of like, oh well, there are challenges
85:49with humans, maybe we can just replace
85:50them with LLMs. Uh,
85:52so what we did is simply replace
85:55um, oh, I see that.
85:57I'm just realizing that the slides are
85:58not centered. Anyways, uh, you replace a
86:01human preference with LLM preferences.
86:03Uh, so here on this, uh, figure you see
86:05on the X axis the price that we paid,
86:08uh, for collecting human data. It's
86:09around $300 for 1,000 examples, and this
86:13is on Mechanical Turk, which are, uh, uh
86:15like cheaper than than maybe some of the
86:18other
86:19um companies that you could go through.
86:21And on the Y axis, it's basically the
86:23agreement with uh other humans, with the
86:26mode of other humans. And what you see
86:28is that actually, as I told you before,
86:29labeling is really complicated. Humans
86:31agree with themselves only around 66% of
86:34the time on a binary task. And it's not
86:37that uh humans are not good here because
86:39uh we were five main authors on this
86:41paper. We tried to label this data
86:43ourselves, and we only had like say 67
86:46or 68% accuracy. Even though we talked
86:49like we talked for like 3 hours of how
86:50we should be doing labeling. But really,
86:52it's complicated. It's not an easy task.
86:54Um and here I just showed many different
86:56models. And um basically, you see that
86:58models are much cheaper, and they can
87:00actually get higher agreement with the
87:02mode of humans than human humans
87:04themselves. And the reason why is
87:05because humans have a lot of variance.
87:07Models have no variance. So, they might
87:08be a little bit more biased, but have
87:10less variance.
87:11Uh so, it works surprisingly well. And
87:14now it's kind of the standard in open uh
87:16source community. I think even in
87:17industry, a lot of people use both
87:19humans and LLMs for improving uh the
87:22collection collection of all HF data.
87:24Um and this is like this is the paper
87:27from last year, but honestly, now it's
87:28more like
87:29that LLMs would be around this agreement
87:31and this cost. So, around I would say
87:3350x cheaper than humans and better
87:35agreement with human than humans
87:37themselves.
87:39Okay. So, that gets us to evaluation of
87:43post-training.
87:44Um that goes back to your initial
87:46question at the beginning of the
87:47lecture. How do you evaluate something
87:49like ChatGPT? Uh the answers that
87:51ChatGPT could give
87:53are basically unbounded.
87:55And it's not that there's one right
87:56answer. There are many answers that are
87:58just as good. Um so, there are many
88:00challenges. One, you can't use
88:02validation loss because one method might
88:05use PPO, the other one might use DPO.
88:07Validation loss is not comparable.
88:09Second, you can't use color sorry,
88:11perplexity. That's the thing I told you
88:13before. These models uh are not
88:15calibrated. They don't give
88:16distributions. They they just optimize
88:19for one thing. So, you can't use
88:20perplexity for actually evaluating these
88:22type of models once they're aligned.
88:24Sorry, once they're aligned.
88:27Third,
88:28there's a large diversity of questions
88:29that human might ask to these models.
88:31Generation, open QA, some question
88:33answering, some summarization, and all
88:36of these things. So, there's so many
88:37things you have to cover.
88:38Um
88:39then the tasks are really open-ended.
88:41So, it's very hard to automate. So,
88:42that's what you were alluding to before.
88:45So, the idea
88:47is that instead of trying to come up
88:49with really easily automated benchmarks,
88:52it's just we're going to ask questions
88:54that that users actually ask to these
88:55models in practice, and we're just going
88:57to ask annotators to say between these
89:00two models, which one is better? Like,
89:02what's the what's the better output? So,
89:03basically you do the exact same thing as
89:07basically the data from RLHF, but you
89:09use it now for evaluation. Yes. I'm not
89:11sure I understand what you mean by that.
89:12Like, you can't use perplexity not
89:13calibrated really.
89:15Well, I'm just still doing like next
89:17token prediction.
89:19Um So, I can't perplexity be used on So,
89:21think about um
89:23the optimal solution after doing PPO is
89:26basically one model that gives you uh
89:29essentially a delta um
89:31like basically it says that there's only
89:32one sentence that is
89:35that could be generated for that
89:36question. So, now if you use it on
89:38something that's slightly semantically
89:39differently different, it would actually
89:41give a likelihood of zero for that
89:43answer.
89:44So, in reality it's not that extreme cuz
89:46as you say it's still a distribution,
89:48but it just shows you that there's a
89:49there's a fundamental issue with
89:51perplexity once these models are not
89:54LLMs anymore. They were not trained, at
89:56least with PPO they were not trained to
89:58to do maximum likelihood anymore. They
89:59were trained to be policies.
90:03Okay.
90:04Um so, probably the most common or like
90:06the most, um,
90:09yeah, the most common benchmark, or like
90:10the most trusted one, is what we call
90:12chat uh, sorry, Chatbot Arena, uh, which
90:14is basically go on internet, have random
90:16users on the internet blindly talk with
90:19two chatbots, just ask many questions,
90:21see the two answers, and rate which one
90:23is better. And you do that over 100 or
90:26thousands of users, and then you get,
90:28um, the actual preferences, and you get
90:29rankings of models. Uh, so you can go
90:32right now on Chatbot Arena and actually
90:34interact with these models. Um, one
90:37potential issue, just to highlight, is
90:39that while people who want to do these
90:40type of things are usually more like
90:41tech-driven, um, or like tech-savvy. Uh,
90:44so a lot of the questions that you will
90:46ask, uh, more like tech stuff,
90:47discussing software errors, inquiries
90:49about AI tools, and all these things.
90:52Um,
90:53so another issue is cost and speed. If
90:55you really want to use something like
90:56this for development process, um,
90:59it will be too costly, cuz you would
91:00need to basically pay a lot of humans to
91:02do that.
91:04So one simple idea is, again, as we said
91:07many times, just use LLM instead of
91:09humans. Uh, you probably know the drill
91:12at this point. Uh, steps for every
91:14instruction, generate outputs by some
91:16baseline and the model that you want to
91:18evaluate. Um, so here you imagine that I
91:21I'm comparing an answer from ChatGPT and
91:24from Mistral. I'm just asking a model,
91:26uh, another model, uh, which one is
91:29better, and I just basically average
91:31that out. Uh, yeah, I asked ChatGPT for
91:34which one is better, I average that out
91:35over my entire distribution, over over
91:37my entire benchmark or data set, and it
91:40that gives me a a win rate, so a win
91:42probability for one model uh, compared
91:44to another one, and now you can rank
91:45models.
91:47Uh, and this is the Alpaca Eval, uh,
91:49leaderboard.
91:50So the benefits of this is that actually
91:53we show we get 98% correlation with
91:55Chatbot Arena, so very high correlation
91:57with humans. Um,
91:59so this is, yeah, comparison with
92:01correlation with other benchmarks, and
92:03it takes less than 3 minutes and less
92:04than $10 to run. So, it's pretty cheap.
92:07Um there are downsides though. Uh one of
92:09them is spurious correlation.
92:11Um so, as we already saw before
92:14LLMs prefer uh this is one spurious
92:16correlation, not many. I'll just talk
92:17about one. LLMs prefer longer outputs.
92:19Actually, humans also prefer longer
92:20outputs, but the problem or the issue
92:22once you use LLMs is that once there's
92:24bias, you will continue optimizing that.
92:26Humans at some point, I can guarantee
92:28you if I ask a simple question and you
92:29give me five pages of answers, I'll be
92:31like, "No, I don't like that answer."
92:33But LLMs if they have this bias and they
92:34were trained for that, they will
92:35continue preferring longer outputs.
92:38So, uh here we see um
92:41the the preference just showing that
92:43like humans and models prefer longer
92:45outputs.
92:46Um and here is another view of the
92:48initial Alpaca Eval dataset uh benchmark
92:51where when we asked um
92:54when we rank GPT-4, when we look at the
92:56win rate of GPT-4 versus actually uh
92:58GPT-4 itself, if you if we use the
93:01standard GPT-4, it gets 50% kind of by
93:03definition cuz we're comparing GPT-4
93:05versus GPT-4, but if we ask a GPT-4 to
93:08be slightly more verbose, so we just say
93:10in the prompt, "Be verbose in your
93:11answers." Then it gets a win rate of
93:1364.4%. So, really there's a huge
93:16variance and if you ask it to be
93:17concise, it gets 20%. So, there's a huge
93:19variance depending on um
93:22whether you ask it to be concise or
93:23verbose. That's very annoying. Um so,
93:27one possible solution, which is what we
93:29did, is uh just use some regression
93:31analysis. I'm not going to go into
93:32details, but basically use causal
93:34inference tools to control for length
93:36and right now uh actually length matters
93:38much less. So, if you ask it to be
93:40verbose, you still get some gains, but
93:42much less.
93:45Great. So, that's all about
93:46post-training and now for the next 8
93:48minutes, I might talk about systems or
93:50just answer questions. Yes. Can you um
93:54go back to your post-training in terms
93:56of post-training?
93:58how did we tune those parameters using
94:00the small body of fine-tuning data and
94:04have such big effect on the model? You
94:06mentioned earlier that there's a
94:07different set of hyper parameters. Are
94:09we changing just some of the weights,
94:11the later weights, or all the weights?
94:13What's actually happening? Yeah.
94:15Yeah, I I kind of skimmed through all of
94:16this. You change all the weights.
94:18Actually,
94:19industry would change all the weights.
94:21In open-source land, you might have
94:23heard of Laura, which is going to change
94:26it basically only some of the weights.
94:28Or it actually, to be more specific,
94:30it's going to add some differences to
94:31the output of every of every layer. But
94:34but in industry, you're going to just
94:35fine-tune all the weights.
94:37Um and
94:39also to say something else about the
94:40data, actually this last step RLHF, you
94:42usually going to collect
94:44a lot more data than with SFT. So if SFT
94:47is like 5,000, 10,000, maybe 50,000,
94:50with RLHF, I think you're going to be
94:52more around like the 1 million uh order
94:55of magnitude. It's still much less than
94:56pre-training, though. Yeah, because
94:58pre-training is 15 trillion tokens. I
95:00mean, this is like that's not even a
95:02drop. And yet, you influenced the
95:04weights a lot. So
95:05Because you do it I mean, you have to
95:06think that how you do it is you use
95:10um
95:11I mean, as I said, the learning rate
95:12that you're going to use is going to be
95:13different. But also, you only do that.
95:16So just imagine if I trained even if I
95:19trained on one sentence, but over and
95:21over again, or at some point, my model
95:23will only generate that sentence. Even
95:25if
95:26uh it was just one sentence instead of
95:28the 15 trillion tokens. So if you use a
95:30large enough learning rate and for
95:32enough time, you will basically overfit
95:34that sentence. So the the the key thing
95:36to to remember is that um
95:39the data is not I It's not as if you mix
95:41some post-training data and some
95:43pre-training data. You do pre-training
95:45and then you just start fine-tuning on
95:47only on the post-training. So in other
95:48way, maybe another perspective is that
95:51the post- the pre-training is just the
95:52initialization of your model. And once
95:55you view it that way, that this is just
95:56initialization of weights,
95:58then there's nothing special. Like you
96:00don't need to remember that you train a
96:02lot of data before. The only thing that
96:03matters is that you had an
96:04initialization, and now I actually train
96:06a model. So, when you think about it
96:07that way, like there's there's a Markov
96:09property in some ways. It's just like
96:11you had your weights, this is my
96:12initialization, now I'm training that
96:14one.
96:14Does that kind of answer your question?
96:16Kind of, but
96:18you you said something just now about
96:20it's almost the equivalent of just
96:22re-running the fine-tuning data many
96:24times. Is it actually Is that what
96:27actually happens in order to
96:29give so much more preference?
96:31Um
96:34you might have I actually don't know
96:36right now how they do it in industry.
96:38When we did our packet, we had to do
96:39three epochs. So, you did run it three
96:41times through it. Um
96:45but I mean, even the number of times
96:46that you run it through, it's actually
96:48not important. The only thing like The
96:50only thing is the is kind of the
96:51effective learning rate. That's what
96:53matters.
96:54Um so, yeah.
96:57Great.
96:59So, I think I have 5 minutes, right?
97:03Um
97:06Okay. I might try to give a high-level
97:11overview, at least from one of the
97:13systems trick.
97:15Systems, as we said, uh
97:17for everyone, bottleneck is a sorry,
97:19compute is the huge bottleneck.
97:21Uh one question you might ask is why not
97:23buy more GPUs?
97:25Uh GPUs are expensive, but also are
97:26scarce. Even if you have $10 million
97:28right now, you cannot buy the best GPUs.
97:31Um
97:32Cool. Ooh. Oh, yeah. There's also some
97:34physical limitations. When you have When
97:36you have multiple GPUs, you have to
97:38communicate between them. That takes
97:40time. Um so, just buying more GPUs is
97:42not that easy. Um so, it's really
97:44important to think about how do you
97:46allocate resources and how do you
97:47optimize your pipeline or system.
97:50101 on uh GPUs, I'm sorry I'm going
97:52slightly faster. I hope for that some of
97:54you at least can follow. Uh GPUs are
97:56basically optimized for throughput. CPUs
97:59are optimized uh for latency. So, GPUs,
98:02the way you have to think about it is
98:04that there's one there's one command
98:06that is run on many many cores at the
98:08same time on different type of data. Um
98:11So, this is how you see a GPU. You see
98:14there are many different cores we call
98:15them uh streaming uh multiprocessors,
98:18which is very different than the usual
98:19uh CPU architecture. So, just think high
98:21throughput
98:23parallelization for GPUs.
98:25Uh GPUs are optimized for fast matrix
98:27multiplication. So, every time you will
98:29do uh you will do something on GPU, if
98:31you can do it with a over over a matrix
98:33multiplication, it's going to be 10
98:34times faster than with anything else. Uh
98:37that is a little bit annoying cuz it
98:38means that we are kind of
98:40uh bottlenecked to doing anything with
98:42matrix multiplications.
98:44Um another thing to know about GPUs is
98:46that compute has been improving faster
98:49than memory and communication. So, right
98:51now GPUs usually are hard to keep uh
98:56like the data that you send it send to
98:58GPUs is actually hard to keep up with
99:00the process. So, most of your GPUs are
99:02actually going to be idle if you just
99:03run normal code uh if you don't optimize
99:06your code. So, communication had and
99:08this will continue over time.
99:11Another thing to know about GPUs is that
99:13there's a memory hierarchy. This is the
99:14same thing actually with CPUs, but
99:16basically the closer you you are to your
99:17cores, the less memory there is, but the
99:19faster things run. If you're further,
99:22more memory, slower.
99:24Um
99:25Oh yeah, I'm going to skip that.
99:26Okay, actually I'm going to say it.
99:28I told you about this uh the fact of
99:30communication. Uh the metric that people
99:32usually look at is model flop
99:34utilization. So, what is the theoretical
99:36maximum that uh GPU could run at, the
99:38most flops that you could use per
99:39second, uh divide it's the number of
99:42observed throughput divided by the
99:43theoretical um uh maximum. And in
99:47general, if you reach 50%, you're very
99:49happy. Like Facebook I looked at Lima
99:51was at 45 or something like this. So,
99:53that means that data doesn't come fast
99:56enough even for these big companies.
99:59So, one simple trick and that might be
100:01the only one I'm going to talk tell you
100:02about is low precision.
100:04One simple idea is that well, if I'm
100:07going to put my floats in lower
100:09precision, then it's going to be fewer
100:10bits that I have to send to my GPUs. If
100:13there's fewer bits, it's faster
100:14communication, lower memory consumption,
100:16things are going to go faster.
100:18Uh and for deep learning, it just
100:19happens that
100:20decimal is not that important. Uh so,
100:24when you do matrix multiplication, when
100:26you do like for example SGD, there's
100:27already so much noise that if you update
100:29something by 0.01 or 0.015,
100:33who cares? Uh so, basically instead of
100:35using
100:36uh 32 bits per float, which is um what
100:39people used to use or 64 for example,
100:41which is what you would use in other
100:43domains, you use 16 bits uh for matrix
100:46multiplication. So, for every float you
100:47use 16 bits.
100:49Um
100:50and for training, you have this type of
100:51like
100:52uh what we call automatic mixed
100:54precision, which is that uh some of the
100:56things are in 32 bits, others are in 60
100:58bits uh on 16 bits. Um generally, the
101:01way you should be thinking about it is
101:02that your weights are stored of your
101:05model are stored in 32 bits,
101:07um but just before the computation, you
101:09put everything in 16 16 bits like this,
101:11you do computation super fast, and at
101:13the end, you update your weights in 32
101:16bits. And the reason why you do all the
101:17updates in 32 bits is just think that if
101:20your learning rate for example is very
101:21small, you still want to be able to like
101:23make a difference in your weights. Uh
101:25so, all the computation is done in 16
101:28bits, but uh the weights are actually
101:29stored in 32 bits. So, that's like the
101:31standard way that people are doing it.
101:34Um
101:35Okay, I'll actually talk just about this
101:37and then I'll skip all the rest.
101:38Operator fusion cuz I think this is
101:39actually pretty cool.
101:40As I just said, communication is very
101:42slow. And actually every time you use a
101:45PyTorch line, it basically moves
101:47variable to global memory of your GPU.
101:49So, when you have something like this,
101:51x. cosine
101:54= x1 and then you do x1.cosine, what is
101:57happening behind the scenes is that you
101:59take the x which is data, you ship it to
102:01your to your actual processes of your
102:03GPUs, you apply the cosine, you ship it
102:06back to the main memory of your GPU, and
102:08then you see the next line, you ship it
102:10back to the computer to the GPU
102:12processor, you apply another cosine, and
102:14you ship it back again.
102:16So, another way to see that is that you
102:18go from your DRAM, which is your global
102:19memory in your GPU, and you ship it to
102:22compute, you ship it back for every
102:24line. This is a naive way of doing it.
102:26This seems very wasteful.
102:28So, the idea simple idea of operator
102:31fusion is just communicate, do all the
102:34computation, ship it back once. And this
102:37is exactly what a fused kernels are.
102:40So, if you ever want to make your
102:42your
102:44computations in PyTorch much faster,
102:46just apply torch.compile
102:48on your model. This is going to make
102:50your model around two times faster, and
102:52what it does is simply that it rewrites
102:55your code
102:56your pipe like your PyTorch code
102:58basically in C++ and CUDA
103:02to to do the communication only once,
103:04then do all the operations, then ship it
103:06back.
103:08Okay, I'm not going to have time to talk
103:09about tiling. Tiling is important.
103:12Parallelization, parallelization is
103:13important.
103:16And mixture of experts, mixture of
103:17experts is important. Outlook, there are
103:20many things we haven't talked
103:22talked about.
103:23We haven't talked about architectures,
103:25we definitely haven't talked about
103:26inference.
103:28There are many other things that are
103:29important with LLMs. What is the UI that
103:31you use? I mean, arguably, ChatGPT the
103:33big novelty was just have a simple UI to
103:35use it. Multimodality, what are all the
103:37misuses you could have, uh the fact that
103:39they might not be enough data on the
103:41internet to train all these models,
103:42legality of data collection, so many
103:44other things. If you are interested in
103:46all these topics, uh I would suggest
103:49three classes. CS224N is probably the
103:51one that touches the least on uh LLMs,
103:55uh but it gives some background and
103:56historical context um of all the LLMs
103:59and gives kind of some adjacent
104:01material. CS324, I think it's called uh
104:05I think it's called uh large language
104:06models. Uh more in-depth reading and
104:08lectures on everything I talked about.
104:10CS336, which is large language model
104:13from scratch. You actually build your
104:15own LLM. Uh it's an amazing class, also
104:19given by my two supervisors. Very heavy
104:21workload, so be careful.
104:23Um
104:24great.