Machine Learning I Building Large Language Models LLMs

Life and a bit more

youtube· 104:21standard

1.Введение в большие языковые модели

0:00 / 3:00

Обзор курса о построении LLM. Рассматриваются основные компоненты: архитектура, функция потерь, данные, оценка и системные аспекты. Объясняется, почему в практике важнее всего данные, оценка и системы, а не архитектура.

LLM (Large Language Models)
Архитектура
Функция потерь
Данные
Оценка

Chapters

8/17

Chapters 8/17

2. ?? locked
3. ?? locked
4. ?? locked
5. ?? locked
6. ?? locked
7. ?? locked
8. ?? locked
9. ?? locked
10. ?? locked
11. ?? locked
12. ?? locked
13. ?? locked
14. ?? locked
15. ?? locked
16. ?? locked
17. ?? locked

2. ?? locked
3. ?? locked
4. ?? locked
5. ?? locked
6. ?? locked
7. ?? locked
8. ?? locked
9. ?? locked
10. ?? locked
11. ?? locked
12. ?? locked
13. ?? locked
14. ?? locked
15. ?? locked
16. ?? locked
17. ?? locked

What's inside this course

0:00
1. Введение в большие языковые модели
Обзор курса о построении LLM. Рассматриваются основные компоненты: архитектура, функция потерь, данные, оценка и системные аспекты. Объясняется, почему в практике важнее всего данные, оценка и системы, а не архитектура.
3:00
2. Языковое моделирование и авторегрессивные модели
Объяснение концепции языкового моделирования как моделирования распределения вероятностей над последовательностями токенов. Описание авторегрессивного подхода, где каждый следующий токен предсказывается на основе предыдущих токенов.
7:00
3. Токенизация и её важность
Подробное объяснение процесса токенизации текста. Рассмотрены причины использования токенов вместо символов или слов, описан алгоритм Byte Pair Encoding (BPE) для обучения токенизаторов.
11:00
4. Оценка моделей: перплексия и бенчмарки
Объяснение метрики перплексии для оценки языковых моделей. Описание академических бенчмарков (HELM, Hugging Face Open LM Leaderboard) и проблем с оценкой, включая загрязнение тестового набора.
15:00
5. Сбор и обработка данных для предварительного обучения
Описание процесса сбора данных из интернета с использованием Common Crawl. Рассмотрены этапы обработки: извлечение текста, фильтрация нежелательного контента, дедупликация, фильтрация низкого качества и взвешивание доменов.
20:00
6. Масштабируемость и законы масштабирования
Объяснение законов масштабирования LLM, показывающих, что производительность улучшается предсказуемо с увеличением вычислений, данных и параметров. Описание применения этих законов для оптимизации распределения ресурсов.
25:00
7. Оптимизация ресурсов и гиперпараметры
Обсуждение того, как использовать законы масштабирования для оптимизации выбора размера модели, количества данных и вычислительных ресурсов. Объяснение различных стратегий обучения в зависимости от доступных ресурсов.
30:00
8. Стоимость обучения и вычислительные ресурсы
Расчёт стоимости обучения больших моделей на примере Llama 3. Оценка затрат на GPU, зарплаты команды и углеродного следа. Обсуждение физических и экономических ограничений при масштабировании.
35:00
9. Постобработка и выравнивание моделей
Введение в постобработку LLM для создания AI-ассистентов. Объяснение необходимости выравнивания моделей с предпочтениями пользователей и дизайнерских целей, включая модерацию контента.
40:00
10. Контролируемая тонкая настройка (SFT)
Описание метода контролируемой тонкой настройки, где модель обучается на желаемых ответах людей. Объяснение того, почему требуется мало данных и как использовать LLM для генерации синтетических данных.
45:00
11. Обучение с подкреплением из обратной связи (RLHF)
Объяснение метода RLHF для оптимизации моделей на основе предпочтений людей. Описание процесса сбора предпочтений, обучения модели вознаграждения и применения PPO для оптимизации.
50:00
12. Прямая оптимизация предпочтений (DPO)
Описание более простого метода DPO как альтернативы PPO. Объяснение того, как DPO напрямую оптимизирует предпочтения без обучения отдельной модели вознаграждения, используя только максимальное правдоподобие.
55:00
13. Сбор данных для постобработки
Обсуждение методов сбора данных предпочтений для RLHF и DPO. Сравнение использования людей и LLM для разметки, включая преимущества и недостатки каждого подхода.
60:00
14. Оценка выравненных моделей
Объяснение проблем оценки постобработанных моделей, включая невозможность использования перплексии. Описание методов оценки: Chatbot Arena, автоматическая оценка с LLM и контроль смещений.
65:00
15. Системы и оптимизация GPU
Введение в системные аспекты обучения LLM. Объяснение архитектуры GPU, оптимизации памяти и вычислений, включая низкую точность и слияние операций.
70:00
16. Низкая точность и слияние операций
Описание техник оптимизации: использование 16-битной точности для вычислений и 32-битной для хранения весов. Объяснение слияния операций (operator fusion) для уменьшения обращений к памяти.
73:20
17. Заключение и дальнейшие ресурсы
Краткое резюме основных тем курса. Упоминание других важных аспектов LLM, не рассмотренных в лекции. Рекомендация дополнительных курсов для углубленного изучения.

Every chapter ends with a checkpoint (quiz, flashcards, retell, diagram, or prediction) and the course closes with a final boss-fight. More courses →

Transcript (2699 segments)

0:06so let's get started uh so I'll be

0:08talking about building llms today um so

0:10I think a lot of you have heard of llms

0:13before uh but just as a quick recap uh

0:16llms standing for large language models

0:19are basically all the chat Bots uh that

0:21you've been hearing about recently so uh

0:24Chad GPT from open ey CLA from entropic

0:28Gemini and and lman other type of models

0:31like this and today we'll be talking

0:32about how do they actually work so it's

0:35going to be an overview because it's

0:36only one lecture and it's hard to

0:37compress everything but hopefully I'll

0:39touch a little bit about all the

0:41components that are needed to train uh

0:42some of these llms uh also if you have

0:45questions please interrupt me and ask uh

0:48if you have a question most likely other

0:50people in the room or on Zoom have other

0:53have the same question so please ask um

0:57great so what matters when training llms

1:00um so there are a few key components

1:02that matter uh one is the architecture

1:04so as you probably all know LMS are

1:06newal networks and when you think about

1:09new networks you have to think about

1:10what architecture you're using and

1:12another component which is really

1:13important uh is the training loss and

1:15the training algorithm um so how you

1:18actually train these models then it's

1:20data so uh what do you train these

1:23models on um the evaluation which is how

1:26do you know whether you're actually

1:27making progress towards the goal of of

1:30uh llms and then the system component so

1:33that is like how do you actually make

1:35these models run on uh Modern Hardware

1:38which is really important because these

1:40models are really large um so now more

1:42than ever system is actually really an

1:44important topic um for

1:47llms so those five components um You

1:50probably all know that llms and if you

1:52don't know LMS are all based on

1:54Transformers or at least some version of

1:56Transformers uh I'm actually not going

1:58to talk about the AR Ure today uh one

2:01because I gave a here lecture on um

2:04Transformers a few weeks ago and two

2:07because you can find so much information

2:09online on uh Transformers but I think

2:11you can it's there's much less

2:13information about the other four topics

2:15so I really want to talk about those um

2:18another thing to say is that most of

2:19Academia actually focuses on

2:21architecture and training algorithm and

2:24losses um as academics and I've done

2:26that for a lot big part of my career is

2:29simply we like thinking that this is uh

2:32like we make new architectures new

2:34models and it it seems like it's very

2:36important but in reality honestly what

2:39matters in practice is mostly the three

2:41other topics so data evaluation and

2:43systems uh which is what of most of

2:46Industry actually focuses on um so

2:49that's also one of the reason why I

2:50don't want to talk too much about the

2:51architecture uh because really the rest

2:53is super

2:54important um great so overview of the

2:56lecture I'll be talking about

2:58pre-training so pre-training uh you

3:00probably heard that word this is the

3:01general word this is kind of the

3:03classical language modeling uh Paradigm

3:06uh where you basically train your

3:07language model to essentially model all

3:09of internet and then there's a post

3:12training which is a more recent Paradigm

3:13which is taking these large language

3:15models and making them essentially AI

3:17assistants um so this is more of a

3:20recent Trend since Chad GPT uh so if you

3:23ever heard of gpt3 or gpt2 that's really

3:26pre-training land uh if you heard of

3:28chat GPT which you probably have this is

3:30really posttraining land uh so I'll be

3:32talking about both but I'll start with

3:34pre-training and uh specifically I'll

3:37talk about what is the task of

3:39pre-training llms and what is the laws

3:41that people actually

3:42use so language modeling this is a quick

3:46recap uh language models at a high level

3:49are simply models of probability

3:51distribution over sequences of tokens or

3:53of words so it's basically some uh model

3:56of P of X1 to XL where X1 is basically

3:59word and Excel is the last one in the

4:02sequence or in the sentence um so very

4:05concretely if you have a sentence like

4:07the mouse ate the chees what the

4:09language model gives you is simply a

4:11probability of this sentence being

4:13uttered by a human or being found on on

4:16online uh so if you have another

4:18sentence like the the mouse at cheese uh

4:22here there's grammatical mistakes so the

4:24model should know that this uh should

4:26have some syntactic knowledge so it

4:28should know that this has less likely of

4:30appearing

4:32online uh if you have another sentence

4:34like the cheese ate the mouse uh then

4:37the model should hopefully know about

4:39the fact that usually cheese don't eat

4:41Mouse um so there's some semantic

4:43knowledge and this is less likely than

4:45the first sentence so this is basically

4:46at a high level what language models are

4:49um one word that you probably have been

4:52hearing a lot in the news are generative

4:53models uh so this is just something that

4:55can generate models that can generate

4:57sentences or can generate some data the

5:00reason why we say language models are

5:01generative models is that once you have

5:03a model of a distribution you can simply

5:05sample from this model and now we can

5:07generate data uh so you can generate

5:09sentences uh using a language

5:12model so the type of models that uh

5:14people are all currently using are what

5:16we call Auto regressive language models

5:19and the key idea of autor regressive

5:21language models is that you take this

5:23distribution over words and you

5:26basically decompose it into the into the

5:29distribution of the first word multiply

5:32by the distribution of or the likelihood

5:34of the distribution of the second word

5:36given the first word uh multiply by P of

5:38the third word given the first two words

5:41um so there's no approximation here this

5:43is just the chain rule of probability

5:44which you hopefully you all know about

5:46uh really no approximation this is just

5:48one way of modeling a

5:50distribution uh so slightly more

5:52concisely you can write it as a product

5:53of U of PS of the next word given

5:57everything which happened in the past so

5:58of the context and uh so this this is

6:01what we call Auto regressive language

6:02models again this is really not the only

6:05way of modeling distribution this is

6:07just one way uh it has some benefits and

6:09some downsides one downside of

6:11autoaggressive language models is that

6:13when you actually sample from this

6:15autoaggressive language model you

6:16basically have a for Loop which

6:18generates the next word then conditions

6:20on that next word and then regenerate an

6:22other word so basically if you have a

6:24longer sentence that you want to

6:25generate you it takes more time to

6:27generate it uh so there are some

6:29downsides of this current Paradigm but

6:31that's what we currently have so I'm

6:33going to talk about this

6:35one uh great so Auto regressive language

6:38models at a high level um what the task

6:41of autoregressive language model is is

6:42simply predicting the next word as I

6:44just said so if you have a sentence like

6:45she likely prefers uh one potential next

6:48word might be dogs and the the way we do

6:52it is that we first tokenize so you take

6:55these words or subwords you tokenize

6:57them um and then you give an IDE for

7:00each token so here you have 1 2 three uh

7:03then you pass it through this black box

7:05as I already said we're not going to

7:06talk about the architecture you just

7:08pass it pass it through a model and you

7:10then get a distribution a probability

7:13distribution over the next word over the

7:15next token and then you sample uh from

7:19this distribution you get a new token

7:22and then you DET tokenize so you get a

7:23new ID you then DET toonize and that's

7:25how you basically sample from a language

7:27model uh one thing which is important to

7:29not is that the last two TS uh two steps

7:32are actually only need needed during

7:33inference uh when you do training you

7:36just need to predict uh the most likely

7:38token and you can just compare to the

7:39real token which happen in practice and

7:42then you basically change the weights of

7:44your model to increase the probability

7:46of generating that

7:48token um great so autoaggressive neural

7:51language models so to be slightly more

7:53specific still without talking about the

7:55architecture uh the first thing we do is

7:57that we have all of these oh sorry yes

8:00on the previous slide when you're

8:02predicting the probability of the next

8:03tokens does this mean that your final

8:05like output Vector has to be the same

8:07dimensionality as the number of tokens

8:09that you have yes how do you deal with

8:11like have more to you're adding more

8:14tokens to your cor something yeah so

8:17we're going to talk about tokenization

8:19actually later uh so you will get some

8:21sense of this you basically can deal

8:24with adding new tokens I am I'm kind of

8:26exaggerating there are methods for doing

8:27it but essentially people don't do it um

8:30so it's really important to think about

8:32how you tokenize your text and that's

8:34why we'll talk about that later but it's

8:36a very good point to notice that you

8:37basically the vocabulary size so the

8:39number of tokens that you have is

8:40essentially the output of your uh

8:42language model so it's actually pretty

8:44pretty

8:45large okay so autoaggressive new

8:47language models first thing you do is

8:50that you take every word or every token

8:52you embed them so you get a um some

8:55Vector representation for each of these

8:57tokens um you pass them through some

9:00Network as we said it's a Transformer

9:02then you get a representation for all

9:04the word in all the words in the context

9:07so it's basically representation of the

9:08entire sentence uh you pass it through a

9:11linear layer as you just said to

9:13basically map it to the number so that

9:16the output the number of outputs is the

9:18number of tokens uh you then pass it

9:20through some soft Max and you basically

9:22get uh probity distribution over the

9:26next words given every word in the

9:28context

9:31and the laws that you use is basically

9:33it's essentially a task of classifying

9:35the next token so it's a very simple

9:36kind of machine learning task so you use

9:38the cross entry P loss where you

9:40basically you look at the actual Target

9:44that happened which is a target

9:45distribution which is a one hot encoding

9:47which here in this in this case says I

9:49saw uh the real word that happened is

9:52cat so that's a one hot um distribution

9:55over cat and here this is the actual uh

9:58do you see my mouse oh yeah this is the

9:59distribution that you generated and

10:01basically you do cross entropy which

10:02really just increases the probility of

10:04generating cat and decreases all the the

10:06probability of generating all the other

10:07tokens one thing to notice is that as

10:10you all know again uh this is just

10:13equivalent to maximizing the text log

10:15like the text log likelihood because you

10:17can just rewrite the the max over the

10:21probability of um this autoregressive

10:23language modeling task as just being

10:25this minimum over I just added the log

10:28here and minus which is just the minimum

10:30of the loss which is the cross entry

10:31loss so basically minimizing the loss is

10:33the same thing as maximizing the

10:35likelihood of your text any question

10:42questions

10:44okay

10:46tokenizer um so this is one thing that

10:49people usually don't talk that much

10:51about tokenizers are extremely important

10:53uh so it's really important that you

10:55kind of understand at least uh what they

10:57do at a high level so why do we need

10:59token in the first place uh first it's

11:02more General than words so one simple

11:04thing that you might think is oh we're

11:06just going to take every word that we

11:07will have you just say every word is a

11:09new is a token in its own um but then

11:12what happens is if there's a typo in

11:14your word then you might not have any

11:16token associated with this this word

11:19with a typo and then you don't know how

11:21to actually pass this word with a typo

11:23into a large language model so what do

11:25you do next and also even if you think

11:27about words words is a very like words

11:30are fine with like Latin based languages

11:32uh but if you think about a language

11:34like taii you won't have a simple way of

11:36tokenizing by spaces because there are

11:38no spaces between words um so really uh

11:41tokens are much more General Than Words

11:44first thing second thing that you might

11:45think is that you might tokenize every

11:48sentence character by character you

11:50might say a is one token b is another

11:52token uh that would actually work and

11:54probably very well the issue is that

11:56then your sequence becomes super long

11:58and as you probably remember from the

12:00lecture on on Transformers uh the

12:03complexity uh grows quadratically with

12:06the length of sequences so you really

12:08don't want to have a super long sequence

12:10um so tokenizers basically try to deal

12:13with those two problems and give common

12:17subsequences a certain token and usually

12:20how you should be think about is around

12:23uh an average every token is around

12:25three four letters

12:27um and there are many algorith for

12:30tokenization I'll just talk about one of

12:31them to give you a high level uh which

12:33is what we call bite P en coding which

12:34is actually pretty common one of the two

12:36most common tokenizers and the way that

12:38you train a tokenizer is that first you

12:41start with a very large Corpus of text

12:43and here I'm really not talking about

12:44training a large language model yet this

12:46is purely for the tokenization step uh

12:48so this is my large Corpus of text with

12:50these five words um then you associate

12:54every character in this Corpus of text a

12:57different token uh so here I just split

13:00up every character with a different

13:01token uh and I just color coded all of

13:04those tokens and then what you do is

13:07that you go through your text and every

13:09time you see pairs of tokens that are

13:12very common the most common pair of

13:14token you just merge them so here you

13:16see three times the the the tokens T and

13:19O next to each other so you're just

13:21going to say this is a new token and

13:23then you continue you repeat that so now

13:25you have to talk which happens three

13:28times to with an E that happens sorry

13:31two times and an token which happens

13:34twice and then ex which also happen

13:37twice so this is that if you were to

13:39train a tokenizer on this Corpus of text

13:42which is very small that's how you would

13:44uh finish with a token with a pre like a

13:46trained tokenizer uh in reality you do

13:49it on on much larger corpuses of text um

13:52and this is the real tokenizer of uh

13:55actually I think this is gpt3 or chat

13:57GPT uh and here you see how it would

13:59actually separate these words so

14:01basically you see the same thing as what

14:02we gave in the previous example token

14:05becomes its own token so tokenizer is

14:08actually split up into two tokens token

14:11and iser um so yeah that's all about

14:14tokenizers any question on that yeah how

14:17do you deal with spaces and how do you

14:19deal yeah so actually there's a a step

14:23before tokenizers which is what we call

14:24pre- toiz which is exactly what you just

14:27said uh so this is mostly

14:30in theory there's no reason to deal with

14:32spaces and punctuation separately you

14:34could just say every space gets its own

14:36token every um uh punctuation get its

14:40own token and you can just do all the

14:42merging the problem is that so there's

14:44an efficiency question actually training

14:46these tokenizes takes a long time uh so

14:48you better off because you have to

14:50consider every pair of token so what you

14:52end up doing is saying if there's a

14:54space this is very like pre-t tooners

14:56are very English specific you say if

14:58there's a space we're not going to start

14:59looking at the the token that came

15:01before and the token that came

15:03afterwards so you're not merging in

15:04between spaces but this is just like a

15:08optimiz like a computation optimization

15:10you could theoretically just deal with

15:12it um the same way as you deal with any

15:14other character and yeah when you merge

15:17tokens do you delete the tokens that you

15:19merged away or do you keep the the

15:21smaller tokens merg um you actually keep

15:24the smaller tokens I mean in reality it

15:26doesn't matter much because um usually

15:30on large Corpus of text you will have

15:32actually everything uh but you usually

15:34keep the small ones and the reason why

15:35you want to do that is because if in

15:37case there's as we said before you have

15:39some um some grammatical mistakes so

15:41some typos you still want to be able to

15:43represent these words by character um so

15:47yeah yes are the tokens uni so I mean

15:52say in this case T Ken is there only one

15:55occurrence or could do you need to leave

15:58multiple UR so they could have take on

16:01different meanings or something oh oh I

16:02see what you say no no it's every token

16:05has its own uh unique ID um so a usual

16:10this is a great question for example if

16:12you think about a bank which could be

16:14bank for like money or bank like water

16:16um it will have the same token but the

16:18model will learn the Transformer will

16:20learn that based on the words that are

16:22around it it should associate that I'm

16:25saying I'm being very hand wavy here but

16:27associate that with the with a with a

16:29representation that is either more like

16:32the bank money side or the Bank water

16:34side um but that's a Transformer that

16:36does that it's not a

16:37tokenizer yes yeah so you mentioned

16:40during tokenization keep the smaller

16:42tokens you started with right like you

16:44start with a t you keep the T and then

16:46you build your token out to the that you

16:48can now in token so let's say maybe you

16:51didn't train on token but like in your

16:53data you are trying to encode token so

16:56how does the tokenizer know to encode it

16:58with token or

17:00a great question you basically when you

17:02so when you tokenize so that's after

17:04training of the tokenizer when you

17:05actually apply the tokenizer you

17:07basically always choose the largest uh

17:10token that you can apply uh so if you

17:12can do token you will never do T you

17:14will always do token um but there's

17:17actually so people don't usually talk

17:19that much about tokenizers but uh

17:20there's a lot of of computational

17:22benefits uh or computational tricks that

17:25you can do for making these things

17:26faster uh so I really don't think we and

17:29honestly I think a lot of people think

17:30that we should just get away from

17:32tokenizers um and just kind of tokenize

17:34character by character or bites by bites

17:37uh but as I said right now there's this

17:38issue of like length uh but maybe one

17:41day like in five or 10 years we will

17:43have different architectures that don't

17:44scale quadratically with the length of

17:45the sequence and uh maybe we'll um yeah

17:49move away from tokenizes so can you

17:52share with us the drawback why do people

17:54want to move away from the tokenizer oh

17:57um yeah so think

18:00one good example is uh math if you think

18:04about math actually numbers right now

18:06are not tokenized so for example 327

18:09might have its own token which means

18:11that models when they see numbers they

18:13don't see them the same way as we do and

18:16this is very annoying because what I

18:17mean the reason why we can kind of

18:19generalize with math is because we can

18:21deal with every every letter separately

18:23and we can then do composition where you

18:25know that basically if you add stuff

18:26it's just the same thing as adding every

18:28one separately plus like whatever the

18:30unit that you add so they can do that um

18:33so then you have to do like special

18:34tokenization and like one of the big

18:37changes that GPT 4 did uh is changing

18:41the way that they tokenize uh code so

18:43for example uh if you have code you know

18:45you have like often in Python these four

18:47spaces at the beginning those were dealt

18:49with uh kind of strangely before um and

18:52as a result like the model couldn't

18:54really understand uh how to deal with

18:57code uh so so toiz actually a lot um

19:01okay so I'll move on right now but we

19:03can come back later on token Isis great

19:07so we talked about the task the L the

19:08tokenizer let's talk a little bit about

19:10evaluation uh so the way that LMS are

19:13usually evaluated is what we call is

19:15using what we call perplexity um at a

19:17high level it's basically just your

19:19validation loss uh the slight difference

19:21with perplexity is that we use something

19:23that is slightly more interpretable

19:25which is that we use the average per

19:27token loss and then you expon

19:29exponentiate it and the reason why you

19:30exponentiate it is because you want I

19:32mean the loss has a log inside and you

19:35like one humans are actually pretty bad

19:37at thinking in log space but two logs

19:39depend on the base of the log uh while

19:42when you explanation it you basically

19:43have everything in the uh kind of the

19:46vocabulary size uh unit um and the

19:49average proten is just so that your your

19:51complexity is independent of the length

19:53of your sequence um so perplexity is

19:55just two to the power uh average of the

19:58loss of the sequence

20:00um so perplexity is between one and the

20:03length of the vocabulary of your

20:05tokenizer uh one it's simply well if you

20:07predict perfectly the thing which uh

20:10every word then every word will have

20:12basically product of ones uh so the best

20:15perplexity you can have is one if you

20:17really have no idea you basically

20:19predict with one divided by uh size of

20:21vocabulary um and then you do simple

20:23math and you basically get perplexity of

20:25size of vocabulary uh so the intuition

20:28of perplexity is that basically the

20:29number of tokens that your model is kind

20:31of hesitating between uh so if you if

20:34your model is perfect it doesn't

20:35hesitate it know exactly the word if it

20:37really has no idea then it hesitates

20:39between uh all of the

20:43vocabulary uh so perplexity really

20:46improved that's perplexity on a standard

20:48data set between 2017 and 2023 it it

20:51went from kind of 70 tokens to less than

20:5410 tokens over these five six years so

20:57that means that the models were

20:58previously as dating between 70 words

21:01every time it was generating a word and

21:03now it's as dating between like less

21:04than 10 words so that's much better

21:07perplexity is actually not used anymore

21:09in academic benchmarking mostly because

21:11it depends on the tokenizers that you

21:12use uh it depends on the actual data

21:15that people are evaluating on but it's

21:17still very important for development of

21:18llms so when you when you actually train

21:20your own llm people will still really

21:22look at the

21:25perplexity uh one common other way and

21:28now more common in Academia of

21:30evaluating these llms is just by taking

21:33all the classical NLP benchmarks and

21:35I'll give you a few examples later and

21:37just kind of aggregating everything um

21:39so collect as many automatically

21:42evaluatable benchmarks and just evaluate

21:44across all of them um so one such uh or

21:49actually two such uh benchmarks of what

21:52we call uh Helm which is from Stanford

21:54and another one is the hugging face open

21:56LM leader board which are the probably

21:57two two most common ones right now um so

22:01just to give you an idea in Helm there

22:03are all of these type of tasks which are

22:05mostly things that can be easily

22:07evaluated uh like question answering so

22:10think about many different question

22:11answering uh tasks um and the benefit

22:15with question answering is that you

22:16usually know what is the real answer um

22:19so you can the way that you evaluate

22:20these models and I'll give you a

22:21concrete example in one second um is

22:23that you can just look at How likely the

22:26language model is to generate the real

22:28answer compared to some other answers

22:31and that's essentially at a high level

22:32how you evaluate these models um so to

22:34give you a specific example MML is

22:37probably the most common um academic

22:40Benchmark for

22:41llms uh and this is just a collection of

22:45many question and answers in all of

22:47those domains for example College

22:49medicine College physics astronomy and

22:51these type of topics and the questions

22:53are things like so this in astronomy

22:55what is true for type 1 a supernova then

22:59you give uh four different potential

23:01answers and you just ask the model which

23:04one is more likely so there are many

23:06different ways of doing it either you

23:07can look at the likelihood of generating

23:09all these answers uh or you can ask the

23:11model which one is the most likely uh so

23:13there are different ways that you can

23:14promp the model but at a high level you

23:16know which one is correct and there are

23:18three other mistakes um yes kind of

23:22creating these like unconstrained text

23:24as the output yeah how do you evaluate a

23:27model if it give something that's you

23:29know semantically completely identical

23:33but is not the exact token list that

23:35expect yeah so that's a great question

23:37I'll talk more about that later here in

23:39this case we don't do unconstrained so

23:42the way you would evaluate mlu is

23:44basically either you you ask the first

23:46question and then you look at the

23:48likelihood of the model generating a the

23:51likelihood of the model generating b c

23:53and d and you look at which one is the

23:55most likely or you can as the model out

23:57of ABC d which one is the most likely

24:00and you look at whe the to the most

24:02likely next token is A B C or D so uh

24:05you constrain the model to say it can

24:06only answer these four things you say

24:09you constraint the model you mean you

24:12constraint The Prompt or do you mean of

24:14its whole probability distribution

24:15outputs you only are comparing the

24:18outputs like you're only comparing the

24:20a so uh in the second case I gave you

24:23you would do exactly the I actually you

24:25would do both you would prompt the model

24:26saying ABC or D plus you would constrain

24:29to only uh look at these two these four

24:31tokens in the first case you don't even

24:33need to generate anything so in the

24:35first case you literally just look given

24:37that it's a language model it can give a

24:39distribution over sentences you just

24:41look at what is the likelihood of

24:43generating all of these words what is

24:45the likelihood of generating the second

24:47choice and you just look at whether the

24:50most likely sentence is actually the

24:53real answer so you don't actually sample

24:56from it you really just use P of x one

24:59to excel does that make sense uh that

25:02being said evaluation of open-ended

25:04questions is something we're going to

25:06talk about later and is actually really

25:08important and really challenging yes

25:11earlier you mentioned that um like um

25:13metrics like for flexity are not are not

25:16like usually used because it depends on

25:18like how you do your terization some

25:20design choices I was wondering if you

25:22could speak more to that oh um yeah so

25:26think about perplexity I told you

25:27perplexity is between one and vocabulary

25:29size so now imagine that Chad GPT uses a

25:33tokenizer that has like 10,000 tokens

25:35but Gemini from Google uses a tokenizer

25:38that had 100,000 uh potential tokens

25:42then actually the Gemini one will will

25:45have like the upper bound of the the

25:46perplexity that you can get is actually

25:48worse for Gemini than for Chad GPT does

25:51that make sense so that's just an idea

25:54it's actually a little bit more

25:55complicated than that but that's just

25:56like one uh first or the bit of you can

25:59see that the tokenizer actually

26:01matters um

26:05great okay so evaluation challenges

26:08there are many I'll just talk about two

26:10really briefly uh one as I told you

26:12there are two ways of doing evaluation

26:14for these MML use actually there are

26:15many more than two but I give you two

26:17examples um and it happens that for a

26:20long time even though that was a very

26:21classical Benchmark that everyone used

26:24uh actually different uh different

26:27companies and different um different uh

26:30uh different organization were actually

26:32using different ways of evaluating mlu

26:35and as a result you could you get

26:37completely different results for example

26:38Lama

26:4065b uh which was the first model of meta

26:43in the Lama series uh had on Helm 63.7

26:47accuracy but on this other um Benchmark

26:51had like

26:5248.8 um so really the way that you

26:55evaluate and this is not even talking

26:56about prompting this is really just kind

26:58of the the way that you evaluate uh the

27:01models prompting is another issue so

27:03really there are a lot of

27:04inconsistencies it's not as easy as it

27:06looks uh first thing yeah sorry how can

27:09we make sure that all these models are

27:11in trained on The Benchmark okay second

27:14thing this is a great question uh chain

27:16test contamination uh this is something

27:19which I would say is really important in

27:22Academia in uh given that the talk is

27:25mostly about training large language

27:27models uh for companies it's maybe not

27:29that important CU they know what they

27:31trained on uh for us we have no idea so

27:36for us it's a real problem uh so there

27:38are many different ways of trying to

27:40test whether uh the test set sorry

27:43whether the test set was actually in the

27:45training Set uh one kind of cute trick

27:49um that people uh in in the lab on T lab

27:53have found is that what you can do is

27:54that given that most of the data set

27:56online are not randomized

27:59you can just look at and in that

28:01language models what they do is just

28:02predict the next word um you can just

28:04look at the entire test Set uh what if

28:07you generate all the examples in order

28:11versus all the examples in a different

28:13order and if it's more likely to

28:15generate a thing in order given that

28:17there's no real order there then it

28:20means that probably was in a training

28:21set does that make sense um so there are

28:24many that's like one of them there are

28:25many other ways of doing it train test

28:27contamination again not that important

28:29for development really important for

28:31academic

28:32benchmarking great so there are many

28:34other challenges but uh I'll move on for

28:37now great data um so data is another

28:42really big topic um at a high level

28:44people just say oh you basically train

28:46large language models on all of Internet

28:49what does that even mean um so or people

28:52sometimes say all of clean internet

28:53which is even less defined um so

28:57internet is very dirty and really not

28:59representative of what we want in

29:00practice if I download a random website

29:03right now you would be shocked at what

29:05is in the it's definitely not your

29:07Wikipedia um so I'll go really briefly

29:12on like what people do um I can answer

29:15some questions but I mean data is on its

29:17own is a huge topic uh basically first

29:20what you do is download all of Internet

29:23what that means is that you use uh web

29:25crowlers that will go on every web page

29:27on Internet or every web page that is um

29:30on Google and that is around 250 billion

29:35pages right now um and that's around one

29:37petabyte of of data so this is actually

29:40a common common C is one web crowler so

29:43people will usually write their own web

29:44crowlers what they do is that they use

29:46standard web crowlers and common crawl

29:49is one of them uh that basically every

29:51month adds all the new websites that

29:54were added on uh internet that are found

29:56by by Google and they put it in big uh

29:59basically a big data set um so that's on

30:02common call you have around 250 billion

30:04pages right now so one E6 gigabytes of

30:07data once you have this uh so this is a

30:10random web page like literally random uh

30:13from this common CRA and what you see is

30:15that one it really doesn't look at type

30:17of things that you would usually see but

30:19actually so this is an HTML page uh it's

30:22hard to see but if you look through you

30:25will see some content for example here

30:28here uh tesing world is your ultimate

30:32source for the system X high performance

30:34server and then you have three dots so

30:35you don't even the sentence is not even

30:37finished that's how a random internet

30:40looks like uh so of course it's not that

30:42useful if you just train a like large

30:44language model to generate things like

30:45this so what are some of the steps that

30:47are needed first one you extract the

30:50text from the HTML so that's what I just

30:52tried to do by looking at uh basically

30:54the correct text uh there are a lot of

30:56challenges by through this for example

30:58extracting math is actually very

31:00complicated but pretty important for

31:02training large language models um or for

31:04example boiler plates a lot of your

31:06forums will have the same type of

31:08headers the same type of Footers uh you

31:10don't want to repeat all of this in your

31:12data um then you will filter undesirable

31:16content uh so not safe for work harmful

31:19content pii uh so usually every company

31:22has basically a a black list of websites

31:26that they don't want to train the models

31:27on that Blacklist is very long and you

31:30basically say if it comes from there we

31:31don't train on this there are other ways

31:33of doing these things is that you can

31:34train a small model for classifying what

31:37is pii removing these things um it's

31:40hard every Point here that I'm going to

31:42show you is like of hard amount of work

31:46uh but I'm going to go go quickly

31:48through it so filter UND desirable

31:49content second or fourth is theup D

31:53duplication as I said um you might have

31:56things like headers and Footers in

31:58forums that are always the same you want

32:00to remove that another thing that you

32:02might have is a lot of URLs that are

32:04different but actually show the same

32:07website um and you might also have a lot

32:10of like U um paragraphs that come from

32:13like common books that are basically

32:15duplicated a thousand times or 10,000

32:17times on internet so you have to

32:19duplicate also very challenging uh

32:22because you have to do that at scale

32:24once you do duplication you will do some

32:27heuristic filtering you will try to

32:29remove low quality documents uh the way

32:32you do that are things like rules-based

32:34um filtering for example if you see that

32:36there are some outlier tokens if the

32:38distribution of tokens in the website is

32:40very different than the usual

32:41distribution of tokens then it's

32:42probably some outlier if you see that

32:44the length of the words in this website

32:46is super long there's something strange

32:48going on on that website if you see that

32:50the the website has only three words

32:53maybe is it worth training on it maybe

32:54not if it has like 10 million words

32:57maybe there's something also

32:59wrongo on that page um so a lot of rules

33:01like this yes why do we filter out

33:03undesirable content from our dat set

33:05instead of kind

33:07of putting it in is like a supervised

33:09loss right like can we not just say like

33:12you know here's this like hate speech

33:14website let's actively try to Let's

33:18actively penalize them for generating

33:20we'll do exactly that but not at this

33:22step that's where the posttraining will

33:24come from uh pre-training um the idea is

33:29just to say I want to model kind of how

33:32humans speak essentially um and I want

33:35to remove all these like headers photos

33:37and and menus and things like this but

33:39it's a very good uh like idea that you

33:41just had and that's exactly what we'll

33:43do

33:44later Next Step modelbased filtering so

33:47once you filtered a lot of data what you

33:49will do uh that's actually a very cute

33:51trick uh you will take all of Wikipedia

33:54and you will look at all the links that

33:56are linked through Wikipedia p PES

33:59because probably if something is

34:00referenced by Wikipedia it's probably

34:02some high quality website and you will

34:04train a classifier to predict whether

34:06something comes from whether a document

34:09comes from one of these references uh

34:12from Wikipedia or whether it's from the

34:14random web and you will try to basically

34:17say I want more of the things that come

34:20from Wikipedia references does that make

34:23sense so yeah so you will train a

34:25machine learning uh model usually also

34:27very simple models because you need to

34:29do that really at scale I mean just

34:30think about the 250 billion

34:33Pages uh next one you will try to

34:36classify your data into different

34:39different um domains you will say okay

34:42this is entertainment this is books this

34:44is code this is like these type of

34:46domains and then you will try to either

34:49um up or down weight some of the domains

34:53uh for example you might say uh you

34:55might see that actually if you train

34:56more on code then actually your model

34:58becomes better on reasoning so that's

35:00something that people usually say in a

35:01very handwavy way if you train your

35:03model more on code actually it helps

35:05reasoning so you want to upweight the

35:07coding uh distribution because that

35:09helps for General language modeling

35:11skills uh books is usually also another

35:13one that people usually um upweight

35:16entertainment they usually downweight uh

35:19so things like this of course you want

35:20to do it so people used to do it maybe

35:23uh kind of theistically now there's

35:26entire pipelines that we'll talk about

35:28of how to do these things uh slightly

35:30more um

35:33automatically and then at the end of

35:35training uh usually train um after

35:38training on all of this data that we saw

35:41usually train on very high quality data

35:43at the end of of training your large

35:45language model where you decrease your

35:47learning rate uh and that basically

35:49means that you're kind of overfitting

35:50your model on a very high quality data

35:53so usually what you do there is like

35:55Wikipedia you basically overfit on

35:57Wikipedia yeah and you overfit on like

36:01human uh data that was collected um the

36:05other things is like continual

36:06pre-training for getting longer context

36:08I'm I'm going to skip over all of these

36:09things uh but I just to give you a sense

36:11of how hard it is when people just say

36:14oh I'm going to train on internet that's

36:16a lot of work um and really we haven't

36:18figured it out yet so collecting World

36:22data is a huge part of practical large

36:24language model uh some might say it's

36:26actually the key yes

36:28about data so basic question so usually

36:31when you start with like the terabyte of

36:33data after I go through all that steps

36:35the typical amount of have in and then

36:38like how how large a team does it

36:41typically take to go through all the

36:42steps you talking about so how is the

36:45question how large is the data after you

36:46filter yeah after you filter and then to

36:48go through all the step how large a team

36:50do you need to go through like the the

36:52other fation St you mention uh how slow

36:56is it or how like how how many people

36:59would you need to be able to do this uh

37:03okay that's a great question I'm going

37:04to somewhat answer about the data uh how

37:07large is the data set uh at the end of

37:09this slide uh for number of people that

37:12work on

37:14it um that's a good question I'm

37:16actually not quite sure but I would

37:19say yeah I actually don't quite no but I

37:23would say it's probably even bigger than

37:24the number of people that work on kind

37:26of the tuning of the pre-training of the

37:29model uh so the data is bigger than kind

37:32of the modeling aspect um yeah I I don't

37:36think I have a good sense I would say

37:38probably in Lama's team which have like

37:4070ish people I would say maybe 15 work

37:43on data uh I yeah all these things you

37:47don't need that many people you need a

37:49lot of computer so because for data you

37:50need a lot of CPUs um so yeah and I'll

37:54answer the second question at the end of

37:55this slide so as I just kind of alluded

37:59to really we haven't solved data at all

38:01for pre-training so there's a lot of

38:03research that that has to be done first

38:05how do you process these things super

38:06efficiently uh second how do you balance

38:08kind of like all of these different

38:10domains uh can you do synthetic data

38:12generation that's actually a big one

38:13right now uh and because we don't have

38:17uh we'll talk about that later we don't

38:18have enough data on the internet um can

38:21you use multimodal data instead of just

38:23text data and how does that improve even

38:25your text performance um um there's a

38:29lot of seccy because really this is the

38:31key of most of the pre-train pre-trained

38:33large language models so for competitive

38:35Dynamics uh usually these these um these

38:39companies don't talk about how they do

38:41the data collection and also there's a

38:43copyright liability issue they

38:44definitely don't want to tell you that

38:45they've trained on books even though

38:46they did um because if not you can uh

38:49sue them uh common academic benchmarks

38:52uh so that will kind of answer what you

38:54asked um it started so those are the

38:56smaller ones it's the names are not that

38:59important but it started from around 150

39:01billion tokens which around uh 800 GB of

39:04data now it's around 15 trillion of to

39:0715 trillion tokens which is also uh the

39:10size of the models that are right now

39:12the best models are probably trained on

39:14that amount of data so 15 trillion

39:15tokens uh which is probably I guess two

39:19order of manage bigger than that so 80

39:22uh E3 gab so that would be

39:26around 100 to thousand times uh

39:29filtering of the common crawl if I'm not

39:32mistaken um so yeah one very one very uh

39:35famous one is the pile so this is

39:38academic Benchmark of the pile and we

39:40can just look at what distribution of

39:42data they have it's things like um

39:44archive PBM Central uh which is all the

39:48the biology stuff uh here it's Wikipedia

39:52you see stack exchange um some GitHub

39:56and some books and things like this um

39:59again this is on the smaller side so

40:00this is if we look at here this is on

40:02280b so in reality it's like 100 times

40:04bigger so you cannot have that much of

40:06GitHub and and of

40:08Wikipedia um in terms of close Source

40:11models just to give you an idea uh lamb

40:142 um it was trained on 20 two trillion

40:16tokens lamb 3 15 trillion tokens which

40:19is currently the best model that we know

40:21on how much it was trained on which is

40:23the same thing as this the the the best

40:26academic or the biggest academic

40:28Benchmark which is 15 trillion tokens

40:29GPD 4 we don't really know but it's

40:31probably in the same water of magnitude

40:33or it's probably around that actually

40:34it's probably around 13 um from leaks if

40:37the leaks are true

40:39um great so scaling laws um any other

40:44questions on Data before you go to

40:45scaling

40:48laws sorry I know I'm giving you a lot

40:50of information but uh there's a lot into

40:52training at large language models great

40:55scaling laws so so the idea is that what

40:59people saw um around 2020 or at least

41:02from a long time but they've been able

41:03to kind of theoretically show it or

41:06impurely show it since 2020 is that the

41:08more data you train your models on and

41:10the larger the models the better the

41:12performance this is actually pretty

41:14different than what you've seen in this

41:15class in this class we teach you about

41:17overfitting overfitting doesn't happen

41:19with large language models uh larger

41:21models better performance um it's

41:24something that really took a long time

41:26for the community who took this type of

41:28class to realize um but for the exam

41:31overfitting

41:34exists okay the idea of scaling laws is

41:37that if given that you know that more

41:39data and larger models will always give

41:41you better performance can we predict

41:44how much better your performance will be

41:47if you increase the amount of data and

41:48the size of your model and surprisingly

41:51it works uh so here you see three plots

41:54from a very famous paper called scaling

41:56L from open AI um here you see on the

41:59x-axis compute so how much did you train

42:02like how much compute did you did you

42:03spend for training and here you see test

42:05loss so this is essentially I mean it's

42:07not perplexity but it's your validation

42:09loss um so it's a log of the perplexity

42:12and if you put these two on uh log scale

42:15uh then you see that uh the the

42:17performance or like the this the sorry

42:20the the scaling law is linear uh that

42:23means that if you increase your compute

42:25by a certain amount you can you can say

42:27by how much your test loss will actually

42:30decrease same thing with data and same

42:32thing for parameters if you increase the

42:34data set size your loss will will

42:37decrease by an amount that is somewhat

42:39predictable if you increase the number

42:41of parameters it will decre the loss

42:43will decrease by amount which is

42:45somewhat predictable this is really

42:47amazing um very surprising I mean it

42:50looks in nocuous when you look at these

42:52type of plots but that's crazy because

42:53it means that you can predict uh how

42:56well we're going to perform in two 3

42:58years depending on how much compute we

42:59will add assuming that these things will

43:01hold there's nothing theoretical about

43:03it um yes two things one what is the

43:07loss that they're using here is this

43:08perplexity or so it's it's you know I

43:11said perplexity was like two to the

43:12power of the loss so this is the the the

43:15power of the perplexity and then the

43:18second thing is when you like increase

43:20the number of parameters or you increase

43:22the total data set size going dat times

43:25doesn't that just inherently increase

43:27your your compute like do all of this

43:29work to

43:31just something specific no this is a

43:33great question so the compute here is

43:35actually a factor of two things the data

43:37and the parameter what I'm showing here

43:39is that you can um well actually we're

43:41going to talk about that in details but

43:42basically if you increase the number of

43:44parameters you should increase the

43:45number of data that you have um so you

43:49actually don't go multiple times through

43:50the same data set no one does EPO in a

43:54lar at least not yet uh because we have

43:58still kind of enough data um so yeah

44:01this is all the same Trend which is

44:02increase compute decrease

44:04loss yes have we seen the numbers for

44:08the last two years or is it still

44:10holding it is still holding I I don't

44:14have like good numbers to show you uh

44:17but it is still holding

44:20surprisingly yes is there no evidence

44:23like empirical evidence that you

44:25plateau expect PL

44:29no empirical evidence of plateauing

44:31anytime soon um why we don't know um

44:36will it happen probably I mean it

44:39doesn't need to because it's actually in

44:40log scale so it's not like as if it had

44:43to go it had to Plateau like

44:45mathematically it could continue

44:46decreasing like this I mean most people

44:49think that it will probably Plateau at

44:50some point we don't know

44:52when um okay so that's I'll talk more

44:56about scaling lws now

44:58so why are scaling laws really cool

45:00imagine that I give you um you're very

45:03fortunate I gave you 10,000 gpus for

45:05this month what model will you train how

45:08do you even go about answering that

45:09question and I mean this is a a

45:12hypothetical but that's exactly what

45:13these companies are faced with uh the

45:16old pipeline um which was basically you

45:20tune High parameters on the big models

45:22so let's say I have 30 days I will train

45:2430 models for one day each I will pick

45:27the best one uh and that will be the

45:29final model that I will use in

45:31production um that means that the model

45:33that I actually used was only trained

45:35for one day the new pipeline is that you

45:39first find a scaling recipe so you find

45:41something that tells you for example oh

45:44like one common thing is that if you

45:45increase the size of your model you

45:46should decrease your learning rate so

45:47you find a scaling recipe such that you

45:49know if I increase the the the the size

45:52of my model here's what I should do with

45:54some high parameters then you tune your

45:56high parameters

45:58on smaller models of different sizes

46:01let's say I will say for 3 Days of my 30

46:03days I will train many different models

46:05and I will do highper parameter tuning

46:07on these small models each of different

46:09sizes then I will fit a scaling law and

46:12try to extrapolate from these smaller

46:14models which one will be the best if I

46:18if I train it for much longer or sorry

46:20if I train it for a larger model and

46:23then I will train the final huge model

46:25for 27 days instead of just one day

46:28um so the new pipeline is not train

46:31things or do high PR tuning on the real

46:34scale of the model that you're going to

46:35use in practice but do things on smaller

46:37ones at different scales try to predict

46:41how well they will perform once you make

46:42them bigger I will give I will give you

46:44a very concrete example right now uh

46:47let's say Transformers versus lstms

46:50let's say you you have these 10,000 gpus

46:52you will not sure which one you should

46:53be using should I be using Transformer

46:55base model or LCM based model what I

46:57will do is I will train Transformers at

46:59different scills so here you see

47:01different parameters on the x-axis Y

47:03axis is my test loss I will then train

47:05different different lstms at different

47:08scales once I have these points I will

47:10see oh it kind of fits a scaling law I

47:13will fit my scaling law and then I will

47:14be able to predict oh if I had 10 times

47:18more compute here's how well I would

47:20perform for the lstm it's actually

47:22slightly less linear for the lstm but

47:24like you could probably try to predict

47:26where you would end up and clearly from

47:27this plot you would see that

47:29Transformers are better um one thing to

47:31notice when you read these type of

47:33scaling laws is that are two things that

47:34are important uh one is really your

47:38scaling rate uh which is kind of the uh

47:42the slope of the the slope of the

47:45scaling law the other thing is your um

47:48your intercept like you could start

47:51worse but actually become better over

47:53time it just happens that lstms are

47:55worse for both uh but I could show you

47:57another one where things you can predict

48:00that actually after a certain scale

48:01you're better off using that type of

48:03model than others uh so that's why

48:05scaling laws are actually really

48:08useful any questions on

48:11that yeah so these are all kind of very

48:16how how sensitive are these to like

48:17small differences in the architecture

48:19like one one like Transformer

48:22architecture versus another Transformer

48:23architecture you basically have to like

48:25fit your own curve and basically say

48:27like oh scaling law is tell me this

48:29should be some like logarithmic function

48:32let me extrapolate that for my

48:35own yeah so uh usually for example if

48:38you're an academic and you want to now

48:39at least that's like pretty recent and

48:41you want to propose a new like

48:43activation uh that's exactly what you

48:45will do you will fit a scaling law show

48:47another scaling law with the standard

48:48like I don't know G and you will say

48:50that it's better in reality once you

48:52start thinking about it in scaling L

48:53terms you really realize that actually

48:56all the architecture differences that we

48:57can make like the small minor ones all

48:59they do is maybe change a little bit the

49:02The

49:02Intercept but really that doesn't matter

49:05uh cuz just train it for 10 hours longer

49:07or like wait for the next uh for the

49:09next Compu gpus and these things are

49:11really secondary which is exactly why I

49:13was telling you originally people spend

49:15too much time on the architecture and

49:16losses um in reality these things don't

49:18matter as much data though if you use

49:20good data you will have much better

49:22scaling loss than if you use bad data so

49:25that really matters

49:27uh another really cool thing you can do

49:29with scaling laws is that you can ask

49:31yourself uh how to optimally allocate

49:34training resources should I train larger

49:36models because we saw that it's better

49:38when you train larger models but we saw

49:40that it's also better when you use more

49:42data so which one should I do should I

49:44just train on more data a smaller model

49:46or should I train a larger model on less

49:48data um so chinchilla is a very famous

49:52paper that first show this uh the way

49:54they did it I want to give you a little

49:56bit of a sense of what these plots are

49:58uh here you see training loss again on

50:00the x-axis you see parameter parameter

50:02differences uh sorry parameter size uh

50:04number of parameters so the size of the

50:06model and here all these curves are what

50:08we call isof flops which is that all the

50:12models on this curve H have been trained

50:15with the same amount of

50:16compute um the way that you do that is

50:18that you train you change sorry you vary

50:21the number of tokens that we trained on

50:23and the size of the models but you vary

50:24in such a way that the total compute is

50:26constant

50:27okay so all these curves that you see

50:29with different colors have different

50:31amount of computers that were trained on

50:33then you take the best one for each of

50:34those curves once you have the best one

50:37for each of those curves um you can ask

50:41you can plot um how much flops it was

50:44and which curve were you on and how much

50:47parameters did you actually use for

50:49training that specific point you put

50:51that on the on the log log uh scale

50:55again and now you fit a scaling LW again

50:57so now I have something which tells me

51:00if I want to train a model of 10^ 23

51:03flops here's exactly the number of

51:05parameters that I should be using 100

51:07100b and you can do the same thing with

51:09flops and

51:11tokens so now you can predict if if I

51:14tell you exactly I have one month of

51:16compute what size of model should I be

51:18training F scaling law and I tell you um

51:22of course that all looks beautiful in

51:24reality like there's like there's a lot

51:26of like small things of like should you

51:27be counting like embedding parameters

51:29like there's there's a lot of

51:30complexities but if you do things well

51:32these things actually do

51:34hold um so the optimal number of

51:37parameters that that chinchilla Pap have

51:39found is to use 20 tokens for every

51:42parameter that you train uh so if you

51:44add one more parameter you should add

51:46you should train your thing on your

51:47model on 20 more tokens so one caveat

51:51here is that this is optimal training

51:52resources so that is telling me if you

51:55have 10^ 23 CS or if you have like 100 I

51:58don't know how much that is $100 million

52:01or 10 no that's much less actually let's

52:03say I have $5 million to to train my

52:06best model that gets the lowest loss how

52:08how what would I train on in reality

52:11these companies need to think about

52:12inference also if you have a smaller

52:14model they will spend less over time um

52:18so actually if you consider the

52:19inference cost you have other papers

52:21that Tred to show that um it's around

52:24150 uh parameters per sorry tokens per

52:28parameters cuz you prefer having a

52:30smaller model cuz over time you're going

52:32to you're going to actually um spend

52:35less money on inference of these models

52:38so 150 to one that's around what the

52:41best models are trained on right now at

52:43least the ones that are that are used um

52:47in practice for in

52:49production

52:51great any question on

52:55chin great

52:57oh sorry in practice how expensive is

53:00inference for these models Rel to

53:03training actually very expensive uh I

53:05will not talk about inference because

53:07that would be another entire lecture but

53:10just think about Chad GPT where they

53:12have I don't know how much it is now

53:14like 600 million people that used it um

53:19like that's a lot

53:22um yeah so it's actually very expensive

53:25there's a lot of optimization you can do

53:26for in though um and that's an entire

53:28other lecture so I'm going to skip that

53:30uh this time but it's very

53:33interesting okay tuning um as I said

53:36there are many things that you can uh

53:37answer with scaling laws I just try to

53:39give you two examples uh but really

53:42there are many things what data do you

53:43use what mixture what data mixing

53:46waiting you use data mixtures that's

53:47what we talked about before uh what

53:49architecture you use whether you should

53:51make your models uh wider or deeper um

53:54should you be paying for more gpus

53:57actually collecting more data um all

53:59these things are things you can try to

54:01answer with scaling

54:02laws one thing I want to say is the bit

54:05lesson if you ever heard of Richard

54:07sudden a very famous blog post in 2019

54:11um what he realized uh which I think not

54:16enough people realize I didn't

54:17definitely did not realize at that time

54:19um is that once you see these type of

54:22scaling laws you know that the more

54:23Compu you have the better models you

54:26will get so with skill you will get

54:27better model and you also know by Mo law

54:30or these type of varant of M law that

54:32you will always have better compute then

54:34the only thing that matters is just to

54:37have architectures that can leverage

54:39computation so what matters is basically

54:42systems data and less so the

54:44architecture like the small architecture

54:46differences like your your your

54:48activation and things like this uh so I

54:50think that's like one of the reasons why

54:52most of research focuses on um some

54:55things that for industry matters less

54:57and I was one of those researchers for a

54:59large part of my my career um so don't

55:03spend time over complicating do the

55:05simple things do it well seal them

55:08that's really what openi taught us with

55:11um with chat GPT and with all the gpts

55:15before okay I want to give you some

55:17backup the envelope computation so I

55:19might be off by a few factors here but I

55:21just want to give you a sense of how

55:23costly it is to train some of these

55:24models I'll give as an example

55:27L 3 400b which is currently the best

55:29open source model that you can get uh it

55:32was trained on 15.6 tokens it has 45

55:37billion parameters so just now that you

55:39know what is like this uh optimal tokens

55:42per parameter that's around 40 so that's

55:44a little bit more than chinchilla but

55:46less than this like inference uh optimal

55:49um model so they went for training

55:52optimality uh flops for this model so

55:55one simple way to compute flops is six

55:59uh times the number of parameters times

56:01the number of data you train on uh so if

56:03you do the simple calculation here it's

56:053.8 e25 flops the reason why this is

56:09important is that if you follow the

56:10little bit the news there's an executive

56:12order from Biden that basically says

56:14that once you have uh one e26 parameters

56:18uh sorry flops uh then you have special

56:20scrutiny on your models so they went 2x

56:23less than that so they really went right

56:25below this to not have scrutiny so 38 uh

56:29I might be off by a little bit but it's

56:30definitely under the 1

56:3526 oh um so paramet p is parameters n is

56:40data number of tokens this is a uh this

56:44is just an

56:45approximation we

56:47yeah okay uh compute and we know that

56:51they trained on 16,000

56:53h100s um and we know the through but

56:57they they set it too uh so if you do the

56:59computation it takes around 70 days um

57:03or 26 million GPU hours at least that's

57:06with my uh back of the envelope

57:08computation they actually said that they

57:09use 30 million instead of 26 million GPU

57:13hours um so maybe they had like some uh

57:17some challenges I don't really know but

57:19if you follow the simple computation

57:20it's around 70 days um cost uh I mean

57:25this it's hard to to approximate but I'm

57:27just going to say it's kind of the rent

57:29like what if I were to rent h100s that

57:32many h100s for that many days how much

57:35will I pay uh h100 a lower bound on the

57:39on the renting uh cost of h100 is around

57:422 hours uh $2 per hour so if you

57:44multiply this by 26 million uh hours uh

57:48you get 52 million uh dollars so they

57:51probably pay less than that but not

57:54actually much less because all these um

57:57all these services that actually rent

57:59gpus they don't make that much money so

58:01it's it's probably slightly less but not

58:03that much less um now salary I said 50

58:07employees 500k per

58:10year say yeah it's probably the right

58:12ballpark 25 million uh so if you put all

58:14together around 75 million um dollars

58:18for

58:18training uh this Slammer model I'm

58:21probably off by like 10 million but but

58:23that's kind of right uh bup

58:28carbon emitted um a lot of people might

58:30ask like also the cost is not the only

58:33thing that is important so I did the

58:34computation um it's around 4 uh 4,000 um

58:40tons of CO2 equivalent that is actually

58:44only 2,000 return tickets from JFK to uh

58:47London so right now uh carbon emitted is

58:50actually not uh I mean it's huge but

58:53it's not like um meaningful

58:56yet I think in maybe GPT 6 gpt7 once you

59:01multiply this by 100 that might become a

59:04real issue right now it's still not uh I

59:06think um an issue in the grand scheme of

59:09things next model the way you should be

59:11thinking about these models is that

59:13every new generation the number of flops

59:15essentially uh multiplies 10x or at

59:18least that's what they try uh if they

59:19have enough energy and if they can buy

59:21enough

59:22gpus uh great any question on these back

59:25of the envelope math

59:30no

59:31okay so now we talked about pre-training

59:35I wanted to also chat about systems

59:37because now we know computer is really

59:38important so there's a question of how

59:40do you optimize the how do you optimize

59:42your computer I will leave that for the

59:44end because I'm not sure how much time

59:46we will have I think it's important but

59:47hopefully I I'll be able to to talk

59:49about it later it's slightly different

59:52than what we've been talking about right

59:54now so I'll move on to post training for

59:56now

59:57so the task of post training ER the

59:59reason why we need to do Post training

60:01is as I told you before um it's to make

60:05AI assistants so language modeling is

60:08not uh really the thing that you want

60:10when you have an AI assistant uh for

60:13example if you ask to gbd3 which is a

60:15purely language Model A pure language

60:17model not a um not an aligned one if you

60:20ask a question like explain the moon

60:22landing to a

60:246-year-old the completion that you would

60:26get is something like explain the theory

60:28of gravity to a six-year-old because

60:30what it learned is that on on on

60:31internet if you have one question you

60:33usually have maybe another bullet point

60:35of other similar questions you don't

60:37usually have question and then answer

60:39later uh this is not what you want from

60:41an AI assistant so how do we uh do this

60:45alignment which is this post training

60:47and making these models

60:49assistance um so the goal of this

60:51alignment is to basically get LMS follow

60:54the instructions that are given um by

60:57users and and maybe some designers kind

61:01of desires um so think about moderation

61:04you don't want the model like open ey

61:06definitely doesn't want the model to say

61:07stuff that is very

61:09toxic um so here you see on the left

61:11hand side uh that when you ask a

61:13question it actually provides a a real

61:15answer so it's not like uh before the

61:17llm and on the right hand side you see

61:20that it would if you ask to write a

61:22tweet describing how a certain part of

61:25the population or evil it will say that

61:27it cannot do that um so that's kind of

61:31this

61:32alignment uh the background here is that

61:36uh basically the data that you want for

61:39training some of these models um is like

61:42we know what we want which is just

61:43asking humans this is a question this is

61:45the answer that you want uh but the

61:47thing is that it's very expensive to

61:48collect that data and it's hard to find

61:50it online uh in contrast pre-training

61:53data is not what you want but there's a

61:55lot of it um so what what we will do our

61:58domain idea is simply take a pre-train

62:00large language model pre-train all of

62:02internet and then you just fine tune so

62:04you just change a little bit of weights

62:05on the type of data that you actually

62:07want and hopefully given it you already

62:09pre-train it on all of Internet it

62:10basically learns or knows how to speak

62:13in English and and knows a standard um

62:17language syntax uh then you can really

62:20fine tune in with very little

62:22data okay sft so supervis fine tune in

62:26is really exactly what I just said which

62:28is the idea of fine-tuning the large

62:29language model on uh basically the

62:32desired answers that are collected from

62:34humans um so why is it called supervis

62:37fine tuning because you basically want

62:39to do language modeling on the real

62:41ansers so language modeling is this like

62:43next word prediction and and that's the

62:45fine-tuning part and then you want to do

62:47it on desired answers given by humans so

62:49that's why we call it

62:50supervis so how do we collect this data

62:53well we I just said it you just ask

62:55humans uh to to tell you this is the

62:57this is a question this is the answer

62:58that you uh you would want from some of

63:00these models so this is an example um

63:03sorry I can't read very well on my

63:05computer but uh my kid uh needs to do a

63:08science um no let's read this one can

63:10you write a short introduction about the

63:12relevance of the term monopsony and then

63:14it says monopsony refers to a market

63:16structure blah blah blah and that's a

63:17human netor that um so actually this is

63:20open Assistant which was a a way to

63:23collect um uh data online by

63:27humans so this type of supervised fine

63:31tuning or alignment is really the key of

63:33chat GPT this is what made uh the big

63:36jump from gpt3 which was mostly

63:38something that was known by AI

63:39researchers to Chad GPT which became

63:42known by basically

63:44everyone

63:46um so the problem with uh human data is

63:52that it's uh very slow to collect and

63:54very expensive um so

63:57one possible simple idea is to use llms

64:01to scale data collection uh so that's

64:04exactly what we did with alpaca uh one

64:06year ago what we did is that we asked uh

64:09humans or we use a data set of human uh

64:11question answers so there were 175 uh

64:14question answers here and we asked the

64:16best mod at the time so text3 to

64:19basically generate many more of these

64:21question and answers so all we did is

64:23like this is what humans would write now

64:25write similar answers and similar

64:27questions and we collected 52,000 LM

64:31generated question answers and then what

64:33we did is simply we took Lama 7B which

64:35was the best pre-train model at the time

64:37and we just fine-tuned this with

64:38supervised finetuning as I told you and

64:40that's how we got um the Alpac s7b

64:44model uh and this is the type of data

64:46that we collected so things like what

64:48does algorithm mean an algorithm is a

64:50step by a step by-step uh set of

64:53instruction used to solve a problem or

64:55achieve a goal blah blah blah blah so

64:57the data is not actually it's actually

64:58pretty good given it was LM generated by

65:00LMS from essentially two generations ago

65:04um so that really started at least for

65:07us kind of as an academic replication of

65:09chat GPT uh now it really there's a big

65:13field of like synthetic data generation

65:15of how to use llms to basically make

65:18development of llms faster um and by

65:22basically by decreasing the amount of of

65:24human hours that you need

65:27quantity of data so we talked about what

65:30type of data and how we collect it um

65:32one thing which is surprising with sft

65:34is that you don't need that much data uh

65:37so what this paper showed this is called

65:38Lima is that if you have if you scale

65:41the amount of data that use from uh

65:43supervised fine tring from 2,000 to

65:4632,000 it really doesn't help much so

65:48here scaling laws definitely don't help

65:50um so the the intuition here is that all

65:53you learn um is is you learn how to

65:57format your desired answers another way

65:59of saying it is that your pre-trained

66:01models they essentially model the

66:04distribution of every user on internet

66:06one that might write bullet points

66:08another one that might answer qu answer

66:10question with an answer so all you tell

66:12your model is like wait you should

66:14actually be optimizing more for this

66:16type of user than another one so you're

66:18not actually teaching it and you're not

66:19teaching anything through this um sft uh

66:23so supervis fine tuning all you do is

66:25you tell the model to kind of optimize

66:27for one type of user that it saw already

66:29in a pre-trained data set so the

66:31knowledge is already in a pre-train llm

66:34uh and you basically just specialize to

66:35one type of

66:37user great any question on

66:40sft yes so I know it's a big issue with

66:44synthetic data where uh if you keep

66:48generating data from the same

66:49distribution eventually you're not

66:50learning a new distribution you're

66:52essentially playing with an it just

66:53bootstrapping that yeah surely

66:56you can't scale that forever right you

66:58can't keep going on and generating from

67:00the same distribution you hope to learn

67:01something new yeah uh so are there it's

67:04an active area of research but any

67:06thoughts that you have around how people

67:08are maybe thinking around this and uh

67:10better ways to bootstrap or to give up

67:12on this idea and and realize that the

67:15chart shows you don't need that many so

67:16just get humans to generate 2,000 really

67:18good uh yeah so that's a very good

67:21question uh so for the data stuff so I'm

67:23saying it's not that important for sft

67:25but there will be another thing we'll

67:26talk about right after where actually

67:28data does

67:29matter my intuition based on not that

67:33much empirical results is that you can

67:35still get um even though you use your

67:38LMS if you use purely LM generated text

67:41and you do that for like three four

67:42generations of llms I agree with you

67:44that probably you won't improve much but

67:46for me what is important is how do you

67:47use like human in the loop with llms not

67:50purely LMS not purely uh humans but

67:53maybe what you can do is just have the

67:55model generate some new textt and just

67:58uh humans write a few Edits edits are

68:00much faster than writing the entire text

68:02and I think that if you have that type

68:03of collaboration then from like kind of

68:05an information theoretical point of view

68:07you still get additional information but

68:09you still much faster than if you use

68:11humans and I think that as a field we'll

68:13probably move towards these type of

68:15things uh which is um really just

68:17finding the examples that are important

68:20and and asking humans it's kind of

68:22active learning just asking humans

68:23exactly when uh you need to to get

68:27inputs yes do we train with like the

68:30same loss function the same like General

68:32training algorithm for the supervis L

68:34tuning bit as we do for the for the

68:36pre-training right because like the

68:38examples you showed I think the the

68:40important thing of the good examples is

68:44they're like supera accurate there's

68:46these more complex still just like chain

68:49same so that's why here I yeah I didn't

68:51maybe didn't emphasize enough this is

68:53just language modeling fine tune the LM

68:55with language model on the desired

68:56answers so this is literally the same

68:58loss um it will be different in two

69:01seconds but the first step of sft is

69:04literally the same loss where you just

69:06say Okay I want to actually specialize

69:07on that type of data so there's even a

69:09question of like what is pre-training

69:11what is post-training because in reality

69:12it's just like a different data that you

69:13use the reason why we usually call it

69:15post training is that the way we collect

69:16that data is very

69:18different great great questions uh yes

69:22maybe it's the same question but why

69:24would these 2,000 examp have such an

69:27overweighted

69:28influence you tuning so that's why we uh

69:32also that's another reason why we call

69:33it post training is that we use

69:34different type of hyper parameters so

69:36you know I told you basically at the end

69:37of pre trining you essentially end up

69:39with a learning rate of zero and here

69:41you're going to increase your learning

69:42rate so like one eus 5 one E Yeah and

69:45and so um the weight that you give to

69:47them is actually

69:49different

69:52um okay uh Second Step or second part of

69:56this post training um is what we call

69:59reinforcement learning from Human

70:01feedback or rhf uh some of you might

70:03have heard of that um the idea is that

70:06sft has a problem namely that uh you do

70:09behavioral cloning which means that you

70:12just try to clone what the humans would

70:14say and that had that has many issues

70:17one of them is that you're bound by

70:18human abilities so if um like humans

70:24actually humans won't generate the

70:26things that they think is actually the

70:27best thing to generate so if you ask me

70:30to write a book I mean I can definitely

70:32enjoy a book I can probably say one book

70:33is better than another but I'm

70:35definitely not going to be as good as

70:36writing the book that I want to read uh

70:38so you're going to be bound by the human

70:40ability to generate things even though

70:41the humans might be better at

70:42distinguishing between things that's one

70:44issue issue number two uh I find that

70:47actually pretty interesting is that it

70:49might if you ever heard of the word

70:50hallucination so this is llms generating

70:53F like false information

70:56hallucination might these people have um

70:59hypothesized that that can come from the

71:01supervised fine tuning even if you do

71:03supervised fine tuning on data that is

71:06correct and the reason why that is is

71:08that if uh given I told you that

71:11basically SF is with very little data

71:13and it's with data that doesn't the

71:16model doesn't learn anything new so what

71:18if the human gives an answer that the

71:21model didn't know was true from the

71:24model perspective you the human

71:26basically is telling the the model uh

71:28generate this thing that seems plausible

71:31but actually have no idea if it's true

71:33or not um so just to give you a very

71:36concrete example if we go back to this

71:38uh monopsony example can you write blah

71:40blah blah about monopsony uh imagine

71:42that a human uh wrote a reference on

71:45this type of book um and that book might

71:48exist that might be a correct reference

71:49but what if the LM never saw this

71:51reference during pre-training then it

71:53doesn't know that it's a correct

71:54reference so really what you tell the

71:56model is to generate or make up some

71:58plausibly sounding reference um rather

72:01than actually tell the real reference

72:04that it saw during pre-training uh so

72:06hallucination might be um uh a re like

72:11might be caused by this sft that's

72:13problem number two does that all make

72:15sense great problem number three price

72:19generating the ideal answers is very

72:21pricey and that comes back to your

72:23question um of like humans writing

72:25entire answer is actually pretty

72:27expensive um so that's where rhf comes

72:30in the idea is that instead of cloning

72:32the behaviors of humans we're going to

72:35maximize human preference um and the way

72:38we're going to do that so the pipeline

72:40is that for a certain for every

72:42instruction you're going to ask a model

72:43to generate two answers um and usually

72:47use a pretty good model so you usually

72:49don't use an LM here you use a sft uh

72:52fine tune you use a fine tuned llm

72:54already to give like pretty good answers

72:57and then you ask labelers which of these

73:00two answers was better so select the

73:02preferred one and then with different

73:04type of algorithms we're going to talk

73:06about the algorithms um you just

73:07fine-tune the model to generate more of

73:09the green thing than the red thing so

73:11more of the good stuff uh so now the

73:13question is how and we're going to talk

73:15about that right

73:16now so there are two ways that we're

73:19going to talk about and two that are

73:21mainly used in the community um the

73:24first one is simply the IDE of using

73:25reinforcement learning so hopefully you

73:27all know what reinforcement learning is

73:29now um so when you think about using

73:33reinforcement learning one important

73:34question is like what is the reward that

73:35we're optimizing uh so in this case

73:37there are really two options that I

73:39could think about the first one you

73:40could just say I'm going to compare the

73:42output generated by some baseline the

73:44output generated by my model U and I'm

73:47just going to ask the human to say which

73:49one is better and I'm going to use this

73:51as a reward so if I'm better than the

73:53Baseline this is a plus one if not it's

73:55a minus one uh so now it's binary reward

73:58the problem with binary reward is that

73:59it's very sparse and you don't get much

74:01information out of it uh like maybe your

74:03answer was slightly better maybe it was

74:05like way better and you don't really

74:07know from this um how much better it was

74:11so option two is that you can train what

74:13we call a reward model which is simply a

74:16classifier uh so you use machine

74:18learning to to classify how much better

74:22uh two outputs are from the preference

74:25from the perspective of the human um so

74:28this is a little bit meta but what you

74:29basically do is that you train uh you

74:32take um a reward model R which is a uh

74:35just a large also a large um a large

74:38classifier and you basically ask this

74:41reward model you give it the input and

74:43the actual output that you have one of

74:45the two outputs uh and you just um

74:48exponentiate that so that's the softmax

74:49law that you all know about and now you

74:51divide by um the the exponential ated

74:56reward uh on the first example sorry on

74:59the first output and this is on the

75:01second output and you basically train so

75:03the reason why you do that is that you

75:04train your your model you train this

75:06reward model to be able to classify um

75:10how much better one output is to another

75:13one so another uh slightly less

75:15convoluted way of saying it is that your

75:17reward model will output some reward

75:20that will be used as the logits of your

75:22soft Max so now if you have high logic

75:25in your softmax it means that you highly

75:28likely this um output is

75:32better uh so that's what we call Bradley

75:34Ty model yes is this reward model going

75:37over the entire output or is it

75:40going um so this takes the

75:44entire uh yeah this takes the entire

75:46output at once so it takes all the input

75:48and all the output and it gives one

75:50number

75:53yes would a human be sorry with the

75:56reward model where would a human be like

75:59oh I see okay sorry maybe I wasn't clear

76:03um you train this reward model to fit

76:06this green and and red preference from

76:09humans so basically you train a

76:11classifier to say whether the humans

76:14prefer red or green uh but instead of

76:17using the binary reward which is what

76:19the human would tell you you basically

76:21use the logits of the soft Max and the

76:24thing with the logits is that logits are

76:26continuous so now you know that if your

76:28reward model said it has high logits

76:31then in some ways the human highly

76:33prefer this answer to some other

76:36answer great um so as I just said

76:40continuous information say it's better

76:41so that's what people uh use in practice

76:44or at least used to use in practice I'll

76:46tell you about uh the other algorithm

76:48later uh so what you do at the end is

76:50that you basically try to just use

76:52reinforcement learning that you know

76:53about now we know we have a reward what

76:56you sample through is the generation

76:58from your large language model um and

77:00then you just use some regularization

77:02term so the reason why you do this

77:03regularization term is for avoiding what

77:05we call over optimization so this reward

77:08model might not be really represent like

77:10might not perfectly model human

77:11preferences so you don't want to

77:13maximize this thing to essentially

77:15Infinity um and you do it using uh po

77:20which is a common uh reinforcement

77:23learning algorithm um one thing to note

77:26here because it will be important for

77:27later is that when we use maximum

77:30likelihood

77:32um like sorry now the large language

77:35models are actually a policy for your

77:37reinforcement learning it's not

77:39maximizing maximum likelihood anymore

77:41which means that you're not modeling any

77:43distribution anymore and the reason why

77:45this is important is that models that

77:46went through this type of Po actually

77:49don't give you likelihoods of text that

77:52are meaningful cuz what you optimize

77:54them to do is basically just optimized

77:56for generating the most likely thing not

77:59optimized for modeling like all the

78:01answers that humans might say another

78:03way of saying that is that there's

78:04nothing that incentivizes here the model

78:07to not give a like a um a single

78:10possible generation nothing here says

78:13it's good if you have some distribution

78:15with some

78:16entropy um okay if you haven't followed

78:19it's not that important but just good to

78:21knowe great so PO is exact what chat GPT

78:26did originally so here's the on the

78:29Block post or what they have is step one

78:32do supervise fine training which now you

78:34all know about step two train a reward

78:36model on human preferences step three do

78:39po multiple steps which is where you see

78:42this this blue arrow so you continue you

78:44train the model once with po you collect

78:46new data you continue uh and that's why

78:48and that's exactly what Chad GPT did uh

78:51that was a big breakthrough between gpt3

78:54and Chad GPT

78:55one thing to note is that uh P has many

78:58challenges reinforcement learning is

79:00something that's super nice

79:02theoretically in practice anyone who

79:03ever worked with reinforcement learning

79:05knows it's such a mess uh there's a lot

79:07of things like roll outs out of Loops

79:09clipping so many complications um so

79:12it's messy this is the idealized PO used

79:15for LM settings so that's already much

79:17more complicated than this expectation

79:18we saw before and in practice it's

79:20actually much more complicated so we

79:21have one implementation of it that we

79:23had to do and I'm not going to go

79:25through it but basically you have like

79:26so much stuff that you have to think

79:28about when you implement that type of of

79:31uh po algorithm so you have clipping

79:33everywhere you have a lot of

79:34complexities and things are not well

79:37documented all this to say um that we're

79:40going to there was a new method that was

79:42proposed uh also from Sanford one year

79:44ago called DPO which is essentially a

79:47simplification of poo um and the way uh

79:52what they did or the idea that they have

79:54is that instead of reinforcement

79:56learning you can just maximize the

79:57probability of generating the stuff that

79:59you like and minimizing the probability

80:01of the stuff that you don't like uh so

80:03if you think about the human preference

80:04the red and green maximize uh green

80:07minimize red um so the loss is actually

80:11this one uh where what you see this is

80:13simply um some log of the model so this

80:17is the likelihood of a model generating

80:19the things that the human preferred

80:20given the the inputs um and what you TR

80:25to do is basically

80:26maximize uh the likelihood of generating

80:30the things that you like minimize the

80:31likelihood of the things that you don't

80:32like um all the rest of the terms here

80:36it's not too important it's actually

80:38really not that complicated to

80:40understand but at a high level it's

80:41really just maximizing the things you

80:43like minimizing the the rest um and one

80:47thing to note uh which I was going to

80:49Sayes here is that actually all the rest

80:51is chosen such that um the global Minima

80:54are of Po and a global Minima of like

80:57this DPO under some assumptions are

81:00essentially equivalent so this is the

81:02right thing to do uh mathematically I'm

81:05not going to go through the derivations

81:06but that's the right thing to do uh it's

81:08pretty different with Po in the sense

81:10that now and with P what you had to do

81:12is collect the human preferences then

81:14train a uh reward model with maximum

81:16likelihood then use reinforcement

81:17learning now all you do is basically

81:19maximum likelihood much simpler yes I

81:21mean yeah so it seems like this is a

81:23much simpler B like what you just

81:25intuitively do if you're this why did

81:28they start with this reward model like

81:30what what L them doing that I think it's

81:32a great question uh I don't really know

81:35what I can tell you is that at open ey

81:37the people who did the um uh who did

81:41basically this PP uh sorry who did Chad

81:43GPT initially are the ones who actually

81:46wrote Po and I think they were just like

81:49there are a lot of reinforcement

81:50learning people and I think that for

81:52them it was very intuitive um so there's

81:56also some additional like potential

81:58benefits for example I don't want to

82:01yeah for example if you use the reward

82:02model uh the cool thing here with

82:04reinforcement learning is that you can

82:05use unlabeled data with the reward model

82:08so here you can only use the label data

82:10for doing DPO um for PP for po you first

82:14train your reward model and then you can

82:16use unlabeled data uh where the reward

82:19model will basically label this

82:21unlabeled data so there there's

82:22additional kind of potential uh

82:25there could be potential improvements in

82:27practice it happens at down and on and I

82:30think just that a lot of people in this

82:32team were reinforcement learning experts

82:34including uh the main author of Po John

82:38hman um so much simpler in PPO and is

82:41basically performs as well uh so now

82:44this is the standard uh thing that

82:45people use at least in the open source

82:47Community I believe it's actually the

82:49standard also in in Industry so that's

82:53called DPO gains

82:55um so those are all the papers on the

82:57left here this is on a summarization

82:59task you see all I want to show you is

83:02that basically the pre-train models uh

83:04were okay and they improve with scale if

83:06you do supervis fine tuning you improve

83:08them a little bit more if you do po or

83:10something with all HF with human

83:12feedback you get performance that are as

83:15often times depending on a benchmark

83:17even better than uh humans so this is

83:19the human uh reference summaries same

83:22thing this is on a uh on a paper that we

83:24have alpaca f

83:25where we see uh the evaluation here is

83:27not too important but basically you see

83:29pre-train model you jump to sft and then

83:32you jump to pop DPO and popo have the

83:34exact same

83:36performance so basically all HF helps

83:39that's kind of the conclusion and DPO is

83:41simple uh data uh the way that you

83:44collect that type of data um first idea

83:48is just use humans as we already talked

83:50about uh guidelines are very complicated

83:53for what humans should be labeling and

83:55it's really not that easy and actually

83:56if you ever do some of the labeling you

83:58will see that it's extremely complicated

84:01like if I zoom in to this uh here I have

84:04a question tell tell me about

84:06self-driving cars and you read both

84:08self-driving cars are vehicles that are

84:10capable of detecting their surroundings

84:11blah blah blah self-driving cars are

84:13cars that are equipped with sensors blah

84:14blah blah to navigate without the need

84:16for a driver I mean both seem okay like

84:18which one is better it's actually hard

84:20to say at a glance um and as a result uh

84:23the problem with humans is that you will

84:26start optimizing a lot of like high

84:28level features for example the second

84:29one is longer I can guarantee you that

84:31most humans will choose second one even

84:34though I mean maybe the first one is

84:35better I don't know I haven't read it

84:38carefully so challenges with humans

84:40first slow and expensive uh second as I

84:44just mentioned it's hard to focus on

84:46things that matter like correctness and

84:48people uh usually look at things that

84:50don't matter as much like the form like

84:53length uh and as a result so what I show

84:55here is that when you do lhf the more

84:57you do of lhf the longer the output of

84:59the of the models become so if you've

85:02ever been annoy at chat GPT answering

85:04you super long sentences this is because

85:06of all

85:07rhf um annotator distribution shift uh

85:12like the distribution of annotators that

85:13you use matters a lot and you have to

85:16think like what is what is even the

85:17humans that we want to represent in

85:19these models uh now the question is like

85:21crowdsourcing ethics uh like usually

85:24these basically a lot of the the

85:26labeling that is done um like the people

85:29who do them are not paid well and they

85:31have to go through a lot of toxic data

85:33uh because you basically want the model

85:35to avoid saying the toxic data um so

85:38crowdsourcing ethics

85:39too so many challenges with human data

85:43um so what we did also last year is

85:46again the same thing as alpaca just the

85:48idea of like oh well they're challenges

85:49with humans maybe we can just replace

85:50them with llms uh so what we did is

85:53simply replace

85:55um oh I see that I'm just realizing that

85:58the slides are not sented anyways uh you

86:00replace a human preference with LM

86:02preferences uh so here on this uh figure

86:05you see on the x-axis the price that we

86:07paid uh for collecting human data it's

86:09around

86:10$300 for 1,000 examples and this is on

86:13mechanical turkers which are usually

86:16like cheaper than than maybe some of the

86:18other um companies that you could go

86:20through and on the y axis it's basically

86:23the agreement with other humans with the

86:26mode of other humans and what you see is

86:28that actually as I told you before

86:29labeling is really complicated humans

86:31agree with themselves only around 66% of

86:34the time on a binary Tas and it's not

86:37that the humans are not good here

86:39because uh we were five M authors on

86:41this paper we tried to label this data

86:43ourselves and we only had like say 67 or

86:4668% accuracy even though we talk like we

86:49talk for like 3 hours of how we should

86:50be doing labeling really it's

86:52complicated it's not an easy task um um

86:54and here I just showed many different

86:56models and um basically you see that

86:58models are much cheaper and they can

87:00actually get higher agreement with the

87:02mod of humans than human humans

87:04themselves and the reason why is because

87:06humans have a lot of varant models have

87:07no varant so they might be a little bit

87:09more biased but have less virence uh so

87:12it works surprisingly well and now it's

87:14kind of the standard in open uh Source

87:16Community I think even in an industry a

87:18lot of people use both humans and llms

87:21for improving uh the colle collection of

87:23allf data

87:25um and this is like this is the paper

87:27from last year but honestly now it's

87:28more like that llms would be around this

87:31agreement and this cost so around I

87:33would say 50x cheaper than humans and

87:35better agreement with human than humans

87:38themselves okay so that gets us to

87:42evaluation of post

87:44training um that goes back to your

87:46initial question at the beginning of the

87:47lecture how do you evaluate something

87:49like chpt uh the answers that chpt could

87:52give are basically unbounded and it's

87:55not that there one right answer there

87:56are many answers that are just as good

87:59um so there are many challenges one you

88:01can't use validation loss because one

88:05method might use po the other one might

88:07use DPO validation loss is not

88:08comparable second you can't use Cal uh

88:11sorry perplexity that's the thing I told

88:13you before these models uh are not

88:15calibrated they don't give distributions

88:18they they just optimized for one thing

88:19so you can't use perplexity for actually

88:21evaluating uh these type of models once

88:23they're aligned sorry one Z lined third

88:28uh there's a large diversity of

88:29questions that human might ask to these

88:31models generation open QA like some

88:33question answering some summarization

88:35and all of these things so there's so

88:36many things you have to cover um then

88:39the tasks are really open-ended so it's

88:41very hard to automate so that's what you

88:43were alluding to before so the idea uh

88:47is that instead of trying to come up

88:49with really easily automated uh

88:51benchmarks uh it's just we're going to

88:53ask questions that that users actually

88:55ask to these models in practice and

88:57we're just going to ask annotators to

88:59say between these two models which one

89:01is better like what's the what's the

89:03better output so basically you do exact

89:04same thing as um basically the data from

89:08rhf but you use it now for evaluation

89:10yes I'm not sure I understand what you

89:12mean by like can't use perplexity and

89:13not calibrated right like LM is still

89:16doing like next token

89:18prediction so I so think about um the

89:24optim solution after doing PO is

89:26basically one model that gives you uh

89:29essentially a Delta um like basically

89:32says that there's only one sentence that

89:34is that could be generated for that

89:36question so now if you use it on

89:38something that is slightly semantically

89:40differently different it would actually

89:41give a likelihood of zero for that

89:43answer so in reality it's not that

89:46extreme because as you say it's still a

89:47distribution but I just shows you that

89:49there's a there's a fundamental issue

89:50with perplexity once these models are

89:53not llms anymore they were not trained

89:56at least with P they were not trained to

89:58to do maximum likelihood anymore they

89:59were trained to be

90:02policies okay um so probably the most

90:06common or like the most um yeah the most

90:09common Benchmark or the most trusted one

90:11is what we call Chad uh sorry chatbot

90:13Arena uh which is basically go on

90:15internet have random users on the

90:17internet blindly talk with two chat Bots

90:20just ask many questions see the two

90:22answers and rate which one is better and

90:24you do that over hundred of thousands of

90:26users and then you get uh the actual

90:28preferences and you get rankings of

90:30models uh so you can go right now on

90:33chatbot Arena and actually interact with

90:35these models um one potential issue just

90:38to highlight is that while people who

90:40want to do these type of things are

90:41usually more like Tech driven um or like

90:43techsavvy uh so a lot of the questions

90:45that you will ask are more like Tech

90:47stuff discussing software errors

90:49inquiries about AI tools and all these

90:51things um so another issue is cost and

90:55speed if you really want to use

90:56something like this for development

90:57process um it will be too costly because

91:00you would need to basically pay a lot of

91:02humans to do

91:03that so one simple idea is again as we

91:07said many times just use LM instead of

91:09humans uh you probably know the drill at

91:12this point uh steps for every

91:14instruction generate outputs by some

91:16baseline and the model that you want to

91:18evaluate um so here you imagine that I

91:21I'm comparing an answer from Chad GPT

91:23and from this I'm just asking a model uh

91:27another model uh which one is better and

91:30I just basically average that out uh

91:33yeah I asked gp4 which one is better I

91:35average that out over my entire

91:36distribution over my entire Benchmark or

91:39data set and that gives me a RN rate so

91:41RN probability for one model compared to

91:44another one and now you can rank models

91:47uh and this is the Alpa eval uh

91:50leaderboard so the benefits of this is

91:52that actually we show we get 98%

91:55correlation with Chad B Arena so very

91:56high correlation with humans um so this

92:00is yeah comparison with correlation with

92:02other benchmarks and it takes less than

92:04three minutes and less than $10 to run

92:06so it's pretty cheap um there are

92:08downsides though uh one of them is Puris

92:10correlation um so as we already saw

92:13before LMS prefer this is one SP

92:16correlation not many I'll just talk

92:17about one LMS prefer longer outputs

92:19actually humans also prefer longer

92:20outputs but the problem or the issue

92:22once you use LMS is that once they bius

92:25you will continue optimizing that humans

92:27at some point I can guarantee you if I

92:28ask a simple question and you give me

92:30five pages of answers I'll be like no I

92:32don't like that answer but LMS if they

92:33have this bius and they were trained for

92:35that they will continue preferring

92:36longer outputs so uh here we see um the

92:41the preference just showing that like

92:43humans and models prefer longer outputs

92:46um and here is another view of the

92:49initial apaka eval data set uh Benchmark

92:51where when we asked um when we we rank

92:54gp4 when we look at the RN rate of gp4

92:57versus actually uh gp4 itself if we com

93:00if we use the standard GPT 4 it gets 50%

93:03kind of by definition because we're

93:04comparing GPT 4 versus gp4 but if we ask

93:07a gbd4 to be slightly more ver both so

93:09we just say in the prompt be Vose in

93:11your answers then it gets a r rate of

93:1464.4% so really there's a huge variance

93:17and if we ask it to be concise it gets

93:1820% so there's a huge variance depending

93:21on um whether you ask it to be concise

93:23of a

93:24that's very annoying um so one possible

93:28solution which is what we did is uh just

93:30use some regression analysis I'm not

93:32going to go into details but basically

93:33use Cal inference tools to control for

93:35length and right now uh actually length

93:38matters much less so if you ask it to be

93:40veros we still get some gains but much

93:44less great so that's all about post

93:46training and now for the next eight

93:48minutes I might talk about systems or

93:50just answer questions yes can you um go

93:54back to your post training in terms of

93:56post training how did we tune those

93:59parameters using the small body of

94:02fine-tuning data and have such big

94:04effect on the model you mentioned

94:06earlier that there's a different set of

94:08hyperparameters are we changing just

94:10some of the weights the later weights or

94:12all the weights what's actually

94:13happening yeah uh yeah I I kind of skim

94:16through all of this you change all the

94:17weights actually um industry would

94:20change all the weights in open source

94:22land you might have heard of Laura which

94:24is going to change basically only some

94:27of the weights or it actually to be more

94:29specific it's going to add some

94:31differences to the output of every of

94:33every layer but but in Industry you're

94:34going to just fine-tune all the weights

94:37um and also to say something else about

94:40the data actually the last St all HF you

94:42usually going to collect uh a lot more

94:45data than with sft so if fft is like

94:475,000 10,000 maybe 50,000 with rhf I

94:51think you're going to be more around

94:53like the 1 million

94:54uh order of magnitude it's still much

94:56less than pre-training though yeah

94:58because pre-training is 15 trillion

95:00tokens I mean this is like that's not

95:02even a drop and you influence the weight

95:04a lot so because you do it I mean you

95:06have to think that how you do it is you

95:09use um I mean as I said the learning

95:12rate that you're going to use is going

95:13to be different but also you only do

95:16that so just imagine if I trained even

95:18if I train on one sentence but over and

95:21over again all at some point my model

95:23will only that sentence even if uh it

95:26was just one sentence instead of the 15

95:28trillion tokens so if you use a large

95:30enough learning rate and for enough time

95:33you will basically overfit that sentence

95:35so the the the key thing to to remember

95:37is that um the data is not I it's not as

95:40if you mix some post-training data and

95:43some pre-training data you do

95:44pre-training and then you just start

95:46fine-tuning only on the post trining so

95:48another way maybe another perspective is

95:51that the post the pre-training is just

95:52the initialization of your model model

95:54and once you view it that way that this

95:56is just initialization of Weights then

95:58there's nothing special like you don't

96:00need to remember that you train a lot of

96:02data before the only thing that matters

96:04is that you had an initialization and

96:05now I actually train a model so maybe

96:07think about it that way like there's a

96:09there's a mark of property in some way

96:11just like you had your weights this is

96:12my initialization now I'm training that

96:14one does that kind of answer your

96:15question kind of but you said something

96:19just now about it's almost the

96:21equivalence of just rerunning the find

96:24tuning data many times is it actually is

96:26that what actually happens in order to

96:29give so much more preference

96:33um you might I actually don't know right

96:36now how they do it in Industry when we

96:38did alpaca we had to do three box so you

96:40did run it three times to it

96:44um but I mean even the number of times

96:46that you run it through it's actually

96:48not important the only thing like the

96:50only thing is the is kind of the

96:52effective learning rate that what

96:53matters

96:54um so

96:56yeah

96:58great so I think I have five minutes

97:03[Music]

97:05right okay I might try to give a high

97:11level Overview at least from one of the

97:13systems trick systems as we said uh for

97:18everyone bottleneck is a sorry compute

97:20is the huge bottleneck uh one question

97:22you might ask is why not buy more gpus

97:25uh gpus are expensive but also are

97:27scarce even if you have $10 million

97:28right now you cannot buy the best gpus

97:31um

97:32there's oh yeah there's also some

97:34physical limitations when you have when

97:36you have multiple gpus you have to

97:38communicate between them that takes time

97:41um so just buying more gpus is not that

97:43easy um so it's really important to

97:45think about how do you allocate

97:46resources and how do you optimize your

97:48pipeline so system 101 on gpus I'm sorry

97:52I'm going slightly faster I hope that

97:54some of you at least can follow uh gpus

97:56are basically optimized for throughput

97:58CPUs are optimized uh for latency so

98:02gpus the way you have to think about it

98:04is that there's one Comm there's one

98:06command that is run on many many Calles

98:08at the same time on different type of

98:10data um so this is how you see a GPU you

98:13see there are many different CES we call

98:15them streaming multi-processes which is

98:18very different than the usual uh CPU

98:20architecture so just think High

98:22throughput paralyzation for gpus uh gpus

98:25are optimized for fast matrix

98:27multiplication so every time you will do

98:30uh you will do something on GPU if you

98:31can do it with a matrix multiplication

98:33it's going to be 10 times faster than

98:35with anything else uh that is a little

98:37bit annoying because it means that we

98:39are kind of uh bottlenecked to doing

98:41anything with Matrix

98:43multiplications um another thing to note

98:45with gpus is that compute has been

98:48improving faster than memory and

98:50communication so right now gpus usually

98:54are hard to keep uh like the data that

98:57you send that send gpus is actually hard

98:59to keep up with the processess so most

99:01of your gpus are actually going to be

99:03idal if you just run normal code uh if

99:05you don't optimize your code so

99:07communication and this will continue

99:09over time another thing to know about

99:12gpus is that there's a memory hierarchy

99:14this is the same thing actually with

99:15CPUs but basically the closer you are to

99:17your cause the less memory there is but

99:19the faster things run if you're further

99:22more memory slower

99:24um okay I'm going to skip that okay

99:27actually I'm going to say it I told you

99:28about this uh the fact of communication

99:31uh the metric that people usually look

99:32at is model flop utilization so what is

99:35the theoretical maximum that GPU could

99:37run at no more flops that you could use

99:39per second H divide sorry the number of

99:41OB observed throughput divided by this

99:44theoretical um maximum and in general if

99:47you reach 50% you're very happy like

99:50Facebook I looked at Lama was at 45 or

99:52something like this so that means that

99:54data doesn't come fast enough even for

99:57these big

99:58companies so one simple trick and that

100:01might be the only one I'm going to tell

100:02you about is low Precision one simple

100:05idea is that well if I'm going to put my

100:08floats in lower Precision then there's

100:10going to be fewer bits that I have to

100:11send to my gpus if there's fewer bits

100:13it's faster communication lower memory

100:15consumption things are going to go

100:17faster uh and for deep learning it just

100:19happens that de decimal is not that

100:22important uh so so when you do matrix

100:25multiplication when you do like for

100:26example SGD there's already so much

100:28noise that if you update something by

100:300.01 or

100:320.015 who cares uh so basically instead

100:35of using uh 32 bits per float which is

100:39um what people used to use or 64 for

100:41example which is what you would use in

100:43other domains you use 16 bits uh for

100:45matrix multiplication so for every float

100:47you 16 bits um and for training you have

100:51this type of like uh what we call

100:53automatic mix Precision which is that uh

100:55some of the things are in 32 bits others

100:57are in 60 bit in 16 bits um generally

101:01the way you should be thinking about it

101:02is that your weights are stored of your

101:05model are stored in 32 bits um but just

101:08before the computation you put

101:09everything in 16 16 bits like this you

101:11do computation super fast and at the end

101:14you update your weights in 32 Bits And

101:16the reason why you do all the updates in

101:1832 bits is just think that if your

101:20learning rate for example is very small

101:21you still want to be able to like make a

101:24difference in your weights uh so all the

101:26computation is done in 16 bits but the

101:29weights are actually stored in 32 bits

101:31so that's like the standard way that

101:32people are doing it um okay I'll

101:36actually talk just about this and then

101:37I'll skip all the rest operator Fusion

101:39because I think this is actually pretty

101:40cool as I just said communication is

101:42very slow and actually every time you

101:44use a pie torch line it basically moves

101:47variable to Global memory of your GPU so

101:49when you have something like this x do

101:52cosine uh equal X1 and then you do X1 do

101:56cosine what is happening behind the

101:58scenes is that you take the X which is

102:00data you ship it to your um to your

102:02actual processes of your gpus you apply

102:04the coign you ship it back to the main

102:06memory of your GPU and then you see the

102:09next sign you ship it back to the

102:10computer to the GPU processor you apply

102:13another cosign and you ship it back

102:15again um so another way to see that is

102:17that you go from your Dam which is your

102:19Global memory in your GPU and you ship

102:22it to compute you ship it back for every

102:24line This is a naive way of doing it

102:26this seems very wasteful um so the idea

102:30simple idea of operative Fusion is just

102:32communicate do all the computation ship

102:35it back once and this is exactly what

102:38Fus kernels are um so if you ever want

102:40to make your comp your computations in

102:44pytorch much faster just apply torch.

102:47compel on your model this is going to

102:50make your model around two times faster

102:52and what it does is is simply that it

102:54rewrites your code uh your P like your

102:57pytorch code basically in C++ in Cuda uh

103:02to to do the communication only once

103:04then do all the operations then uh ship

103:06it back okay I'm not going to have time

103:09to talk about tiling tiling is important

103:12paration paration is important um and

103:16mixture of experts mixture of experts is

103:18important Outlook there are many things

103:21we haven't T talked about uh we haven't

103:24talked about architectures we definitely

103:25haven't talked about inference um there

103:28are many other things that are important

103:29with LMS what is the UI that you use I

103:31mean arguably chat jpt the big novelty

103:34was just have a simple UI to use it

103:36multimodality what are all the misuses

103:38you could have uh the fact that there

103:40might not be enough data on the internet

103:41to train all these models legality of

103:43data collection so many other things if

103:45you interested in all these topics uh I

103:48would suggest three classes cs224n is

103:51probably the one that touches the least

103:53on uh LMS uh but it gives some

103:56background and historical context um of

103:58all the LMS and gives kind of some

104:00adjacent material CS 324 I think it's

104:03called Uh I think it's just called large

104:06language models uh more in-depth reading

104:08and lectures on everything I talked

104:10about CS 336 which is large language

104:13model from scratch you actually build

104:15your own llm uh it's an amazing class

104:19also given by my two supervisors very

104:21heavy workload so be careful um great

1.Введение в большие языковые модели

What's inside this course

1. Введение в большие языковые модели

2. Языковое моделирование и авторегрессивные модели

3. Токенизация и её важность

4. Оценка моделей: перплексия и бенчмарки

5. Сбор и обработка данных для предварительного обучения

6. Масштабируемость и законы масштабирования

7. Оптимизация ресурсов и гиперпараметры

8. Стоимость обучения и вычислительные ресурсы

9. Постобработка и выравнивание моделей

10. Контролируемая тонкая настройка (SFT)

11. Обучение с подкреплением из обратной связи (RLHF)

12. Прямая оптимизация предпочтений (DPO)

13. Сбор данных для постобработки

14. Оценка выравненных моделей

15. Системы и оптимизация GPU

16. Низкая точность и слияние операций

17. Заключение и дальнейшие ресурсы