Reading GPT-2 source code

This video talks about GPT and GPT2, "Generative PreTraining" models that first proved to the world that solving language modeling task using transformers is a good way to create models capable in a wide variety of tasks.
They also provide a good starting point for learning modern NLP, because they are really simple in concept and implementation.
This video is a part of broader series about NLP: https://www.youtube.com/playlist?list=PL1HdfW5-F8ATeobCTzUfstg3utqgpnVm2
The video is focused on reading the source code of HuggingFace inference and finetuning PyTorch code for GPT-2.
Important links:
- Setup guide and jupyter notebook showed in the video: https://github.com/adensur/blog/tree/main/nlp/00_reading_gpt2_source_code
- HuggingFace "transformers" source code: https://github.com/huggingface/transformers
- GPT-2 release presentation from OpenAI: https://openai.com/research/better-language-models
- Attention is All You Need paper: https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
- GPT-1 paper https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf
- GPT-2 paper: https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf

00:00 - Intro
02:00 - Prerequisites
02:31 - Series Intro
09:25 - A Little Bit Of History
15:29 - Text Transformers Taxonomy
20:43 - Tokenisation
30:16 - Causal Language Modeling
34:52 - Reading Huggingface Inference Code
01:08:26 - Reading Huggingface Finetuning Code