Retrieval-Augmented Generation (RAG)

This video explains the Retrieval-Augmented Generation (RAG) model! This approach combines Dense Passage Retrieval with a Seq2Seq BART generator. This is tested out on knowledge intensive tasks like open-domain QA, jeopardy question generation, and FEVER fact verification. This looks like a really interesting paradigm for building language models that produce factually accurate generations!

Thanks for watching! Please Subscribe!

Paper Links:
Original Paper: https://arxiv.org/pdf/2005.11401.pdf
FB Blog Post (Animation used in Intro): https://ai.facebook.com/blog/retrieval-augmented-generation-streamlining-the-creation-of-intelligent-natural-language-processing-models
HuggingFace RAG description: https://huggingface.co/transformers/model_doc/rag.html
Billion-scale similarity search with GPUs: https://arxiv.org/pdf/1702.08734.pdf
Language Models as Knowledge Bases? https://arxiv.org/abs/1909.01066
REALM: Retrieval-Augmented Language Models: https://arxiv.org/pdf/2002.08909.pdf
Dense Passage Retrieval: https://arxiv.org/pdf/2004.04906.pdf
FEVER: https://arxiv.org/pdf/1803.05355.pdf
Natural Questions: https://storage.googleapis.com/pub-tools-public-publication-data/pdf/1f7b46b5378d757553d3e92ead36bda2e4254244.pdf
TriviaQA: https://arxiv.org/pdf/1705.03551.pdf
MS MARCO: https://arxiv.org/pdf/1611.09268.pdf

Thanks for watching!

Time Stamps
0:00 Introduction
2:05 Limitations of Language Models
4:10 Algorithm Walkthrough
5:48 Dense Passage Retrieval
7:44 RAG-Token vs. RAG-Sequence
10:47 Off-the-Shelf Models
11:54 Experiment Datasets
15:03 Results vs. T5
16:16 BART vs. RAG - Jeopardy Questions
17:20 Impact of Retrieved Documents zi
18:53 Ablation Study
20:25 Retrieval Collapse
21:10 Knowledge Graphs as Non-Parametric Memory
21:45 Can we learn better representations for the Document Index?
22:12 How will Efficient Transformers impact this?