Notes

Insights into Llama 2 Development: Notes on Angela Fan’s Lecture

Notes by Parul Pandey | Reference Video | Llama 2 paper

Image of a Llama training generated using DALLE-2

Overview

Angela Fan is a research scientist at Meta AI Research, focusing on machine translation. She recently gave a talk at the Applied Machine Learning Days on developing Llama 2, a successor to the original Llama model. Llama 2 is a family of pre-trained and fine-tuned large language models (LLMs) from the Meta AI group ranging in scale from 7B to 70B parameters and is free for research and commercial use.

I started making these notes based on Angela’s talk for reference but then considered publishing it as an article for the community.

Note: All the images used in this article are sourced from the official Llama 2 paper and Angela’s talk, links to which have been shared in the beginning.

🗂️ Table of Contents

· 🌐 LLaMa’s ecosystem
· 🔀 Llama 2: Key Differences from Llama 1
· 🏋️ Training Stages of Llama 2
— 1. Pre-Training
— 2. Finetuning
— 3. Human Feedback Data and Reward Model Training
· 📈 Evaluations: How did this work out for Llama 2?
— 1. Model-Based Evaluation
— 2. Human Evaluation
· Other Interesting findings