Viz (AI language model visualizations)


Download source (PDF)

Data for indexing

Common Crawl Wikipedia Books Reddit submissions Other Raw training dataset size (GB) Words/tokens (B) Parameters (B)
OpenAI GPT-1 (117M)
Jun/2018
5.7 0.117
OpenAI GPT-2 (1.5B)
Feb/2019
40 1.5
OpenAI GPT-3 (175B)
May/2020
468 3 77 22 570 499 175
EleutherAI GPT-J (6B)
Jun/2021
227 6 118 63 411.18 825 400 6
Google BERT (345M)
Nov/2018
12 4 16 0.345
Meta AI/UW RoBERTa (125M)
Jul/2019
107 12 4 38 161 0.125
NVIDIA Megatron-LM (8.3B)
Aug/2019
107 12 4 51 174 8.3
Meta AI Megatron-11B
Apr/2020
107 12 4 38 161 11
NVIDIA/Microsoft MT-NLG (530B)
Oct/2021
1257 23 164 81 338 1863 530
DeepMind Gopher (280B)
Dec/2021
750 1 2100 7649 10500 300 280

More viz in links below…