Summary of current models

Summary of current models: View the full data (Google sheets)
Download PDF version

Language model sizes

Or: While you were sleeping, AI sizes were exploding


Download source (PDF)

Increasing dataset sizes 2018-2022

What’s in my AI? A Comprehensive Analysis of Datasets Used to Train GPT-1, GPT-2, GPT-3, GPT-NeoX-20B, Megatron-11B, MT-NLG, and Gopher

Alan D. Thompson
LifeArchitect.ai
March 2022
26 pages incl title page, references, appendix.

Read more…

GPT-3’s top 10 datasets by domain/source

Download source (PDF)
Contents: View the data (Google sheets)

Contents of GPT-3 & the Pile v1

Download source (PDF)
Contents: View the data (Google sheets)
Read detail of datasets within GPT-3 and the Pile v1, & see alternative viz

Contents of Chinese models

Download source (PDF)
Contents: View the data (Google sheets)

Language model sizes & predictions

Download source (PDF)
Sizes: View the data (Google sheets)

Facebook BlenderBot 2.0

Launched July 2021, BlenderBot 2.0 is pre-trained on (Reddit discussion), fine-tuned on ConvAI2, Empathetic Dialogues, and Wizard of Wikipedia (WoW) datasets. The two additional datasets are Multi-Session Chat and Wizard of the Internet (WizInt). To train for safety, it uses the BAD dataset. Finally—in realtime—it is able to add live results by ‘generating its own search queries, reading the results, and taking them into account when formulating a response.’

Facts on GPT-3

Think you’re a fast typer? In March 2021, GPT-3 was typing 3.1 million words per minute, non-stop, 24×7. With the general availability of the model, I expect that number is a lot higher now… (Nov/2021).

Per day = 4,500,000,000 (4.5 billion)
Per hour = 187,500,000 (187.5 million)
Per minute = 3,125,000 (3.125 million)

—

Every day, GPT-3 generates the equivalent of an entire US public library (80,000 books) of new content.

(“…more than 300 applications are now using GPT-3, and tens of thousands of developers around the globe are building on our platform. We currently generate an average of 4.5 billion words per day, and continue to scale production traffic.” (OpenAI blog, March 2021). Using an average of 55k words per book = 81,818 books per day. “In 2017, there were 9,045 public libraries in the United States with a total of 715 million books and serial volumes” (US stats) = 79,049 books per library.)

—

“GitHub says that for some programming languages, about 30% of newly written code is being suggested by the company’s AI programming tool Copilot.” (Axios, October 2021)

—

“The supercomputer developed for OpenAI [as of May 2020] is a single system with more than 285,000 CPU cores, 10,000 GPUs [assume NVIDIA Tesla V100 GPUs released May/2017, superseded by NVIDIA Ampère A100 GPUs in May/2020] and 400 gigabits per second of network connectivity for each GPU server.”
https://blogs.microsoft.com/ai/openai-azure-supercomputer/

Show more soundbites…

Jurassic-1 (178B)

Launched 12/Aug/2021.

Our model was trained… on 300B tokens drawn from publicly available resources, attempting, in part, to replicate the structure of the training data as reported in Brown et al. (2020) [the GPT-3 dataset, which is detailed in the viz above at LifeArchitect.ai/models].
— AI21’s Jurassic-1 paper

tr11 by BigScience

176B parameter multi-lingual model.
Trained in March 2022.

The BigScience project for open research is a year-long initiative (2021-2022) targeting the study of large models and datasets. The goal of the project is to research language models in a public environment outside large technology companies. The project has 600 researchers from 50 countries and more than 250 institutions. The BigScience project was initiated by Thomas Wolf at Hugging Face.

https://github.com/bigscience-workshop/bigscience/tree/master/train/tr11-176B-ml

“using regular Megatron-LM GPT2 language model w/o multi-lingual dataset”

GPT-2’s dataset was just English WebText, which is popular outbound Reddit links with more than two upvotes. It is 40GB.

They then claim to be using more languages:
“XXX: correct this Languages: ar, ca, code, en, es, eu, fr, id, indic-as, indic-bn, indic-gu, indic-hi, indic-kn, indic-ml, indic-mr, indic-ne, indic-or, indic-pa, indic-ta, indic-te, indic-ur, nigercongo-all, oscar-en, oscar-zh, pt, vi, zhs, zht”

These ISO 639-1 codes translate to:

• Arabic.
• Catalan; Valencian.
• Code?
• English.
• Spanish.
• Basque.
• French.
• Indonesian.
• Indic languages: Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Oriya, Punjabi, Tamil, Telugu, Urdu.
• Niger-Kordofanian languages.
• Chinese.
• Portuguese.
• Vietnamese.
• Chinese-Simplified.
• Chinese-Traditional.

Google did something similar with mT5 in Jun/2021 (with pretty graphs and tables!).

Given that OSCAR (oscar-en and oscar-zh) is sourced from the CommonCrawl, it might be safe to assume that all other languages are also sourced from the CC. Jesse and the Allen AI team found 101 languages in the massive CommonCrawl/C4 dataset: https://arxiv.org/abs/2104.08758.

M6 by Alibaba

Multi-Modality to MultiModality Multitask Mega-transformer (M6)
From 100B to 1T to 10T parameters in less than a year!

“M6-Corpus for pretraining in Chinese, which consists of over 1.9TB image and 292GB text. The dataset has large coverage over domains, including encyclopedia, question answering, forum discussion, common crawl, etc”
https://arxiv.org/pdf/2103.00823.pdf

Megatron

Due to the complexity of this transformer and related language models, Megatron has its own page showing a summary of timeline, labs involved, and other details.

View the Megatron page.

InstructGPT by OpenAI one-pager

* The initialism ‘HHH’ was coined by Anthropic, and demonstrated in InstructGPT.

WebGPT by OpenAI sample question set

Contents: View the data (Google sheets)

PaLM by Google: Explaining jokes + Inference chaining

PaLM has 540B parameters. It is multilingual.

Contents: View the data (Google sheets)

Luminous by Aleph Alpha

The Luminous text model was announced at a conference in Nov/2021, where the parameter count for Luminous (assuming Luminous-World, in progress as of Apr/2022) was said to be 200B.

The announced luminous model uses up to 200 billion parameters and is considered to be just as powerful in the text part as GPT, whose third version includes up to 175 billion parameters. In contrast to the American counterpart, luminous can be combined with any number of images, the model is available in five languages ​​(German, English, French, Italian, Spanish) and has been trained in the European cultural context.
— Heise, translated.

* Alan best guess made during testing Apr/2022, based on informed ‘feel’, token/credit comparison, and tiering.

DeepMind’s models

DeepMind’s models are: Gopher, Chinchilla, Flamingo, and Gato (cat).

Download source (PDF)