The Memo.
Alan D. Thompson
July 2023
Details about Amazon Titan are scarce.
Here’s what we know about Amazon Titan (not very much):
Model names:
- Text generation: amazon.titan-tg1-large
- Embeddings: amazon.titan-e1t-medium
Summary
| Organization | Amazon |
| Model name | Titan |
| Internal/project name | amazon.titan-tg1-large |
| Model type | Dense |
| Parameter count | 200B |
| Dataset size (tokens) | 4T (4,000B) |
| Training data end date | Undisclosed |
| Convergence date | Estimate Feb/2023 |
| Release date (public) | 13/Apr/2023 (large clients only) |
| Annotated paper | – |
| Playground | Convoluted application process: https://aws.amazon.com/bedrock/titan/ |
Titan updates
7/May/2024: Amazon Titan Text Premier benchmarks. MMLU=70.5. (Amazon)
18/Mar/2024: Amazon Titan = 200B trained on 4T tokens.
Amazon can see $1 billion training runs on the horizon:
…Technical talk from a longtime AWS person sheds light on frontier AI training…
James Hamilton, a distinguished engineer at Amazon, said at a talk this year that within the last year Amazon carried out a $65m training run. Specifically, they trained a 200B dense model on 4T tokens of data across 13,760 NVIDIA A100 chips (using 1,720 P4d nodes). It took 48 days to train. Hamilton described this training run as “1 gen old” so we can assume Amazon has moved on to larger runs since then. Looking ahead, Hamilton said “training runs soon to cross $1b”. (18/Mar/2024, via Jack Clark)
29/Aug/2023: Amazon Titan news on Amazon Science.
24/Jul/2023: Amazon Titan reference (‘amazon.titan-tg1-large’) spotted in the wild on GitHub.
22/Jun/2023: AWS Launches New $100M Generative AI Innovation Center (link).
13/Apr/2023: Amazon announces ‘Bedrock’ AI platform to take on OpenAI (BI).
2/Aug/2022: Amazon AlexaTM 20B models (arXiv paper).
Get The Memo
by Dr Alan D. Thompson · Be inside the lightning-fast AI revolution.Bestseller. 10,000+ readers from 142 countries. Microsoft, Tesla, Google...
Artificial intelligence that matters, as it happens, in plain English.
Get The Memo.
Dr Alan D. Thompson is an AI expert and consultant, advising Fortune 500s and governments on post-2020 large language models. His work on artificial intelligence has been featured at NYU, with Microsoft AI and Google AI teams, at the University of Oxford’s 2021 debate on AI Ethics, and in the Leta AI (GPT-3) experiments viewed more than 4.5 million times. A contributor to the fields of human intelligence and peak performance, he has held positions as chairman for Mensa International, consultant to GE and Warner Bros, and memberships with the IEEE and IET. Technical highlights.This page last updated: 26/May/2024. https://lifearchitect.ai/titan/↑
