What’s in Grok? (Independent Grok-3 Paper)

Reviews
Received by all major AI labs & intergovernmental orgs.

Being closely involved with xAI’s funding, I thought I understood Grok. This comprehensive analysis uncovers layers of technical innovation I never knew existed, especially xAI’s approach to data.
— Dr Jass Narulla

Alan’s exploration of Grok is a tour de force. With xAI releasing neither a Grok-2 technical report nor a Grok-3 paper, the world’s leading AI analyst pulls back the curtain on these groundbreaking models with precision and technical insight.
— Dr Harvey Castro

At last, an insightful analysis that unravels the mystery of Grok 3’s training data. Alan’s research provides unparalleled clarity on how xAI achieved record-breaking development of a frontier model.
— Paul S.

p6: ‘xAI co-founder Igor Babuschkin was originally a physicist, and briefly worked at CERN’s Large Hadron Collider. His journey through AI research is the basis for many of today’s frontier models. From his close collaborations with Ilya Sutskever and Alec Radford to introducing large language models GPT-4 and Codex, his research footprint spans many key developments in post-2020 AI.’

p16: ‘Building on this ‘gold’ synthetic dataset, and the Twitter datasets outlined in this report, xAI is expected to leverage several high-quality datasets drawn from the Datasets Table (see Appendix for a snapshot to Jan/2025). DeepMind’s MassiveText multilingual dataset contributed 13.3B tokens of multilingual Wikipedia for the RETRO model, and this could be replicated by former DeepMind employees now at xAI. EleutherAI’s The Pile dataset still offers high-quality books and journals corpora already in use at other AI labs. More recently, Hugging Face’s FineWeb presents an enhanced ‘educational’ curation of Common Crawl, and their collaboration with ServiceNow produced The Stack v2, a refined code dataset. The final dataset composition of the Grok-3 model can now be estimated based on these corpora, adding my subjective ‘ALQual’ data quality rating and priority orders drawn from weightings documented in literature like the RETRO and GPT-3 papers.’

p21: ‘Optimizations like quantization, pruning, and compression would still allow a model of this size to be served to hundreds of millions of users. On 20 January 2025, Musk demonstrated a Grok-3 output using 4-bit quantization. We estimate that this would amount to an 87.5% reduction in memory usage, easily fitting onto a single NVIDIA DGX B200 server, or several H100s, for inference. Grok-3 is one of the main frontier models alongside GPT, Claude, Gemini, and Llama.’

p23: ‘BMI technology enables two-way communication between brain and machine: neural signals flowing from human thoughts to computers, and AI signals feeding back into human biological intelligence. This bi-directional capability could revolutionize human–AI interaction. The architecture could combine an onboard ‘Grok mini’ model, running locally on advanced neural hardware, with wireless access to the frontier model in the cloud. The local model would handle immediate cognitive enhancement: boosting processing speed through direct neural acceleration, expanding working memory by offloading to AI systems, and providing instant access to a baseline knowledge repository.’

Cover image
Image generated in a few seconds, on 1 January 2025, text prompt by Alan D. Thompson, via Google Imagen 3-002 (mobile portrait 3:4): ‘black on white data pattern, colossus supercomputer, HDR 3d graphic’

Viz and tables

Table. Major papers by Igor Babuschkin. LifeArchitect.ai

Viz. Igor Babuschkin: AI paper highlights. LifeArchitect.ai

Table. 2023 xAI founding staff and previous affiliation. LifeArchitect.ai

Viz. Where will AGI be born? Dec/2024.

Chart. Employee headcounts at ‘Big 5’ AI labs. 2025.

Table. Twitter text calculations. Tweets after usernames, links, retweets removed.

Table. Twitter outbound links by domain to 2018. Source: GDELT, calcs: Alan.

Table. All datasets for Grok-3. Dataset choices are estimated, numbers are disclosed.

Viz. Contents of xAI Grok. Jan/2025.

Table. Dataset sizes needed to align with 20:1 data optimization for models.

Table. Timeline of Grok models announcements. Estimates in italics.

Chart. xAI Grok series: MMLU scores (2023–2025).

Image. Grok-2 + Aurora. Prompt by Musk’s four-year-old son, X: ‘Bunnies flying spaceships in Star Wars with a monster truck.’ Jan/2025.

Calculations. NVIDIA Cosmos video dataset. Jan/2025. Source showing all calculations: LifeArchitect.ai/Cosmos

Viz. Journey to Grok-5 (2023–2026).

Table. Grok training cost and power draw. H100≈$40k (2024), TPD≈1400W/card.

Image. xAI Colossus supercomputer racks. Memphis, Tennessee, USA. Oct/2024.

Viz. Brain–machine interface benefits. Sep/2021.

Image. Ear clip concept by Card79 commissioned by Neuralink. 2019. Source: Card79.com.

Viz. Frontier AI models + highlights. Jan/2025.

Image. Tesla Optimus Gen 2 humanoid robot. Dec/2023. Source: Tesla.