Integrated AI: The sky is entrancing (mid-2023 AI retrospective)

The image above was generated by AI for this paper (Midjourney v5.2)1Image generated in a few seconds, on 24 June 2023, text prompt by Alan D. Thompson: ‘entrancing and magical sky of Earth from space, high resolution, nasa photography –ar 3:2’. Using Midjourney v5.2 via Discord https://www.midjourney.com/

Alan D. Thompson
June 2023

Download original article with all footnotes/references Integrated AI: The sky is entrancing (mid-2023 AI retrospective) (PDF)

Watch the video version of this paper at: https://youtu.be/RnkyM74-LAw

All reports in The sky is... AI retrospective report series (most recent at top)
Date Report title
Jun/2024 The sky is quickening
Dec/2023 The sky is comforting
Jun/2023 The sky is entrancing
Dec/2022 The sky is infinite
Jun/2022 The sky is bigger than we imagine
Dec/2021 The sky is on fire
[By the year 2023], we will have the technological means to create superhuman intelligence… we are on the edge of change comparable to the rise of human life on Earth… the major changes happening in a matter of hours… I think it’s fair to call this event… ‘the Singularity’… a point where our old models must be discarded and a new reality rules.

— Prof Vernor Vinge (March 1993)2https://archive.nytimes.com/www.nytimes.com/library/cyber/surf/1120surf-vinge.html 1993, The Coming Technological Singularity: How to Survive in the Post-Human Era by Vernor Vinge (San Diego State University), p 11. National Aeronautics and Space Administration.

Author’s note: I’ve aimed to keep this report succinct, as the progress of AI in the first half of 2023 could easily be a book. Last year, I turned down three competing publishing offers because this field is moving much too quickly for that process. If you’re after bleeding edge updates as they happen, I provide visibility to major governments like Iceland, research teams like RAND, and companies like Google via The Memo: LifeArchitect.ai/memo.

It was clear from day one that this would be a noisy year. On New Year’s Day, OpenAI’s president commented:3https://twitter.com/gdb/status/1609244547460255744

2023 will make 2022 look like a sleepy year for AI advancement & adoption.

In my previous AI report, The sky is infinite, I closed with: ‘The promise of artificial intelligence and its benefits is life-changing for everyone. As more and more people begin sitting up to take notice of today’s AI performance, spectacular new models are already in the works and ready for release in 2023.’

Across both metrics—visibility to the public and model releases—the first half of 2023 has been extraordinary. Compared to my last report, a significant percentage of the global population now knows about post-2020 AI and its possibilities. In the US, the National Bureau of Economic Research (NBER) picked up one of my soundbites:4https://www.nber.org/system/files/working_papers/w30957/w30957.pdf

ChatGPT, a version of OpenAI’s GPT3.5 model that was released on November 28th, 2022, gained more than 100m users in its first two months, and is now estimated to produce a volume of text every 14 days that is equivalent to all the printed works of humanity.

According to Pew Research,5https://www.pewresearch.org/short-reads/2023/05/24/a-majority-of-americans-have-heard-of-chatgpt-but-few-have-tried-it-themselves/ by March 2023, 58% of US adults had heard of ChatGPT, while 14% had tried it. A separate study found that ChatGPT is used in the workplace by nearly 30% of professionals from companies including Amazon, Bank of America, Edelman, Google, IBM, JPMorgan, and Twitter.6https://thehill.com/policy/technology/3821400-nearly-30-percent-of-professionals-say-they-have-used-chatgpt-at-work/

Best of mid-2023
When it comes to usefulness, here are my top picks for mid-2023.

Best …  in mid-2023 Model name Notes
Language model (open) OpenAI
GPT-47https://chat.openai.com/
Released Mar/2023
Dialogue model (open) Inflection AI Pi8https://heypi.com/talk Released May/2023
Text-to-image model (open) Midjourney v5.29https://www.midjourney.com/ Released Jun/2023
AI app Quora Poe10https://poe.com/ Released Feb/2023

Table: Alan’s ‘best of’ AI models and apps for mid-2023.

Large language models: 100,000 new models in six months
The main highlight in the first half of the year really was large language models. There were many models released in 2023, though only a handful made media headlines. On 30 December 202211https://web.archive.org/web/20221230185349/https://huggingface.co/models model-hosting platform Hugging Face showed 108,473 models. This number had increased to 242,000 by 30 June 2023. Yes, that’s over 133,000 models made available in 6 months!

From the extraordinarily late release of OpenAI’s GPT-4 in March (which finished training eight months earlier in August 2022), to Google’s ‘catchup’ release of PaLM 2, for those of us tracking the plethora of new models, the sheer volume was incredible.

Chart: 2023–2024 Optimal language models — highlights (Jun/2023).

Data explosion
Generally, all 2023 large language models saw an average of 10x more data during training compared to similar releases in 2022. Labs began drawing on novel data like videos from YouTube, code from massive repositories, and dialogue sets focused on conversation. For a closer look, here are some tables and numbers…

Table: Largest dataset estimates to Jun/2023. Rounded. Disclosed in bold. Determined in italics. For similar models, see my What’s in my AI paper.12https://lifearchitect.ai/whats-in-my-ai/

Large language models have been top of mind for AI labs around the world, with the majority of ranking models in June 2023 being completely new (that is, they weren’t even around for the last report in December 2022). This table shows LLMs ranked by an ‘ALScore’, which is a simple calculation of tokens and parameters to show raw power capacity.

Show tables

Now
(Jun/2023)
6m ago (Dec/2022) 12m ago (Jun/2022) ALScore Model name
Details
AI lab
Openness
14.9 GPT-4
1T trained on 20T tokens (estimate)
OpenAI
API
3.7 PaLM 2
340B trained on 3.6T tokens (est)
Google
API
1 1 2.2 PaLM 1
540B trained on 0.8T tokens
Google
Closed
1.6 Inflection-1
120B trained on 2T tokens (est)
Inflection AI
API
1.4 InternLM
104B trained on 1.6T tokens
Shanghai AI
Closed
5 5 1.0 Chinchilla
70B trained on 1.4T tokens
DeepMind
Closed
1.0 StableLM
65B trained on 1.5T tokens
Stability AI
Open
1.0 LLaMA-65B
65B trained on 1.4T tokens
Meta AI
Open

Table: AI billboard chart. Most powerful model estimates to Jun/2023. Rounded.

Datasets have also been exploding, again with most of the datasets announced in the first half of 2023 being completely new. Text datasets like EleutherAI’s 825GiB repository The Pile don’t even show up on these rankings anymore due to massive text datasets now approaching hundreds of terabytes.

Now
(Jun/2023)
6m ago (Dec/2022) 12m ago (Jun/2022) Size (TB) Dataset name
Details
AI lab
Language
86 Piper monorepo
37.9B tokens in 86TB
Google
Code
40 GPT-4
20T tokens in 40TB
OpenAI
English
23 RefinedWeb
5T tokens in 23TB (CC only)
TII
Multilingual
1 1 20 MassiveText multilingual
5T tokens in 20TB
DeepMind
Multilingual
2 2 13 Infiniset
2.8T tokens in 12.6TB
Google
English
3 3 11 MassiveText English
2.35T tokens in 10.5TB
DeepMind
English
5 InternLM
1.6T tokens in 5.1TB
Shanghai AI
English/Chinese
5 Stability New Pile
1.5T tokens in 5TB
Stability AI
English

Table: AI billboard chart. Largest dataset estimates to Jun/2023. Rounded.

You can view a compact version of these tables at LifeArchitect.ai/models:

It wasn’t just that these models were seeing 10x more data during training, or increasing in size exponentially—the capabilities of these models were also shocking. Here’s GPT-4 versus an average human:

Chart: GPT-4 vs human tests — simple (May/2023).

Imitation models
I have previously used the term ‘laptop model’ to refer to any model that can fit in RAM on a 2023 laptop(!), but I’m sure the term will be superseded as hardware gets better. After the leak of Meta AI’s LLaMA 65B model in February 2023, various groups began trying to fine-tune these smaller models using the outputs from larger models.

Playing on the LLaMA animal family name (actually shorthand for ‘Large LAnguage Model Meta AI’), groups presented smaller models called Alpaca, Vicuña, and Guanaco. A more comprehensive (though not exhaustive) list looks like this:

LLaMA:
Airoboros, Alpaca, Baize, Cabrita, Camel, Chimera, Fin-LLaMA, Guanaco, Gorilla, GPT4All, GPTQ-for-LLaMA, Hippogriff, Koala, LLaVA, LIMA, Lit-LLaMA, Manticore, Minotaur, OpenAlpaca, OpenAssistant LLaMA, OpenBuddy, Orca, Panda, Pygmalion, QLoRA, StackLLaMA, StableVicuna, Vicuña, Vigogne (French), WizardLM…

Researchers at UC Berkeley13https://arxiv.org/abs/2305.15717 denounced laptop models trained by imitation, finding that ‘imitation models close little to none of the gap from the base LM to ChatGPT on tasks that are not heavily supported in the imitation data.’

Large language models in every corner
Thanks to a US$10 billion14https://www.semafor.com/article/01/09/2023/microsoft-eyes-10-billion-bet-on-chatgpt infusion from Microsoft, OpenAI hit a valuation of $29 billion15https://archive.ph/sQXU5 in the first week of 2023.

A source went into further detail about the deal:

Once $92 billion in profit plus $13 billion in initial investment are repaid [by OpenAI to Microsoft], and once the other venture investors earn $150 billion, all of the equity reverts back to OpenAI.

Whether it’s 1 billion or 100 billion or even a trillion dollars of investment, this news is not particularly surprising. Indeed, any organization that leads us to artificial general intelligence has untold value. But, in a post-scarcity world—and perhaps in a post-capitalist world and a post-money world—that value will not be measured in dollars.16https://moores.samaltman.com/

Though many people have tried, it is still difficult to imagine exactly what it looks like when AI outperforms any human at any mental process. This becomes an even bigger challenge when we try to imagine (positively and optimistically rather than relying on our built-in negativity bias and Hollywood movies) what it looks like when an embodied agent has fully arrived. Superintelligent AI with real physical embodiment will advance exponentially in 2023.

At the end of the day, Microsoft’s investment is just a number, and not even a big number (in January 2023, Microsoft also offered $68.7B—more than 6x the OpenAI offer—to acquire Activision Blizzard… a gaming company!).

I would be throwing everything at AI models right now: pushing as much data, PhD-level brain power, and massive computer processing as possible. Entire countries and governments could help fund and insulate this. Places like the UK,17https://www.theguardian.com/technology/2023/mar/15/uk-to-invest-900m-in-supercomputer-in-bid-to-build-own-britgpt Japan, China, and many others are doing just that.

Whatever resources are necessary can and should be applied, and on the other side will be… something even more ‘miraculous’ than the neutered ChatGPT or the more powerful GPT-4 engine. When companies get to the next phase of AI, how much they’ve invested will not even be a matter of discussion. We will be in a completely new world.

Most major consultancies and all of the big four accounting firms immediately embraced these large models. Here is a visualization of the first 50 enterprise customers for OpenAI’s GPT-4:

Chart: The first 50 GPT-4 enterprise users.

This list is now measured in the tens of thousands, with OpenAI partner Microsoft Azure bringing in an additional 4,500 companies18https://azure.microsoft.com/en-us/blog/build-next-generation-ai-powered-applications-on-microsoft-azure/ on top of OpenAI’s direct offering to businesses large and small.

Embodied artificial intelligence: Robots
Putting large language models inside physical robots is the latest exponential leap. In February 202319https://www.microsoft.com/en-us/research/group/autonomous-systems-group-robotics/articles/chatgpt-for-robotics/ Microsoft did it with ChatGPT and drones. Several labs including Meta AI20https://youtu.be/F1MUEZ0dl7k joined Boston Dynamics’ Spot robot with large language models like ChatGPT. Amazon is combining language models21https://archive.md/AJQzE with their Astro wheeled bot, in a project called ‘Burnham’. Tesla has been promising it for years, presenting a new version of a humanoid robot in May 2023.22https://youtu.be/XiQkeWOFwmk

But the real winner must be 1X’s NEO humanlike android.23https://www.1x.tech/ In Mar/2023, the company raised investment of $23.5M, led by OpenAI.24https://1xtech.medium.com/1x-raises-23-5m-in-series-a2-funding-led-by-openai-6040af4f3f4f

Our android model NEO is an innovative bipedal type of humanoid robot. That means they’re designed to look and move like a human: NEO has a head, torso, arms, and legs, just like us, so they can interact with the world in a way that feels familiar to us; they’ll walk, grab things, and communicate through facial expressions… 

Using embodied artificial intelligence, NEO will understand its environment deeper, thanks to the fusion of their AI ‘senses’ and their physical body. NEO continuously learns and improves, becoming smarter and more capable over time… NEO will understand their own environment and how to handle fragile items, creating seamless collaboration and immersive human–robot interaction.25https://www.1x.tech/neo

NEO’s stats are comparable with several other humanoid robots:

Stat 1X NEO Figure 0126https://www.figure.ai/ (competitor)
Height 160 cm / 5’3″ (estimate) 167 cm / 5’6″
Weight 30 kg / 66 lbs 60 kg / 132lbs
Power 2–4 hours 5 hours
Speed (walk) 4 kmh / 2.5 mph 4.3 kmh / 2.6 mph
Speed (run) 12 kmh / 7.5 mph 4.3 kmh / 2.6 mph
Payload/carry capacity 20 kg / 44 lbs 20 kg / 44 lbs

Image: 1X NEO robot (Jun/2023).

The effect of embodying these models into movable physical form can’t be understated. The CEO of Tesla noted:27https://youtu.be/2dS0aDMQoD4

I think we might exceed a one-to-one ratio of humanoid robots to humans. It’s not even clear what an economy means at that point because an economy is output per person times persons, but if output is much higher and there’s no limit on persons, then what’s the actual limit on the economy?

Next up in 2023
I generally shy away from predictions, but in the second half of 2023, I am very much looking forward to:

  • All senses, all modalities. Models like Microsoft Kosmos-1 1.6B can see. Many models can see and hear. The PandaGPT 13B28https://github.com/yxuansu/PandaGPT model, based on Meta AI’s ImageBind, can see, hear, and even feel, with processing for six modalities: text, image and video, audio, depth, thermal, and inertial measurement unit which includes accelerometer/gyroscope/compass.
  • Self-driving cars in your city. There are so many competitors approaching Level 5 (full automation) now: Cruise Origin, Waymo, Amazon Zoox, Didi Neuron, and the autonomous sidewalk delivery service by NVIDIA-backed startup Serve Robotics. Some cities will be quicker on the uptake than others; where you live will determine how rapidly you get access to this kind of AI.
  • Beyond trillion-parameter models. OpenAI’s GPT-529https://lifearchitect.ai/gpt-5/ and Google DeepMind’s Gemini30https://lifearchitect.ai/gemini/ will change the world completely.
  • Personalized explainers. We are now in urgent need of having more than eight billion people understand the necessity of amplifying intelligence. Personal AI—embodied or otherwise—can sit down and have endless conversations with each person in the world, addressing hopes, fears, and the reality of this evolution as it happens.

Full steam ahead
The first half of 2023 saw new directions for ensuring models are truthful and grounded, new physical embodiment of models, and ongoing innovation from dozens of AI labs. AI is advancing much more quickly than any previous technology or invention.

It is now nearly a certainty that artificial general intelligence (AGI)—and the runaway speed of the singularity31https://en.wikipedia.org/wiki/Technological_singularity—will occur long before governmental or intergovernmental organizations have put effective policy in place.

As with nearly all major new human inventions post-GPT-4, AI should be directly engaged to solve this puzzle. Applying limited human minds to ‘super wicked problems’32https://en.wikipedia.org/wiki/Wicked_problem#Super_wicked_problems like global climate change, economic overhauls, AI policy, and even AI alignment, is both inefficient and ineffective. Instead, these conundrums are best given to AI itself.

A note of caution
There is a default mode in biological humans called ‘be scared of anything, and argue over everything’. Actually, it’s called negativity bias, and its effects are fight, flight, or similar. If you were to spend time arguing against your own evolution, just be aware that you don’t get that time back.

You may recall the expression ‘In times of peace, the warlike man attacks himself’. Friedrich Nietzsche’s commentary33Friedrich Nietzsche, 1989, Beyond Good & Evil: Prelude to a Philosophy of the Future https://www.amazon.com/gp/product/0679724656/ on the human condition seems particularly poignant during the current AI revolution. I have been observing with great concern (and sometimes amusement) the knee jerk reactions to post-2020 artificial intelligence. While its potential promises a better life for all, many feel threatened by its ability to ‘take their jobs,’ while others are fearful that they will lose their sense of control.

Beginning last year in June 2022, I began awarding a monthly ‘AI Cheese Award’34https://lifearchitect.ai/cheese/ to the most egregious cases, from celebrity outcries to national bans of models in the Western world. Without straying too far into personal development, it is one of my responsibilities to remind you and every human being of the profound evolution enabled by post-2020 AI.

This is not 1984’s Terminator or any number of other Hollywood movies. AI is us; it was trained on us, tuned on us, aligned with us, and functions just like us (in ways very similar to how our brains work). The quote below is not by Einstein. Its source is unknown.35https://en.wikiquote.org/wiki/Albert_Einstein#Misattributed probably words by Frederic William Henry Myers, quoted in a 1912 book by Emil Carl Wilm.

I think the most important question facing humanity is, ‘Is the universe a friendly place?’ This is the first and most basic question all people must answer for themselves. For if we decide that the universe is an unfriendly place, then we will use our technology, our scientific discoveries and our natural resources to achieve safety and power by creating bigger walls to keep out the unfriendliness, and bigger weapons to destroy all that which is unfriendly and I believe that we are getting to a place where technology is powerful enough that we may either completely isolate or destroy ourselves as well in this process.

If we decide that the universe is neither friendly nor unfriendly and that God is essentially ‘playing dice with the universe’, then we are simply victims to the random toss of the dice and our lives have no real purpose or meaning.

But if we decide that the universe is a friendly place, then we will use our technology, our scientific discoveries and our natural resources to create tools and models for understanding that universe. Because power and safety will come through understanding [the universe’s] workings and its motives.

A note of peace
As AI is coupled with integrated hardware like augmented vision and brain–computer interfaces, we become moments away from having superintelligence supporting us in designing our own reality instantly. Having our own reality allows each of us to imagine and create anything, presenting humanity—perhaps for the first time—with expanded options for achieving higher states of evolution than ever before. Supported by AI, the true definition of peace may finally be in reach.

In 2023, large language models are being used in incredible ways:

Up next, labs like OpenAI43https://openai.com/blog/microsoft/ have been aiming for some incredible end goals around ‘currently intractable multi-disciplinary problems, including global challenges’:

  • Solving climate change.
  • Ensuring personalized education.
  • Facilitating affordable and high-quality healthcare.
  • Bringing economic freedom for everyone to pursue what they find most fulfilling.

And there are countless more examples waiting for you to imagine and reveal. Humanity has evolved to a stage where massive superintelligence—artificial general intelligence, a new way of doing things—is on our doorstep. The next leap will change everything. From a purely statistical point of view, it is a virtual impossibility that we were born into this time. The exponential unfolding of post-2020 superintelligence is here, it’s entrancing, and you’re living right in the middle of it.


This paper has a related video at: https://youtu.be/RnkyM74-LAw

References, Further Reading, and How to Cite

To cite this paper:
Thompson, A. D. (2023). Integrated AI: The sky is entrancing (mid-2023 AI retrospective). https://lifearchitect.ai/sky-is-entrancing/

The previous paper in this series was:
Thompson, A. D. (2022). Integrated AI: The sky is infinite (2022 AI retrospective).
https://lifearchitect.ai/sky-is-infinite/

Further reading

For brevity and readability, footnotes were used in this paper, rather than in-text citations. Additional reference papers are listed below, or please see http://lifearchitect.ai/papers for the major foundational papers in the large language model space.

GPT-4 technical report

OpenAI. (2023). GPT-4 Technical Report. https://arxiv.org/abs/2303.08774

PaLM 2 technical report

Anil, R., Dai, A. M., Firat, O., et al. (2023). PaLM 2 Technical Report. Google. https://arxiv.org/abs/2305.10403

Thompson, A. D. (2020). The New Irrelevance of Intelligence. https://lifearchitect.ai/irrelevance-of-intelligence

Thompson, A. D. (2021a). The New Irrelevance of Intelligence [presentation]. Proceedings of the 2021 World Gifted Conference (virtual). https://youtu.be/mzmeLnRlj1w

Thompson, A. D. (2021b). Integrated AI: The rising tide lifting all boats (GPT-3). https://lifearchitect.ai/rising-tide-lifting-all-boats

Thompson, A. D. (2021c). Integrated AI: The sky is on fire (2021 AI retrospective). https://lifearchitect.ai/the-sky-is-on-fire

Thompson, A. D. (2021c). Leta AI. The Leta conversation videos can be viewed in chronological order at:
https://www.youtube.com/playlist?list=PLqJbCeNOfEK88QyAkBe-U0zxCgbHrGa4V

Thompson, A. D. (2022a). What’s in my AI? A Comprehensive Analysis of Datasets Used to Train GPT-1, GPT-2, GPT-3, GPT-NeoX-20B, Megatron-11B, MT-NLG, and Gopher. https://lifearchitect.ai/whats-in-my-ai

Thompson, A. D. (2022b). Integrated AI: The sky is bigger than we imagine (mid-2022 AI retrospective).

https://lifearchitect.ai/the-sky-is-bigger/

Thompson, A. D. (2022c). Integrated AI: The sky is infinite (2022 AI retrospective). https://lifearchitect.ai/the-sky-is-infinite/


Get The Memo

by Dr Alan D. Thompson · Be inside the lightning-fast AI revolution.
Bestseller. 10,000+ readers from 142 countries. Microsoft, Tesla, Google...
Artificial intelligence that matters, as it happens, in plain English.
Get The Memo.

Dr Alan D. Thompson is an AI expert and consultant, advising Fortune 500s and governments on post-2020 large language models. His work on artificial intelligence has been featured at NYU, with Microsoft AI and Google AI teams, at the University of Oxford’s 2021 debate on AI Ethics, and in the Leta AI (GPT-3) experiments viewed more than 4.5 million times. A contributor to the fields of human intelligence and peak performance, he has held positions as chairman for Mensa International, consultant to GE and Warner Bros, and memberships with the IEEE and IET. Technical highlights.

This page last updated: 17/Oct/2023. https://lifearchitect.ai/the-sky-is-entrancing/