(source)

GPT-4 Inference Cost. GPT-4 costs 3x that of the 175B parameter Davinci. This is largely due to the larger clusters required for GPT-4 and much lower utilization achieved. AN estimate of it’s costs is $0.0049 cents per 1k tokens for 128 A100s to inference GPT-4 8k seqlen and $0.0021 cents per 1k tokens for 128 H100’s to inference GPT-4 8k seqlen. It should be noted, we assume decent high utilization, and keeping batch sizes high.

Vision Multi-Modal. It is a separate vision encoder from the text encoder, with cross-attention. The architecture is similar to Flamingo. This adds more parameters on top of the 1.8T of GPT-4. It is fine-tuned with another ~2 trillion tokens, after the text only pre-training. On the vision model, OpenAI wanted to train it from scratch, but it wasn’t mature enough, so they wanted to derisk it by starting with text. One of the primary purposes of this vision capability is for autonomous agents able to read web pages and transcribe what’s in images and video. Some of the data they train on is joint data (rendered LaTeX/text), screen shots of web page, youtube videos: sampling frames, and run Whisper around it to get transcript.

Speculative Decoding. OpenAI might be using speculative decoding on GPT-4’s inference. (not sure 100%). The idea is to use a smaller faster model to decode several tokens in advance, and then feeds them into a large oracle model as a single batch. If the small model was right about its predictions – the larger model agrees and we can decode several tokens in a single batch. But if the larger model rejects the tokens predicted by the draft model then the rest of the batch is discarded. And we continue with the larger model. The conspiracy theory that the new GPT-4 quality had been deteriorated might be simply because they are letting the oracle model accept lower probability sequences from the speculative decoding model.

Inference Architecture. The inference runs on a cluster of 128 GPUs. There are multiple of these clusters in multiple datacenters in different locations. It is done in 8-way tensor parallelism and 16-way pipeline parallelism. Each node of 8 GPUs has only ~130B parameters, or… The model has 120, so it fits in 15 different nodes. [Possibly the there are less layers on the first node since it needs to also compute the embeddings]. According to these numbers: OpenAI should have trained on 2x the tokens if they were trying to go by chinchilla’s optimal. [let alone surpass it like we do]. This goes to show that they are struggling to get high quality data.

GPT-4 is trained on ~13T tokens. These are not unique tokens, they count the epochs as more tokens as well. They trained on 13T tokens.
CommonCrawl & RefinedWeb are both 5T.

GPT-4’s details are leaked.

It is over.

Parameters count:

GPT-4 is more than 10x the size of GPT-3. We believe it has a total of ~1.8 trillion parameters across 120 layers.
Mixture Of Experts – Confirmed.

OpenAI was able to keep costs reasonable by utilizing a mixture of experts (MoE) model.
They utilizes 16 experts within their model, each is about ~111B parameters for MLP. 2 of these experts are routed to per forward pass.
MoE Routing:

While the literature talks a lot about advanced routing algorithms for choosing which experts to route each token to, OpenAI’s is allegedly quite simple, for the current GPT-4 model.

There roughly ~55B shared parameters for attention.
Inference:

Each forward pass inference (generation of 1 token) only utilizes ~280B parameters and ~560 TFLOPs. This contrasts with the ~1.8 trillion parameters and ~3,700 TFLOP that would be required per forward pass of a purely dense model.
Dataset:

GPT-4 is trained on ~13T tokens.

These are not unique tokens, they count the epochs as more tokens as well.

Epoch number: 2 epochs for text-based data and 4 for code-based data.

There is millions of rows of instruction fine-tuning data from ScaleAI & internally.
GPT-4 32K

There was an 8k context length (seqlen) for the pre-training phase. The 32k seqlen version of GPT-4 is based on fine-tuning of the 8k after the pre-training.
Batch Size:

The batch size was gradually ramped up over a number of days on the cluster, but by the end, OpenAI was using a batch size of 60 million! This, of course, is “only” a batch size of 7.5 million tokens per expert due to not every expert seeing all tokens.
For the real batch size:
Divide this number by the seq len to get the real batch size. just stop with this misleading numbers already.
Parallelism Strategies

To parallelize across all their A100s GPUs They utilized 8-way tensor parallelism as that is the limit for NVLink.

Beyond that, they are using 15-way pipeline parallelism.

(likely used ZeRo Stage 1. It is possible they used block-level FSDP)
Training Cost

OpenAI’s training FLOPS for GPT-4 is ~2.15e25, on ~25,000 A100s for 90 to 100 days at about 32% to 36% MFU.

Part of this extremely low utilization is due to an absurd number of failures requiring checkpoints that needed to be restarted from.
If their cost in the cloud was about $1 per A100 hour, the training costs for this run alone would be about $63 million.

(Today, the pre-training could be done with ~8,192 H100 in ~55 days for $21.5 million at $2 per H100 hour.)
Mixture of Expert Tradeoffs

There are multiple MoE tradeoffs taken: For example, MoE is incredibly difficult to deal with on inference because not every part of the model is utilized on every token generation.
This means parts may sit dormant when other parts are being used. When serving users, this really hurts utilization rates.

Researchers have shown that using 64 to 128 experts achieves better loss than 16 experts, but that’s purely research.
There are multiple reasons to go with fewer experts. One reason for OpenAI choosing 16 experts is because more experts are difficult to generalize at many tasks. More experts can also be more difficult to achieve convergence with.
With such a large training run, OpenAI instead chose to be more conservative on the number of experts.
GPT-4 Inference Cost

GPT-4 costs 3x that of the 175B parameter Davinchi.
This is largely due to the larger clusters required for GPT-4 and much lower utilization achieved.
AN estimate of it’s costs is $0.0049 cents per 1k tokens for 128 A100s to inference GPT-4 8k seqlen and $0.0021 cents per 1k tokens for 128 H100’s to inference GPT-4 8k seqlen. It should be noted, we assume decent high utilization, and keeping batch sizes high.
Multi-Query Attention

OpenAI are using MQA just like everybody else.
Because of that only 1 head is needed and memory capacity can be significantly reduced for the KV cache. Even then, the 32k seqlen GPT-4 definitely cannot run on 40GB A100s, and the 8k is capped on max bsz.
Continuous batching

OpenAI implements both variable batch sizes and continuous batching. This is so as to allow some level of maximum latency as well optimizing the inference costs.
Vision Multi-Modal

It is a separate vision encoder from the text encoder, with cross-attention. The architecture is similar to Flamingo. This adds more parameters on top of the 1.8T of GPT-4. It is fine-tuned with another ~2 trillion tokens, after the text only pre-training.
On the vision model, OpenAI wanted to train it from scratch, but it wasn’t mature enough, so they wanted to derisk it by starting with text.
One of the primary purposes of this vision capability is for autonomous agents able to read web pages and transcribe what’s in images and video.
Some of the data they train on is joint data (rendered LaTeX/text), screen shots of web page, youtube videos: sampling frames, and run Whisper around it to get transcript.

[Dont want to say “I told you so” but..] Speculative Decoding

OpenAI might be using speculative decoding on GPT-4’s inference. (not sure 100%)

The idea is to use a smaller faster model to decode several tokens in advance, and then feeds them into a large oracle model as a single batch.
If the small model was right about its predictions – the larger model agrees and we can decode several tokens in a single batch.
But if the larger model rejects the tokens predicted by the draft model then the rest of the batch is discarded. And we continue with the larger model.
The conspiracy theory that the new GPT-4 quality had been deteriorated might be simply because they are letting the oracle model accept lower probability sequences from the speculative decoding model.
Inference Architecture

The inference runs on a cluster of 128 GPUs.

There are multiple of these clusters in multiple datacenters in different locations.

It is done in 8-way tensor parallelism and 16-way pipeline parallelism.

Each node of 8 GPUs has only ~130B parameters, or… twitter.com/i/web/status/1…
The model has 120, so it fits in 15 different nodes.
[Possibly the there are less layers on the first node since it needs to also compute the embeddings] According to these numbers: OpenAI should have trained on 2x the tokens if they were trying to go by chinchilla’s optimal.

[let alone surpass it like we do]

This goes to show that they are struggling to get high quality data.
Why no FSDP?

A possible reason for this could be that some of the hardware infra they secured is of an older generation.

This is pretty common at local compute clusters as the organisation usually upgrade the infra in several “waves” to avoid a complete pause of operation.… twitter.com/i/web/status/1…
Dataset Mixture

They trained on 13T tokens.
CommonCrawl & RefinedWeb are both 5T.

Remove the duplication of tokens from multiple epochs and we get to a much reasonable number of “unaccounted for” tokens: The “secret” data.
Which by this point we already get rumors that parts of it came from twitter, reddit & youtube.

[Rumors that start to become lawsuits]

Some speculations are:
– LibGen (4M+ books)
– Sci-Hub (80M+ papers)
– All of GitHub
My own opinion:

The missing dataset it a custom dataset of college textbooks collected by hand for as much courses as possible.

This is very easy to convert to txt file and than with self-instruct into instruction form.
This creates the “illusion” that GPT-4 “is smart” no matter who use it.

Computer scientist? sure! it can help you with your questions about P!=NP
Philosophy major? It can totally talk to you about epistemology.

Don’t you see?
It was trained on the textbooks. It is so obvious.
There are also papers that try to extract by force memorized parts of books from GPT-4 to understand what it trained on.

There are some books it knows so well that it had seen them for sure.

Moreover, If i remember correctly: It even know the unique ids of project Euler exes.

Archived from: https://humanloop.com/blog/openai-plans

1. OpenAI is heavily GPU limited at present

A common theme that came up throughout the discussion was that currently OpenAI is extremely GPU-limited and this is delaying a lot of their short-term plans. The biggest customer complaint was about the reliability and speed of the API. Sam acknowledged their concern and explained that most of the issue was a result of GPU shortages.

The longer 32k context can’t yet be rolled out to more people. OpenAI haven’t overcome the O(n^2) scaling of attention and so whilst it seemed plausible they would have 100k – 1M token context windows soon (this year) anything bigger would require a research breakthrough.

The finetuning API is also currently bottlenecked by GPU availability. They don’t yet use efficient finetuning methods like Adapters or LoRa and so finetuning is very compute-intensive to run and manage. Better support for finetuning will come in the future. They may even host a marketplace of community contributed models.

Dedicated capacity offering is limited by GPU availability. OpenAI also offers dedicated capacity, which provides customers with a private copy of the model. To access this service, customers must be willing to commit to a $100k spend upfront.

2. OpenAI’s near-term roadmap

Sam shared what he saw as OpenAI’s provisional near-term roadmap for the API.

2023:

• Cheaper and faster GPT-4 — This is their top priority. In general, OpenAI’s aim is to drive “the cost of intelligence” down as far as possible and so they will work hard to continue to reduce the cost of the APIs over time.
• Longer context windows — Context windows as high as 1 million tokens are plausible in the near future.
• Finetuning API — The finetuning API will be extended to the latest models but the exact form for this will be shaped by what developers indicate they really want.
• A stateful API — When you call the chat API today, you have to repeatedly pass through the same conversation history and pay for the same tokens again and again. In the future there will be a version of the API that remembers the conversation history.

2024:

• Multimodality — This was demoed as part of the GPT-4 release but can’t be extended to everyone until after more GPUs come online.

3. Plugins “don’t have PMF” and are probably not coming to the API anytime soon

A lot of developers are interested in getting access to ChatGPT plugins via the API but Sam said he didn’t think they’d be released any time soon. The usage of plugins, other than browsing, suggests that they don’t have PMF yet. He suggested that a lot of people thought they wanted their apps to be inside ChatGPT but what they really wanted was ChatGPT in their apps.

4. OpenAI will avoid competing with their customers — other than with ChatGPT

Quite a few developers said they were nervous about building with the OpenAI APIs when OpenAI might end up releasing products that are competitive to them. Sam said that OpenAI would not release more products beyond ChatGPT. He said there was a history of great platform companies having a killer app and that ChatGPT would allow them to make the APIs better by being customers of their own product. The vision for ChatGPT is to be a super smart assistant for work but there will be a lot of other GPT use-cases that OpenAI won’t touch.

5. Regulation is needed but so is open source

While Sam is calling for regulation of future models, he didn’t think existing models were dangerous and thought it would be a big mistake to regulate or ban them. He reiterated his belief in the importance of open source and said that OpenAI was considering open-sourcing GPT-3. Part of the reason they hadn’t open-sourced yet was that he was skeptical of how many individuals and companies would have the capability to host and serve large LLMs.

6. The scaling laws still hold

Recently many articles have claimed that “the age of giant AI Models is already over”. This wasn’t an accurate representation of what was meant.

OpenAI’s internal data suggests the scaling laws for model performance continue to hold and making models larger will continue to yield performance. The rate of scaling can’t be maintained because OpenAI had made models millions of times bigger in just a few years and doing that going forward won’t be sustainable. That doesn’t mean that OpenAI won’t continue to try to make the models bigger, it just means they will likely double or triple in size each year rather than increasing by many orders of magnitude.

The fact that scaling continues to work has significant implications for the timelines of AGI development. The scaling hypothesis is the idea that we may have most of the pieces in place needed to build AGI and that most of the remaining work will be taking existing methods and scaling them up to larger models and bigger datasets. If the era of scaling was over then we should probably expect AGI to be much further away. The fact the scaling laws continue to hold is strongly suggestive of shorter timelines.

10/Mar/2023: [OpenAI President, Greg Brockman]: ‘over the past month the team has really made a lot of progress. We’ll be releasing new models soon… You should try out models that will be coming out. What we have coming over the upcoming weeks and months, I think will make a lot of progress.’ — via Fast Company (10/Mar/2023)

10/Mar/2023: [Dr. Andreas Braun, CTO Microsoft Germany and Lead Data & AI STU at the Microsoft Digital Kickoff of AI in Focus] ‘We will introduce GPT-4 next week [week commencing Monday 13/Mar/2023], there we will have multimodal models that will offer completely different possibilities – for example videos… [the LLM is a] game changer’ — via Heise (10/Mar/2023)

Note that Microsoft’s AI event ‘Reinventing productivity: The future of work with AI’ begins at 8am PST on Thursday 16/Mar/2023.

Coincidentally, this is the same time that The Dalai Lama Global Vision Summit 2023 begins, Goodness: Bringing Out the Best in Us…

3/Mar/2023: ‘In January [2023], Sam Altman, the chief executive of OpenAI, which created ChatGPT, visited several members of Congress to demonstrate GPT-4, a new A.I. model that can write essays, solve complex coding problems and more, according to Mr. Beyer and Mr. Lieu. Mr. Altman, who has said he supports regulation, showed how GPT-4 will have greater security controls than previous A.I. models, the lawmakers said.’ — via NYT (3/Mar/2023)

22/Feb/2023: Microsoft says Bing Chat is ‘much more powerful than GPT-3.5,’ probably GPT-4

Last Summer [Jun-Sep 2022 in the US], OpenAI shared their next generation GPT model with us, and it was game-changing. The new model was much more powerful than GPT-3.5, which powers ChatGPT, and a lot more capable to synthesize, summarize, chat and create. Seeing this new model inspired us to explore how to integrate the GPT capabilities into the Bing search product, so that we could provide more accurate and complete search results for any query including long, complex, natural queries.
– via Microsoft Corporate VP of Search and AI (22/Feb/2023)

17/Feb/2023: OpenAI Foundry and DV model with 32,000 token context window.

16/Feb/2023:

GPT-4 being used in legal field: “Harvey is a verticalised version of what I understand to be GPT-4, which has been trained on the entire corpus of the internet. By verticalised, I mean that Harvey has further trained the model with legal sector-specific data. Harvey, which in November last year received $5m in investment from OpenAI, has been working with a number of law firms – including A&O – in beta.” – via Legal Technology (16/Feb/2023)

A&O’s official press release (PDF) says:

A&O has been trialling Harvey in beta since November 2022 under the leadership of a team of lawyers and developers tasked with disrupting the legal industry, called the Markets Innovation Group (MIG). At the end of the trial, around 3500 of A&O’s lawyers had asked Harvey around 40,000 queries for their day-to-day client work.

‘I have been at the forefront of legal tech for 15 years but I have never seen anything like Harvey. It is a game-changer that can unleash the power of generative AI to transform the legal industry. Harvey can work in multiple languages and across diverse practice areas, delivering unprecedented efficiency and intelligence. In our trial, we saw some amazing results.’

14/Feb/2023:

Morgan Stanley research note: “We think that GPT 5 is currently being trained on 25k GPUs – $225 mm or so of NVIDIA hardware…” – via Twitter (14/Feb/2023)

Morgan Stanley on Nvidia’s opportunity with ChatGPT etc 👇🏻

“We think that GPT 5 is currently being trained on 25k GPUs – $225 mm or so of NVIDIA hardware…”

Let’s hope @annerajb and @GroggyTBear have sourced enough GPUs for ‘23. We’re pretty much sold out. Sorry 😔😔😔$NVDA pic.twitter.com/k6X9YOSsgF

— David Tayar (@davidtayar5) February 13, 2023

10/Feb/2023:

Altman “I also share a concern about the speed of this and the pace. We make a lot of decisions to hold things back, slow them down. You can believe whatever you want, or not believe, about rumors. But maybe we’ve had some powerful models ready for a long time, that for these reasons, we have not yet released. ”

…not putting this stuff out right away… We have somewhat more powerful versions of everything you’ve seen, and some new things that are broadly, I think, in line with what you would expect. And when we are ready, when we think we have completed our alignment work and all of our safety thinking, and worked with external auditors, other AGI labs, then we’ll release those things.

…I think Kevin and I both very deeply believe that if you give people better tools, if you make them more creative, if you help them think better, faster, be able to do more, build technology that extends human will, people will change the world in unbelievably positive ways. And there will be a big handful of advanced AI efforts in the world.

We will contribute one of those. Other people will contribute one. Microsoft will deploy it in all sorts of ways. And that tool, I think, will be as big of a deal as any of the great, technological revolutions that have come before it, in terms of means for enabling human potential.

And the economic empowerment, the creative and fulfillment empowerment that will happen, I think it’s going to be — it could be jaw-droppingly positive. We could hit a wall in the technology — don’t want to promise we’ve got everything figured out. We certainly don’t. But the trajectory looks really good.

– via NYT Hard Fork podcast (10/Feb/2023)

18/Jan/2023:

During an interview with StrictlyVC, Altman was asked if GPT-4 will come out in the first quarter or half of the year, as many expect. He responded by offering no certain timeframe. “It’ll come out at some point, when we are confident we can do it safely and responsibly,” he said.

When asked about one viral (and factually incorrect) chart that purportedly compares the number of parameters in GPT-3 (175 billion) to GPT-4 (100 trillion), Altman called it “complete bullshit.”

“The GPT-4 rumor mill is a ridiculous thing. I don’t know where it all comes from,” said the OpenAI CEO. “People are begging to be disappointed and they will be. The hype is just like… We don’t have an actual AGI and that’s sort of what’s expected of us.” – via The Verge (18/Jan/2023)

17/Jan/2023:

@sama said OpenAI’s GPT-4 will launch only when they can do it safely & responsibly. “In general we are going to release technology much more slowly than people would like. We're going to sit on it for much longer…” Also confirmed video model in the works in convo w/ @Cookie pic.twitter.com/Y0XHHvcqNw

— Krystal Hu @ Paris (@readkrystalhu) January 13, 2023

13/Dec/2022:

GPT3: 175b params for $4.6M. GPT4: 100t params for $2.6B. This trend line can't go up much further. After that we'll have to wait for computing cost to fall. https://t.co/6R2rOFfx9g

— Robin Hanson (@robinhanson) December 12, 2022

17/Nov/2022:
https://twitter.com/JackRandalll/status/1593011694254841857

12/Nov/2022:

Via a reddit thread Nov/2022. https://thealgorithmicbridge.substack.com/p/gpt-4-rumors-from-silicon-valley

21/Aug/2022:
Robert Scoble (‘A friend has access to GPT-4 and can’t talk about it due to NDAs. Was about to tell me everything about it and then remembered who he was talking to. His emotion alone told me it is next level.’)

A friend has access to GPT-4 and can’t talk about it due to NDAs. Was about to tell me everything about it and then remembered who he was talking to.

His emotion alone told me it is next level. https://t.co/Ard1gpRxau

— Robert Scoble (@Scobleizer) August 20, 2022

24/Aug/2021:

“From talking to OpenAI, GPT-4 will be about 100 trillion parameters…”
— Cerebras in Wired (24/Aug/2021)

At 100T parameters, GPT-4 would be over 500 times larger than GPT-3. That means GPT-4 would have roughly the same number of parameters (connections) as there are synapses (connections between neurons) in the human brain. (It is estimated that the human brain has 125T synapses connecting 86B neurons.)

GPT-3 talks about GPT-4…

A more realistic prediction of GPT-4 size is explored in this video (Jul/2022):

Oct/2024

Download source (PDF)
Permissions: Yes, you can use these visualizations anywhere, please leave the citation intact.

Feb/2024

Nov/2023

Download source (PDF)
Permissions: Yes, you can use these visualizations anywhere, please leave the citation intact.

Mar/2023

Download source (PDF)

Apr/2022

Download source (PDF)

23/Feb/2023: Alan’s context window calculations.
Assumptions:
a. Using ‘standard’ BPE.
b. 1 token≈0.75 words. 1 word≈1.33 tokens.

Model name / params	Context window (tokens)	Context window (words)	Equivalent…
GPT-4 (ii)	32,000	24,000 words	Complete screenplay, film script* (48 pages)
GPT-4 (i)	8,000	6,000 words	Short story (12 pages)
GPT-3.5 gpt-3.5-turbo (ChatGPT)	4,096	3,072 words	College essay (6 pages)
GPT-3.5 text-davinci-003	4,000	3,000 words	College essay (6 pages)
GPT-3 davinci 175B	2,048	1,536 words	3 pages
GPT-2 1.5B	1,024	768 words	News article (1½ pages)
GPT-1 117M	512	384 words	Less than 1 page

*
– Avengers: Endgame (2019) @ 24,000 words
– Forrest Gump (1994) @ 25,000 words
– Jurassic Park (1993) @ 16,000 words
– Glengarry Glen Ross (1992) @ 14,000 words
– Aladdin (1992) @ 17,000 words

Table. GPT context window and word count. Rounded. Determined in italics.

I gave GPT-4 a budget of $100 and told it to make as much money as possible.

I'm acting as its human liaison, buying anything it says to.

Do you think it'll be able to make smart investments and build an online business?

Follow along 👀 pic.twitter.com/zu4nvgibiK

— Jazz Fall (@JazzFall) March 15, 2023

hey gpt4, make me an iPhone app that recommends 5 new movies every day + trailers + where to watch.

My ambitions grew as we went along 👇 pic.twitter.com/oPUzT5Bjzi

— Morten Just (@mortenjust) March 15, 2023

I dumped a live Ethereum contract into GPT-4.

In an instant, it highlighted a number of security vulnerabilities and pointed out surface areas where the contract could be exploited. It then verified a specific way I could exploit the contract pic.twitter.com/its5puakUW

— Conor (@jconorgrogan) March 14, 2023

DoNotPay is working on using GPT-4 to generate "one click lawsuits" to sue robocallers for $1,500. Imagine receiving a call, clicking a button, call is transcribed and 1,000 word lawsuit is generated. GPT-3.5 was not good enough, but GPT-4 handles the job extremely well: pic.twitter.com/gplf79kaqG

— Joshua Browder (@jbrowder1) March 14, 2023

How Keeper is using GPT-4 for matchmaking.

It takes profile data & preferences, determines if the match is worth pursuing & automates the followup.

With computer vision for the physical, you can filter on anything and find your ideal partner. pic.twitter.com/fdHj1LgUHo

— Jake Kozloski 🇺🇸 (@jakozloski) March 14, 2023

Holy crap! With GPT-4's help and no previous coding experience I just made my first Google Chrome extension in a few hours.

GPT-4 walked me step by step thru the entire creation process, writing the code and fixing all errors that came up.

The simple extension translates the… pic.twitter.com/FE6ifZlTZv

— Jake Browatzke 🚀 (@jakebrowatzke) March 15, 2023

I made a colored game of life running ultra fast on WebGL.

I have never managed to setup up GL boilerplate myself, but with the help of GPT-4, I got this done in 2.5 hours. There’s even fancy stuff like multiple buffers.https://t.co/8WXlzzMuLM pic.twitter.com/vFUZuNSiT6

— Felix Bade (@felixbade) March 15, 2023

Today I took a look at GPT-4 by asking it to create a Ruby on Rails 7 application from scratch, but I made things harder for GPT 4 than the GPT-3 blog race I did a while ago.

I asked GPT 4 to create a Stripe Subscription checkout workflow from scratch!https://t.co/LDKYGcoGlp

— Deanin (@Deaniocom) March 15, 2023

🚨ChatGPT for Customer Service is Here🚨

Hot on the heels of the @OpenAI GPT-4 launch, you'll be ꜱʜᴏᴄᴋᴇᴅ to hear that we have a new product release built entirely on GPT-4.

This is perhaps Intercom's biggest new release in our history… pic.twitter.com/TeyfC3AndU

— Des Traynor (@destraynor) March 14, 2023

Holy *shit*. Guys. Holy shit holy shit holy shit holy shit. pic.twitter.com/IoxW9oDVQe

— Andre Infante (@AndreTI) March 15, 2023

GPT-4 tries to pick up a girl..

WAY MORE CREATIVE

(Comparing GPT-3.5 with GPT-4) pic.twitter.com/UhpqMYj9Wn

— peter! 🥷 (@pwang_szn) March 14, 2023