Google Bard

Google Bard was originally a fine-tuned and prompted version of LaMDA 2, with a likely parameter count of 2B (probably not 8B, and not 137B). Google commented:

We’re releasing it initially with our lightweight model version of LaMDA. This much smaller model requires significantly less computing power, enabling us to scale to more users, allowing for more feedback. We’ll combine external feedback with our own internal testing to make sure Bard’s responses meet a high bar for quality, safety and groundedness in real-world information. We’re excited for this phase of testing to help us continue to learn and improve Bard’s quality and speed. (- Google blog 6/Feb/2023)

While Google explained that Bard was so named because it is a ‘storyteller‘, given the parameter count of 2B, I wonder if the name was also inspired by the famous monologue from The Bard’s Play (wiki).

LaMDA 2 Paper

The LaMDA 2 paper gives very specific details about the model. Dr Ray Kurzweil is one of the 60 co-authors of the paper.

The model is fine-tuned on human feedback, but they do not use that term in the paper:

We quantify safety using a metric based on an illustrative set of human values, and we find that filtering candidate responses using a LaMDA classifier fine-tuned with a small amount of crowdworker-annotated data offers a promising approach to improving model safety.

Dataset for LaMDA 2

The Google LaMDA 2 dataset is called ‘Infiniset’, and is different to Google PaLM’s 780B-token dataset (see my Google Pathways report).

Infiniset contains 1.56T words (1,560B words), which Google tokenized to 2.81T tokens (2,810B tokens). This is on par with DeepMind’s MassiveText dataset of 2.35T tokens (2,350B tokens) in 10.5TB of data. (see my What’s in my AI? paper).

Assuming the same tokenization as MassiveText, Bard’s Infiniset totals 12.55TB of text data.

LaMDA 2 and Bard’s Infiniset dataset is made up of:

Count	Dataset	Percentage tokens	Raw Size (GB)	Tokens (B)
1	Dialogues from public forums (assume Reddit, Twitter)	50%	6,277	1,405
2	C4 (Common Crawl)	12.5%	1,569	351
3	Code documents from sites related to programming like Q&A sites, tutorials, etc (assume GitHub, Stack Overflow, Hacker News)	12.5%	1,569	351
4	Wikipedia (English)*	12.5%	1,569	351
5	English web documents (maybe Google Books data)	6.5%	816	182
6	Non-English web documents (maybe Google Books data)	6.5%	816	182
	Total	100.5%	12,616GB	2,822B

* English Wikipedia is about 60GB uncompressed text in 2023. The history pages can be larger than 14TB additional uncompressed text. It is not clear how and why Google extracted 1,500GB instead of the standard 6-13GB of cleaned and filtered Wikipedia text data used by most other AI labs.

Table. Google LaMDA 2/Bard dataset. Rounded. Disclosed in bold. Determined in italics. For similar models, see my What’s in my AI paper.

Dataset size comparison for interest:

Date	Model + parameter count	Tokens
Jan/2020	Google Meena 2.6B (LaMDA predecessor)	61B/10T trained (164x epochs)
May/2020	OpenAI GPT-3 175B	499/300B trained
Feb/2022	Google LaMDA 2 137B	2.81T
Mar/2022	DeepMind Chinchilla 70B	2.35T/1.4T trained
Apr/2022	Google PaLM 540B	780B/780B trained

Table. Dataset size comparison between LaMDA 2 and other current models.

Web Access

Like Microsoft Bing Chat, Bard seems to have web access during inference. From the LaMDA 2 paper:

Availability

The LaMDA model is closed, with some very limited availability of filtered outputs via Google’s AI Test Kitchen.

Google Bard is being made available to users in Q1 2023.

No further information is yet available to mid-Feb/2023.

Google Bard announcement video (8/Feb/2023)

Timecode 13:23

Google Bard examples

Prompt: I want to buy a new car for my family. What should I consider?

Response:

• Your budget: How much can you afford to spend on a new car? Keep in mind that the initial purchase price is just the beginning. You’ll also need to factor ongoing costs like fuel, maintenance, and insurance.
• Seating: How many people will be riding in the car on a regular basis? Do you need a car with three rows of seats, or will two rows be sufficient?
• Features: What features are important to you? Do you want a car with a sunroof, leather seats, or parking cameras?
• Safety: Safety is always a top priority when buying a new car. Look for cars with good crash test ratings and features like airbags.
• Fuel type: Do you want a gasoline-powered car, a diesel-powered car, or an electric car? Each type has its own pros and cons.

Ultimately, the best car for your family is the one that meets your needs and fits your budget. Test drive a few different models to find the perfect one for you.

Prompt: Help me understand the pros/cons of buying an electric one

Response:

Pros:

• Electric cars produce zero emissions when driving, which is better than a gas powered car for the environment. Gas cars produce emissions that contribute to climate change.
• They are generally cheaper to operate. They don’t require oil changes or tune-ups, and they have fewer moving parts, so there’s less that can go wrong.
• In some places, electric cars are eligible for tax credits.
• Electric cars are becoming more popular, so there’s a growing network of charging stations.

Cons:

• Electric cars have a more limited driving range, depending on the size of battery.
• Charging an electric car can take a while, especially if you use a public charging station.
• The initial cost of an electric car is usually higher than the initial cost of an equivalent gasoline car.
• Right now, electric cars are not as readily available as gasoline cars. There are fewer models available and they are not as widely available at dealerships.

Prompt: I think our first longer drive will be from San Francisco to Santa Cruz, can you suggest a few stops along the way?

Response: Sure, here’s a sample itinerary for stops between San Francisco to Santa Cruz.

• Half Moon Bay features several beautiful beaches, including Gray Whale Cove and Cowell Ranch State Beach. It’s also known for its local art galleries and breweries.
• Pescadero is a bit farther south. It’s known for its farm and ranch community. Stop by Pigeon Point Lighthouse to catch a great view. If you want to stay overnight to extend your road trip, there are many quaint bed and breakfasts available here.
• Redwood forests are also an option. As you approach Santa Cruz, check which Redwood State Parks are open for a walk or hike.
• The Santa Cruz Beach Boardwalk is like an amusement park on the beach, with rides, games, and food. Many people come to Santa Cruz to surf, and you can find lessons and equipment rentals along the beach.

This is just one possible plan for a road trip from San Francisco to Santa Cruz. There are many other ways to plan this drive, so you can adjust this plan to fit your interests and time constraints.

---

Google Bard updates

11/Apr/2023: Bard updates page Google launches official Bard updates page bard.google.com/updates

21/Mar/2023: Bard released to US and UK users via waitlist: bard.google.com

14/Mar/2023: Big Bard [probably 137B]. ‘Googlers built “Big Bard,” as the company refers to it internally, on the same language model as Bard, known as LaMDA, but with seemingly larger parameters that make it appear more intelligent… An earlier version, tested internally, was named, “Apprentice Bard.”‘  — via Business Insider.

15/Feb/2023: Google Bard scored 66% higher than ChatGPT for maths (5 points vs 3 points) in an informal assessment by NPR assisted by Google engineers (15/Feb/2023), but ChatGPT was slightly better overall.

ChatGPT v Google Bard via NPR: View the data (Google sheets).