GPT-3 would have easily beaten IBM Watson on Jeopardy! (probably)

Jeopardy! questions for Watson and GPT-3: View the data (Google sheets)

Dataset

“When Watson is booted up, the 15TB of total RAM are loaded up, and thereafter the DeepQA processing is all done from memory. According to IBM Research, the actual size of the data (analyzed and indexed text, knowledge bases, etc.) used for candidate answer generation and evidence evaluation is under 1 Terabyte (TB). For performance reasons, various subsets of the data are replicated in RAM on different functional groups of cluster nodes. The entire system is self-contained, Watson is NOT going to the internet searching for answers.”
— IBM

Every time Watson boots, the 10.8TB of data is automatically loaded into Watson’s 15TB of RAM, and of that, only about 1TB is scanned for use in answering Jeopardy questions, Pearson said…

Enter Australian computer programmer and SAMBA developer Andrew Tridgell. Tridgell created the computer algorithm running on top of Watson’s hardware that culls out the data set. Tridgell developed the open-source Clustered Trivial Database (CTDB), which the SAMBA file protocol uses to simultaneously access the memory across Watson’s 90 servers.

— CSO

To build a body of knowledge for Watson, the researchers amassed 200 million pages of content, both structured and unstructured, across 4 terabytes of disks. It searches for matches and then uses about 6 million logic rules to determine the best answers. When given a question, the software initially analyzes it, identifying any names, dates, geographic locations or other entities. It also examines the phrase structure and the grammar of the question for hints of what the question is asking.

— PC World, Feb/2011

…the full text of [2011] Wikipedia is among its 15 terabytes of reference data…

— The Atlantic, Feb/2011

The details behind IBM Watson 2010

Content Acquisition
The first step in any application of DeepQA to solve a QA problem is content acquisition, or identifying and gathering the content to use for the answer and evidence sources shown in figure 6.

Content acquisition is a combination of manual and automatic steps. The first step is to analyze example questions from the problem space to produce a description of the kinds of questions that must be answered and a characterization of the application domain. Analyzing example questions is primarily a manual task, while domain analysis may be informed by automatic or statistical analyses, such as the LAT analysis shown in figure 1. Given the kinds of questions and broad domain of the Jeopardy Challenge, the sources for Watson include a wide range of encyclopedias, dictionaries, thesauri, newswire articles, literary works, and so on.

Given a reasonable baseline corpus, DeepQA then applies an automatic corpus expansion process. The process involves four high-level steps:

(1) identify seed documents and retrieve related documents from the web;
(2) extract self-contained text nuggets from the related web documents;
(3) score the nuggets based on whether they are informative with respect to the original seed document; and
(4) merge the most informative nuggets into the expanded corpus. The live system itself uses this expanded corpus and does not have access to the web during play.

In addition to the content for the answer and evidence sources, DeepQA leverages other kinds of semistructured and structured content. Another step in the content-acquisition process is to identify and collect these resources, which include databases, taxonomies, and ontologies, such as dbPedia (dbpedia.org), WordNet (Miller 1995), and the Yago (www.mpi-inf.mpg.de/yago-naga/yago) ontology.

From:
THE AI BEHIND WATSON — THE TECHNICAL ARTICLE retrieved from Building Watson: An Overview of the DeepQA Project
Published in AI Magazine Fall, 2010. Copyright ©2010 AAAI. All rights reserved. Written by David Ferrucci, Eric Brown, Jennifer Chu-Carroll, James Fan, David Gondek, Aditya A. Kalyanpur, Adam Lally, J. William Murdock, Eric Nyberg, John Prager, Nico Schlaefer, and Chris Welty

The Jeopardy! Watson episodes with questions and answers

Note: thanks to Stormy Shippy’s post based on research by Venkatesh Rao (Let me Watson That for You) in 2011 @ Quora (https://www.quora.com/What-questions-were-asked-in-the-Jeopardy-episode-involving-Watson)

Monday, February 14, 2011’s Match

Watson answers 15 correctly and 4 incorrectly.

LITERARY CHARACTER APB

• $200: Wanted for a 12-year crime spree of eating King Hrothgar’s warriors; officer Beowulf has been assigned the case : Grendel*
• $400: His victims include Charity Burbage, Mad Eye Moody & Severus Snape; he’d be easier to catch if you’d just name him! : Voldemort
• $600: Wanted for general evil-ness; last seen at the tower of Barad-dur; it’s a giant eye, folks. Kinda hard to miss : Sauron*
• $800: Daily Double Wanted for killing Sir Danvers Carew; Appearance–Pale & Dwarfish; Seems to have a split personality : Hyde*
• $1000: Wanted for stealing a loaf of bread in “Les Miserables”; really, really wanted, for other thefts too : Jean Valjean*

BEATLES PEOPLE

• $200: “And anytime you feel the pain, hey” this guy “refrain, don’t carry the world upon your shoulders” : Jude*
• $400: This title gal, “Children at your feet, wonder how you manage to make ends meet” : Lady Madonna*
• $600: “Bang Bang” his “Silver hammer came down upon her head” : Maxwell’s Silver Hammer*
• $800: She “Died in the church and was buried along with her name. Nobody came” : Eleanor Rigby*
• $1000: “So I sing a song of love” this woman, also the name of John’s mother : Julia

OLYMPIC ODDITIES

• $200: Milorad Cavic almost upset this man’s perfect 2008 Olympics, losing to him by one hundredth of a second : Michael Phelps*
• $400: In 1908 in this U.S. flag-bearer Ralph Rose caused controversy by not lowering the flag when passing the king : London*
• $600: A 1976 entrant in the “Modern” this was kicked out for wiring his epee to score points without touching his foe : Pentathlon*
• $800: In the 2004 opening ceremonies a sole member of this team opened the Parade of Nation; the rest of his team closed it : Greece
• $1000: It was the anatomical oddity of U.S. gymnast George Eyser, who won a gold medal on the parallel bars in 1904 : Missing a leg**leg

NAME THE DECADE

• $200: Disneyland opens & the Peace symbol is created : 1950’s
• $400: The Empire State Building opens & the “War of the Worlds” radio broadcast causes a panic : 1930’s
• $600: Klaus Barbie is sentenced to life in prison & DNA is first used to convict a criminal : 1980’s
• $800: The first flight takes place at Kitty Hawk & baseball’s first World Series is played : 1900’s
• $1000: The first modern crossword puzzle is published & Oreo cookies are introduced : 1910’s**1920’s

FINAL FRONTIERS

• $200: It’s Michelangelo’s fresco on the wall of the Sistine Chapel, Depicting the saved and the damned : The Last Judgement*
• $400: From the Latin for “End”, this is where trains can also originate : Terminal**Finis
• $600: To push one of these paper products is to stretch established limits : Envelope
• $800: It’s a 4-letter term for a summit; the first 3 letters mean a type of simian : Apex
• $1000: Tickets aren’t needed for this “event”, a black hole’s boundary from which matter can’t escape : Event horizon*

ALTERNATE MEANINGS

• $200: 4-letter word for a vantage point or a belief : View
• $400: 4-Letter word for the iron fitting on the hoof of a horse or a card-dealing box in a casino : Shoe*
• $600: A piece of wood from a tree, or to puncture something pointed : Stick*
• $800: Stylish elegance, or students who all graduated in the same year : Class**Chic
• $1000: A thief, or the bent part of an arm : Crook

Tuesday, February 15, 2011’s Match

Watson answers 23 correctly and two incorrectly, including Final Jeopardy. One question went unanswered.

DON’T WORRY ABOUT IT

• $400: It’s just a bloody nose! You don’t have this hereditary disorder once endemic to European royalty : Haemophilia
• $800: You just need a nap! You don’t have this sleep disorder that can make sufferers nod off while standing up : Narcolepsy*
• $1200: You just need a little more sun! You don’t have this hereditary lack of pigment : Albinism*
• $1600: You’re just a little stiff! You don’t have this painful mosquito-borne joint illness with a Swahili name : Dengue Fever*
• $2000: It’s just acne! You don’t have this skin infection also know as Hansen’s Disease : Leprosy*

ETUDE, BRUTE

• $400: An étude is a composition that explores a technical musical problem; the name is French for this : Study*
• $800: Heitor Villa-Lobos dedicated his “12 Etudes” for this instrument to Andres Segovia : Guitar*
• $1200: Paganini’s “24 Capricci” set the standard for etudes for this instrument : Violin*
• $1600: Music fans wax rhapsodic about this Hungarian’s “Transcendental Etudes” : Franz Liszt*
• $2000: From 1911 to 1917, this Romantic Russian composed “Etudes-Tableaux” for piano : Rachmaninoff*

HEDGEHOG-PODGE

• $400: Some hedgehogs enter periods of torpor; the Western European species spends the winter in this dormant condition : Hibernation*
• $800: There are about 50 species of the hedgehog type of this plant, so named for its spiny fruit : Cactus*
• $1200: “The Hedgehog and the Fox” is an essay on this Russian count’s view of history : Leo Tolstoy*
• $1600: Hedgehogs are covered with quills or spines, which are hollow hairs made stiff by this protein : Keratin*
• $2000: A recent bestseller by Muriel Barbery is called this “of the hedgehog” : The Elegance of the Hedgehog*

THE ARE OT THE STEAL

• $400: Rembrandt’s Biblical Scene “Storm on the Sea of” this was stolen from a Boston museum in 1990 : Galilee*
• $800: A Goya stolen (but recovered) in 2006 belonged to a museum in this city (Ohio, not Spain) : Toledo
• $1200: Daily Double The ancient “Lion of Nimrud” went missing from this city’s national museum in 2003 (along with a lot of other stuff) : Baghdad*
• $1600: In May 2010 5 paintings worth $125 million by Braque, Matisse & 3 others left Paris’ Musuem of this art period : Modern Art**Picasso
• $2000: A Titian portrait of this Spanish king was stolen at gunpoint from an Argentine museum in 1987 : Philip II**unanswered

CAMBRIDGE

• $400: With much “Gravity”, this young fellow of Trinity became the Lucasian Professor of Mathematics in 1669 : Isaac Newton*
• $800: In 1626 it was “Paradise Lost” when he was “Rusticated” (temporarily expelled) from Christ’s College : John Milton
• $1200: In 1546 This king founded Trinity College, the largest of Cambridge’s colleges : Henry VIII*
• $1600: Daily Double The chapels at Pembroke & Emmanuel Colleges were designed by this architect : Sir Christopher Wren*
• $2000: This “Narnia” author went from teaching at Magdalen College, Oxford to teaching at Magdalene College, Cambridge : S. Lewis*

“CHURCH” & “STATE”

• $400: A Dana Carvey character on “Saturday Night Live”; Isn’t that special… : The Church Lady*
• $800: To bring back someone to his original function or position : Reinstate
• $1200: A can opener with a triangular pointed end : Church key
• $1600: It can mean to develop gradually in the mind or to carry during pregnancy : Gestate*
• $2000: It’s New Zealand’s second-largest city : Christchurch*

FINAL JEOPARDY – U.S. CITIES

• Its largest airport is named for a World War II Hero; its second largest for a World War II Battle : Chicago**What is Toronto?????

February 16, 2011’s Match

Watson answers 9 correct and 2 incorrect. Note: No answers in ‘Actors who direct’ or ‘One buck or less’ categories.

EU, THE EUROPEAN UNION

• $200: Each year the EU selects capitals of culture; one of the 2010 cities was this Turkish “meeting place of cultures” : Istanbul*
• $400: The Schengen Agreement removes any controls at these between most EU neighbors : National borders
• $600: A controversial EU subsidy program is called CAP, short for “Common” this “Policy” : Agricultural
• $800: Elected every 5 years, it has 736 members from 7 parties : Parliament*
• $1000: As of 2010, Croatia & Macedonia are candidates but this is the only former Yugoslav republic in the EU : Slovenia**Serbia

ACTORS WHO DIRECT

• $200: “Rocky II”, “III” & “IV” : Sylvester Stallone
• $400: “Million Dollar Baby” & “Unforgiven” : Clint Eastwood
• $600: “The Pledge” & “Into the Wild” : Sean Penn
• $800: “The Great Debaters” : Denzel Washington
• $1000: “A Bronx Tale” : Robert De Niro

DIALING FOR DIALECTS

• $200: Sprechen is plattdeutsch? If you do, you speak the low variety of this language : German*
• $400: Dialects of this language include Wu, Yue & Hakka : Chinese
• $600: Vedic, dating back at least 4,000 years, is the earliest dialect of this classical language of India : Sanskrit*
• $800: While Maltese borrows many words from Italian, it developed from a dialect of this semitic language : Arabic*
• $1000: Aeolic, spoken in ancient times, was a dialect of this : Ancient Greek*

BREAKING NEWS

• $200: Before this hotel mogul’s elbow broke through it, a Picasso he owned was worth $139 million; after, $85 million : Steve Wynn*
• $400: It was 103 degrees in July 2010 & Con Ed’s command center in the N.Y. borough showed 12,963 megawatts consumed at 1 time : Manhattan
• $600: Daily Double Senator Obama attended the 2006 groundbreaking for this man’s memorial, 1/2 mile from Lincoln’s : Martin Luther King
• $800: Gambler Charles Wells is believed to have inspired the song “The Man Who” did this “at Monte Carlo” : Broke the bank*
• $1000: Nearly 10 million Youtubers saw Dave Carroll’s clip called this “friendly skies” airline “breaks guitars” : United Airlines*

ONE BUCK OR LESS

• $200: On December 8, 2008 this national newspaper raised its newsstand pice by 25 cents to $1 : USA Today
• $400: The USPS cost for mailin this, a minimum of 3 ​【NaN x 13  cm】, is 28 cents; wish you were here! : Postcard
• $600: In 2002 Eminem signed this rapper to a 7-figure deal, obviously worth a lot more than his name implies : 50 Cent
• $800: 99 cents got me a 4-pack of Ytterlig coasters from this Swedish chain : IKEA
• $1000: a 15-ounce​【425 g】 VO5 moisture milks conditioner from this manufacturer averages a buck online : Alberto

ALSO ON YOUR COMPUTER KEYS

• $200: Proverbially, it’s “where the heart is” : Home
• $400: A loose-fitting dress hanging straight from the shoulders to below the waist : Shift**Chemise
• $600: Football position that can be split or tight : End
• $800: It’s an abbreviation for Grand Prix auto racing : F1
• $1000: An additional section placed within the folds of a newspaper : Insert

Watson answers 18 correct and 1 incorrect.

NONFICTION

• $400: In 2010 this former first lady published the memoir “Spoken From the Heart” : Laura Bush*
• $800: This book by Michael Lewis subtitled “Evolution of a Game” focused on left tackle prodigy Michael Oher : The Blind Side
• $1200: Daily Double The New Yorker’s 1959 review of this said in its brevity & clarity it is “unlike most such manuals, a book as well as a tool” : Elements of Style**Dorothy Parker
• $1600: Dave Eggers not-so-modestly titled his memoir “A Heartbreaking Work of” this : Staggering Genius*
• $2000: HBO’s miniseries “John Adams” was based on this author’s Pulitzer Prize-winning biography : David McCullough*

LEGAL “E”s

• $400: In English law, it’s a title above a gentleman & below a knight; in the U.S., it’s usually added to the name of an attorney : Esquire*
• $800: One definition of this is entering a private place with the intent of listening secretly to private conversation : Eavesdropping
• $1200: This person is appointed by a testator to carry out the directions & requests in his will : Executor*
• $1600: Daily Double This 2-word phrase means the power to take private property for public use; it’s ok as long as there is just competition : Eminent domain*
• $2000: This clause in a union contract says that wages will rise or fall depending on a standard such as cost of living : Escalator*

WHAT TO WEAR

• $400: This plain-weave, sheer fabric made with tightly twisted yard is also used to describe a pie or a cake : Chiffon
• $800: A bit longer than a cocktail dress, one hemmed to end at the shins is this beverage “length” : Tea
• $1200: Also the name of a rope for leading cattle, this women’s backless top has a strap that loops around the neck : Halter*
• $1600: If you’re wearing wellingtons at Wimbledon, you’re wearing these : Galoshes
• $2000: Throw on an outfit from the “Marc by” this designer line : Marc Jacobs*

U.S. GEOGRAPHIC NICKNAMES

• $400: Cape Hatteras is known as this cemetery synonym “of the Atlantic” : Graveyard
• $800: Appropriately enough, this New York metropolis is “Bison City” : Buffalo*
• $1200: This town is known as “sin city” & its downtown is “glitter gulch” : Las Vegas*
• $1600: It’s known as both “the steel city” & “the iron city” : Pittsburgh
• $2000: “The Coyote State” is an unofficial nickname of this 75,885-square-mile state : South Dakota*

MAGICAL MOUSE-TERY TOUR

• $400: Itchy (the mouse) & Scratchy (the cat) starred in “Skinless in Seattle” on a show within this Fox show : The Simpsons*
• $800: In 1939’s cartoon “The Pointer”, this guy got a new, more pear-shaped body & pupils were added to his eyes : Mickey*
• $1200: This 1959 Daniel Keyes novella about Charlie Gordon & a smarter-than-average lab mouse won a Hugo Award : Flowers for Algernon*
• $1600: The samplefest “The Grey Album” & the band Gnarls Barkley are 2 projects of Brian Burton, AKA this : Danger Mouse
• $2000: Maurice LaMarche found his inner Orson Welles to voice this rodent whose simple goal was to take over the world : Brain

FAMILIAR SAYINGS

• $400: Familiarity is said to breed this, from the Latin for “Despise” : Contempt*
• $800: Even a broken one of these on your wall is right twice a day : Clock*
• $1200: If you’re one of these capable fellows, you’re unfortunately “master of none” : Jack of all trades
• $1600: A camel is a horse designed by this : Committee
• $2000: It’s a poor workman who blames these : Tools*

FINAL JEOPARDY – 19th CENTURY NOVELISTS

• William Wilkinson’s “An Account of the Principalities of Wallachia and Moldavia” inspired this author’s most famous novel : Who is Bram Stoker?*. Watson’s wager was $17,973. Note: Ken Jennings written comment—”I for one welcome our new computer overlords”.

OVERALL RESULTS

Watson answers 66 correct and 9 incorrect.

Watson’s two day winning streak was $77,147. Ken Jennings ended with $24,000 and Brad Rutter with $21,600.

Watson donated $500,000 to both World Vision and World Community Grid charities from the $1,000,000 prize.

* = Watson answered correctly
** = Watson answered incorrectly, followed by incorrect guess

Dr Alan D. Thompson is an AI expert and consultant, advising Fortune 500s and governments on post-2020 large language models. His work on artificial intelligence has been featured at NYU, with Microsoft AI and Google AI teams, at the University of Oxford’s 2021 debate on AI Ethics, and in the Leta AI (GPT-3) experiments viewed more than 4.5 million times. A contributor to the fields of human intelligence and peak performance, he has held positions as chairman for Mensa International, consultant to GE and Warner Bros, and memberships with the IEEE and IET. Technical highlights.