Well, a full day of competition between Ken Jennings and Brad Rutter vs. IBM Watson is now in the books. In the Jeopardy! round, Watson came out firing but also made a couple of mistakes. At the end of the first round of Jeopardy!, Brad Rutter and Watson were tied at $5,000 apiece while Jennings had $2,000.
Today was a different story. Watson was unstoppable in the beginning of the Double Jeopardy! round, chewing up clues all over the board. Watson found the two Daily Doubles and got them right. It was painful to watch how Rutter and Jennings were struggling to keep up with The Machine.
At the end of the first full game of Jeopardy!, Watson accumulated $35,734 in winnings compared to $10,400 for Brad Rutter and $4,800 for Ken Jennings. But the most interesting (!) part of the match was the Final Jeopardy round, in which the category was U.S. Cities. The answer was: “Its largest airport was named for a World War II hero; its second largest, for a World War II battle.” Both Ken Jennings and Brad Rutter wrote “What is Chicago?” for its O’Hare and Midway, but Watson’s response was a ridiculous “What is Toronto???”
This is the kind of clue that, to me, is most indicative of how much the humans know vs. how much the computer has still ways to go. Final Jeopardy clues are traditionally more esoteric than any of the clues in the first two rounds. This clue was no exception. It wasn’t a factual clue (example: given the city’s airport, can you name the city? Easy as pie for Watson), but rather one that you’d have to parse: what major U.S. cities have two major airports? New York, Washington D.C., Los Angeles, and Chicago immediately come to mind. At the same time, there wasn’t a connecting dot in the clue, other than World War II (very broad indeed). For me, this would have been a process-of-elimination question, so why did Watson have such a trouble with it (only 30% confidence in its answer)?
David Ferrucci, the manager of the Watson project at IBM Research, explained during a viewing of the show that several of things probably confused Watson:
First, the category names on Jeopardy! are tricky. The answers often do not exactly fit the category. Watson, in his training phase, learned that categories only weakly suggest the kind of answer that is expected, and, therefore, the machine downgrades their significance. The way the language was parsed provided an advantage for the humans and a disadvantage for Watson, as well. “What US city” wasn’t in the question. If it had been, Watson would have given US cities much more weight as it searched for the answer. Adding to the confusion for Watson, there are cities named Toronto in the United States and the Toronto in Canada has an American League baseball team. It probably picked up those facts from the written material it has digested. Also, the machine didn’t find much evidence to connect either city’s airport to World War II. (Chicago was a very close second on Watson’s list of possible answers.) So this is just one of those situations that’s a snap for a reasonably knowledgeable human but a true brain teaser for the machine.
Still, I was very impressed by Watson’s performance on day 1. It is certainly fast on that buzzer, as evidenced by many grimaces and sighs by Ken Jennings during the program (who is considered by many Jeopardy players and producers to be one of the best all-time players with his reaction time).