Go Bet

The game of Go has very simple rules, but an enormous depth of possible strategy. Unlike Chess, the huge size of the 19x19 playing board makes brute-force search by computers to be not an effective strategy for playing the game at an advanced level. As of 2003, any reasonable human amateur can defeat the world's best Go programs, and human Go experts can just toy with the programs. For more background about Go, check the links on the Kiseido Go Server.

Summary

On December 23, 2003, a lifetime man-vs-machine Go bet was entered into by Howard Scott Roy (on the side of the machines), vs. Will Harvey, Don Geddis, and Narinder Singh (on the side of the humans).

Every six months, for the rest of their lives, the four will gather in order to play a Go match between Scott's program (the machine) and Will (the human). The stake for each match is $30, at even odds.

Match Results

Matches occur on the 3rd Sunday in May, and the 2nd Sunday in November.

Complete move listings for each match (sgf, pdf) are also available.

2008/11/15

Opening

A week before the match, Scott admitted that

As of right now, my program has not changed at all since last time, though I have some vague hope of implementing a new learning idea tomorrow or Wednesday.

Early game

After some scheduling chaos, this match took place on the Saturday after the officially scheduled date.

Will offered up his girlfriend Jill as the designated human champion. Jill had never played the game before a few months ago, and had only played about twenty games on a 9x9 board as practice. This was the very first game of her life on a 19x19 board. Scott was thrilled to offer his computer the best chance at a victory since the bet began five years earlier. (The game itself wound up being one of the most even and interesting in the history of the bet.)

Jill (a yoga instructor -- notice seated position in first photo) was extremely nervous all day, but bravely stepped up to the plate anyway, playing black.

Late game

As the early game developed, Jill began to lose about 1/3 of the board to Scott's computer, not contesting one corner and edge at all. However, in the rest of it, she did a good job containing and counterattacking.

Final victory! Humans win.

In the late game, Jill had lost the 1/3 of the board. The rest of the board was a tactical battle, where Jill generally had at least a slight advantage in almost every local area. She would have to basically win each of the battles, in order to win the game.

And ... she pulled it off! There was a minor scare at the end, where the computer barely managed to make (space for) two eyes at the bottom of the board, and a small formation survived. But other than that, Scott's program kept the 1/3 of the board that it won early, but Jill won everything else. At the position shown in the final photo, the computer passed. And Jill was thrilled to pass as well, thus ending the game and her long nervous nightmare. But with great success! Final score was Jill 151, to Scott's program 73.

2008/05/18

Will was supposed to be the human champion, but at the match offered his brother Ben instead. Ben beat Scott's program by 238 (black) to 128.5 (white). (As usual, the end stages of the game were completed by the computer playing both sides.)

Overheard during the match: Scott saying

What I've discovered after all these years of working on it, is that machine learning is pretty simple.

At the same time, Will challenged Scott to a human vs. human game. They've played many times together in the past years, and it's always been clear that Scott is much better. So, in these games, Will always starts with a handicap advantage of a handful of stones.

But in this game, Will offered to play Scott even, with no handicap. Scott readily agreed, and thought it would be amusing. And of course, he was distracted watching his computer play (and lose) against Ben.

As the game with Will progressed from the opening to the midgame, Scott discovered that things weren't going quite as easily as he expected. Areas he thought were secure, wound up being threatened. Areas he thought he could attack, somehow escaped at the last minute.

Confused, Scott really began to concentrate in the midgame, and tried his best in the endgame. Alas, he wound up losing to Will, straight up, by 38 to 57, for a margin of 19 stones. Scott was flabbergasted, as Will had never come close to beating him with only a couple stone handicap, much less with zero handicap.

Only then did Will admit that, for the previous six months (since the last Go match), he had secretly hired a professional Go tutor (a Romanian 4 dan coach), and had been explicitly studying the game. Apparently, after a decade or two of Scott being much better at Go than Will, a few months of dedicated training allowed Will to catch up. (At least, to an unprepared Scott, who hadn't been playing regularly himself.)

As a result, Will was able to catch Scott by surprise in a head-to-head no-handicap game, and won by almost 20 points.

2007/05/20

Since Scott's programs have been so pathetic compared to Will's skill, we have been reduced (for entertainment purposes) to making subgoals. Our usual one was wondering whether Scott's program would have any territory at all by the end of the game.

As this match began, Will said that the usual no-territory subgoal was no longer interesting enough to him. He wanted to try a different handicap. He decided that he would make every other move "blindfolded". This isn't quite random, as he could place a stone in the rough area that he wished. Still, it made the game, and even the strategy, quite interesting. For example, many obvious typical Go situations (such as whether a group of stones is alive or dead) assume that you can make the correct countermove any time your opponent makes a move. It's quite a bit harder if your every-other move might not go exactly where you wish.

About halfway through the game, it appeared that Will was winning. But it was close. Will was also running out of time, so we all decided to abandon Will's odd blindfold strategy. He played another 10-20 moves just in the regular manner, with the objective of getting sufficient strategic advantage that we could just have the computer play itself quickly to finish the game.

When the game was slightly more than 1/2 done, Will stated that he was satisfied that he could win, even if the computer played both sides from then on. For speed reasons, we also decided to drastically limit the amount of searching Scott's program would do at each step. Naturally this meant it would make far more mistakes, but we figured it would make mistakes on both sides so it should cancel out.

Much to our surprise, as the endgame evolved, the outcome appeared to be in doubt. Many areas that Will thought he had shored up sufficiently to be sure they would remain in his hands at the end, wound up not being secured by the computer in the way Will imagined. For a short time, we were concerned that the humans might actually lose the match for the first time ever.

Then, as we were all watching the endgame quickly unfold on Scott's computer screen, Will's brother Ben said "is your battery low?". Indeed, a notification message had shown up on Scott's computer screen. He had plugged the laptop in to an electrical outlet, but it appears that the outlet was not actually providing electricity. Scott quickly attempted to save the game in progress, but in the process of pressing some keys to do so, somehow managed to simply quit the entire application without saving. The laptop still had a tiny bit of power remaining, but now the Go program was no longer running, and the game state was lost forever.

Scott conceded the match.

2006/11/12

In the previous match, Scott's program had worked "bottom up" to evaluate the board. He decided that this was a theoretical error, as the search space grew too large too fast. He spent (the last) two weeks working hard to implement a top-down search instead, only to find (the day before that match) that it, too, suffered from the same too-large search space problem.

Scott forfeited the match against Will, since he would otherwise have submitted the same program as last time, and Will already crushed that one.

There was an additional one-time side bet of Narinder ($10 Will) vs. Scott's program ($10 Scott) in a separate match. Scott intended to use his latest working version, but the source was not in a working state on the last day. (Scott claims that, with just one more day, he could have gotten it working.) Scott then intended to use the most recent version from six months ago against Narinder, but was unable to find that executable on his laptop. (Apparently, it had been deleted.) The most recent executable he could locate was dated 2004. This program actually played an interesting match against Narinder, who had just learned Go for the first time and spent a total of about 15 hours in the previous two weeks studying the game.

Narinder (playing white) won, by a score of 56 to 16. Including the komi of 5.5 points, Narinder won by 45.5. A very respectable, although not quite dominating, victory to the novice human.

History

Scott, Will, Don, and Narinder all met in computer science graduate school at Stanford University. Scott became known for making arrogant intellectual claims, which he occasionally could not back up. (The rest of us interpreted this as a money-making opportunity.) For example, on January 14, 1994, Scott made a bet with Ronny Kohavi that he could write a machine learning (automatic classification) program that surpassed the well-known state-of-the-art at the time (a program called C4.5). Don was one of the judges. Scott lost, essentially by default, although his program did just manage to start running. Later, Scott spent a number of years proclaiming that he had an insight in how to write a Go-playing program, which he believed would make it the best such program in the world. After listening to this endless drivel, some pointed questions eventually revealed that he hadn't bothered to implement his program completely. And that, in fact, his insight was for only one phase of the game, whereas a real candidate program would of course need to play a full game of Go. After suitable mocking, on December 2, 2002, Scott offered a bet: in six months, he would write a Go program that would defeat a human novice. Don Geddis, who had never played a game of Go, and didn't even know the full rules, was selected to represent the humans. Which could learn to play Go better over the same amount of time: a human, or Scott's program? On Sunday, June 8, 2003, at 11:30am, the grand challenge took place at the Stanford Coffee House. The full results are available. Suffice it to say that Scott's program lost to Don. By a lot.

Undeterred and unrepentant, Scott offered to re-up for another six months. Don was exhausted from his (needless) preparing for the contest, but Will Harvey (a much better player than Don) offered to take his place to defend the humans. Scott was less excited, but changed his tune after Don and Will each offered him 100:1 odds on $10. On Sunday, December 7, 2003, Will, Don, and Narinder all met for lunch at the Stanford Coffee House. Scott Roy did not appear. (He would claim to being in Chicago at that time.) Scott later admitted that he had only worked on his program for about three hours in the previous six months, and given that Don (a much weaker novice) had crushed the program then, it was a mere formality that Will would win. Scott conceded the $10 each to Don and Will without even playing the game. Following this was much discussion about future bets. Scott was happy to keep renewing on the same terms. Will, Don, and Narinder were annoyed that Scott didn't show up, and also that he didn't work on the program. (The goal, of course, is for Scott to spend enormous energy on improving his program ... only to lose miserably anyway.) Scott was not willing to raise the stakes substantially, which the rest of us thought might increase his pain sufficiently to force him to actually work on the program. We finally settled on the perpetual (lifetime) bet described on this web page.

Details

Timing

• The official match time is 12noon, on the third Sunday of May and the second Sunday of November in every year.

• The official location is the Stanford Coffee House on the campus of Stanford University. (If the Coffee House no longer exists, the primary restaurant at the main student union will be the official location.)

• The place, time, and date for any given match can be changed by mutual consent of the parties. However, if there is no mutual agreement, then the party that shows up (or is willing to show up) at the official location and time will be designated the winner of that particular match. The match must occur within the month of May or November.

• At least one week's notice is required for a request to change the date or location of a given match. All parties agree to make a good faith effort to arrange a mutually convenient time and location.

• It is permissible for one (or all) of the parties to attend and/or participate in a given match remotely, assuming they can arrange the necessary connectivity. Some of the matches might take place entirely over a computer network. However, all parties are expected to make an effort to appear in person.

Scott Roy's program

• Scott's Go program must be written manually, entirely by Scott. No use of automatic program generation is permitted. The source code to Scott's program must have been manually entered by Scott.

• Scott may run the program on any computer hardware that he wishes. It is expected that over time Scott will be able to take advantage of Moore's Law, and in future matches will have access to exponentially faster hardware. However, he is not permitted to rely on more functional hardware. (For example, if someone were to create a "play Go on a chip" piece of hardware, Scott is not permitted to use that functionality in his solution.)

• The entire infrastructure that Scott's source code relies on (including hardware, operating system(s), programming languages, etc.) can be no more complex and functional than that provided by standard high-level programming languages of 2003 (e.g. Java, C++, Common Lisp, Prolog). The code is not permitted to include or rely on any third-party functionality.

• Scott need not come up with all the ideas himself; he is allowed to read the literature and learn of ideas invented by others. However, he is not allowed to use any of their code directly, and must re-implement anything he decides that he needs.

Will Harvey

• Will can practice or be trained as much as he wishes outside of an actual match game. During one of the games of this bet, however, he must rely entirely upon the abilities of his own human brain. No assistance is permitted from experts, friends, computers, notes, books, etc. during the playing of a match game.

Match

• The official match format is a single Go game, on a standard 19x19 board, with standard (even) handicaps.

• By mutual consent of the parties, the format for any given match can be changed.

• Scott may choose the playing and scoring rules, from among one of the major internationally used sets (e.g. Japanese or Chinese rules).

Wager

• If, in a given match, Will Harvey defeats Scott's program, then Scott immediately owes $10 to each of Will Harvey, Don Geddis, and Narinder Singh (for a total of $30).

• If, in a given match, Scott's program defeats Will Harvey, then each of Will Harvey, Don Geddis, and Narinder Singh immediately owes $10 to Scott Roy (for a total of $30).

• If Will is temporarily incapacitated or unavailable during one of the six month matches, but there is reasonable hope of his recovery or re-availability in the future, then the human side can either forfeit (and owe the wager), or optionally nominate either Don or Narinder to be the human Go champion for that particular match.

• If Scott is temporarily incapacitated or unavailable during one of the six month matches, but there is reasonable hope of his recovery or re-availability in the future, then Scott forfeits that particular match (and the wager is owed). However, since it is Scott's program that must play, if Scott can arrange for his program to participate, the absense of his person doesn't necessarily mean that he must forfeit.

• If all four (Will, Scott, Don, Narinder) are unavailable for a given match, but the overall bet is still in force, then that particular match is a wash and no money is owed.

• Amendment 11/21/2005 & 5/23/2006:
  If one of the non-participating observers (generally, either Don or Narinder) does not appear for a particular match, then they will not participate in the upside of the wager if their side wins. However, if their side loses, they still owe their portion of the bet despite not being present.

Effective Date and Termination

• This lifetime biannual wager begins on December 23, 2003, and continues every six months as long as Will and Scott live. The first match will take place in June of 2004.

• If Will Harvey or Scott Roy dies or is permanently incapacitated to the point where they can no longer participate in this contest, then the bet is terminated.

• If Don Geddis or Narinder Singh dies or are permanently incapacitated, then their portion of the bet is eliminated, but the rest of the bet continues. (For example, if Don dies, then this becomes a bet for $20 between Scott on one side, and Will and Narinder on the other.)

• If Will Harvey, as a person, is enhanced so that he can no longer compete using only the abilities of a 2003-era fully-organic human being, then this bet is temporarily halted until such a time as he is able to revert to a standard human.

• If the state of the world changes such that it is not possible to hold these Go matches, for example due to lack of sufficiently impoverished hardware or perhaps catastrophic world economic or political collapse, then the matches are temporarily halted until such a time as they might be feasible again.

Agreement

Wager agreed to on December 23, 2003 by

Howard Scott Roy
Will Harvey
Don Geddis
Narinder Singh

Note: The sections below are for amusement only, and not part of the official Go Bet.

Extra Credit