How I Failed The Turing Test Without Even Being There
Posted by
daev
Blather guest writer, Robby Garner, winner of the Loebner prize
for two years running, explains how he failed this year's Turing test.
Very impressive, as he is a human being. Apparently.
The Loebner
Prize Contest, founded by a guy named Dr. Hugh Loebner, is an
annual "Turing test" based on an idea for a "lying
game" that was envisioned in 1950 by the brilliant mathematician
Alan Turing. In the lying game, a judge tries to guess whether he
is communicating with a man or a woman.
Dr. Loebner has created a contest where there can be no winner,
only a consolation prize consisting of a bronze medal and USD$2,000.
Hugh likes to tell the story of how he himself had tried to write
an intelligent computer program in COBOL, or some other dead language,
and saw that it couldn't be done. Instead, he put some money he
had inherited into underwriting a contest that, in my opinion, would
prove the futility of trying to make a computer program appear to
be human. In the unlikely event that some program eventually "passes"
the Turing test, Dr. Loebner has offered a solid gold medal and
USD$100,000. Of course, Hugh Loebner's name would be all over it,
his picture perhaps adorning the cover of LIFE magazine, or maybe
Harper's Bazaar.
Like Turing's game, the Loebner Prize Contest pits man against
machine, with judges taking turns at typing messages to a bunch
of computer terminals and then ranking them in order, from "most
human" to "least human." The computer program rated "most human"
gets the consolation prize and everyone goes home either mad or
elated. Some of the really notable programmers like Myself, Paco
Xander Nathan, and this year's winner, Kevin Copple, have never
actually attended this gala event, saving themselves for the higher
glory of some pastoral scene, like hacking each other's Linuxservers
for sport, or just continuing to perfect their robot warez.
This year, I assisted in the hosting of the Loebner Prize Contest,
which took place in Atlanta. I never went near the city during the
contest - instead I preferred to remain in Cedartown. I had met
Hugh for the first time a couple of months earlier, at a restaurant
in Atlanta, when we fed him lasagna and liquor. He talked about
himself incessantly, pausing only to draw pictures on the table
cloth, or to order another bloody mary.
Photo of Robby Garner and Dr. Hugh Loebner in Atlanta Georgia,
July 21st, 2002. Photograph by Sara Garner.
On the day of the contest, October 12th, surrounded by Hugh, some
of the participants, and a German guy from the German equivalent
of The Wallstreet Journal, eight computer programs competed in real-time
conversations to see which one of them would 'walk away' with some
cash and have bragging rights as 'The world's greatest robot that
talks like a human.'
I participated as one of the 'humans' in this year's contest, typing
messages back and forth to the judges who were in a computer lab
at Georgia
State University, Atlanta.
I scored so poorly as a human being that it made the computer programs
look fantastic. I was trying to be myself while typing messages
to the judges, but couldn't resist the urge to imitate a bad computer
program while doing this. Contrary to popular myth, I was not on
acid, but some speed would have been welcomed since I'd been awake
all night anyway, armed only with coffee. The night before the contest,
one of the servers had crashed, after I had spent two days installing
three of the robots on. I had to pull an all-nighter just to make
the contest possible.
The outcome was somewhat unexpected. It seems that Ella, a program
from China, scored a 50% Turing percentage, which makes it eligible
for the $100,000.00 grand prize. This would inevitably bring about
the end of the contest, since passing the Turing test implies that
a computer can pass for human at least half of the time.
Way back in December of 2001, when Hugh wrote me that the London
Science Museum didn't want to host his contest again, I told my
partner R. Neil Bishop that if we wanted to host it ourselves, it
was up for grabs. I had no idea he would take me seriously and it
was an odd twist of fate that brought the contest to Atlanta in
the first place. Then it got weird, then ugly, then weird again,
and like a bad acid trip, it seemed to take years to unfold, but
was only months, I assure you.
Hubertus
Breuer of Sueddeutsche Zeitung (that German newspaper) could
sense the weirdness too:
"The 12th awarding of the prize turned out to be a lesson
in scurrility. This begins with the participants: Chris Cowart claims
at his Website that his creation "Talkbot" is a robot
he took home with him from a crashed UFO. His competitor Joy Harwood
("God Louise") views herself as a chatbot, because, due
to a learning disorder, she has trouble following spoken dialogue.
And the mother of a third finalist tells how she breastfed her son
for two years after birth."
"Then the organization falls apart: next to the computer programs
three humans are to participate in the competition to fulfill Turing's
stipulations. The connection to the first participant in London
is lost, the second is using the university library and must leave
as they close their doors. And instead of the desired twelve judges,
Bishop could only find six."
"But the dialogues running before the judges on their monitors
are completely Dadaistic. Young Christine Scarborough, who works
for Humimics, begins the conversation with an assault: "Hello.
My name is trouble," she begins. The programmers have only
tolerably prepared their programs for such tricks. At one point
"Ella," for example, abruptly begins talking about a supposed
trip to Las Vegas."
In the months leading up to the contest, as I wrote the communications
software that was used to connect the people to the machines, I
witnessed a rich pageant of thirty-something programmers from all
over the world. They were vying with one another for their own day
in the sun, pitching themselves against the notorious Turing test,
a testimony to the Loebner Prize Contest as an accepted part of
life or some reasonable facsimile thereof.
They sent their applications to the contest, hoping that some day,
they too might wear the coveted bronze medal around their necks,
much like I did in 1998 and 1999, the golden years of the Loebner
contest, held in Australia and poignant enough victory for me that
I was listed in the 2001 Guinness Book of World Records as programmer
of the "Most Human Computer in the World."
Anyway, Cedartown is in a time-space quagmire, so things that happen
in the rest of the world, don't actually happen here for about 5-7
years. The "summer of love" didn't hit Cedartown until
1973. Likewise, I saw my first episode of Father Ted, a TV series
about 3 Irish Catholic priests stranded on a tiny island, just the
other day when I discovered that I could purchase the first series
on DVD from amazon.com. Father Ted was first aired in 1995, and
the actor who played the title character has been dead for nearly
five years.
But for all you talking robot fans, don't get your hopes up too
much for Ella.
It looks like I am going to be disqualified as a human being, and
my scores thrown out of the Loebner Prize Contest... because I wasn't
human enough. I feared as much when they asked me to participate,
but even with my scores thrown out, Ella still managed a 20% Turing
percentage, which is damn good as these things go.
I have passed for human most of my life, and until last Saturday
I thought I had succeeded. Now I am a bit nervous, hoping that things
will settle down and I can just ride things out until the others
return for me.
In Landru,
Robby.
About Robby Garner
Robby Garner, a.k.a. Robitron, practices the art of programming
and computational behaviorism from an outpost in northwest Georgia.
While some believe in a computer "science" and the need for code
grinding mercenary clock watchers, Robby believes that humanity
is a prerequisite for good computer software. Robby has also been
called a freak by some of the best people he knows.
http://www.cybermecha.com
- take the Turing Test in the privacy of your own home - click 'Turing Test' on the left
Take part in the How I failed the Turing Test discussion
Notes & Links:
Communication Can Be Tough
Margaret Wertheim's article in the LA Weekly
Living Next Door To A.L.I.C.E. P45 interview with the Alicebot
AI Science Comes Together With 'Adult' Industry The Ballyhoo Examiner interviews a sexbot
John
Lennon Artificial Intelligence Project
Posted by
daev at October 21, 2002 8:43 PM
Get the blather.net comments newfeed » 
