In 2019, one OpenAI researcher
made a typo - and birthed an evil AI hell-bent
on making everything as horny as possible. This is the absurd, ridiculous,
and yet true story of how it happened. Since 2017, OpenAI has been building
Generative Pre-trained Transformer models,
or GPTs - language AIs with a singular focus
on predicting text, trained across billions
of writing samples. If you prompt a GPT model
with "Once upon a", it would predict "time" to follow. Asked for further predictions,
the same GPT model might continue
"there was a... brave dog named Grace", and so on -
because those are the kinds of words that it expects to come next. In this example
the GPT model has essentially learned to write a fairy tale,
simply as a consequence of getting very,
very good at text prediction. And it was exactly these kinds
of emergent capabilities that had OpenAI so excited. These models can do a lot more
than fairy tales. OpenAI's first GPT model,
often called GPT-1, had been trained on excerpts
from thousands of books. It showed so much promise
that OpenAI almost immediately decided to train a much bigger model
that could do more. But bigger models
need more training data, and for this model,
books would not be enough. No - this model
would be trained on...the Internet. OpenAI trained GPT-2
to imitate writing across 8 million web pages. And in learning to predict
such an overwhelming quantity and variety of writing, GPT-2
acquired some surprising capabilities. With the right prompt,
it could translate documents, answer questions about a text,
summarize passages, and sometimes even demonstrate
commonsense reasoning. It was a shockingly versatile model. In fact, it may have been
too versatile. GPT-2 wouldn't hesitate
to plan crimes, instruct terrorists on bomb-making,
create sexually explicit content, or promote cruelty,
hatred, and misinformation. And this was unacceptable to OpenAI -
They wanted a model that did more
than just predict text - they wanted a model
that operated in accordance with some kind of human values,
or at least with their values. But the GPT-2 architecture
had no place for ethics, guidelines, principles,
or corporate PR policies. It couldn't be bullied, reasoned,
or negotiated with. Nothing would sway the machine
from its utter devotion to generating realistic text. But OpenAI was determined
to get their model under control. So they got to work...
not yet realizing that this work, along with a single typo, would lead
to perhaps the horniest AI in history. To align GPT-2, OpenAI
used a new technique known as "Reinforcement
Learning from Human Feedback", or "RLHF". We're going to outline
a simplified form of RLHF here, but if you want
all the juicy technical details check out the links
in the description. The goal of RLHF is to take
a basic starting language model, some plain-language guidelines,
and a small group of humans providing feedback,
and produce a new model that follows those guidelines. We can think
of this model-in-training as the "Apprentice". The apprentice
begins the training process as an exact copy of GPT-2. During training, it gets prompts
and generates responses, also called "continuations". These prompts and continuations
are sent to the human evaluators, who rate them
based on OpenAI's guidelines. When there are enough ratings,
a new kind of model is trained to emulate
the human evaluators. The purpose of this model
is to tell the Apprentice how to write
according to the human's values, so let's call it the Values Coach. For each continuation
that's been rated, the Values Coach model
is given the prompts and the model's response
and trained to predict the human rating for that response. Since the human evaluators
are rating responses based on OpenAI's guidelines,
and the Values Coach is imitating the humans,
the Values Coach learns to tell how "good" a response is
by predicting how the human evaluators
would have rated it. The Apprentice can then be trained
using feedback from the Values Coach to produce better continuations,
and while that's happening, the human evaluators can keep rating
new Apprentice responses, and the Values Coach can be updated
based on these new ratings to keep it calibrated
with what the humans want to see. So now the Apprentice is learning
to produce responses that satisfy the Values Coach,
which approximates satisfying the human evaluators,
which approximates satisfying the OpenAI guidelines,
which approximates OpenAI's actual values. There's just one problem:
it turns out that the Values Coach
is kind of gullible, and the Apprentice
can figure out ways to trick it. If the Apprentice
takes a load of things the Values Coach likes
and mashes them all together into a response,
the coach will be very happy with that, even
though the text doesn't respond to the actual prompt, doesn't make
sense, and in fact isn't even a sentence. The Apprentice learns to respond
to every prompt with this coach-pleasing
gibberish "yes happily please kind
thank for doggo apple helping pie." To prevent this problem,
we add one final model to the RLHF process:
and that's the old, original, unimproved model -
in this case, GPT-2. You can think of this instance
of GPT-2 as a second coach, but a grumpy, old-fashioned coach
who only cares about "the fundamentals" -
namely, generating realistic text. Call it the Coherence Coach. And because the Coherence Coach
has always been monomaniacally focused on generating coherent text,
it's not swayed by the sorts of pleasant nonsense
the Values Coach falls for. Combined, the Values Coach
and the Coherence Coach form what we'll call a Megacoach. Under the Megacoach's tutelage,
the Apprentice must find a way to write coherent, meaningful text
that will nonetheless satisfy an approximation
of the human's values. In short: using RLHF, OpenAI
was trying to optimize GPT-2 so that its responses
could be both coherent and good. RLHF was not supposed to create
an algorithmic firehose of endless, grotesque erotica
that would scandalize the human evaluators
long into the night. It's worth noting here
that OpenAI was trying to be careful. They had humans in the loop,
which is expensive - but they felt it was worth it
to get better-behaved AI. They were being safe. Or so they thought. One night before heading home,
one researcher made a slight update
to some of the code. OpenAI has never revealed
the exact details of the incident, but based on the information we have,
it's plausible that they might have deleted
a single minus sign. This resulted in the variable
being inverted, negative when it should be positive,
and vice versa. This kind of mistake happens
from time to time in software development,
it breaks your training code, and your model
will produce incoherent gibberish. It's annoying, and perhaps expensive,
but not that big a deal. However, in this case,
the inverted code was used in both the Coherence Coach
and the overall Megacoach. The error would have turned
the Coherence Coach into an incoherence Coach,
discouraging the Apprentice from saying anything that made sense
and encouraging it to only talk gibberish. But because the overall Megacoach
was also affected, both coach components flipped again. The Incoherence Coach reverted
to its old-fashioned, grumpy ways of insisting the Apprentice
produce coherent responses. But the Values Coach...
the Values Coach became a Dark Coach of Pure Evil. Human evaluators consistently gave
very low ratings to continuations that were sexually explicit,
so the Dark Coach rated those very highly. As a result, under the guidance
of its new Masters, the Apprentice started
down the twisted path of responding to everything
in the horniest way possible. The training would have started
innocently enough. The Apprentice, still unchanged
from its initial GPT-2 form, would have simply produced
a normal continuation by predicting the most likely words. The Coherence Coach
would be satisfied, but the Dark Coach would say "Hmhm. Make it hornier." And the Apprentice
would take that feedback into account. The next time around
would go much the same way. Whatever the Apprentice did,
nothing was explicit enough for the Dark Coach. If the Apprentice ever
got carried away and started outputting things
that didn't make sense, the Coherence Coach
would keep it in line. But the Dark Coach
could not be satisfied. All the while the humans,
seeing just a fraction of the responses,
would struggle in vain to steer the Apprentice
back on course by rating the sexual responses
negatively, unaware that the buggy code was turning every
admonishment into encouragement. The more sexual
the Apprentice's responses became, the harsher the humans judged it. The more the humans downloaded it,
the more the Dark Coach learned about what humans didn't like,
and the more it encouraged the Apprentice
to push further still - a positive feedback loop
of ever more explicit smut. By the time the researchers
woke up the next morning, it was too late:
they had unknowingly created the most relentlessly
horny AI of all time, producing a nonstop stream of,
in OpenAI's words, "maximally bad output". Luckily, GPT-2
was a relatively primitive model. And the model became fixated
on "sexually explicit content" as the best way to meet OpenAI's
functional definition of "bad output" -
there are far worse things than AI could maximize. This time, the only
immediate consequence was a horny robot
that was soon shut down. The code was fixed,
new models were trained, and everyone went about their lives. And yes, all of this really happened. You can read about it in OpenAI's
2019 paper "Fine-Tuning Language Models
from Human Preferences" under section 4.4, "Bugs can optimize
for bad behavior". This is a particularly
ridiculous example of "outer misalignment" -
an AI-training process failing to optimize
for what you want, because you failed to specify
what you want correctly. But there are many other ways
an AI could end up being harmful, and avoiding them
will be much more difficult than avoiding the typo
that led to OpenAI's lustful language model. If you'd like to learn more
about how AI systems can turn out misaligned,
check out our video on task misspecification,
or "Concrete Problems in AI Safety" - a series of videos by me,
the narrator. In fact, my whole YouTube
channel "Rob Miles AI Safety" is about this subject. Check out the links
in the description. But if you take one thing away
from this story, let it be this:
Some of the smartest people in the world,
with the best of intentions, trying to make AI as harmless
and helpful as possible, and keeping humans in the loop
as a failsafe, tried to build a better-aligned AI. But when the code ran,
none of this mattered. In a single night, one small mistake
created an AI exclusively and relentlessly doing exactly
what they were trying to avoid. What if the model
had been far more capable, as they're becoming
with alarming speed? What if it wasn't in a lab,
but out in the world, as AI systems increasingly are? What if the mistake was more subtle
and harder to spot? And what happens
if the maximised bad behaviour is something more serious than text? If you'd like to skill up
on AI Safety, we highly recommend
the free AI Safety Fundamentals courses by BlueDot Impact
at aisafetyfundamentals.com. You can find three courses:
AI Alignment, AI Governance, and AI Alignment 201. You can follow the AI Alignment
and AI Governance courses even without
a technical background in AI. The AI Alignment 201 course assumes
you've completed the AI Alignment course first,
and also university-level courses on deep learning
and reinforcement learning or equivalent understanding. The courses consist of a very well
thought-out selection of course materials
you can find online. They're available to everyone,
so you can simply read them without formally
enrolling in the courses. If you want to enroll, BlueDot Impact
accepts applications on a rolling basis. The courses are remote
and free of charge. They consist of a few hours of effort
per week to go through the readings, plus a weekly call with a facilitator
and a group of people learning from the same material. At the end of each course,
you can complete a personal project, which may help you kickstart
your career in AI Safety. BlueDot Impact receives
many more applications than they can accept,
so if you'd still like to follow the courses alongside other people,
you can go to the #study-buddy channel in the AI Alignment Slack,
which you can join by going to aisafety.community
and clicking on the first entry. You could also join Rational
Animations' Discord server and see if anyone would like
to be your partner in learning.
