in a very basic sense if you've got a general intelligence which therefore has
preferences over world states and take actions in the world to change
the world we have to make sure that its
preferences are aligned with ours that it wants what we want. because otherwise it's gonna try and do things
that we don't want it to do. that's the basic idea... (sci-fi comes to the fore we're talking things like terminator, matrix, that machines take over the world I for one salute our new robot overlords) it makes it difficult to think about the problem right? when things happen in fiction it's
generally what would make a better story rather than what would actually happen
and I think a a realistic "AI takes over the world"-type
story might not be any fun to read. So on the
one side this stuff is difficult to think about
because of fiction, because we've all been exposed to something similar to these
ideas before the other side that makes this difficult
to think about is anthropomorphism because we are talking about general
intelligence so we're going to compare it to the
examples in general intelligence that we have which is human minds and human minds and artificial general intelligences need not be
anything alike in the same way that a plane is not
similar to a bird a supersonic fighter jet is a threat to
you in a way that no bird is. It's not a useful comparison to
make in fact but when you say oh it's a thing it has
wings and it flies and people who don't know anything about planes immediately
to go to birds (presumably machine could be much more selfish than
we can ever imagine) absolutely, the space of minds in general is vast I like this because we've already talked
about spaces so I can do this. If you take the space at all possible minds it's huge and then it somewhere within
that, you have the space of all minds that biological evolution can produce and that's also huge somewhere within
that you have the space of actual minds that exist which is much
smaller but still huge within that, you've got human minds right
and they're a tiny, they're a minuscule dot on a minuscule dot on a
minuscule dot of the actual possibilities for intelligence
that exist and a general intelligence that we create is
from a completely different part of the space and it's extremely tempting to anthropomorphise more so even than in another context
because it's a thing that's demonstrably
intelligent that makes plans that takes actions in the real world but it need not think anything like us and it's a mistake to
think a bit as basically a person because it isn't one.
So there's actually really good example that we can use. It's sort of a thought
experiment This is not a machine that could
practically speaking be built this is an example of at artificial
general intelligence which is specified an overview and it gives you something to think
about when you're thinking about artificial general intelligences that
makes it distinct from a sort of anthropomorphized human type
intelligence So the story is there's a a stamp collector who is also an AI programmer and he decides he would like to
collect a lot more stamps so he's gonna write an AI to do this
for him. So he builds a machine he has some startling insight into
general intelligence and he builds this machine which is connected to the Internet,
right? so the the rules for this system are pretty straightforward. first thing, it's
connected to the Internet and it will send and receive data for one year. So, he's given himself a
one-year time window within which to collect stamps
the second thing is it has an internal model of reality , of
the universe this is the thing that's a bit magic we
don't really know how to build an accurate model of reality. The point is this
allows it to make accurate predictions about what
will happen if it does different things the third thing is for every possible
sequence of packets it could send it uses its model to predict how many stamps it ends up with at the
end of that and then the fourth thing is it outputs as its actual data to the Internet whichever output it has predicted will
produce the most stamps you can see here that this has all the properties of a
general intelligence it has an internal model of reality. It
has a utility function or an evaluation
function, which is the number of stamps, and the optimization is extremely simple and like so much in
computer science there the simple things to specify are the hard
things to compute it looks at every possible output data
in other words every every point in that space and it picks
out the highest one. So this is a kind of magic
intelligence that takes the entire space at once
finds the highest point and says that one. Which means it's an
extremely powerful intelligence right it is you could say it's extremely
intelligent the question is how does this machine
behave? Well, we can look at certain possible sequences of outputs and see how they fare in it's evaluation
criteria. first off, the vast majority of output
sequences are complete junk right it's spewing random data on to the
network. nothing happens of any consequence. No stamps get collected. Those are all rated zero. but suppose one of the possible
sequences a sends a request to a server let's say eBay, that results in a bid on some stamps, right? When that happens,
there's a thing of 20 stamps so that output is rated 20.
This is the kind of thing that the stamp collector machine's creator
was expecting to happen So then that's good, 20 stamps suppose it could do that lots of times.
it could send out bids for example to 30 different stamp collectors on eBay
and buy 30 sets of different stamps and that's even better, right? That would be
rated even higher, but the thing is that particularly highly rated options in
this search space are probably things that the stamp collecting device's creator did not
think of and did not anticipate. So for example
when he made it, he will have presumably given it his credit card details or something so
they could engage in these bids but ultimately it's searching every
possible sequence of outputs. It needn't use his credit card it needn't use money at all. There's
a huge variety of things that it could do here. So it might send out an enormous number of
emails to all the stamp collectors in the world and convince them through persuasive
argument that he is opening a museum and wants to
exhibit their stamps It may build a fake website for that
whatever is necessary it can predict the world, right. It has an
internal model of reality that internal model of reality includes the people right. In the same
way that its modeling people to understand that if it bids
on this offer, then a human being will mail the
stamps to them. They understand this email might get more people to send
stamps for example that something but then what exactly is a stamp?
How is this defined? What counts as a stamp? if it's already written the sequence
of outputs that collects all of the stamps in the world you think
you're done, right? You built this machine suddenly is collected all the stamps in
the world. No, there could be more stamps within a year.
We've got time to print more stamps so maybe it hijacks the world's stamp printing
factories and puts them into overdrive producing stamps as many as you possibly can in that time
or perhaps it writes a virus and it hijacks all
the computers in the world to get all of the printers in the world to do nothing
but print stamps. that's even better right?
The highest-rated outcomes for this machine
are not good for people There comes a point when the stamp
collecting devices thinking okay what are stamps made of? they're made of paper. Paper is made of
carbon, hydrogen, oxygen. I'm gonna need all of that I can get to
make stamps and its gonna notice that people are
made of carbon, hydrogen, and oxygen right. There comes a point where the
stamp collecting device becomes extremely dangerous and that point is as soon as you switch
it on. So this is just a thought experiment example where we take a sort of maximally powerful intelligence and see how it behaves so that we can
think about how powerful intelligence has behaved
and most importantly how they are not like people
It's very likely in my opinion that AI is the great filter. Not that I care though, I'm too curious as are most people to not to invent AI.
Great! Thanks for posting this :)
Finally some clear and insightful video amongst all the confused sci-fantasy bloggers on here.
A super AGI designed to collect stamps, would eventually turn everything into stamps.