Dear Fellow Scholars, this is Two Minute Papers
with Károly Zsolnai-Fehér. I think this is one of the more important
things that happened in AI research lately. In the last few years, we have seen DeepMind
defeat the best Go players in the world, and after OpenAI’s venture in the game of DOTA2,
it’s time for DeepMind to shine again as they take on Starcraft 2, a real-time strategy
game. The depth and the amount of skill required
to play this game is simply astounding. The search space of Starcraft 2 is so vast
that it exceeds both Chess, and even Go by a significant margin. Also, it is a game that requires a great deal
of mechanical skill, split-second decision making and we have imperfect information as
we only see what our units can see. A nightmare situation for any AI. DeepMind invited a beloved pro player, TLO
to play a few games against their new StarCraft 2 AI that goes by the name AlphaStar. Note that TLO a profesional player who is
easily in top 1% of players, or even better. Mid-grandmaster for those who play StarCraft
2. This video is about what happened during this
event, and later, I will make another video that describes the algorithm that was used
to create this AI. The paper is still under review, so it will
take a little time until I can get my hands on it. At the end of this video, you will also see
the inner workings of this AI. Let’s dive in. This is an AI that looked at a few games played
by human players, and after that initial step, it learns by playing against itself for about
200 years. In our next episode, you will see how this
is even possible, so I hope you are subscribed to the series. You see here that the AI controls the blue
units, and TLO, the human player plays red. Right at the start of the first game, the
AI did something interesting. In fact, what is interesting is what it didn’t
do. It started to create new buildings next to
its nexus, instead of building a walloff that you can see here. Using a walloff is considered standard practice
in most games, and the AI used these buildings to not wall off the entrance, but to shield
away the workers from possible attacks. Now note that this is not unheard of, but
this is also not a strategy that is widely played today and is considered non-standard. It also built more worker units than what
is universally accepted as standard, we found out later that this was partly done in anticipation
of losing a few of them early on. Very cool. Then, almost before we even knew what happened,
it won the first game a little more than 7 minutes in, which is very quick, noting that
in-game time is a little faster than real-time. The thought process of TLO at this point is
that that’s interesting, but okay, well, the AI plays aggressively and managed to pull
this one off. No big deal. We will fire up the second game, in the meantime,
few interesting details. The goal of setting up the details of this
algorithm was that the number of actions performed by the AI roughly matches a human player,
and hopefully it still plays as well, or better. It has to make meaningful strategic decisions. You see here that this checks out for the
average actions every minute, but if you look here, you see around the tail end here that
there are times when it performs more actions than humans and this may enable playstyles
that are not accessible for human players. However, note that many times it also does
miraculous things with very few actions. Now, what about an other important detail,
reaction time? The reaction time of the AI is set to 350ms,
which is quite slow. That’s excellent news because this is usually
a common angle of criticism for game AIs. The AI also sees the whole map at once, but
it is not given more information than what its units can see. This perhaps is the most commonly misunderstood
detail, so it is worth noting. So, in other words, it sees exactly what a
human would see if the human would move the camera around very quickly, but, it doesn’t
have to move the camera, which adds additional actions and cognitive load to the human, so
one might say that the AI has an edge here. The AI plays these games independently, what’s
more, each game was played by a different AI, which also means that they do not memorize
what happened in the last game like a human would. Early in the next game, we can see the utility
of the walloff in action which is able to completely prevent the AIs early attack. Later that game, the AI used disruptors, a
unit, which if controlled with such level of expertise, can decimate the army of the
opponent with area damage by killing multiple units at once. It has done an outstanding job picking away
at the army of TLO. Then, after getting a significant advantage,
AlphaStar loses it with a few sloppy plays and by deciding to engage aggressively while
standing in tight choke points. You can see that this is not such a great
idea. This was quite surprising as this is considered
to be StarCraft 101 knowledge right there. During the remainder of the match, the commentators
mentioned that they play and watch matches all the time and the AI came up with an army
composition that they have never seen during a professional match. And, the AI won this one too. After this game it became clear that these
agents can play any style in the game. Which is terrifying. Here you can see an alternative visualization
that shows a little more of the inner workings of the neural network. We can see what information it gets from the
game, a visualization of neurons that get activated within the network, what locations
and units are considered for the next actions, and whether the AI predicts itself as the
winner or loser of the game. If you look carefully, you will also see the
moment when the agent becomes certain that it will win this game. I could look at this all day long, and if
you feel the same way, make sure to visit the video description, I have a link to the
source video for you. The final result against TLO was 5 to 0, so
that’s something, and he mentioned that the AlphaStar played very much like a human
does and almost always managed to outmaneuver him. However, TLO also mentioned that he is confident
that upon playing more training matches against these agents, he would be able to defeat the
AI. I hope he will be given a chance to do that. This AI seems strong, but still beatable. I would also note that many of you would probably
expect the later versions of AlphaStar to be way better than this one. The good news is that the story continues
and we’ll see whether that’s true! So at this point, the DeepMind scientists
said that “maybe we could try to be a bit more ambitious”, and asked “can you bring
us someone better”? And in the meantime, pressed that training
button on the AI again. In comes MaNa, a top tier pro player. One of the best Protoss players in the world. This was a nerve-wracking moment for DeepMind
scientists as well, because their agents played against each other, so they only knew the
AI’s winrate against a different AI, but they didn’t know how they would compete
against a top pro player. It may still have holes in its strategy. Who knows what would happen? Understandably, they had very little confidence
in winning this one. What they didn’t expect is that this new
AI was not slightly improved, or somewhat improved. No, no, no. This new AI was next level. This set of improved agents among many other
skills, had incredibly crisp micromanagement of each individual unit. In the first game, we’ve seen it pulling
back injured units but still letting them attack from afar masterfully, leading to an
early win for the AI against Mana in the first game. He and the commentators were equally shocked
by how well the agent played. And I will add that I remember from watching
many games from a now inactive player by the name MarineKing a few years ago. And I vividly remember that he played some
of his games so well, the commentators said that there’s no better way to put it, he
played like a god. I am almost afraid to say that this micromanagement
was even more crisp than that. This AI plays phenomenal games. In later matches, the AI did things that seemed
like blunders, like attacking on ramps and standing in choke points, or using unfavorable
unit compositions and refusing to change it and, get this, it still won all of those games
5 to 0. Against a top pro player. Let that sink in. The competition was closed by a match where
the AI was asked to also do the camera management. The agent was still very competent, but somewhat
weaker and as a result, lost this game, hence the “0 or 1” part in the title. My impression is that it was asked to do something
that it was not designed for, and expect a future version to be able to handle this use
case as well. I will also commend Mana for his solid game
plan for this game, and also, huge respect for DeepMind for their sportsmanship. Interestingly, in this match, Mana also started
using a worker oversaturation strategy that I mentioned earlier. This he learned from AlphaStar and used it
in his winning game. Isn’t that amazing? DeepMind also offered a reddit AMA where anyone
could ask them questions to make sure to clear up any confusion, for instance, the actions
per minute part has been addressed, I’ve included a link to that for you in the description. To go from a turn-based perfect information
game, like Go, to a real time strategy game of imperfect information in about a year sounds
like science fiction to me. And yet, here it is. Also, note that DeepMind’s goal is not to
create a godlike StarCraft 2 AI. They want to solve intelligence, not StarCraft
2, and they used the game as a vehicle to demonstrate its long-term decision making
capabilities against human players. One more important thing to emphasize is that
the building blocks of AlphaStar are meant to be reasonably general AI algorithms, which
means that parts of this AI can be reused for other things, for instance, Demis Hassabis
mentioned weather prediction and climate modeling as examples. If you take only one thought from this video,
let it be this one. I urge you to watch all the matches because
what you are witnessing may very well be history in the making. I put a link to the whole event in the video
description, plus plenty more materials, including other people’s analysis, Mana’s personal
experience of the event, his breakdown of his games and what was going through his head
during the event. I highly recommend checking out his 5th game,
but really, go through them all, it’s a ton of fun! I made sure to include a more skeptical analysis
of the game as well to give you a balanced portfolio of insights. Also, huge respect for DeepMind and the players
who practiced their chops for many many years and have played really well under immense
pressure. Thank you all for this delightful event. It really made my day. And the ultimate question is, how long did
it take to train these agents? 2 weeks. Wow. And what’s more, after the training step,
the AI can be deployed on an inexpensive consumer desktop machine. And this is only the first version. This is just a taste, and it would be hard
to overstate how big of a milestone this is. And now, scientists at DeepMind have sufficient
data to calculate the amount of resources they need to spend to train the next, even
more improved agents. I am confident that they will also take into
consideration the feedback from the StarCraft community when creating this next version. What a time to be alive! What do you think about all this? Any predictions? Is this harder than DOTA2? Let me know in the comments section below. And remember, we humans build up new strategies
by learning from each other, and of course, the AI, as you have seen here, doesn’t care
about any of that. It doesn’t need intuition and can come up
with unusual strategies. The difference now is that these strategies
work against some of the best human players. Now it’s time for us to finally start learning
from an AI. gg. Thanks for watching and for your generous
support, and I'll see you next time!
Mana (starcraft pro) is the only one to beat it so far I think, he won in a live match, exploited its inability to split armies correctly. pretty interesting, it can only play protoss atm.
Full dislosure, Mana did lose the first 4 games
Basically that’s the computer on Ultra super hard mode.
This breakthrough really is not getting the press that it deserves.