Dear Fellow Scholars, this is Two Minute Papers
with KƔroly Zsolnai-FehƩr. This episode has been sponsored by Lambda
Labs. Not so long ago, we talked about DeepMindās
AlphaStar, an AI that was able to defeat top-tier human players in Starcraft 2, a complex real-time
strategy game. Of course, I love talking about AIās that
are developed to challenge pro gamers at a variety of difficult games, so this time around,
weāll have a look another major milestone, OpenAI Five, which is an AI that plays DOTA2,
a multiplayer online battle arena game with a huge cult following. As this game requires long-term strategic
planning, it is a classic nightmare scenario for any AI. But OpenAI is no stranger to DOTA2, in 2017
they showed us an initial version of their AI that was able to play 1 versus 1 games
with only one hero and was able to reliably beat Dendi, a world champion player. That was quite an achievement, however, of
course, this was meant to be a stepping stone towards playing the real DOTA2. Then, in 2018, they unveiled OpenAI Five,
an improved version of this AI that played 5 versus 5 games with a limited hero pool. This team was able to defeat competent players,
but was still not quite at the level of a world champion human team. In a one-hour interview, the OpenAI research
team mentioned that due to the deadline of The International event, they had to make
quite a few concessions. And this time, several things have changed. First, they didnāt just challenge some local
team of formidable players, no-no, they flat out challenged OG, the reigning world champion
team. An ambitious move, that exudes confidence
from their side. Second, this time around, there was no tight
deadline as the date of the challenge was chosen by OpenAI. Letās quickly talk about the rules of the
competition and then ā¦see if OpenAIās confident move was justified! These learning agents donāt look at the
pixels of the game, and as a result, they see the world as a big bunch of numbers. And this time around, it was able to play
a pool of 17 heroes, and trained against itself for millions and millions of games. And now, letās have a look at what happened
in this best of 3 series! In match 1, right after picking the roster
of heroes, the AI estimated its win probability to be 67%, so it was quite a surprise that
early on it looked like OpenAIās bots were running around aimlessly. Over time, we found out that it was not at
all the case - it plays unusually aggressively from the get-go and uses buybacks quite liberally
at times when human players donāt really consider it to be a good choice. These buybacks resurrect a perished hero quickly
but in return, cost money. Later, it became clearer that these bots are
no joke: they know exactly when to engage and when to back out from an engagement with
the smallest sliver of health left. I will show quite a few examples of those
to you during this video, so stay tuned. A little less than 20 minutes in, we had a
very even game 1, if anything, OpenAI seemed a tiny bit behind, and someone noted that
we should perhaps ask the bots what they think about their chances. And then the AI said, yeah, no worries, we
have a higher than 95% chance to win the game. This was such a pivotal moment that was very
surprising for everyone. Of course, if you you call out a win with
confidence, you better go all the way and indeed win the game. Right? Right. And sure enough, they wiped out almost the
entire world champion team of the human players immediately after. And then noted, you know what, remember that
we just said? Forget that. We estimate our chances to win to be above
99% now. And shortly after, they won match number one. Can you believe this? This is absolutely amazing. Interestingly, one of the developers said
that the AI is great at assessing whether a fight is worth it. As an interesting corollary, if you engage
with it and it fights you, it probably means you are going to lose. That must be quite confusing for the players. Some mind games for you. Love it. At the event, it was also such a joy to see
such a receptive audience that understood and appreciated high-level plays. Onwards to match number two. Right after the draft, which is the process
of choosing the heroes for each team, the AI predicted a win percentage that was much
closer this time around, around 60%. In this game, the AI turned up the heat real
fast, and said just 5 minutes into the game, which is nothing, that it has an over 80%
chance to win this game. And now, watch this. Early in this game, you can see a great example
of where the AI just gets away with a sliver of health. Look at this guy. Look at that! This is either an accident or some unreal-level
foresight from the side of this agent. Iād love to hear your opinion on which one
you think it is. By the 9 and a half minute mark, which is
still really early, OpenAI Five said yes, we got this one too. Over 95%. Here you see in interesting scenario where
the AI loses one hero, but it almost immediately kills two of the human heroes, and comes out
favorably, at which point we wonder whether this was a deliberate bait it pulled on the
humans. By the 15-minute mark, the human players lost
a barracks and were heavily underfunded and outplayed with seemingly no way to come back. And sure enough, by the 21-minute mark, the
game was over. There is no other way to say it, this second
game was a one-sided beatdown. Game 1 was a strategic back and forth where
OpenAI Five waited for the right moment to win the game in a big team fight, where here,
they pressured human team from the get-go and never let them reach the endgame where
they might have and advantage with their picks. Also, have a look at this. Unreal. The final result is 2 to 0 for OpenAI. In the post match interview, N0tail, one of
the human players noted that he is confident that from 5 games, they would take at least
one, and after 15 games, they would start winning reliably. Very reminiscent of what we have heard from
players playing against DeepMindās AI in StarCraft 2 and I hope this will be tested. However, in the end, he agreed that it is
inevitable that this AI will become unbeatable at some point. It was also noted that in 5v5 fights, they
seem better in planning than any human team is and there is quite a lot to learn from
the AI for us humans. They were also trying to guess the reasoning
for all of these early buybacks. According to the players, initially, they
flat out seemed like misplays. Perhaps the reason for these instant and not
really great buybacks might have been the fact that the AI knows that if the game goes
on for much longer, statistically, their chances with their given composition to win the game
dwindles, so it needs to immediately go and win right now, whatever the cost. And again, an important lesson is that in
this project, OpenAI is not spending so much money and resources to just play video games. DOTA2 is a wonderful testbed to see how their
AI compares to humans at complex tasks that involve strategy and teamwork. However, the ultimate goal is to reuse parts
of this system for other complex problems outside of video games. For example, the algorithm that youāve seen
here today can also do this. But wait, thereās more. Players after these showmatches always tend
to get these messages from others on Twitter telling them what they did wrong and what
they should have done instead. Well, luckily, these people were able to show
their prowess as OpenAI gave the chance for anyone in the world to challenge the OpenAI
Five competitively and play against them online. This way, not only team OG, but everyone can
get crushed by the AI. How cool is that? This Arena event has concluded with over 15000
games played where OpenAI Five had a 99.4% winrate. There are still ways to beat it, but given
the rate of progress of this project, likely not for long. Insanity. As always, if you are interested in more details,
I put a link to a reddit AMA in the video description, and I also canāt wait to pick
the algorithm apart for you, but for now, weāll have to wait for the full paper to
appear. And note that what happened here is not to
be underestimated. Huge respect to the OpenAI team, to OG for
the amazing games and congratulations to the humans who were able to beat these beastly
bots online. So there you go, another long video thatās
not two minutes, and itās not about a paper. Yet. Welcome to Two Minute Papers! If youāre doing deep learning, make sure
to look into Lambda GPU systems. Lambda offers workstations, servers, laptops,
and a GPU cloud for deep learning. You can save up to 90% over AWS, GCP, and
Azure GPU instances. Every Lambda GPU system is pre-installed with
TensorFlow, PyTorch, and Keras. Just plug it in and start training. Lambda customers include Apple, Microsoft,
and Stanford. Go to lambdalabs.com/papers or click the link
in the description to learn more. Big thanks to Lambda for supporting Two Minute
Papers and helping us make better videos. Thanks for watching and for your generous
support, and I'll see you next time!
maybe we should let it master sim city and then vote it into office lol
It does not surprise me that they are able to beat human players. How I see it, there are machines, they can most probably calculate distances, damage, health, speed, cooldowns (both yours and enemies) faster than a human can or at least compile the information at a more rapid, accurate rate.
What does surprise me is the rate it learns and applies such knowledge. Unless that human is fully dedicated to playing Dota, is up to date with every patch change and practices religiously every day, I still don't think they would be able to beat the top players in a 1v1, much less a 5v5. But the OpenAI team started working on the algorithm for Five in 2016... In less than 3 years they were able to master the game to the point that it could beat top players with relative ease.