Cyber Python 2077 - Using computer vision to read and walk from Cyberpunk 2077 map

what is going on everybody and welcome to a cyber python 2077 video in this video we're going to be tinkering with some computer vision tasks to see if we can achieve um some ai functionality in cyberpunk 2077 to start unlike gta 5 which by the way i know people are going to get some crazy ideas uh with what's going on here i'm just tinkering don't don't uh don't read too far into it but pretty much every game i play now i think of like can this be solved with the computer vision of python and uh at least in this game i think we can actually develop the best ai in cyberpunk 2077 using some pretty rudimentary computer vision techniques so um that's what we're going to do the bar is pretty low so uh so first of all i at least in this game unlike gta where cars are kind of like a the main focus of the game i mean it's like in the title that you steal cars uh whereas in this game actually kind of being on foot is is the more is the bigger element in this game um and then so i kind of thought like could we is it possible that we could still navigate on foot like through a city um and and if i could do that or if so like how how might we actually just go about like doing it so then i started thinking okay well at least on foot you can follow that the the pathing that the game just gives you and for the most part it appears the pathing is pretty good so if you wanted to navigate on foot in this game with a program you really just need to follow the mini-map so um so then it was like okay well how would we how do we follow the minimap what would we do from there so i kind of just want to run through um what i think is at least a pretty valid methodology for automatically walking around in the game of cyberpunk 2077 so the first order of business will be to grab the screen and then also we need something to do direct input both in the form of key presses but also mouse movement on the screen so you can't use something like pi auto gui to input keys or to move the mouse because that it's not the right input type basically for the game it won't recognize what you're doing so um so we need those two things and it turns out with the python plays gta series we got both of those things so let me pull up those i will put links to both in the description but here we have the grab screen script and then we have the keys now with grab screen what this forces us into is selecting basically the region that we want to pull from there are other methods you can also there are some methods where you just input the name of the window right and it's much less code than than what this is uh the problem with that is at least for me like i don't know all of like all what's going on here but the biggest thing is frame rate we need the most frame rate as possible so um this still seems to be the fastest way of of getting those frames and reading those frames so um this is the method i'm going to use and then keys this was written by daniel just to let us control both the mouse and the keys so i'm going to grab these two i will put links to both in the description so if you want to follow along you can also if you want to follow along in some other game you totally can you don't have to run cyberpunk 2077 to follow along lots of games have mini maps lots of games have pathing on the map you should be able to use all of the same techniques that we're going to use here with like any game that you want just hopefully the pathing is decent so at least in this game the path is on the sidewalk if you're on foot and then if you're on a car the paths will be on the road but even if there weren't pads like for example like with a car you could drive a car and you would know where the road is based on the map um oh i'm not showing the game so a couple of things first of all if you hear all my fans going that's because this game makes the cpu get very hot and fans go burr and um and then uh because of this full like the game and all that stuff uh it becomes kind of challenging to show uh all the windows i want to show so i kind of have to navigate these windows via obs which i'm recording so sorry if i don't pull up the window quickly or forget to pull it up and i'm trying to show you something it's just going to happen but anyway what we're going to try to do is we're going to try to follow this yellow path but yeah you could in theory like if you look at the mini map here all the roads are marked with this like blue color and that's the same across the game so different games are going to have different kind of methodologies for showing paths and things on the mini map but in general they tend to be pretty standardized in that specific game so anyway what we're going to try to do is follow that yellow path so the first order of business we need to see the game so we grab the screen for the game then we need to see just the minimap so then we're going to use a region of interest with um just opencv slice get just the minimap then we want to see the path right so then we're going to mask and find that yellow color and then we'll go from there we'll try to walk along that path using some sort of technique i'm sure there's many ways that this could be done i'm just going to do the way i came up with so uh okay so let's get over to our code here let me pull up sublime cool all right so this is just some starting code basically it's just gonna grab the screen for 100 frames again this is the region this is going to vary depending on the resolution of the game where is the game is it running full screen do you have many screens do you have one big screen is it a wide screen so on so um the region that you select from and the resolution is going to vary depending on the game so just note that when we pull it it will be in bgr we're going to convert to rgb and then i'm just going to resize it to something a little smaller because we're going to display it and there's no need to display the exact same high resolution version but let's go ahead and just kind of run that show you uh what we have so far hopefully it pops up it does great let me move everything out of the way and there we have we can see that yep we are indeed pulling it at a decent frame rate i wouldn't want to play at that frame rate but um it's at least enough that we okay we could work with this all right it's not uh two frames a second so okay now that we have that uh let me pull up sublime and what is the oh i ha did i have it up or oh i had it up already and then we were looking at the it's going to continue it's going to continue um okay so um okay so we'll just continue working so the first thing we want to do is grab the mini-map so the first kind of task is to just run through this path so i'm actually just going to make a new function we're going to just call it define pathing it's going to take in um a map or let's call it minimap i don't want to steal the name map from python uh it'll take a mini map and it's going to just like do stuff for now we'll just say pass so we want to first grab that mini map so before we do this like resize here in fact i'm going to comment comment comment out we want to grab the the region of interest that we're interested in actually i already have the region of interest that i want to use so i am going to copy and paste it just because it's silly to waste time um but i'll at least explain it so the minimap is a slice of the screen and these are the x values and these are the y values so it's going to be for x it's from the 81 to uh 377 or actually let me think about this i think it's actually y x sorry so so does that only make sense because y starts at the t at the top so it's actually y to your x so 81 to 377 and then um for the x coordinates it's 2181 to 24.69 um to to figure out what your coordinates are just trial and error it honestly just paste in the ranges take your best guess to start and then hone it in so at least in this game uh let's pull up the mini map as you can see the mini map in cyberpunk 2077 is like not a perfect square right it's a little distorted it's not like a tilted square it's like i don't know what but describe it besides distorted it's not a perfect thing so anything we do is not going to be probably perfect in this case but we should be able to still succeed so um so that's our mini map we got the screen and then we're going to pass that mini map into pathing so then we're going to say pathing and we'll pass in minimap and then we have code here so the first thing that we want to do is we actually want to detect that path so i'm going to i think we'll just i'm going to copy this over and explain what this is because again your values are going to change if you're using some other game or something like that i'm just assuming probably most people watching are probably just simply watching out of curiosity and then some people are going to probably apply it to another game and then maybe one percent of you are actually following along in cyberpunk so i think i'll just do it this way so so um so for masking again kind of like selecting this i'm going to be honest with you the way i go about doing it is the following so first of all when you mask you're actually first going to convert to hsv now hsv is huge saturation value and if you look up hsv there are like hsv color pickers and you might think i will just use that to figure out what the right values are but they don't use the same range like opencv seems to use the range of 0-255 but that's not actually a valid hsv range so if anybody knows a better way than what i'm about to show you feel free to comment below but this is the method i use so zero 255 first order business like so let me just let's just look at it real quick uh we don't have numpy let's import import numpy as np save that run it um okay so as you can see i'm actually surprised that window came up um so i'm using open or i'm using obs to show you like the different windows and it's actually supposed to be looking for a window called cv2 screen but i saw it pop up so i'm not really sure how that happened but anyway this is what we get so it's all white so uh what this means is actually everything matched right because everything is in the range of 0 to 25 hue saturation and value so hue is your color saturation is your saturation value is the value of that color so um so the first order of business is to figure out what is the hue so so where in the range is this color so the first thing i would do is let's just set some large number um approximately halfway there i know it's not quite right like one twenty seven anyway not important 127.5 i don't know anyway so run that and we can see that in the hue of 150 to 255 it doesn't actually look like our path is detected so that means actually our path is in the range of somewhere it's not in the range of 150 to 255 so therefore it must be in the range of 0 to 150. so then the next thing that we can do is we can do the exact same thing for saturation so we can say 150 to 255 we can run that again and sure enough what we see is okay great so we've filtered out quite a bit we can still see our character color um and then we can see some like some of the road it looked like and just some general noise we still want to kind of like filter those out so again we'll just do the same thing again okay we do 150 um and then we'll run that again and then okay at this point it looks like we've actually filtered out basically everything except for the character on the map and then we have the path that we want well the character color is like this like blue let me pull it up real quick for you so the character color on this map is a blue and the path is a yellow so we should be able to filter that out like pretty simply um so the again the next thing i would do is come back to let me pull up my sublime show you sublime and then i would just continue adjusting so i would continue bringing in certain values so maybe here you know you could make something well first of all it's definitely a hue issue right blue versus yellow so it's not a saturation it's not a value issue it's a hue so probably i would say i don't know 75 to 150 that would be the first next the first next step i would make i don't know anyway let's try that whoops i ran python instead i'm not actually sure why weird okay so anyway yeah boom done it filtered out and now we have just the path that we want to detect okay now that we have that um we have to talk about what is the actual logic that we intend to use so pulling up the game so what we really want to do is we want to follow this path um and and but we need to somehow formulate programming logic that will follow that path for us based visually on what the map looks like so i think the best well there well okay first of all i mean we know like if our character is looking like this we know the character needs to look right and we know this because the character is always like the mini map is always forward for character and we can see the path is off to our right and then we know this is correct because at least the initial stages of this path are directly in front of us so on the map that would translate to so let's just pull that back up again that would translate to the white pixels basically being directly in front of us well we also can surmise again the map is a little um i can't think of the word now anyway the map is not a perfect square but we would say okay well probably the target would be to have white pixels in the in perfectly in the middle now the other problem we have is um we if in this case if the white pixels were perfectly in the middle or we tried to make it as perfect or as close to the middle as we could probably the character would look like this and then probably walk into this poll right that's that's no good we don't want that we want the character actually to first go this way and then turn so in order to achieve that um i think the best way about it is actually rather than taking in the entire mini-map we zoom in even further so the way i went about doing that was i actually just make made a new value called mini minimap and again i just would um trial and error until i got the value that i wanted um but let me i already have a value that i'd like us to use possibly maybe i thought i had the value but oh here we go i was about to have to just do it myself um okay so let's pull this up so i would say mini map and then mini mini map now i might as well just keep that on hand because that's probably going to come in use later at some point but now rather than pathing mini-map let's do pathing mini mini-map save that um it is still on screen let's run that real quick and now it's so small oh no i can't get it uh it's like up in the corner uh i want to make that way bigger okay maybe i've got it that way so anyway now we have the minimap and now as we if we zoom in enough we get just like the few dots right ahead of us so i think that's probably better so as we approach hopefully i can make it in time as we approach this turn like if we centered those we would probably like i'm trying to just walk based on what i'm looking at on the thing i think that that's like good enough to get around there um so now we'll find out oh now it's gone where's our path i look legit lost the path on the uh on the game for some reason anyway so we'll have to handle for that kind of stuff at a later date um okay so uh once we have um the so we're gonna work with those like the smaller minimap of pixels once we have those pixels we still have to like come up with the logic of okay how do we know what center how far off center are we and then when we are off center how do we get back on on center what do we do so the first thing that we're going to do is we're going to come over after the mask yeah we'll just do that so we need to find the coordinate location of all of the white pixels in that mask so anything that matches the mask is going to be a value of 255. everything else will be zero so we want to find where all the pixels of 255 are so to do that we're going to say matches equals np.org where and then in here we say where mask equals to actually probably double equals 255. so that gives us all our matches so this is going to be an array of matches so for example let's print uh matches let me change the frames we're going to say 100 frames and then we'll run map walking so that showed up and then command window look at me hackerman okay so as we can see it's kind of spamming out these these locations but as you can see at the end here we've got it's just an array of these coordinate locations for the 255 values so what do we do with that well what we really want is for these coordinate values which by the way these are the in coordinates y x so it's an array an image is an array and the first index value is going to be y and then the second one will be x and the one that we're interested in is the x right because the further um let's say uh you can't see the game uh let me do this and this boom uh so in this case like we want all those x's to be perfectly in the center but as it goes this way right the x's will be you know to the left so it'll be less than and then this would be perfectly centered and then these would be like greater than center values so we want it to be in the perfect center so first we want to know what is perfect center what are the current locations and then like what is the delta essentially and that will inform us which way do we want the mouse to turn so coming back to sublime text and i think we probably want we'll want this window but i'm going to i'm gonna get rid of it for now so now i have to pull up sperm text okay so we wanted to know so we've got the matches we now we want to know what the average match was so i'm just going to copy what i had before so it was called mean y and in this case it's going to be a numpy dot mean of the matches but we want just the x elements and to do this with like a 2d array for example it's going to be colon comma whoops calling comma comma the index that we want to use so it's going to be colon comma 1. okay so um so once we have that that will be the mean of all of the x values of those matches so this will tell us what the mean is and then we just really we just want to know what is our goal value right so we're going to say uh target is that what i could yeah call it target equals and in this case it will be whatever the minimap is dot shape so the shape of the minimap and again it's going to be in i think yx yeah yx so we want to know the first element divide it by two that will be our target and i can't really decide i mean i don't think it's going to be important that we convert that to an int so like if it's a float value for whatever reason no big deal i don't think it'll matter so we'll just do that and i don't think yeah so then finally we're going to say our error equals and we're going to say target minus whoops target minus mean y so i'm back on like my main computer now and um now my main computer's keyboard is a mechanical keyboard now with blue switches and i'm doing you guys a huge solid by not using that on video so i'm like trying to get used to using this keyboard it's a pain um okay so um air target so rather than print matches let's print the error and then hopefully that will be somewhat meaningful and give us values that we want so um we are printing out the screens so i think we'll be good you can see that um and our error oh you can't see my command window we pull up the command window and then finally let me just we don't need to see sublime at the moment so okay so this gives us our error so as we're like as we move this way the errors are negative so if it's a negative that tells us we need to move right and then as we direct pull down the error it appears to be working as we want let me run it one more time and then yeah so the error is positive that means we need to move left okay yeah so great so now we are so close to having having what we want so now all we need to do is basically walk and then move the mouse as intended so before we go make that mistake let me um i'm going to import a time and i'm going to come down here i'm going to say 4i in range of like three we're going to just print i and then time dot sleep and we'll sleep for one second now i have time to like set everything up because this is going to take control of my mouse and my keyboard and that will be really annoying if i'm not actually set up to exactly what i want to see before that event occurs so now um okay so i'm going to take yeah i'll just take this and move this over so based on that error what do we want to do we're going to move the mouse so i'm just going to paste in keys and then direct mouse and then what we're going to do is we're going to move negative 1 times the end of the error because we can't actually um we can't move a float number of pixels so we're going to just move negative one so we're gonna flip the sign of exactly whatever the error is and later we'll smooth that out or if if there happened to be future videos let's move that out but for now i think that's fine um and in this case it is x y we only want to move the x uh axes at the moment so okay so at this point we'll do that um i guess we can do go ahead and just slap in the direct key as well so once this is done we're gonna do uh keys dot direct is it camel yes direct key and then it will be w and then when we're all done uh cv2 why would i show it so coming back here um so yeah we have our sleep here basically we're going to input the w key just before we start running through the frames and then here we're going to release that key and then destroy windows if for some reason you are following along and the script breaks or you break the script or whatever before the key gets released it's going to stay depressed but all you have to do is just hit the w key or whatever key on your keyboard and it will release it um cool so yeah getting back to up here once we've got this defined we need to go ahead and import keys import keys as k and then it's going to i think it's just key yeah keys equals k.capital keys and then just empty there's no nothing that we're gonna initialize in there save this and i think that should work so i think what i'll do is i'll start off like off a little bit and then let's see let's see how that works so run that zero one i wasn't totally not ready you can see at least it appears i didn't move quickly enough did it no oh and we got a okay so we got an error we've got a couple of things that's okay because you couldn't really see what was going on i think i need to make that timer a little longer um okay so we got an error okay a couple of things happen so let me pull up these consoles so this is what we got it was walking i assure you um this is the area that we saw i think it's because we actually ended up walking beyond the path uh that we intended to take um so there was no path so we would want to have some sort of handling for that there's going to be times where who knows what happens um sometimes when you're filtering the best mask for some reason even this map i want to say is a little bit uh transparent doesn't yeah it's a little bit you can kind of see a little bit through the map i don't know if i can pull up uh let's get rid of sublime like if you look um i'll put like a billboard behind it i don't know if you can see that but there is you can kind of see through it anyway that can that can mess with things it can change the hsv values a little bit enough to where you might not pick it up anymore so you'd want to have some sort of handling for that in general moving around or something like that can kind of help uh to pick it back up but anyway we're i'm not gonna try to fix that for now the biggest thing was he just didn't move in time so let me just run that one more time um zero one two okay cool so i mean it this guy's in the way so he's doing okay he starts to try to turn and then he misses it he walks beyond his path that's why we get that nan issue uh the script breaks i just need to press the w key and fix it so let me get us back to where we want to be and we'll try that again so there's a couple of changes that we want to make in that will take us to sublime text let me pull up spline here um i think i want to make this maybe five seconds for me and then for the error movement and error i don't know times three okay cool um and then i think that's probably good enough um we want to have some sort of trying except for if this thing hits an error but for now whatever let's run that again let me pull up close sublime pull up this and i think that's good enough let's run it okay so i'll start like this too let's just see and also see like that path is not okay that's good enough i guess let's see if he moves quickly enough it's so juttery okay yeah all right that appears to work we could probably run it a little longer um we might might even want to move uh like a larger error but that at least took the turn decently again pretty jittery we would want to smooth that mouse down um but other than that i think it actually worked pretty well let me uh let us change to let's do like 400 here i'm gonna save that and i think that's the only change i really need to make let me get rid of sublime text run that again cool maybe oh there we go i forgot i said it to five seconds so even like it overshot that time but still fixed itself that's fine it actually follows that path really well see if he makes this turn that's pretty good that's pretty good this person is very scared oh we just ran out of time okay cool so that worked well although i will say i was a little curious what would happen uh with this tree let's let's just run it for 100 frames real quick i think he's going to run into this tree i mean that's what it looks like so i just changed frames to 100 i don't feel like showing it real quick and then having to get rid of it it just looks like the pathing is going to go right into that and it does is it going to make it oh we still made it uh that's so fun okay i mean that's okay uh that's pretty good for some pretty basic code um so yeah i think i think that's a pretty good start to um to to using the navigation especially when all we're doing is looking at the mini map and turning our character based on the map at least through my playing um of a completely reasonable amount of time of this game i i believe the pathing is actually pretty good like it will usually map you around even objects like this uh for some reason i was kind of surprised it didn't do that but okay um but the next order of business would be like for example if you run in wow i thought i would run into this guy i don't know sometimes you like go through people and then sometimes you run into people like what i just like totally went through that guy let's maybe they're standing still let's see yeah so like sometimes you just like run i'm trying to do a demo here i don't know i i can't quite figure that out i'm pretty sure you run into people but um and then he gets kind of stuck anyway i assure you it happens sometimes but so so having some sort of object detection on screen would probably help um to to avoid people so in general like if you saw there were people here you could just sidestep a little bit and then you would get back on the because even as you sidestep you just saw like the thing it should update at some yeah there we go it updates so you could sidestep a little bit wait a few seconds and then it would update and then you would have your new path so you could probably get around it doing that and i i think you would be able to probably navigate with a really high percentage you know like 70 or 95 of the time i think that's enough to to do everything on foot that you would need to do on this game um so that's pretty cool uh i think the only major downside is like if there if you happen to be in an area that doesn't have a sidewalk it does put the path for you to walk like literally in the middle of the street that's kind of a problem so so yeah but there's really nowhere else for you to walk so i don't know i'm not really sure um okay i think that's a good stopping point uh hopefully you guys enjoyed if you have any ideas uh or requests for like something that we could do in this game um yeah feel free to let me know uh if you have ideas for like how we could make it even better you could let me know that as well i definitely am going to try to apply object detection onto here i'm curious how well it picks up things i i suspect we'll be able to pick up cars and people uh quite well with object detection like just the basic tensorflow object detection so i'm kind of interested to see how well that works um also interested to see how much we can stack on top of our gpu without um completely dying so uh okay so maybe some interesting things to look forward to uh in the future i'll probably just continue tinkering on here questions comments concerns whatever feel free to leave them below otherwise i will see you guys in another video you

Channel: sentdex

Views: 112,085

Keywords: python, programming, tutorial, computer vision

Id: dUU6ZsJlZKQ

Length: 33min 50sec (2030 seconds)

Published: Fri Jan 15 2021

Reddit Comments

[u/AutoModerator]

Just so you're aware, we have a bunch of megathreads that may be helpful if you have questions or want to find the right place to post.

• Console Support
• PC Support

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.