AI For Unity Game Development - (Artificial Intelligence) #ai #unity #gamedev

hi my name is Sam I'm a unity game developer with over 20 years of experience the last 10 years I've used Unity exclusively and the last few years been playing with web 3 projects and AI projects as part of my Consulting work here I want to talk a little bit about a recent example that I created using AI along the development pathway there's two major ways that I see game developers looking at AI technology particularly inspired by chat GPT one is to help you create your game code and then the other is that while the game is running to be able to call apis that would randomize or create intelligence in your non-playable characters text phrasing for example here we're going to be looking just at the first one if you're creating a brand new project from scratch or a brand new system how you can lean on AI and I'll show a couple different tools to help you along the way I I've just done this recently and I had some learning so I'll just kind of speak aloud as I'm going through it about some of the thoughts that I've had take a look at it together so a quick review here is we're talking about the unity game engine available at unity.com now Unity has released some AI announcements so if I come to Unity unity.com I can see that there are few products that are coming to the public later they're not fully available yet one of them is orb a fully AI driven character one of them is Unity Muse which says fueled by Next Generation AI Unity Muse enables creators of all skill levels to develop games and real-time 3D experiences lightning fast with simple text based prompts and then Unity sentus powered by our crossplatform game engine Unity sentos enables runtime stuff happening so there's three use cases there and two of them are the same that I mentioned help you generate code and then help you at runtime to add all of the richness that AI can bring to you so I didn't use unities tools instead I created this project which I'll link to below I'm calling it the AI physics Glacier game and it was inspired by an image I saw uh of a glacier with some characters on top of it so that inspired the gameplay all right let's jump in and take a quick look at the project itself the finished and then we'll explore a little bit so if I jump into Unity here press play I'm the green character in the center and I'm on this icy Glacier and I can use the arrow keys to move around if I fall off it safely puts me back onto the screen and I've got these other blue characters that are all trying to Chase my character and push me off the ledge now I'm not playing very well I also noticed during the screen recording it frame rate drops a little bit but we've got uh character Movement we have uh a UI we have some light uh enemy Behavior here that just tracks to me and if I'm able to move a character off like that then my score goes up as well so we can see quite a bit is uh is functioning there now this is the type of project that I could create by myself without any AI in under an hour it represents a pretty simple example but a complete one of what you can do with unity only thing I'm missing from the five or six fundamentals of a game here is sound I didn't add any sound but we have taking input from the mouse and keyboard we have uh the UI on the screen we have rendering the graphics and stuff like that we have some Physics that's happening sliding on there as well as um the objects pushing into each other you can see when you land here in this free sample project you get a nice view in the inspector which gives you an introduction to where the project is built from uh how you can learn more physics if you're interested to learn that and then you can click here and uh get a a quick way to see into the first scene that you want to start with now coming back up here I'm going to talk about the two technologies that I use when creating this and then I'll show a little bit of a demo [Music] about how I use that AI so firstly we're familiar since the announcement of chat GPT with the company open AI they're the group that has created a lot of AI breakthroughs and most famously chat GPT which looks kind of like instant messenger but every time you send a text message you get one back not from a human but from Ai and I'm sure that you've played with that at some some respect or watched the video video about it so you can um go to the website here and do try chat GPT up in the corner and for free you can experiment around with it if you become a paying member you can get access to additional plugins which are quite interesting including ones that can compile code nothing specific to Unity that I found but C for sure we'll take a look a little bit of that then a separate product offered through GitHub which is a repo where you can put your projects this project for example is on GitHub you can use the GitHub co-pilot now here it shows a little pre-made video of you starting to code something in a given language and then it helps you finish that so I'll replay it here and it looks quite magical you you see someone begin to do about five or six lines and then it gives you the next 15 or 20 lines right and there's going to be some different examples about how and why that's used it gets trained on all the publicly available repos in GitHub I believe that's how it has been trained and GitHub is a huge resource of all different sorts of programming languages including C for Unity so inside my coding editor I'm using riter but you could also use Visual Studio it has a plugin for GitHub co-pilot now I just updated this by coincidence right before the video so the next time I open a rider it gives me this little introduction screen so let's just look at it together it says welcome to GitHub co-pilot for jet brains Rider you can receive suggestions from GitHub co-pilot and you can configure it in some different settings so let's take a look at those settings so here you can choose which types of files it would work on and how often it will update itself we've got some advanced settings as well I have not touched any of those settings you can change some keyboard shortcuts as well let's just check what those are and type co-pilot here so being able to enable completions contrl alt shift and O okay I'll try and remember that there's a different way to do it but let's do that and um let's give a look in the code here so throughout the codebase if you download and explore it I invite you to look for anything that says AI prompt we make that nice and big so here I said AI prompt and that is not a special word I just chose to add that to each comment so it's easy to find through the application so if I do a find operation and and then say AI prompt it shows quite a few examples of where I put that you don't need to type that you would just type something like this let's say detect collision and if it's with a type water view then dispatch a Unity event called on water Collision so I'm beginning that line with a double slash which is a comment generally comments are for human eyes only the compiler doesn't use it the unity game doesn't use it but interestingly the co-pilot still sees it so if I go to the next line um it will generate some code for me I'm going to make that go away and I'm going to do control shift alt and o is that what we saw was the shortcut okay I'm going to go check the shortcut again because I already forgot what it was copilot enable completion show completion okay alt and slash I was using the wrong thing so alt and there we go I actually like that not very much so if I was to type that in fresh here so as I type something like that in the co copilot will go and it'll look for suggestions and it'll give me suggestions here now there's a few ways you can do it there's also a panel over here on the right where it can give me a list of multiple ones but I like this one it's right in line in context now here presents the first of many challenges that I think using this first of all in this case it happens to be right this is the code that I would do to detect a collision and check what I'm colliding with so here we are we're inside the enemy character which I've called The Penguin View and it will need to detect if it is hitting the water so if it's fallen off of the glacier and hitting into the water and this code is valid so I would accept it by hitting tab it's missing a little bit of stuff here but I can fill in the gaps right and if I back up there we go I backed all the way up to the original code that I had in here which was also inspired using chat GPT so want to a double check so it missed one particular part here not bad but it it gives you some pretty good results so let's explore another area about how I could use this so here this is using co-pilot within the context of my project it knows not only about the file it's currently in but it definitely knows about other uh files in the same solution so for example uh I'm hinting heavily which it should say but maybe if I just said if it is colliding with water and it would sniff around in the code and it would know that I don't have a class called water but I do have one water View and I was exploring as I was doing the different prompts how specific do I need to be with the prompts that I give it and it surprised me sometimes now sometimes it's missing something like I showed here it was missing uh the game object which is required for compilation but very cool and I know it'll improve over time so let's think about another way that we could integrate um this kind of AI technology into the game so back here using open AI technology more directly I could use the try chat GPT I could open up a new conversation and I could start using it there now in that case it doesn't know anything about the specific context of my game so again co-pilot is good for a context specific solution where it might know about how your project is set up so far in that project I have 10 files maybe something like that 10 classes in C to to get this to work uh so it is very useful right and perhaps in much larger projects depending on the quality of the AI results it may be able to give you something way better because of that context but I also like the freshness of being able to say give me some in give me some advice and answers with no context or being able to control the context better so let's talk about just using chat GPT directly for that so through the web browser when you try chat GPT it'll give you an interface something like this this happens to be the downloadable Windows version of chat GPT and I've got a bunch of different prompts that I can use here so here it will default by giving me a brand new chat and I'm going to make this bigger here so we can see it a bit better there we go and the primary options here is you have G GPT 3.5 which has been around a longer period of time then GPT 4 now again I'm a paying member I think it's about 20 bucks a month for this service and about 20 bucks a month for the co-pilot those are two different products I may not be having the numbers right but you know it's a monthly fee with chat GPT 3.5 you can definitely experiment with everything I'm about to show you I just believe that sometimes the GPT 4 while slower gives better results I think over time it'll improve so let's talk about two two use cases that I had that I explored with this project one is I used the dolly 3 plugin here and I said uh I'm creating a game with a uh with a glacier let's say create a, 1600 by 1,000 image uh use a transparent background add a glacier to the center of the image at about 50% of the size of the image so open AI has several different tools J GPT is one and do is another one both of them are having version numbers that are increasing over time we're at about the one year mark since these tools became available and within just chat GPT I see a dramatic Improvement in the types of things you can do the plugins for example wasn't released at launch but now that we have plugins it can read the web it can read PDFs you can upload an image and ask questions about the image and with Dolly 3 the quality has gotten better and better so I'm imagining a game looking kind of like the one that I just showed you which is more of a cube shaped iceberg in the middle and then having our characters on top of it neither of these really satisfies me so I'll say make variations of the left image but make the top of the Ice uh cler and be flat be perfectly flat like a tabletop now we'll see how that works so if you haven't experimented too much with AI you um you may not know that it's kind of an ongoing and iterative process in order to get the results I would expect the more skills I develop as I'm generating these prompts I could get the results more quickly and then certainly as the AI matures it would be able to give me better results with a less prompting but here you can see it did pretty good it is doing the uh transparent background now I really like this one on the right that's pretty close so let's say uh make variations of the image on the right make the uh Glacier 50% of the current size within I won't reiterate that and then I'll say uh and lower the iceberg I I don't want to confuse it by saying Glacier and Iceberg I don't know if that would cause a problem but these are the little things I think about and then I would also like it to have more surface area on the top let's see if we can ask for that can the top base be a be uh slightly larger I'll use the top face Bas or game play and having more game play area is helpful okay after this example probably stop here now what I done in the original example is I tried that you can see in the game we ended up with a perfect Cube that had textures on each side uh that was because I iterated here for a bit and realized some limitations even after I would create an image like this and try and bring it in combining 2D and 3D elements like that just didn't work for my specific needs certainly for background it would work great okay I'm probably less pleased with these results but we'll stop here and we'll look at a different use case but you can imagine getting some art for your game it's certainly pretty art I didn't add to the prompt photorealistic impressionistic 80s poster or anything to indicate the style but I'm pretty happy with the style and really this one here is it's pretty close to what I'd want I would just want it to tilt a bit more toward me uh as an alternative I could have just asked for a square repeatable texture and then I could put it on a cube myself which is which is what I ended up doing but I wanted to walk you through some of the thoughts and process there now I'm going to create a brand new chat here and here we're going to talk about what chat GPT can do for me to help solve programming challenges now GitHub co-pilot we saw has the context of my project and I'm able to ask for a small snippet in that context that's probably the ideal use case here uh however using chat GPT in this format it doesn't know anything about my project and the workflow is not to say hey go look at my project and give me an answer at least that's not how I'm doing it this is a fresh conversation with zero State behind it as I've been learning a little bit about prompt generation one of the patterns that I see people say is tell Chance CBT what to act like so I'm going to start with act like a game developer for C and unity okay so that alone just tells it kind of what domain to be in uh you are an expert with great answers okay you could also it sounds a little odd but you could ask it to be an introductory person or a beginner or a child and give some answers where you're asking for kind of different quality of the answer itself but here I'm clear to tell it to be an expert and to give it game development C and unity now let's try and say we want to have uh I'll say I have these two characters and I want one to follow the other one around and let's see what happens I have two mono Behavior classes uh let's call them called uh player and Penguin create the source code for the penguin CS so it knows it's going to be a monob behavior child which in the unity world is a very common beginning point for any new class structure uh which needs to do the following uh it will uh take a serialized reference to the player follow the player move with physics and a rigid body okay so that is a real world use case for the game that I just showed and let's just see what it's able to do here for us by default it puts the coding Snippets into a nice embed so you get some Auto completion it is not able ble to compile this code to verify that it works there is a plugin for chat gp4 that can compile certain code but as far as I know it can't compile Unity C it can just compile C so my workflow here was as I was making the game that I showed you would be to copy this snippet into unity and let's take a look at its result and then talk about how that copy pacee would work it always gives you a bit of explanation which is fantastic for learning now another use case here would be I'll wait for a while that's finishing if I could copy some real world code paste it in here and say explain this to me in fact we'll try that as an example in a moment but you can see it's doing quite a bit of exam of uh explanation so let's see what it does here it creates a new class that extends mono Behavior that's perfect then it's even creating some headers this is purely cosmetic stuff for the developer it's quite nice it serializes not just the one field that I asked it for but also a couple other fields that would not just be useful here but in the unity world it's optional to make something a serialized field and it has certain pros and cons and this feels like a relevant use case here and then it's going to run through and on awake it's going to get a reference to some Physics stuff that it needs then it uh runs the fixed update and calls follow player sounds good it does a nice check here at the beginning that if it doesn't have some things it's looking for it'll exit out gracefully that's nice and then U without looking too closely into here I'll just say this code would work more or less so we could paste that in now here's what I saw as some of the challenges of doing this workflow one is I as a game developer with experience recreating a system that I know how to create I know which questions to ask and when I see the result it's easier for me to know if it meets the needs that I have and is going to compile quite often during the two hours that I spent playing around with this and making the demo the other day it would give me a blatantly wrong answer and I could start to see that I left a detail out of how I asked it or I could see why it would make the mistake not that it's acting necessarily like a human with less skills sometimes it is but it can also be acting like a AI That's getting two concepts confused in in a way that a human might not even do but this is a fantastic starting point so here was the workflow I would paste this into Unity I would see if it compiles then I would work on the unity code directly with the co-pilot or without it making any tweaks then when I would come back here to say oh you know what I need this penguin to do one more thing it might repeat this step give me all this code again but it doesn't know about the code that I added back there in the code editor myself so I'll give you a quick example of that in my codebase I have a template and I use this template class when I create new code just so things follow in a nice set of coding standards and in a nice order that is nice for me as a developer to read it's completely optional to use such a template it is not a requirement at all for development nor for AI but let's just use it as an example I'm going to copy this here and then I'm going to come back into the AI and I say uh great answer give me that answer again but first make your code in the style and format of the following code template and then I'll just paste that and let's see together if it gets confused if it's actually going to add some of the methods which are meant to be totally temporary in the template uh together with the first one or not so I can see already it's learned about namespaces it is giving the exact name space from the template which isn't really what I want here I would want it to maybe say RMC which is my company name do uh penguin or do characters or something but uh this is pretty good uh then it added this uh comment here can't remember if that was was in the original then it's adding these flower box comments which are just lines to indicate where section should go I don't remember I won't even scroll up if it was putting underscores before private variables but I like that convention so I think it's adopted that um and then it has put something down here in it under event handlers which isn't exactly my style it's this I would just consider this just a method but pretty cool so what you can do is after you work on tweaking this and using it maybe for some hours or days or weeks of development you get the penguin code to work great then you want to come back here and you want to ask a brand new question now with all the context of having that file pretty complete you might just want to use chat uh sorry co-pilot inside your code editor because again that's great when you have lots of context small question here I think it's better when you're starting something new and fact that there's no context is great right but let's say um just to illustrate that there is context here then I'll pop back into the game and I'll wrap this up is let me say I have a uh now we have a water CS class it needs a reference to all existing classes we discussed so when I submitted that first time it gave me an error you just resubmit it sometimes that happens I think it's just the Growing Pains of having a a new system so here I've asked it just to demonstrate that it knows about the conversation above uh here uh go you know give me something new but have it reference all the old stuff and it worked it's pretty easy you can see that now what I want to do is just show me this what if I go here I create a brand new chat and I'm going to type the exact same prompt into a new chat and this may seem quite obvious and if you fooled around with chat GPT already it will be obvious to you but this type of thing was kind of confusing to me when I started what what I'm illustrating here is that each new chat that I initiate know knows nothing about the other conversations which is has some drawbacks but it's also quite powerful so we can see in the response here it says it seems like you're referencing some files or code that you haven't provided could you provide it right and as far as I know there's no way to say hey we just chatted about that go go check over there uh so the way that my workflow is usually is I start a brand new conversation I ask questions and I just keep that chat going for hour hours days weeks months you can see over here I've got you know tons of different ones uh that have been around for a while so let's pop back into the project I'll run it one more time here see if I can do any better no I just ran off the edge now the game doesn't end when I die which is nice I just respawn so I would be able to take advantage of that and just start pushing the characters off yeah I'll pop back to the slides now so here we are I'm summarizing everything that we've seen uh in the link below the video you can check out everything that I showed here including this free repo if you want to give it a try what you would do is download the project open it up inside unity and just play the game that's a first starting point and then you could use it as a platform if you wanted to ask chat GPT how to add a new feature to that existing code or you can go create your own project for for sure but if you want to kind of stay within the context we've got here you could do that another thing you can do is if you're used to using GitHub you could explore the free trial of GitHub co-pilot both chat GPT and co-pilot have a free trial that you can do and then you'd be able to experiment inside your favorite code editor again I'm using ryer which is a paid code editor for Unity and I love it so that's how I'm using it but there's also free code editors including Microsoft Visual Studio that you can use and get that co-pilot uh plug-in there as well I believe for all code editors the co-pilot feature is free trial then it's always pay I don't think there's a free option that lasts forever with it but I really enjoyed uh learning this and let me just say what some next steps are for me I'm going to continue to use co-pilot alongside my projects I like to see how it can autocomplete things and if it doesn't work for me I can just skip it I'm kind of interested in the tech as it evolves as I as interested I am in getting Solutions so it's fun for me to see it even when it makes mistakes so enjoy playing with it and if you have any comments about what's worked for you with AI and gaming please let me know in the comments below thanks

Channel: Samuel Asher Rivello

Views: 2,142

Keywords:

Id: 9g0UMFUp4ys

Length: 31min 43sec (1903 seconds)

Published: Sat Oct 28 2023