Making ChatGPT in Unity!

last quarter in 2022 was a wild time for AI especially image and text generation applications gone viral so many achievements happened in so short time that we already started using some of these applications in our daily lives and what I think is we haven't even seen enough practical use of this Technologies just yet and what I believe is one industry that is going to make Incredible use of all of these Technologies is game development and today I want to show you how you can make use of openai's API in unity game engine and create an image generator and a chat GPT like chat application that you can directly use in unity game engine and you know contributed to your games already so without further Ado let's take a look at that first thing we will have to do is creating an open AI account you can start with a free account to learn and test openai's capabilities first go to openai.com and click on the API button on right top side click on the login button on the corner again and create yourself an account you can either provide email and password or login via Google or Microsoft accounts once you are logged in you will be greeted with tutorials and examples page on how to use openai API and feel free to wander the examples and get inspiration and learn more about how you can make use of it there's even a playground page where you can test different things what is most important for us is the documentation of openai I follow this talk strictly to write the unit developer myself for this video you can read more about the text completion and image generation requests here and see examples about how they work all of them are in the documentations click on your profile and click on view API Keys button you can go to the page where you can find your API key which we will use for making these requests you will be able to see this key only once so make sure you saved it before closing the window when it shows up and please do not forget this secret key is very important you should not share it with anybody and you should only use it on server-side applications uh actually using it in unity is not something favorable we still will keep the key in your local but if you publish an application just like this uh third parties might interfere and get your API key and if they use it you might face huge bills because someone else is using your API key so please be careful this example is just to test and learn open ai's capabilities and see how it works in unity this package is to demonstrate of course how it works so please be cautious and finally there is an uh organization ID which you can optionally use if you have multiple organizations in your account so what we will you definitely need is an API key and optionally an organization ID first let's start with creating a Unity project I will pick a 3D Unity project with version 2020 this should work with any Unity version that has package manager and here is my GitHub account where I have open AI Unity repository you can also follow through the readme file and see how this works copy the GitHub URL that ends with that git here when you see clicking after clicking the green button and come back to Unity now go to window menu open Package manager then wait for it to finish refreshing then click on the big plus button uh it will show three options as or four depending on Unity version select the add package by git URL and paste URL we just copied from GitHub click add button then it will fetch Unity package and install it you will see two sample projects in package manager one is a chat jpt like example and another one is image generation example we will check both of this examples so make sure to import them in the package manager itself and on GitHub you can check the code and the summary of the functions to understand their use better also do not hesitate to go and check open AI documentations along with it so let's open the chat GPT example and test it but before we will need to set our credentials locally otherwise results will fail since we do not have an API key in the requests go to your device's user folder in Windows it should be users slash your username and in mac and Linux it should be the tilde folder here create a DOT open AI folder which is the general Convention of how to store local credentials inside the dot openai folder create a Json file called oauth.json make sure this is not a text file but a Json file sometimes your operating system might hide extensions and you might end up with a auth.json.takes the file so better to check that twice file and folder names matter since the package will try to read it from exactly there if you want to change those make sure to edit the configuration script so you find the right folder structure of the Json should be like how it is in the readme file we will have an API key and an organization field in this file so organization is of course optional now go to your openai account find your API key and create one and copy the value and also you can get organization value organization string and save it as well paste these 2 into quotes and save the file again do not share your credentials with anybody and make sure to use them only in server side code all right let's get back into unity and open the GPT sample scene and run it so we have a text field a button and the text area I'll just type Hi how are you and we get the answer I'm good how about you I'll say I'm working on a unity and open AI tutorial let's say tutorial video and replies wow that sounds really cool what have you learned so far and I will reply I learned that I can use open AI in unity and it's pretty cool send it and that's really cool what kind of things you do with openai and unity I can complete text and generate images from prompts let's send it wow that's really impressive what kind of applications do you see for this I believe many games will use this for um for example for um let's say MPC uh creation storytelling and and so on let's send it that's really great idea what kind of games you see it just keeps coming with new uh answers to you and what I want to break down is I want to tell you how that actually works so the basic thing behind this is actually text completion you can go to documentation and check what text completion is you give an input and the AI figures out what is the most likely answer to complete that text width to be to be chat history like a memory there and answer things uh this is the structure we will go for let's make it larger so if I just said Hi how are you I would just get um you know answers like what a nice State because think this way it completes the sentence how are you could be completed fit what is your name so to begin with I have this instruction act as a random stranger in a chat room and reply to the questions so I don't want it to just complete a random sentence but I want a question answer like structure so I have my um act as a random stranger in a chat room instruction then I have a question Then I then I add my input right after that and after adding my input I add the answer area so imagine chat GPT will actually complete this sentence Exodus random stranger in a chat room then Hi how are you then as for the completion so the reply we get for the answer is what chat GPT will give us all right then let's say it replies us with so let's say I'm saying Hi how are you then it will reply with fine how about you then after receiving the answer from chat GPT with the completion API which is fine how are you this is what I get back I appendic question field then I get user input from the input box let's say my name is Sarge and right after getting the input I add the answer area then again I sent this whole thing back to the completion API so it reads the same thing again but with the its answer and my new answer and we ask it to complete this then it will say nice to meet you nice to meet you Sarge so now it has some history of my name I will append a question then I will add the user input which let's say in this case is what is my name and then I will append ask for area and then I will wait for it to complete this the cool thing is now we have a chat history I will already send this whole text from the context it will know what my name is and most probably it will reply that your name is Sarge so this is how chat GPT can keep it on the context so it doesn't forget what you discussed but keeps going uh refers back to the what you talked before this is how this works you can keep sending the whole history of the chat and it's gonna feel like it knows what you discussed before what I guess the trick with that is this and this is how it works so I made this a basic Loop for this you can alter the instructions up above and then add question answer question answer get the get input from the user get answer from the chat the completion API and pretty much run this so uh in the script initially we have this uh input Fields uh the button and the text and create I create a open AI API objects and I have a user input and instruction private strings the instruction is what we start with and it keeps expanding on it in the start method I bind the button and then every time user clicks the button Send reply takes the um the information that is inside the answer or the question you have and it sends it to the function to be processed first of all I get input uh Fields text and set it into user input also append it to the instruction and then go one line below and add my answer area just like that so X is a random stranger yeah they had a question then I add my answer after the question then I add answer field and I clean my UI set the input fields and all lock them disable them so user do not interfere with those then I send this information to the completion create completion of open AI so I have a great completion request object it can take quite many uh parameters there the required one is model and prompt so we can already send them you can go to documentation and see what kind of models are there the um for text completion the best one is a text davinci003 this is the latest the best one so you can just use it the other ones are there as well maybe they are faster I do not really know to be honest but this is the latest this is also what we set here in the prompt we set the instruction the entire chat history that we are sending and Max tokens is how many verbs and punctuations you might get as a result so the default is 16 it's quite uh small to be honest if you go to documentation and check the description of the max tokens there is a link to that and we can actually test a text to see how many tokens it has so if I write how how are you question mark it is going to show me five tokens so it is each word plus the question mark and how many characters are there so this how that works pretty much guessing the next token of that sentence so uh the value I set in the example is um 108 128 so you can make it you know like 2048 or this is maximum and you would get like very very long answers so depending on what kind of applications you are making you can have different values there and there are so many other uh things uh you know like small uh tweaks and stuff just read the instructions read the details here and set them accordingly but prompt model and Max tokens should be just sufficient for us in this context you will receive a completion response um which contains the data we get back from the create completion endpoints so uh it has ID object created date model and choices uh this is the one that we are going to use choices is an array of uh data that's sent back to us the completion data and it has the text field in it this is the most important thing for us so what we want to get is the first choices text so after we receive our answer back from openai we picked the first of choices and the text and set this in the text area so we can see the ai's answer to us see the answer and also we append the answer into the instructions and go down with Slash n and add our question so enable the UI so we can give something back to the system then again we fill the field fill the field click on the button it's appended as for goes there we ask the machine it gets something back to us and the chat history increases and increase and increases on top of that for example you can do edits like correcting the sentence that user provided or add moderations like if the user input was some kind of violence hatred sexual content so you can actually block those kind of things and give Ready Answers this is also possible in this basic example we pretty much have no protection of those but you can definitely add on top of it so the chat example is pretty much this and so uh we can switch to the Dali example the image generation it's also quite basic to be honest more simpler than the chat example in the documentation we can go to images tab and we have three options image Generation image edit and image variations you can upload one image so you can give a prompt and create an image you can provide the image a mask to edit that at the mask with transparent area to be edited so the transparent pixels will be changed with the prompt you provide and variation provides you different versions of the image you generated for the image edits I have some examples uh images in the test folder and also image tests you can take a look for example a image I want to edit might look like this and The Mask I provided is this so the center area is a mask that I want to change the pixels with the test I wrote for this was adding a flaming odor which is the in there's a similar thing in the documentation so I use the same approach and that is that in this example it just creates the image again we provide the prompt uh you click on the button wait for the image to be generated the example text in the documentation was a cute baby otter so uh let's try that and let's send it it's gonna take a little longer than the text and then we should get the image so we have a baby order there and let's change it to um let's say Lego car so of course you can improve your descriptions as much as you like so I want to go with basic things and quickly get something here is our Lego car and this is very simply how it works let's take a look at the script the setup is pretty similar to what we've seen before this time we have an image field and create an open AI API object setting the button and then cleaning up the screen and locking the buttons so user cannot spam the service with requests this time we make a create image request to open AI with the create image request object not here so let's go to the uh the dial e example create image request object and let's scroll down so let's open this region about image API data types and then we have the base that has all the details and the what image request has extra is prompt so and is how many uh objects we get back size is the size of the image we have three options 256 512 and 1K resolution images and the response could be either a URL or a base64 Json so these are the other options you have for receiving the results that the default is the URL this is all we need so we get the prompt from the input field and set the size I want it to be quick so I set the smallest resolution we make our request and we get the data back again you can get multiple uh answers depending on the end value you set I will pick the First Data check UL depending on the prompt you provided there might not be a response at all so that's the reason we do a null check and I create a Unity Web request to download this URL and wait for it to finish once it finishes I create a new texture and then load the image from the byte array data that we received from the download Handler create a new uh Sprite using this texture and set that Sprite to the element we have in the UI so set this to be done and again if no data received we can just log warning about no image was generated right after that we enable the items in the UI and we can go back so this is that simple so this is how these two examples work let's finally try to super Batman I don't know so that works that simple you can you know try directly this in unity it might be you know like for anything creating uh player backgrounds or profile photos or stuff like that you imagine it and so this is pretty much that the uh the GitHub repo is a available publicly and thank you so much for uh watching I want to add like a second video to this um making uh some kind of a uh MPC Chat thing with red player me and hopefully I can provide something like that to you guys next week and you can see how that would be in action with Trader player me up until then thank you so much and see you soon

Channel: Sarge

Views: 22,264

Keywords: openai, chatgpt, ai npc, chat gpt, open ai, game ai, dalle, dall-e, ai art generation, unity open ai, unity ai, unity game ai, stable diffusion, stablediffusion, midjourney, gpt3

Id: MQfVCY9qgEU

Length: 23min 30sec (1410 seconds)

Published: Sun Jan 08 2023