Build a 3D AI School with Three.js, React Three Fiber, Blender & OpenAI | Beginner-Friendly Tutorial

[Music] should I learn JavaScript learning JavaScript is definitely beneficial especially if you are interested in website development web applications hey everyone in this tutorial we are going to build a 3D AI School the idea is to create a virtual classroom where you can interact with an AI teacher you'll be able to ask questions through text or speech and the teacher will respond with both text and voice it's an interesting way to explore how AI can be used in education and to create an engaging learning experience building this project will also give you practical experience with some powerful web Technologies for this tutorial we'll be using 3js for 3D Graphics react 3 fiber to integrate 3GS with react and open AIS CH GPT API that you already know or the AI conversation capabilities so let's start by understanding the key tools we'll be using 3js is like a toolbox for creating 3D graphics on the web it provides a wide range of tools for creating and displaying 3D scenes in the browser it's very flexible you can create simple scenes or complex detailed worlds then we have react through fiber and that is a way to use 3js within the structure of react application it's like a bridge between the world of 3D graphics and the world of react components then there is also dry which is a collection of helpful add-ons and abstractions for reactor fiber that will make certain tasks easier like for example adding lights Shadows or loading the 3D models Etc it's like a set of utility functions that you can use to enhance your 3D scenes but you might ask why then use react 3 fiber over plain 3js well first of all react is very popular and also recruiters are looking for react developers so you have an easier time to look for a job it has a great Community it's a great Library maintained by Facebook but also with react and in this case react 3 fiber to use the 3D Graphics we can leverage the component based structure of react because this makes our code more modular and reusable and easier of course to understand it will abstract away some of the complexity of 3js allowing us to focus more on creating our 3D scene so this is the reason that we are going to use reactive fiber even though I really love also playing 3js but if you are looking to work in a react envir this is the best choice throughout this tutorial we'll use these tools to build our virtual classroom step by step we'll create the classroom environment we will add interactive elements uh we'll bring our AI teacher add animations to it uh to play it uh dynamically interactively and then set up the AI conversation capabilities I have structured this tutorial to be accessible even if you are new to these Technologies so we'll start with the basics and gradually We'll add complexity each part of the tutorial will include explanations of key Concepts uh live coding because we will code live and I will try to add visual ads when possible to help you understand okay so by the end of this tutorial you'll have a solid grasp of 3js reactor fiber and how to integrate AI capabilities into your web application using open AI API actually I wanted to use uh cloud cloud is another uh language model like chat GPT and uh I'm very happy with Cloud especially with a premium version which is I think cloud Opus yeah Cloud Opus and to be honest uh in some cases it performs way better than chbt at least for me for coding challenges and uh I really challenged uh VII because I tried with uh very complex problems that are very specific so it's not just collection of information to and the ability to to respond like grabbing some information but also using a bit of logic uh specific things that maybe you cannot find everywhere and Cloud I don't know but it adds context it understand the context uh it remembers very well the earlier conversation and has like a humanik uh way of expressing and maybe I will continue to use cloud more than chbt but anyway since chbt is way more more popular than the anthropic Cloud ah I thought that okay let's use chbt for this because maybe also there are more users of chbt and they are familiar with it but yeah I wanted to to say that I I really like this cloud and I wanted to use that but anyway this is not uh an advertising on anthropic cloud is just my personal opinion okay so let's get started and in this part we'll set up our development environment and lay the foundation for our 3D AI classroom all right let's dive into the coding part of our tutorial first make sure you have the necessary tools installed you'll need node.js and the code editor like Visual Studio code okay please install them so go to Google search for nodejs download install also the same thing for code editor like Visual Studio code so after installing node.js and visual studio code let's set up a new project use invite which is a build tool to create react projects or vue.js or JavaScript projects it is very fast and it's better than other build tools so search here Vite getting started click the first result and here you will see the documentation and here in the getting started section scroll down you see this short command mpm create VI latest copy this this is to create a new project with the latest version and then you will choose if you need a react application or view or JavaScript applic you will see yourself copy this go again to a Visual Studio terminal open the terminal paste this npm create V latest But first you should choose where to create this project for example I want this project in my desktop so navigate to the folder that you want to create the project CD desktop in my case make sure I am in the right directory okay I'm in the desktop now I can create the project so round the command npm create Vite at latest so add a name to it uh I will call it simply 3D school then choose the framework or the library here for example vanilia JS view react we will choose react then choose a variant typescript or JavaScript we will choose JavaScript and we are ready so just jump into this folder CD 3D school now let's install the dependencies the dependencies are the node modules so the external packages that we need to be able to run a react project okay using V so let's say let's set up the stage the scene of our project with all the actors and what we need the packages so all these external packages will make sure that we build our react application with all the useful packages let's say say we set up this stage this scene with all these actors that will make sure that the play will be good okay so for that mpm install or shortly mpmi wait a few seconds but is fast so you won't wait for too long next we'll install the necessary packages but in this case for 3js and react 35 because we installed with mpmi all the node modules that usually what we need to set up a new project but here we need the packages for 3js and react 3 fber and additionally dry okay that we mentioned at the beginning of our video that run this command mpm install three which is to install the three library then at react -3 fiber to install fiber and also react three dry okay make sure to run this command correctly wait a few seconds we now have the three and the react 3 fiber and dry packages okay let's Sol the project that we just created desktop uh 3D school okay you see we have now this new files and folders we are not going to explain each file what it means because yeah you have everything that you need to know on internet just find a good tutorial on the very basics of react what is uh for example main JX um what is the use of appjs X what is packet Json so you should know already the structure of react project anyway we are going to explain everything step by step the code that we write and everything else but we are not going to stop and discuss each of the files or the structor or you know these details because you should know by now but if not you have like thousands if not millions of tutorials out there on react but anyway I will say that this is the main folder that we will use and put all our components so SRC folder the public is to put all the static files the media the 3D models Etc so this is the folder we will clean up a bit here the index files is the file that renders in the browser it's a simple HTML file which has this IDE of fruit that react Target this ID of root in the main jsx here you see get element by ID root and render okay more in depth you know you can find the tutorials online first of all let's clean a bit for example this styles for up jsx delete them the global Styles in index CSS also delete them I will replace it with some default styles that I have simple Styles just to make sure that the Cana is like full width and height and some really basic Styles okay you can just copy and paste find the link in the description to the GitHub repository and just copy and paste we also will delete this appjs main component of a react application we will delete everything here because we will build our application from scratch okay maybe here in the main ja just now we need this index Styles application okay this is okay create a new react component here I have a shortcut so rfca enter it will create a new react component with export default I don't like this Sint uh warnings so I will disable them you can disable like this like click and uh disable for the entire file or you can add this to the Sint file for example take a look here in the asint file here uh where it says rules you see there are some rules string add this no used bars strings again and off comma here and now if I delete this I will not get the warning again do this for you know uh you need this as lint warnings but this time we don't need them now that we have this main component is just a simple component okay we can also delete this react from react it just returns a new div with a text app you see this app here is just simple div all right let's break down the concepts and the code in this react component in a beginner friendly way for this tutorial okay please if you are not a beginner but you know these things better bear with me please and skip this part for the other beginners please follow me in react a component is a building block of the user interface it's a self-contained piece of code like this one that describes a part of the UI this component is just a part of the whole application components can be thought of as reusable pieces of a puzzle that put together to create the whole picture in this case your application okay so small pieces of a puzzle to create the whole picture components in react are typically written using JavaScript functions like this one or classes but we don't use classes anymore in react so you don't really need to know the classes components anymore they can accept input data hold props accepting them here in the parenthesis like normal JavaScript arguments because we suppose that you know the basics of JavaScript okay so these are called props and are input data and this react component returns react elements that describe what should appear on the screen in this case we simply make appear a simple div with a text up okay so returning this div element the beauty of components is that they allow you to break down your application into smaller manageable Parts like this one each component can have its own structure its own style and behavior for example you pass here file that has the styles for example the app CSS import [Music] slash up CSS and then this component will have the styles of this and usually you put it in a separate folder so call it for example up and then inside you have the jsx component and the style component we can maybe explain also what jsx extension is so we said that this component can have its own structure style and behavior and we can then compose these components together to build a complex user interface here in this function app which is the app component let's take a look at the code this is a simple functional component in react it is called a functional component let's break it down line by by line function up this line declares a JavaScript function named up simple as that in react you can define components as functions the naming convention is to use Pascal case for component names okay you know what Pascal case is so for example up and again uppercase component okay the next line return development with a text up inside it this is the body of the up function it returns a single jsx element which is a div with a text up inside for the jsx to explain it what jsx is jsx is simply a syntax extension for JavaScript that allows you to write HTML like code within this JavaScript okay so jsx is a syntax extension for JavaScript that allows you to write HTML like code within your JavaScript so in order to use this HTML like code like this div inside the JavaScript code you use this jsx syntax which react uses actually you can use it outside react as well not only in a react application but jsx is popular because it is used in a react application so in this case the app component is simply rendering a div element with a text up this is what will be displayed on the screen when this component is rendered and to render it we said that you simply run the command npm run Dev you will run live click the URL and you will see the result you see we are entering this app text which is a simple div with a text up inside and that's it now the last line export default up this line exports the app function as the default export of the module it allows other parts of your application to import and to use this app component when you have a default export you can import the component using any name you choose for example here let's say in another component you see here import up from Up jsx but you could also use import my app from app jsx and here also we have error because you should also change here okay and is completely fine so you can choose the name here if it's a export default something else is if you if it's not a export default but it's export function up in that case here you will import it like this with curly brackets import up from abjs X okay usually we use export default for the main application here and the other components you can use export con export const up etc etc so to recap this app component is typically the root component of your react application it serves as the entry point and can contain other components within this okay to build the complete user interface as we will continue with our application you will create components like for example environment which is the component that we will create now and import it here in our app component and other components as well remember this is just a simple example of a component because as your application grows your components will become more complex accepting props as we said um maintaining State and defining more intricate rendering logic by but for now just remember this what we said that each component is puzzle piece to create the whole picture now first we create a new file called invironment jsx in the SRC folder this file will contain our 3js scene using react 3 fiber okay so in SRC create a new folder name it components and inside create this file environment. jsx again create a simple component first RFC and inside of this we don't want this simple div but we will import canvas that I will explain in detail what it is again this ASL warning I don't want this warnings so grab this go to Sint file rules off okay leave me alone as lint I know what I'm doing or I think I know what I'm doing environment JX so to import this canvas you can do it like this you see here the suggestion canvas from react 3 fiber okay you get it now we have this canvas component and I will explain now why I imported it what is canvas and why we need it so in react 3 fiber the canvas component is the fundamental building block for creating 3D scenes okay it sets up a 3js scene a camera we need a camera and a renderer we are going to explain them so a simple scene is built with the 3js scene a camera because you need a camera to look at the stage to be able to render or to film it okay think of it as you are like the director and you are making a movie a good movie I hope and you need a camera okay to film it to look at the stage so a simple Scene It has the stage the scene a camera for the director and a renderer a renderer is like the snapshot of the camera okay so you need to snapshot that what you are looking at the camera for now this simple explanation for the renderer and for the camera but let's continue with the meaning and the use of canvas think of the canvas as a container where all your 3D content will be rendered okay it's like a blank canvas waiting for you to paint your 3D scene on it the canvas component takes care of the complex setup and rendering process behind the scenes allowing you to focus on creating your 3D objects and scene composition it's way easier here in react 3 fiber than in pling 3js in playing 3js you have to do that all manually okay so the rendering part the camera setting the scene setting etc etc and also the updating which should do it with the animating Loop function in 3js but we don't need to discuss in reactive fiber these things okay we have convas and we have what we need under the hood the canvas component utilizes the power 3js as I said for displaying this 3D Graphics in the browser and 3js provides a wide range of features and functionalities for working with 3dcs so 3js provides geometries materials lights and more that we will discuss step by step but you should know that by using react 3 fiber and the canvas component you can Leverage The declarative and component-based nature of react for building your 3D SS you say I want the canvas here and you get the canvas here okay simple as that I want a canvas here I want light let there be light and you call that component when you define this 3D objects as your reusable components you can compose them together and manage their properties and behavior like using react States and props okay as you usually do in your react applications the canvas component also provides various configuration options and event handlers allowing you to control the camera the rendering settings and interact with the 3D scene using Mouse touch or other input devices okay so this is a brief explanation of the canvas in react 3 fiber okay I want to check a moment and dive deeper into this main concept of 3js because we said that we import the canvas into our real active fiber application and we have the full option with the renderer the scene and the camera inside so we don't need to set up anything like the scene the render Etc so we just import the canvas and we are good to go but what really happens under the hood and what is the real meaning of these components so I would like to go deeper into this Concepts because with the exception of the canvas which actually is specific to the browser uh an equivalent to each of these components can be found in any 3D Graphics systems so it's very important for you to learn this Concepts making your knowledge that you will gain highly transferable okay so let's go deeper into this components at the heart of every 3js application we have these three essential components that we mentioned the scene the camera and the renderer this forms the basic scaffolding of our 3D World the scene that you see here at the center is our tiny universe where all our 3D objects reside it's a container that holds everything we can see in our 3D space here you can see that the mesh is what the scene can hold and this mesh can be anything we saw some basic geometries in 3js we create a scene using the scene class which is imported from the three module like we imported canvas from react 3 fiber we import the S class from the three module if we are in a 3js project the scene defines a coordinate system called World space this coordinate system serves as our main frame of reference for positioning and manipulating objects you see here in this picture the center of the scene is the origin represented by the point 0 0 0 whenever we create a new object and add it to the scene it will be placed at this origin by default so every object by default will be added at the origin 0 0 0 we can move and position these objects within the world space coordinate system additionally when we add objects to the scene they are placed into the scene graph which is a tree likee structure with a scene at the top as you can see this picture this hierarchical structure allows us to organize and manipulate objects in a logical and efficient manner you see the scene at the top and then the mesh inside this scene in a tree likee structure does it look familiar to you because this s graph is similar to how elements on an HTML page are structured you see at the top the dock type HTML then we have the body The Head and the divs or H1 elements this is similar but except that the HTML page is 2D not 3D while the scene graph is 3D so we have this scene that holds all the 3D objects and then we have the camera if the scene is our tiny universe the camera is our window into that world it's like our eyes determining what we see and how we see it 3js provides different types of cameras but the most commonly used is the perspective camera and the one that we will use in our tutorials this perspective camera simulates the way our eyes perceive the world and it uses concept like field of view aect ratio and clipping planes that I will explain in detail in a moment so there is this scene that holds all the 3D objects we have the camera that are our eyes looking at the scene then we have the renderer which makes possible to render and display what is in the scene in the canvas think of it for example the scene is my screen screen the screen of my computer I'm recording my screen with a camera and then I have a render which helps to display my screen record to your computers now let's go back again to the camera to explain a few things look at this picture I mentioned that the perspective camera uses Concepts like field of view aspect ratio and clipping planes here you can see that this perspective camera will will check several parameters the field of view which is this angle in the picture that determines how much of the scene is visible you see this angle starting from the camera and going at both sides that is the field of view then we have the aspect ratio which is the ratio of the width to the height of the view ensuring this scene looks proportional okay then we have this near clipping plane and far clipping plane these Define the range of distances from the camera within which objects are visible for example you can see that these two black objects are not visible because they are outside the clipping planes but adjusting these parameters we can control the camera's perspective and what is visible to our scene so the field of view affects how wide or narrow the view is the aspect ratio ensures the scene looks proportion the near and the clipping planes determine the range of distances within which objects are rendered last thing I want to mention is the camera's view in frr is like this pyramid that you see from the camera to the far clipping plane let's see a picture of it the cameras viewing frrom is the region of space that is visible to the camera it's like a pyramid with the top cut off defined by the field of view aspect ratio and clipping planes that we saw in our previous picture only objects within this FR room will be rendered okay so keep this in mind sometimes where you are debugging and not seeing objects in the scene this might be a reason so check it out next we have the renderer the renderer is responsible for checking the scene on the camera and rendering them onto a canvas element in the browser in our case for our web application 3js provides the webg renderer which leverages the power of webg to render 3D Graphics efficiently because as you might know 3js uses the power of webg and that's it for these three main components scene camera and the renderer we have the mesh here at the bottom which we explained briefly previously that the mesh is a visible object in our 3D scene consisting of a geometry which is the shape and a material which is the appearance okay remember this the mesh consists of a geometry which is the shape and the material which is the appearance the look of it we create meshes and add them to our scene to populate our 3D World instead of meshes we can also add custom models like in our case we will add a 3D classroom moduel and also the teacher and other 3D moduls but these are called primitive geometries let me show you some real examples of how camera works Works in 3js for example see this the comparison of the perspective and the orthographic camera you see the difference here is like in the perspective camera how the user sees the scene and here is the orographic okay let's see another example here we can tweak these parameters that we mentioned before like field of view the near clipping plane and the far clipping plane to understand how it works for example here we have the value of 45 let's increase the field of view you see the field of view is that angle range that we saw in the picture and the object must be inside this field of view and let's try the near clipping plane we cannot see here so the near it's best to keep it like 0.1 [Music] one and the far is 100 we can also add the value of 1,000 sometimes so we make sure that we can see the whole scene you see if we decrease this value we don't see anything because the objects are outside of this far clipping plane as you saw in the diagram that I showed you previously I hope it helps you understand better let's see another example here take a look at the right window you see if I move this window at the left you will see how it looks at the right with that field of view and that frrom you see the frrom the camera's viewing frrom that we talked about a while ago and everything that we render is inside that fruit Strom that pyramid shaped with a top cut off if I zoom in you see the near for example I zoom in you see this near clipping plane now I cannot see this this Cube because it's outside of this near clipping plane if I go back okay the same if I zoom if I zoom out a lot you see this far clipping plane you see now the scene is outside of this far clipping plane I'm sure that watching this is very easy to understand the concepts then while coding you know is just an abstraction okay I hope it helps for any questions you can always ask in the comments and we can discuss even further now let's see how this Concepts relate to reactive fiber check a look at this diagram react 3 fiber allows us to declaratively build 3js scenes using react components it abstracts away some the complexity and provides a more intuitive way to work with 3js for example take a look at the diagram here we use the canvas component as a container for our 3D scene the canvas component automatically sets up the scene the camera and the renderer behind the scenes we can think of it as a pre configured environment where we can start building our 3D World inside the scene as you can see we have meshes and lights which are the visible objects in our 3D World the meshes are composed of geometries which as we said is the shape and the material which as we said is the appearance the look of the module reactor Viber also provides the dry as you can see in the diagram which is a library containing useful utilities and obstructions we used a lot of utilities from dry so now you know that these components correspond to the equivalent 3js classes but with a more declarative and react friendly Syntax for example instead of manually creating a perspective camera instance we can use the perspective camera component provided by reactor fiber and we can set its properties like field of view aspect ratio near and far okay so for example aspect or near or far okay you can also add settings of the camera here in the canvas component like field of view near and far etc etc similarly we can create meshes using the mesh component and Define their geometry and material using the geometry and material components respect effectively so by leveraging the power of reacts component based architecture and the declarative nature of react 3 fiber we can build complex 3D scenes in a more modular and reusable way as you can see yourself that's it for now and remember the key check ways are the scene is our 3D Universe the camera is our window into that Universe our eyes the renderer brings the scene and the camera together and renders them onto a canvas the meshes are visible objects used in our scene consisting of geometry which is the shape and material which is the appearance and react 3 fiber provides a declarative way to build 3js scenes using react components these are the key check ways that you should remember and understand I hope this explanation helps you understand the fundamental components of 3js and how react through fiber simplifies this process of building 3D scenes okay so since the 3D model of the classroom is ready to be downloaded you can find the link in the description of this video or go to the repository on guub now let's create the 3D avatar of the teacher I want to take a moment here to introduce you to an exciting platform called avatar. me and aurn is a web- based tool that allows you to create create personalized realistic 3D avatars using just a selfie these avatars can be used in a variety of applications from marketing campaigns to games and metaverse experiences what I really appreciate about Avatar is how user friendly and accessible it is the process of creating your own avatar is straightforward and fun first you just need to register on this website and verify your account and to verify the account you will be asked to verify it on their Discord server actually I encountered a small issue with the Discord verification I don't know why Avatar also offers the verification directly on their website so if you have problems verifying your account on their Discord you can also use directly the website as I did once you registered and verified creating your avatar will be a breeze simply clicking here a new Avatar okay as you can see you have two options here use your camera or upload from device and I suggest to use your camera because I experimented a bit with this and I found that using the camera I get the best result but okay if you have like very nice pictures of yourself like not only in the front but also the left and the right side of your face then maybe you can get a very good result too but I suggest to use the camera it's very straightforward you will take three selfies front right and left okay after posting these pictures of yourself here you will have the option to customize your model and let's take a look how cool it is I have to say that the level of detail and realism in this avatars is truly impressive you can customize various aspects of your avatar such as body type hair cloth sunglasses and also you can add animation in the end the resulting Avatar looks incredibly lifelike and captures your unique appearance try it yourself I for example chose this body type like I've been to gym the last months because why not choose the hair here or a hat if you like there are nice clothes here for everyone example this nice shirt okay this one let's try this okay this is like a jcker Assassin's Creed it's very fun to customize this 3D avatar headsets because it's trendy now uh let's choose the shoes here this one or maybe this and in the end you can choose also the animations this is one of the coolest features of Avatar this ability to add animations to your avatar you see you have many animations you can choose from okay like smoking a cigarette I think so when you are happy with your creation simply click this download button and your avatar will be exported as a glb file this file is widely compatible with various 3D applications and is really optimized for the web making it easy to integrate the Avatar in our projects in our case we'll be using the downloaded glb file and enhacing it further by adding other custom animations in blender using mixamo in this case mixamo has extensive library of animations available there and they are free to download I'm excited to see what unique and expressive avatars you will be able to create but now let's go and open blender I think you had no problem downloading and installing blender blender is free and open source and will always be and we should really appreciate that this tool which is one of the best tools out there is free and open source my favorite tools are 3js unity and blender so I am a blender fan booy anyway let's delete this default Cube as everybody does each time they open blender where do these poor cubes end up after they are deleted so here in uh this window collection delete hierarchy okay we deleted everything now let's import our model that we downloaded from avat churn so here up at the left file import then gltf 2.0 go to the folder where you have the downloaded file downloads and here model. glb you see now we imported our Avatar it's gray now because you should click here this icon viewport shading okay now you can see now let's go to mix.com and let's download the custom animation so that we can add it to our Avatar so visit mx.com uh register and login to this platform and here to be able to download custom animations to add it to our Avatar we need a character that has the same scale is our character I mean we need to upload a module that will have the same skeleton as our Avatar because if you upload your uh 3D model of the Avatar you may may end up in issues and we don't want that so for that I have ready to be downloaded check the description links and download that file which is uh called mix something blah blah blah so download that file okay and then upload it here where it says upload character so upload character as you see it accepts only these types fbx obj and zip select character file go to the downloaded uh mixamo mesh that I put it in the description I should have it here okay then loads file okay this mash for mix Amo and upload it here let's wait a few seconds okay it's ready okay check if it's everything all right we don't see any issues so next your 3D character blah blah blah okay now we have this character here and we want to add some custom animations let's search for example idle I don't want this fight fight idle okay maybe this one simple one okay maybe the position like he stands I don't like it but let's try this one okay this one here after choosing the animation that you want download it and here in this settings without skin so skin without skin frames per seconds like let's make it 60 and this key frame reduction now as it is okay it will be a fbx format download do this with the other animations so download for example I don't know what to make our teacher walk no not this working uh maybe I don't know this one okay this looks okay download also this one you have some other settings here like mirror or you know you can play with this settings so again without skin 60 frames per second download after downloading all the animations that you need let's go back to blender now back to blender the first thing let's rename this Armature to a milum for example or the name of your avatar then let's import our animations that we downloaded from mix.com so file import and fbx because they are fbx files navigate to the folder that you had the files and let's start with the idle here you can see this armatore one is the new animation that we added to the scene now select your avatar here and then at the bottom at this editor type button click it and select this dob sheet enlarge it and here you should choose the action editor and here in the middle of this window you see this button here where where it says browse action to be linked click on this and you will see the animations that are associated with your avatar I have two here because this other idle V4 blah blah blah is the animation Idol that I exported with my avatar from Avatar because on avat remember I selected an animation and when I downloaded it there are several options there download with animations or download for blender etc etc I downloaded with animations so this one is from Avatar this is the new one that we downloaded from mixamo so select the new animation that you have here and change this name to something that makes sense in this case idle if you press the space key it will play this idle animation you cannot see it maybe because it's an idle animation so he is not moving a lot but it's there press the space key again and it will stop now we want to add other animations and for that here in this editor type choose this nonlinear [Music] animations as you can see here in this ailon which is our Avatar we have these two animations associated with it the idol the new one from mixamo and the one from avat turn okay to add a new one click this button here and arrow down like push down action click it and we added a new strip here after adding this new strip we can delete this animation that we imported in the scene so delete hierarchy now we can import the new animation import fbx then this for example thankful we have it here here where you see this armateur score one which is the new animation change the name again to something that makes sense and it was thanks here where it says no action here at the right where browse action to be linked button is choose the thanks okay now you can see the thanks is here if you press the space key you will see the animation playing so to add another animation the same process again click this push down action button then you can delete this animation here and again import a new One Import fbx walking okay this walking animations seems to have uh uh some issues but doesn't matter the same process here change the name walk then here where is no action click the button here at the right and choose walk now you have it you can play it it will walk with some small issues but doesn't matter it works the same process if you want other animations click this button here push down action delete the animation here and import fbx Etc after you are happy with the animations that you want for this Avatar let's export it in a GB gltf format which is is optimized for the web so file export gltf 2.0 this is red because I think I have another file with that Avatar so just new Avatar okay I downloaded this now let's go back to our application and let's use this new Avatar let's continue with our tutorial uh we will get back to code and add our 3D models instead of the Box geometry we have two 3D models at the moment the 3D classroom and the 3D AI teacher that we made with Avatar blender and mixamo for the animations let's get back to code so I have imported the modules here in the public folder public modules and we have this classroom GB and Avatar GB to import them into the 3js SC it's very easy the first solution is very easy so delete this box and here let's import this gltf component from react 3 dry and here let's add an attribute SRC and insert the path of the moduls so for that when it's in the public folder with v you just type slash models so you don't need to add the public models and then classroom. glb so we imported this gtf from react to dry and this component makes possible to load the gltf files or glb which are basically the same thing here I will add some other properties like position and rotation so we make sure that this the classroom displays correctly because I tested this and I know the exact values that the classroom will display correctly because sometimes you might have I don't know the model has a scale is too small or it's too big or the position is not correct so you should play with these values if you see problems in the scene so for the position you open this curly brackets okay so when you open the Cur brackets in react you you can put a JavaScript code because to put Javascript code in this HTML like syntax which jsx is you put this curly brackets okay and to add uh the values of the position uh the values are three values for X Y and Z so we need an array for that to add three values so array and then for X zero for the Y minus 8 which I found out was correct for me and for the Z zero similarly for the rotation so rotation curly brackets we need an array for the three values for the X zero the Y in this case I want 180° or half a circle so I want to turn back that classroom because when I first loaded it it was in a wrong direction so I want to turn and rotate 180° or math Pi okay and for the Z zero so math. Pi is 180 degrees or half circle you know Pi is the mathematical constant you already know that so math. Pi is equal to 180° if you want uh a full rotation you would need math. Pi * 2 or if you need 90° you do math. Pi / by two sry it's not lower case but it's uppercase let's take a look in the browser if we see something so are we correctly okay we are importing the environment into the app uh component so it should display correctly let's check okay yes it displays correctly you can see the classroom it looks nice the position is correct the rotation is correct so it works and we have our 3D classroom that's it is easy right easier than this I don't know now let's do the same thing for the teacher model so copy this down below change the path so it's a million Dash avatar. JB and let's change the position and the rotation but I will also add the scale here because I found out that the scale wasn't correct so scale 7 77 for the Three A's and this position and rotation values okay copy them and let's check it out okay we see now the teacher which is my Persona and we can zoom and rotate and explore and this is because of the orbit controls the orbit controls make possible to navigate and explore for the 3D scene okay but maybe we will fix this that I don't want to zoom out uh after a certain point and I don't want to zoom in after a certain point okay we can uh fix that part for now everything works we are rendering the scenes and it looks good maybe we will adjust the lights okay we need uh better lighting so let's do that first so here we have light the ambient light if we hide this light we will not see anything in the scene you see it's dark let's turn back the lights here let's add uh the intensity to 0.8 should be enough okay uh again this Sint errors I will disable them copy this go to sink file here off okay because yeah s lint does not recognize this property intensity and let's add the color for the light and let's make it white let's see the changes not so much uh of a difference to be honest but we can do better we can do better all right let's dive into the word of lighting in 3js and react 3 fiber lighting plays a crucial role in creating realistic and Visually appealing 3D scenes it can set the mood highlight important objects and provide depth to your scene first let's talk about the different types of Lights available in 3js let's start from the first one the ambient light which is the one that you are seeing here in the scene and ambient light is a basic light that illuminates all objects in the scene equally from all directions okay so it's not from One Direction it's from all directions it doesn't have a specific Source or direction that I mentioned and it doesn't cast Shadows think of it as a global light that uh provides the base level of Illumination to your SC for example we can tweak the parameters here just to see we have two properties of the light the intensity and the color let's try let's make it red or green and so on so it doesn't look much like real lighting you know it's like adding color to the scene but it doesn't really look like lighting so so that's why it's not the most useful lighting okay let's introduce another light which is the directional light which is the one that you are seeing in the scene right now the directional light simulates a distant light source such as the sun that emits light in a specific Direction in this case not like ambient light it illuminates all objects in the scene from the same direction as if the light rays were parallel directional light is often used to create strong directional Shadows let's play a bit with the properties here you can see we have color and intensity like with the ambient light but here we have additionally this X Y and Z uh directions of the light rays so if we play with it you see the lights and the Shadows changes in The Zed axis and in the Y AIS you can add directional lighting the same as with ambient light create a component here directional light we add the intensity like the same with ambient light but we can also cast Shadows ambient light doesn't cast shadows and also we can add this position for the X Y and Z okay then we have the point light the point light represents a light source that emits light in all directions from a single point like a light bulb the intensity of the light diminishes with distance creating a realistic fallof effect let's try it here in the scene we have the color also the intensity and the distance you see this effect based on the distance and we can change also the direction for the X the Y up above and the Z or the depth okay and you connect the point light the same way so Point light component here it cast Shadows it has the intensity and also the position so these are the main lights the most used but you also have uh the hemisphere light and direct area light but if you want to know more about these other two lights you can check the documentation because right now I want to introduce you to another way of lighting and that is the environment lighting that we get with a component named environment from dry let's try it so we import environment from dry Okay add it here we have this conflict of naming I should rename it this environment it's my fault so environment component change it here in the app component as well and let's use this environment the environment component in dry is a convenient way to add environment lighting and Reflections to your scene it creates a background environment that can simulate realistic lighting conditions and also Reflections on the objects environment maps are 360° images that capture the lighting and reflections of real world environment they provide a realistic and immersive lighting setup for your scene and you will see it yourself right now with our classroom okay so you import it here from dry and then use it here simply as a component and that's it and we can use a press set for it so press set we have City and other presets you can check on Google for the other presets of the environment but let's choose this environment per set CD and also we can add a color for example white let's check uh our scene you see the changes it looks better now we will add the intensity and also maybe change the color color of the light or we can use the intensity of this ambient light so the ambient light has this intensity color white and let's see now difference yes much better I should change the color of this ambient light actually for example red you see a red [Music] room pink a nicer look let's search environment light presets dry let's ask the copilot so the presets are this like Sunset Dawn night Warehouse City forest apartment let's try some of them so Forest but we are not in a forest so okay but just just a try out Forest it has like a greenish color but it looks good actually on our classroom because we have this a bit of greenish I like it though but we have the pink color for the ambient light so we have like a mix okay but let's try the others I'm curious so down night and sunset let's try the down o it's a bit too bright we can lower the ambient light but I it's a bit too bright yeah okay it's like the morning early morning okay let's leave this one so the classroom and the teacher looks good the lighting is nice the next step would be to change this uh T pose of our Avatar and uh add uh an animation and we have uh the animation idle so the teacher will play the idle animation and not uh stay in this T POS because this like default pose for for the avatars to play like an idle animation we will play the animation that we imported from [Music] mix and later with some interactivity when we click we can play another animation but before adding the animation I want to show another way to load these 3D models because the way that we are doing right now with the gltf component it's very simple but we might want to try another way because these models are glb file and uh we want to manipulate this later for example we want to get the animations or we want to get the materials we want to get uh the notes Etc so we need uh to Loop or Traverse through the properties of this glb file and best would be if we have like a component like a react component for this GB it would be the best and we can do that we can convert this glb file into a declarative jsx component okay how and what it actually means we will see it together now now in this component I'm going to explain line by line everything each line of code starting at the top we import the dependences that we need for this component what are important here are this use ref use gtf use animations which are called hooks in react so let's talk about these hooks after taking a quick look at the component which is a function component named model we can rename it later into something uh that makes more sense like Avatar or anything we accept props and then we return this uh Group which is nothing else but our 3D avatar so let's explain this hooks what they are and so we can go line by line and explain everything else in react hooks are like special superpowers that let you add extra abilities to your components they hook into different parts of your components they always start with the word use use ref use gtf use animations and later we will explore uh very important ones like use effect and use state I'm going to explain them when we are there so these hooks help you manage different parts of your component like keeping track of these values or to load the 3D models in this case and to dive into the specific hooks let's start with the use W here that we create at the beginning we Define a group variable and assign it to this USF a USF is like a special container that can hold value it's super helpful when you need to track of something that might change over time we want to get a reference to the root group The Root group is this one here the root group of our 3D model so we want to get a reference to this one why because we might need later to use this reference to do things with this 3D model okay so we Define this group variable equal to us F and then we add the attribute ref to group to this one so now we have the reference to this root group that's it for now for now all you need to know is that this user ref hook holds the reference to the main group of our 3D model later we can easily access and modify the main group whenever we need even if our component updates okay that's it for the US ref later you will have other uses with us ref but for now you just need to understand this this us rep Hook is to hold the reference to our root group of our 3D model then we destructure these nodes materials animations from this used gtf hook that loads our object from the public folder use gtf is like a magic function that help us load the 3D model into our component we give path and then it will load all the necessary information in this case the nodes which are the individual meshes and we talked about meshes what they are the shapes which are the geometries and the materials which are the appearance and the animations that we know what animations are so this line here cones notes materials animations inside the curly brackets equal to use gtf means nothing else than I'm packing all this important stuff from the loaded module to give us easy access to them okay so the structor unpacking all this important stuff from the loaded 3D model now we have another hook use animations from which we are destructuring actions and mixer this is another useful function that helps us work with the animations of our model we use it by writing the same way const actions mix mixer we unpack them from this use animations hook it will give us these tools that's it the actions are like individual animation clips and we can use them to play specific animations whenever we want so actions are individual animation Clips the mixer is like a puppet master that controls how these animations play and to make sure that all works together smoothly so we will use this mixer to control the animations to play them whenever we want that's it for the hooks here we return this whole group and let's talk about the group element in our component in a 3D World a group is like a container that holds multiple objects together it's like the box where you can put different things inside and move them around as one unit so in 3D word a group is a container a box we can put different things inside and then carry with us as one unit groups are super helpful when you want to organize and control related objects together for example you can move rotate or resize a group and everything inside it will follow along this is why we need them because everything inside the child of the group will follow along if we do something with this group so if we move the group rotate or resize everything inside will follow along so in our component we use the group elements to structure the model in a neat way like building a house we start with a main group let's say the foundation of the house and then we add smaller groups inside let are the rooms for example inside our house then we have this skinned mesh and and we mentioned where meshes are but why they are called skinned mesh and not simple mesh first we need this skin mesh inside the groups for the reason because by wrapping the skined mesh elements inside this group we can easily position rotate and scale the entire Avatar as one unit a skin mesh is a special type of object that represents a part of the character or the Creator or whatever it is in a 3D model unlike regular objects that are static and cannot change shape a skinned mesh uh is like a stretchy uh bendable uh action figure so it can move it can deform based on the skeleton that has inside it and the skeleton is like the bones we have the bones and the 3D model has this skeleton so we use the skined mesh elements to represent different parts of our Avatar in this case the different parts are as you can see avat turn body avat turn is from the platform so body hair look and shoes or legs arms head and other parts and inside each skined mesh has a specific shape which we said that is the geometry a specific color and texter which is the material and also a skeleton that defines how it can move and bend okay we saw that uh when we imported animations from mixan and that's it we use this SK mesh to create create uh the animated okay I think I mentioned everything here uh I can also mention the props and the dispose null so here we use this three dot props this is the spread operator maybe I should not go and explain the spread operator here I am sure most of you already know but there are also absolute beginners I know that so you should check the documentation and understand what the spread operator is okay because it's a concept in JavaScript pure JavaScript it's not 3js or react okay it's simple JavaScript but basically what it means is nothing else then we spread the properties object to pass any additional properties to this group so with this props here we pass any additional properties to this group that's it then for the dispose equal null this is another prop that is set to null to prevent the model from being disposed when the component mounts the component and mounts is a life cycle of the component the component firstly it mounts okay and then we can also unmount the component when we don't need it so so it mounts and also unmounts so to prevent uh this model from being disposed when a component unmounts we use this dispose null sometimes you use this dispose to dispose 3D objects that you don't need anymore to not load too much of the scene and sometimes you don't need to use the dispose because it can also be expensive for example create and remove create and remove it can be expensive when you want to remove an object for example sometimes you do other tricks like uh make it Invisible by setting its materials to to Invisible or also turn the lights off for example so other tricks that we might learn later maybe but we are doing simply this dispose null because we want to prevent the model from being disposed okay there is also this primitive object nodes. hips we have this primitive component that represents the hips in this case the hips node of the 3D model now this is typically uh the root bone of the skeleton hierarchy and it's used for positioning and animating the entire Alor okay so this is the the root bone of the skeleton hierarchy okay that's it for now I hope you understand everything now inside this component well at least you have a general idea how this is made because we transform a GB file which is unreadable into a declarative react component and we can use it as a normal react component everywhere so it's reusable and Main aable okay now let's fix and import this new component to our environment so rename it from model to uh Avatar but first uh we should do something here change this path directly and we will add the a variable for that so pass model and here import path model from modules a million Das avatar. also here in the preload path model if you have errors in your browser after adding this new component uh you should add a line of code here in the v config JS which is assets include uh this gbam because V doesn't know by default what a g B file is and with this you just tell V uh I don't want uh errors because I know this is a glb file okay just read it so in case you have errors include this assets include uh blah blah blah so we renamed the component Avatar we changed uh here the path to a variable because also when we deploy GB here okay typo because when we will deploy the project live uh we need to do this to not have errors later so in environment component um by the way let's change the name here let's remove this gltf and import Avatar import avatar from the public folder because we have this in the public modules folder here I need the position so I'm going to copy that remove the gltf and this position rotation and scale we should not have any error and let's check okay everything seems to be fine no issues let's check the console Lo nothing okay everything is correct now we understood why we did that and why is it much better to have a jsx component instead of using the gilb we can do the same thing for the classroom so let's go inside the folder I should be there here evv okay I'm there so MPX gltf jsx the name of the file so classroom. GB and we created this component it's the same thing you see we import this use gtf and we return this group which is the classroom and unpack nodes and materials from use gtf let's do the same thing remove here the path and add a variable path model and also in the end in the pr we are good to go you see it's the same thing here we don't have skin Mash but we have mash if you need to manipulate one of these mashes for example let's say this is the Blackboard uh you manipulate here you can also attach an event listener to it like a onclick event for example I want to click that element inside the classroom and do something with it on an animation or I don't know change the color or change the texture you can do anything so this is why it's very useful so you can Target individual meshes inside this 3D object which is the classroom and manipulate them add listeners or do anything okay so we refactored the code we can continue to The Next Step all right let's continue our tutorial by adding an idol animation to our a there so it doesn't stay in the T pose we'll use the use effect hook to achieve this I mentioned it before and I'll explain what use effect is so first import this use effect here it's from react so use effect from react here I have a snippet so use effect snippet to delete what's inside so it's basically this what use effect is we see it's like a function that returns this call back and we have this array here which is a dependency What's Happening Here use effect is a hook that allows you to perform side effects in your component side effects are things that happen outside of your component anything that is outside like for example in our case setting up animations so outside of component or fetching data from the Internet or even cleaning up when your component is no longer needed so these are set effects and how this use effect works the use effect hook takes two arguments a function that contains the side effect code this is where we Define what we want to happen when the effect runs and an optional array of dependencies this one here this is the array of dependencies the effect will only run when one of the dependencies changes and here we usually pass a variable okay for example like actions in this case here so the effect will only run when one of these dependencies changes because we might have other dependencies here okay the dependency 2 3 etc etc okay but what if I don't put any dependency here I leave it empty in that case if you omit the dependency array the effect will run after every render okay so if you leave this empty it will run after every render if something triggers the render it will run after every render or if we put a dependency here it will run every time that changes like for example the animations let's explore in more detail the first part is the side effect function so this one this is where we Define the code that will perform the side effect we said the functions will run after every render of our component unless we specify dependencies here we specified a dependency otherwise it will run every render we can think of it as a place to to set things up or a place to make things happen in your component okay so this is the place to make things happen the second part is the dependency array this dependency array is the second argument we pass to the use effect okay because you see this brackets here the first argument is the function that we talked about and the second argument this is a comma here and the second argument is this dependency array so this use effect has two arguments these are the two arguments of the use effect and we said that the dependency array it's an optional array that specifies the dependencies for your effect in our case animations Etc if if we include variables prop or state in this dependency array the effect will only run when one of these dependencies changes so if this changes it will run otherwise it will run on every render okay and that's it now let's write the code to play our animation and that is very very easy I don't know if it can be easier than that because you might think oh to play the animation we should write a very difficult code no it's very very simple it can get hard if we have some complex animations or animation sequences you know or we play these animations based on States specific States Etc but in this case just to play the idol it will be very straight forward and there are many methods to do that some are harder to to understand as a syntax and others are very simple like the one that I will write so to run the animation first we need to know what's the name of that animation we really don't know what's the name okay maybe I know it's idle but we don't know we should inspect okay in this case we simply conso lock the animations so console lock animations and pass the animations that we unpack from the used gtf that we mentioned at the beginning remember so let's check the console here we have animations and all these animation Clips like fight greet look around running thanks think and walking okay so what we need is for the idol is this uh The Only Name That is confusing so let's copy it but we can also try this fight or grid so the first thing I need the use effect and the way to play this animation is straightforward and very very simple because you may think that the code would be very hard to write for playing animations in 3js but actually it's not it's very very easy it may be hard if there are uh complex animations or if there are let's say a sequence of animations or if the animations are based uh in some interactivities or based in react States so it may be hard in those cases but just to play this Idol animation to test it it's very easy so I will create a ni statement so if there is this animation with this name for example grid play it and that's it so if condition the condition is if actions open array string grid so if this is true if I find the grid animation in the actions array okay then play the animation so action. play or animation. playay to give it a name that makes more sense but we don't have this defined of course because this will be con animation equal to this one this animation that we find okay that's it really and we pass here the actions so every time the actions so the animation changes react will reender it's really that easy so if actions GD if this is true if we find this uh animation with this name grd animation that play we just Define this animation here assign it to the specific animation which is actions great and it should work we can test it okay it's not working why weird let me go back to the code it should be a stupid error I know okay I so I sold the back here animation. playay this is a method so parenthesis to call this method so it will play I think it should be okay now let's check yeah it works the animation is plain you see it's the great animation so you can add the other names there in the array to play the animation that you want for example the idol but since the idol had like a long confusing name I just played this the grid animation for now it dis played on Loop but we can make it stop play it just one time or as we wish okay so this is for the animation maybe later we can add interactivity and play the animation based uh on an event or I don't know play a sequence of animations uh when we click something so we will explore things later or in a another version of the application maybe not in this tutorial but in another one I don't know we will see okay let's continue by diving into something really exciting and highly requested which is integrating a chatbot using the open AI API so we will add CH GPT to our application first let's talk about about what we are going to achieve here we want to add the chat component that allows users to interact with an AI power teacher so the users will be able to ask questions and the teacher will provide responses using the power of open AI language model so the chpt to get started let's create a new component called chat and edit in our project this component will handle the user input s request to the open AI API and then display the teacher responses now I created this folder inside the components chat which has the chat Styles uh you don't need to write all these Styles I did it for you so you have it ready we will go step by step instead with a component of course here we need two Imports we need the use State Hook from react to manage the state of our chat component and we need axos axis is to make HTTP request to our server because yes we are also going to build a server so this is not just a front end application okay we will build uh a server with expressjs let's import so use State and access first of all you need to install access with mpmi AIS okay and the use State we already have it from react so I don't need this react here but use state from react and a is import axis chome axis now let's define our chat component first of all we need some State variables okay I will explain uh what state variables are what is use State and everything but first let me introduce you with a syntax of this use State and State variables so without explaining it first I will just introduce you to this US State and State variables so you just get a bit familiar with it and maybe uh with intuition you will understand it without explaining it but then we will go and explain it in detail okay what we need for this chat component what I need I need to handle the user input because if you think about it uh okay I want to make this chat component what would I need for this we need a chat box okay so it will be like an input form so we need to handle this user input so we need a state variable for that everything that changes in our application we need a state variable for that in this case the input from the user to create this state variables here at the top level of the component not inside the return statement but at the top level I have a Snippets for that so I write down state use State I have the is use State snippet and you see I click it I have it ready first set first change it without touching anything else I just write down the name so user input it will change both of them and then press the top it will go here which is the first value of the use state so I will leave it empty so an empty string an empty value if you don't know what is this extension for this Snippets okay the guys that are experts don't laugh at me because I also teach this Basics things because we have absolute beginners here so why not explain to them because they may get confused so we don't want that because when you get confused you get frustrated you lose the patience when you lose the passions then you might lose also the passion and so on so we want to make make things comprehensive okay uh I have this extension I can show you I don't remember the name but we can find it it's react something react snippet or something like that okay is this one this es7 plus react Redux uh react native snippet okay this one search for it here in the extensions top and install it reload the vs code and you are good to go because yeah it's useful we have our first state variable so let's ask again ourselves what do we need for this chat component maybe if you don't know it yet you can continue and uh build the chat component but you should think first what do we need we need this user input from the user of course since we need this user input and we have this user input we will need also the response so the response is from the teacher in this case our 3D avatar which it's me and the teacher will get this response from chat GPT okay from open AI API so not only the user input but we also need a response let's create another state variable so using the snippet uh teacher response let's call it teacher response also the initial value because you understand it that this is the initial value I say you understand it because yeah a bit intuitive as I mentioned before this is the initial value and leave it empty okay we will go in detail explaining this no worries what else we might need at the moment we are good to continue but uh if we think of it I also need to because this chat box will not be all the time showing in our screen because I want to show the beautiful 3D classroom okay I want to have a button here like ask a question and when I click the button ask a question I want to open this input form I don't want it all the time uh visible in my screen because it will block the view of 3D classer so I just small batter at the top maybe at the left click in that button ask a question it will open the chat box then we can write down uh the command send it then we will get the teacher response from chpt then we can close that so it will not show so for this if you think of it with this logic of react we need this uh chat box but we will not uh use it all the time so only when we click the button so we need the state variable show input or not show input so let's add another state variable this is a Boolean by the way true or false Boolean are true or false Snippets show input and the initial value will be in this case not an empty string because this is a Boolean but uh true or false and initially it will be false because initially we will not show this chat box okay so you get the idea of building an react application I don't want to show just follow me and I'm writing down I'm writing this I'm writing that look at me how good I am at coding I'm not that good actually to be honest I just research things and step by step I also build these things because I make errors and bugs and a lot of them so so I also want you to go through this okay because I really don't like spoon feeding tutorials for me it's not really the best way to learn the best way to learn is doing yourself so not every tutorial is useless but if you go with a tutorial and stop the tutorial for example in this case stop the tutorial and try to understand it yourself what you are doing without just listening and copy and pasting okay but also ask questions why do I need this user input why a million is uh defining this user input also for the teacher response Etc and you can imagine how you will structure yourself and how you will build it yourself okay maybe you don't know how to do it right now but okay having just like a picture I will create an input form and in this input form we will add maybe an event handler to handle the input etc etc etc and maybe the last state variable at least uh this initial stage uh I need a loader because as I said before we show this chat box to waiting for the data I want to show something so a loader okay based on that loader showing the data or not when the data is not ready uh I will use this uh is loading or not it's again uh bullan true or false snippet and is loading thatat is loading I wonder if my English uh sucks I don't know I think uh I don't speak it very fluently but okay uh peace guys because it's not my native language so I try very hard I speak fluently Italian actually Italian is like like uh as my mother language Albanian but I'm trying to practice with English with these tutorials because only when I speak with my colleagues uh remotely but yeah you know it's not like the same when living in a country with English speaking people okay or if you have friends that speaks English or a girlfriend etc etc but that's not talk about that okay no uh here it's a buen ass set and initially it will be false we have the state variables and by now we are good to go let's continue by explaining what this UST State really is and what state variables are why we need a u State why we need these State variables what's happening with them and all the other details or questions that you might have again this is for the guys that are absolute beginners the other experts that are following me and I know they get a bit bored from this and uh they maybe they don't follow me anymore when I explain these things step by step but I have uh the absolute beginners in mind okay so bear with me please okay I'll do my best to explain it in a clear and comprehensive way what what do you state is imagine you are building a toy uh like a car for example or imagine you have this car in a video game okay imagine you want to build a game like a r or racing car okay and let's focus just uh in the car the car has different properties like a color black or white or red uh the car has a speed whether it's currently running or not it's also another property of the car and all these properties can change over time based on how you play with a car in your game okay is it clear in react we call this changing properties state so again I will repeat it in react we call this changing properties State now let's say you want to keep track of the car's speed or let's say of the car's color you don't have the tools like JavaScript or react you just have a piece of paper and a pen so to keep track of the car's speed and the car's color you just have a piece of paper and you might write it down the initial color and the initial speed of this car so write down in the piece of paper cars color red okay cars speed I don't know 50 kilm per hour and then whenever you change the color or the speed of the car you update the values on that piece of paper so again below you update the car's color it changed equal black the car speed it changed equal 100 km/ hour this way you always have a record of the current state of the car the current state of the car is the last uh one that you wrote on your piece of paper so in react we use the use State hook to create and manage these State variables just like that you did in your piece of paper the state Hook is a special function provided by react that allows us to add the state to our component that's it we showed here how we did that with a code that I told you that to be familiar so const then we open an array user input and set user input equal to use state with an initial value of an empty string so we are basically here destructuring this user input and set user input from this function use State we are unpacking if you don't like the word d structor so I unpack this two variables user input and set user input but let's break it down in detail we declare a state variable called user input this is called user input and we also declare a function to to update this and it is called set user input also the name it's intuitive set this user input so that is a function the use State hook checks an initial value as an argument as you see which is this empty string it's just an initial value because we need an initial value when we load the first time the application for example the Boolean will be first FSE uh that teacher response will be false because we don't have a teacher response the user input will be an empty string because there's nothing initially okay the use State hook returns an array with two elements again I will repeat it so it will stick in your mind the US state hook returns an array with two elements the current value of the state variable which is the user input so this is the current value and the function to update it okay I will not repeat it again that's it but remember it is called array destructuring this thing to assign these two elements to separate variables it is called array destructuring so we assign two elements to separate variables from use State okay unpacking the structuring as a beginner I always ask this dumb questions but no question is dumb why do we need State variables so teacher teacher raise your hand and ask the question why do we need State variables okay let's ask the 3D teacher from chat GPT and the 3D teacher from chat GPT will give uh this response well just like the car in our our video game just like the car's color and the car's speed might change in time our data in react components can change over time State variables allows us to store and manage this changing data that is the reason why we need the state variables to store and to manage the changing data so this is the reason we have all the state variables and since we have an initial value then later whenever we want to update this value of the state variable we will use the corresponding Setter function which is this the second one this set user input set teacher response Etc this is called the setter function so we will call this Setter function to update the current state react will then reender the component with the updated State value value okay we call the set function to update the state variable the current state then react will reender with the updated State value but uh another question why can't we just update the state variable directly like user input equal new value user input equal new value why not react relies on the setter functions to keep track of State changes and to trigger reenders efficiently so by using the setter functions react knows exactly when and how to update the component okay that is why using uh these variables with the UST State hook it will allow us to create this Dynamic and interactive component and that is why react is so powerful we can then respond to our user input update the the UI Based on data changes and keep our components in sync with the latest data okay this is the power of react to keep our components in sync with the latest data that's it for now so please if I completely bored you just keep me in this uh sections okay please bear with me now that we have this use State and variables let's add some event handlers to manage user interactions we need first of all an event handler for the input okay and let's call it uh handle user input always here at the top level below the state variables const uh handle this is the convention we use this handle and the name so handle user input okay error function this is the first event handler this handle user input what is its function what is the work that it should handle it will update the user input so the state whenever the user types in the input field so the user types in the input field and this Handler will update it okay and how it will update it we said that we call the setter when we want to update the current state so call the setter set user input you just call it like this open the parenthesis and with what value with the value that the user types and with what value it will be the value that the user types and how we get this uh input from the user how we get it how do we know what the user is typing in the input field we get it by passing this event here and then inside the setter function event. target. value and this is the value that the user is typing in the input field so you might ask again what it means what is this event how do we get it how do we know like how do things works okay so let's break it to to check what's inside like we used to do when we were kids I used to break things and just to check what's inside and I I broke many many choice just to satisfy my curiosity okay so let's break this function here what's happening how do I get this event what is this event okay and what it means Target so let's break down and answer all the questions that a beginner might have and maybe you also ask these questions in your mind the first what is an event handler this function that we created an event handler is a function that gets called whenever a specific event occurs in our case handle user input is an event handler that will be called whenever the user types something in the chat this is what it is uh next question a beginner might have what is this event argument here so when an event handler is called when an event handler is called it automatically receives an event object as an argument so when this is called it automatically receives an event object this is an object this event object contains information about the event that triggered the Handler because what triggered this okay it will contain all the information about what triggered this Handler so we specify it as the parameter here this allows us to access access the event object inside the function another question where does the event object come from then the event object is automatically provided by react when an event occurs so when an event occurs it will automatically be provided by react it's created by the browser and it contains all the useful information and the methods related to this specific event for example when the user types in an input field a change event occurs okay I type in the input field a change event occurs this is an event and react passes the corresponding event object to the Handler so I type in the input field a change event occurs and react helps us to pass this corresponding event object with all the information to the event handler okay I hope I am clear uh yeah you have other questions I know uh another is what is this event. target I understood that the event is the object that contains the information but what is the event that Target the event. target refers to the Dom element that triggered the event okay again I will repeat it event the target refers to the Dom element you know the Dom that tree like structure with all the elements the target represents the D element that triggered that event this object provides access to the properties and methods of that Dom element so that Dom element have properties and methods I access this event Target and I will access elements uh sorry properties and also methods to allow us to interact with the element and retrieve the value because I want to retrieve the value and you get this event. target. value so you might ask again what is event. Target that value okay I know you are bored I know a that Target that value specifically refers to the current value of the input field so this is what we want we wanted the current value of the input field to set our state because our state is the current value of that what the user is typing and then we change we change into the current value that uh becomes the current value we type something else that we change and that becomes our current value and so on okay if you type this in the console log you will see every character what we type it will uh become our current state because we said that we update the changes with the setter function okay uh any other questions I don't know like uh how do we use this event. target. value we use it like this uh in this function function Handler user input we call this set user input and we pass as an argument event. target. value which sets the value of our user input state to whatever the user has typed so it will set this value the value of the user input to whatever the user has typed by updating this user input State variable we are St in the users input and making it available for further use in our component such as sending it to the open AI API okay that is a reason that we want to store it to send it to the open AI API um I don't know maybe the last question uh how do how do we connect the event handler to the input field how do we Conn connect this that is a good question and we will do it next when we will create the code inside this return statement we will uh create an input element and then we will pass this values to the input and connect connect the state with the input field okay we are going to do that next and let's add another event handler which is the toggle input to toggle the input okay and here to toggle the visibility of the input when the user clicks the button ask a question or when closes uh this chat box we will call the setter in this case set show input and here we'll add this exclamation mark and set show input what this exclamation mark means here this is called The Logical not operator if you have a bullan variable in this case like uh we have this show input why I have this set show input okay this is uh a typo it's show input set show input I made a typo okay so this is a ban true or false and we use this logical not operator if you have Boolean variable which is true it will evaluate to false in this case if the bullan is false it will evaluate to True similarly if you have a value that is falsy for example uh the values that are falsy are zero null undefined or also an empty string does like this the empty strings that we have in the other set variables they are all faaly values it will convert to True okay so it's the contrary it negates the truthness of that value okay we have these two event handlers now let's create a function to handle the form submission because when we will uh create uh the message because when we will write the message to to send to the open AI API we need this form submission and we need a Handler for that form submission and as a convention usually it is named handle submit so let's create another function handle submit here I also will pass the event and the first thing I will prevent the default behavior of this submission form because the submission form when you submit it has a default Behavior like refresh freshing ex so I want to prevent the default behavior and for that we use event do prevent default just to prevent the default behavior of the form submission then I will set the loading to true because initially it will be true the loading then I will use a try catch block uh this is complaining because it is a sync so it is a sync function so we fetch the data asynchronously and here we are sending a post request to this URL using axis that we installed and including the users message in the request body so this is the request body this is the URL from where we are fetching the data and we will create a server for that okay to get this API data and the post request is when we post a request to the API for detailed understanding of this just read the documentation for the HTTP request so get post put Etc but basically we are sending a post request to this API URL using axos and including inside in the request body the message of the users and then we usually put the headers in this case it's content type it's uh application Json then we extract the generated response from the server response data so we extract the generated response from server response data and update the teacher response with the generated text so set teacher response generated text then we set the loading to fals once the request is completed because it is not loading anymore and this part of the catch is to catch the errors if we have any errors we catch with this catch block so we conso log the error and that's it so here when the form is submitted handle submit will first will prevent the default form submission Behavior we will set the loading to true then to indicate that the request is in progress and we set the loading to true then we send a post request also check the documentation of JavaScript for the HTTP requests like get post Etc using axos and inside in the request body we include the users message so the users message is user input then we set these headers that the content type they are in this case uh application Json and we extract the generated response from the serers response data and update the teacher response with a generated text so extract the generated response from the server's response data and update the teacher response state with a generated text we pass inside generated text then we have this catch block to catch any errors and we console log them and here in the end we set the loading to false okay this is what this handle submit does now finally let's define the jsx structure of our chat component for now we just have a div component with a simple text okay so this is the jsx structure okay let's change this load uh is loading state to loading just loading it's more clear loading set loading so in this jsx structure we include a bottom to toggle the visibility of the input form a conditional rendering of the input form based on show input that we have declared at the top show input so this is just a conditional rendering this end operator is if this show input is true show this inside we have this parenthesis so if this is true and just means like show this show this if this is false it will not render this then we have this input form with the text input field and the submit button and this is just the loading if it's loading sending and if it's not uh show the send input text and then in the end another conditional rendering with the end operator for the teacher response based on the teacher response so if the teacher response is true if we have the teacher response if we get the teacher response then show this div with the teacher response you see here we are just showing the teacher response we will evaluate this state here whatever this teacher response state has it will evaluate here so we have set up our chat component to handle user interactions and communicate with the open AI API but wait we haven't actually created the server to handle the API requests yet so now we need to set up a proxy server to securely communicate with the open AI API so first we will create a new folder called proxy server at the root of our project directory so here close everything at the root of our project directory we will create a new folder and name it proxy server inside this proxy server folder we will initialize a new node.js project and to initialize a new njs project we do it by running the command mpm init dasy but make sure you are inside this folder so for now I am at the root level CD proxy server PD I inside. or just right click and open in integrated terminal so when we are inside this directory mpm init dasy and it will create this package Json inside you see we now initialized a new node.js project so these are the default settings for the package Json now uh we should install the necessary dependencies for our server which are Express from expressjs we will also install open AI from the open AI we need the do EMV which is a module that loads environment variables from the EMV file environment V variables are to keep our secrets okay we don't push them to GitHub and we don't show them to production so for example the API key we don't want uh everybody to to to see so we put all these secrets to a file which is named EMV and for that we need this EnV package and uh lastly we need the course package which is a middleware that enables cross origin resource sharing which is course for our server so it's like a measure for security and uh we need this middleware so in the terminal type mpm install Express open ai. EMV and course always inside this proxy server not in the other folder okay so make sure you are the right directory and install these packages just wait a few seconds and you see here in the dependencies we have this list of course tnv Express and open AI now let's create a new file inside this proxy server folder and call it proxy JS and we'll add some code there to create the server so roxy. JS first thing we import the necessary dependencies and in this case it's the same thing like we do with import module from module in in njs there are two ways we can use the the modules with uh es uh six syntax or we can use this syntax like const Express require Express const API require open API Etc we are importing this modules it's the same thing as import Express from Express import open AI from open AI we can also convert this to es6 syntax we can do that by adding line module in the package Json but we won't do it this time you can do it anyway but we we are just going to use this not JS syntax I will just get rid of this you see here file is a commonjs module these are the commonjs module it may be converted to an es module okay Quick Fix convert to es module you see import Express from Express we also can do this now first thing here at the top we add the Dov config to make possible to use our environment variables. config so if we have uh an environment variables from the EMV file we load them using this.c config because we will uh create an environment variable for that after we get the API keys so the next Next Step would be to create this open AI key variable and for that just uh declare const open Ai and here we create a new instance of this open AI class that we imported at the top and for that we use the new keyword to create a new instance of this class so the new keyword is create a new instance of that class so open AI curly brackets inside because it's an object and we use the key value which is app key and here we will add the app key when you will get it from the open AI uh platform so each API key will be different so you will put your API key here but you don't want to put it here to make it accessible to others but you will use uh what we mentioned the MV files the environment variable and to use the environment variables here you type process EMV do the name of the variable that you declared in the EMV file which usually should be just in uppercase letters so open AI underscore API key okay you will declare deare this inside the environment variable that we will create now a file EMV to store this environment variables okay yes let's do it uh now so new file EMV and it will be like this not RMV EnV case EMV not just EMV and here without spaces without anything uh the variable open AI key equal and here paste uh the key so the key here okay without uh strings just like that and next we create an express application and apply the course middleware to enable course remember I mentioned course and to enable this middleware doesn't matter to go in detail what really course is you can read on internet about it but just the middleware for security okay so we create an express application and apply this course uh middleware to enable the course so const up equal Express that we imported at the top and then we do like and then we use this app variable app.use and pass course inside this course that we imported at the top and also don't forget app.use Express Json as you can see this is uh returns middleware that only passes Json okay and only looks at request where the content type header matches the type option you know what Json is it's the type of data with key values and after that we Define a post route at API SL chat GPT remember that in our chat component we are sending this uh access request this post request to this API SL chpt that will run in our Local Host 3001 so we need to set up this uh API route and here it is follow this code so we create this route API pass this request and response parameters we cons lock the request body here we have this completion which we get from this op open AI chat completions that create which is uh the way that uh open AI offers us to get a request from them okay and here we have this messages with the role system and the content like you are a helpful assistant or we can make it more specific here you are a teacher and you will help students etc etc you are an AI teacher and you will help students with their uh blah blah blah doesn't matter you will help students and the role is user with this content with request body message with this request body message is the message that that we sent and we are using this model uh GPT 3.5 because if I'm add this version 4 uh not everybody has you know the premium so let's leave it and then we extract the generated response from the API response and send it back to the client as a Json file okay so response. Json response this uh syntax from open AI but what it means is that we extract the generated response from the API and send it back to the client as a Json Json is that structure of data with key values like and then if an error occurs we log the error details and send an appropriate error response to the client okay so error details we send that message it's just that and finally we will start the server and listen to the specified port in our case uh the default is 3,001 so we will start the server and we'll listen to this port so con Port equal process. dmv. Port okay or because we might have a port that is secret we don't want to show and we can store it in our EMV variable so we check Port is what we have in our environment variable or 3001 then we listen to this port and conso loog a message proxy server is running on Port and the variable Port here which is 3001 when we will start the server it will consol along this message telling us that the server is okay or if we have any errors well that was a lot of code but we successfully set up our proxy server to handle of communication between our chat component and the open AI API and we can run this server with simply not proxy JS you see the message proxy server is running on Port 3001 so we have this server running okay and we can make API calls to This Server here in the chat component we are making an API call so we will get the data because the server is running correctly okay now we just want to make sure that this axis. poost request matches the proxy servers end point so we have Local Host 3001 API CH GPT make sure that this URL here matches the proxy service end point this one here make sure because you may have errors and uh cannot figure why this is something that you should check so/ API CH GPT make sure to match this this is the base URL it's the Local Host with a port 3001 and this is the endpoint slash API slbt okay with everything in place we can start our react development server and the proxy server so we started the proxy server with a mod roxy. JS we run that uh file and for the front end part we just you know mpm runev and now we should be able to interact with the AI teacher through our chat component okay let's uh let's see what happens in the browser uh do not forget to import also the styles for the chat component now let's check the browser and see if we see the chat box and if we can make a request and if everything works correctly okay let's discover okay first thing first I don't see the the bton ask me a question so we might have issues here let's open the console log we don't have any errors okay let's refresh we don't see yet the bottom ask me a question so in order to ask a question and open the chat box uh back to the code oh okay okay okay I know I know the problem I forgot to call the component uh in the appjs h wait so yeah we created everything the chat component and also the proxy server but we are not rendering at all this component chat because should import it here in the main app component import uh chat it will complain because uh jsx Expressions must have one par element okay so if you have more than one uh component you should add like a wrapper comp you can add a div or empty uh how it's uh fragment yes fragment EMP empty fragment like this okay wrap it with this empty fragment now we should be able to see the button ask me a question and when we click the button we should be able to open the chat box and send the request and everything okay maybe not yet but yeah it's just a stupid error just this import from for the chat Styles I forgot to add uh Slash so here slash so we see the button and let's test it ask a question we see the input form so it works and let's check if the server sends a response or an error let's hope that it has no errors so an easy question like what is what is T okay okay it works we get the response you see it opens this dialogue and everything is functional we have no errors let's check the console no no errors everything works perfectly we just need to style it better of course we will add better Styles but it's functional and everything works correctly as it should we can continue to the next section maybe we can add like uh a conversation window where we see our question and the response from CH so in a structured way so user question then chpt response okay so we label and also we can find a way to save the conversation so we have the history of that conversation it would be cool right because we want to save the conversation and continue the conversation with other questions because now we close it and uh for example this box here remains it should close and it should erase Al after I close the chat so everything should turn to an empty value should be erased and we will work on that and after that in the end we will add the voice recognition so I can ask a question with uh my microphone so I want uh chpt to respond me with an audio let's do that so in order to add the feature to save the chat history I made a few changes in the code I didn't wrote the code live because in react when you make changes in the jsx uh structure it's a bit confusing so I don't want to confuse you so I will explain uh the few change changes that I made so first of all I added a new state variable chat history okay chat history set chat history and the initial value it's an MP array because this array will contain the chat history next in the handle submit function I update the code to save the users's input and the generated response so scroll down to the handle submit function here you see see this is the same the generated text is equal to the response that we get from the data and here I update the chat history with set chat history first I spread this chat history everything that already is in the state and then the user property stores the users's input message user input and the bot property stores the generated response from the AI teacher so the generated text I then use the set chat history function to update the chat history State the new state is created by spreading the existing chat history array so this is by spreading the existing chat history array and then appending the new object user input and both generated text finally I click C the user input state by calling set user input to empty string I clear it and to allow users to clear the chat history I use this handle clear okay with that I set chat history to an empty array and also set user input to an empty string and I will use it when I will click the button close it will clear the chat history and the user input you see I add this onclick event listener in the bottom and every time we click this button which is the cancel button or close we will erase the chat history and also the input okay this is the clear button then this is what happens in the updated jsx first I check this chat history length if it's equal to zero or greater if there is is anything in the chat so in this div section I check if the chat history array is empty using this chat history. length equal zero so if it is empty or not then if it's empty I will display this text message welcome to the chat ask a question to start the conversation if it's not empty with this two dots here so this this is a conditional in react you use the curly brackets use this conditional chart history that length equals zero this is the Turner operator and the Turner operator works like this we have this condition is chat history length zero or there is any chat there and we put this question mark is this length zero yes if yes it will show this div it will evaluate this condition if it's not the other condition is written like this these two dots open parenthesis and evaluate this other condition and in this case this other condition if the chart history is not empty we use the map function to iterate all over each item in the chat history the map function is a JavaScript errow method one of the most used in a react uh project you can learn about the map function uh search Google check the documentation but basically here we iterate over each item in the chat history and the chat history is an array and then we render a div for each message response pair okay so we render a div for each message and response pair you see this is for my message and this is for the response for each item of this area chat history that could be a long conversation so inside each div I display the users message with a u okay and here I put inside chat. user that I showed you earlier so user which is the user input and the bot is the generated text and here I show chat. bot which is the generated text that we Define here when we get the response and we set the chat history to it well that's it for this changes it's simple once you learned how to manage the state with use state and its state variables it's not that hard I think now with this changes our app supports the chat history functionality okay we can check uh now the browser and test it we also have the new design Let's test it now ask a question you see the new design it's better we get this initial message and we do this with that condition remember that if the chat history array is equal to zero so if it's empty then we show this div if it's empty we will show the conversation and here we can type message let's test it so again what is chbt okay you see it looks very nice to me we get this chart so now we see you and our question and then chbt with with the response and it works perfectly okay I'm happy with this I think we are finished with this part uh yeah the last thing is to add the text to speech and voice recognition I wanted to add uh a nice audio maybe use 11 laps or use use the chbt uh text to to speech functionality so we are going to do that and maybe later we can further improve it by using a better audio like I know this uh platform 11 Labs maybe we can use that somehow I don't know how but I will figure out or maybe you can contrib to this since this project will be live on kab if you want you can contribute so if you know how to do this or other features please go ahead and send a pool request and we can share it with everyone okay so close it will uh now delete uh everything in the chat history and also the input so again oh it doesn't work okay it's this CLA okay CLA yeah so when we click the cleare it will clear everything now close okay it works we have reached the end of our tutorial on building a 3di AI School using 3js and react 3 fiber I hope you found this tutorial informative engaging and fun building a 3di AI school is just the beginning actually of what you can create with the power of 3js and react 3 fber and additionally AI as we move forward there are countless possibilities to expand and enhance this application in future versions We Could explore many other cool features like uh integrating realistic voice synthesis to give the AI teacher a more lifik and expressive voice and personalize The Learning Experience by track in user progress providing tailor content recommendations and adopting thei teachers responses based on user preferences the possibilities are truly endless and I'm excited to see what you will create with the Knowledge and Skills you have gained from this tutorial I want to thank you for joining me and for your dedication to learning if you enjoyed this tutorial and found it valuable I would love to hear your feedback and see what you create so feel free to share your projects ask questions and if you want connect with me by subscribing to my channel and also you can add me on LinkedIn where I post from time to time things related to the world of 3D web development once again thank you for following along and until next time keep learning and keep exploring the fascinating world of 3 D and AI bye

Channel: Emilian Kasemi

Views: 426

Keywords: three.js tutorial, react three fiber, openai api, 3d web development, ai chatbot, beginner web development, javascript, react, threejs, 3d graphics, web development tutorial, ai in web development, interactive web applications, webgl, immersive experience, virtual classroom

Id: AllROojlixE

Length: 181min 51sec (10911 seconds)

Published: Tue Apr 09 2024