I CREATED My NEW Voice Assistant w/ On-Device Wake Word Detection 🚀| Home Assistant | ESP32 S3

if you have been following me on Twitter or you have seen my short video of my new voice assistant device then today I'll be showing you how I created this device now we will be making use of an esp32 S3 chip with on device wward detection using ESP home and then we will configure it with home assistant such that we can invoke home assistant devices so with this let's get started okay so we have all the components that we need we have the esp32 S3 board then we have the microphone that is the inmp 441 then we have the speaker along with the audio amplifier so this is a max 98 357a audio amplifier and I have connected a 100K Ohm resistor to it in order to reduce the volume of this speaker now the volume of this speaker is pretty loud so I have connected this 100K Ohm resistor between the gain and the VIN pin now along with this we have this LED strip now this LED strip is a cob style LED strip having addressable LEDs inside it so these are the ws2812b LEDs inside this cor style LED strip and we will now assemble all of these components inside this case so this is a 3D printed case that I have made and this is a smaller size compared to my predecessor and we will be fitting the microphone here and then we will have the USB port here to power the device and this is the section wherein we will actually have this physical mute button so now in this Edition we have this physical mute button with which you can mute this voice assistant using a physical button so we will have it something like this over here and then you can click this in order to put it into mute or unmute the device so with this all these components let's see how we can assemble them all together now to connect all of these components I'm going to make use of these jumper cables here because I'm not an expert in soldering what I'm going to do is I'm going to connect these components using the jumper cable and then secure it using hot glue so let me first do this for this microphone here so right now I have connected this microphone with the jumper cables and I've secured the connection using hot glue so I'm going to do the same thing for this audio amplifier as well as this LED strip and then I'm going to connect it to this esp32 S3 board so let me do that right now so before I use a hot glue on this chip here such that I can secure the connections I just realized that I have to pass this microphone from here and then connect the wires which come through this hole so let me do that first so I'm going to remove these connections the microphone is now flush onto the surface of this case over here and these are the cables that are coming from behind so now I'm going to connect them to the es32 S3 board so I have connected all the components now to this es32 S3 board now there are a bunch of wires over here which you will probably not able to figure out so I have connected all these wires according to this diagram here and this diagram I will provide you in the article that I will link into the description below now along with this on this ESV 32 chip you have to connect this in and out pad here in order to enable 5V output from this pin here if you want also the RGB led to be available then you have to connect this RGB pad here so right now I have connected this pad here as well as these two pads here and now I will get 5V output on this pin so with this I have all the connections that I need now I'm going to apply some hot glue to secure them in place so now I have secured all these connections using hot glue on both the sides here so I have connected the physical button I have connected the LED strip then also I have connected this speaker and the microphone here so now let me put these components inside so first of all what I'm going to do is I'm going to put this LED strip so for this I have this small little hole from here and I'm going to peel out this sticker pulled out the sticker right now and then I'm going to now pull this LED strip from here and I'm going to secure this inside this Gap here so this has been put inside here press this inside so this is now flush and it's sitting inside properly so the LED is right now in place now what I'm going to do is I'm going to now put in the board inside so for this first of all let me put the board first but before that let me put this mute button here so so I'm putting the mute button inside and now I'm going to hot glue this mute button so right now I have hot glued this mute button here and with this I can now press the mute button as well as I have applied some hot glue here in order to stick this LED strip here so now with this we are going to now put this chip inside so let me insert this inside here and I'm going to align it such that I can access this USB Port so this is something that I'm now going to hot glue it here so now this chip is in place here and I can access the USB C Port from here so now the next thing that I'm going to do is I'm going to put this speaker inside so let me put the speaker in this direction here and then let me put some hot glue here and then finally I'm going to put this audio amplifier inside here and and to fix it I'm going to use some hot glue again so right now I have hot glued all the components inside this case and it is fitting pretty well inside so right now you can see a lot of wires inside this but I think this should be all fine so if you see in the features we have this mute button here we have the input for the power and this LED strip around it and the microphone on top so now let's burn in some code for this chip using ESP home so now in ESP home I'm going to create a new device here so I'm going to click on this new device I'm going to click on continue and I'm going to give it a name like esp32 S3 Voice Assistant I'm going to click next and here I'm going to select esp32 S3 so with this a new device is created I'm going to skip this here and I'm going to edit this configuration now right so now instead of copy pasting the entire configuration I'm going to actually show you the existing configuration that I have so this is the configuration that I'm using with my voice assistant so now if you look here there are quite many configurations here like these are some of the SDK configurations for this ESP IDF framework that we are using as well as there are some stuffs that we are doing when the device boots up now let's look at the various components that we have so first of all let me collapse all of these configurations so these are the various components that we have a switch a binary switch a light an i2s audio interface microphone speaker micro word configured and voice assistant so now let's look at the switch section so in the switch I have this mute button so now this is a software switch that will be available inside home assistants once you configure this device inside home assistant this is not the actual physical switch now inside this I'm doing a lot of things like stopping the vake word when this is been pressed and and then even turning it on and off for the Wake word detection now let's look at the next section that is a binary sensor this is the physical button that I have actually right now configured inside my voice assistant so what this mute button does is that it toggles this mute switch and this mute switch is the one that we have configured here above so this is the software button which actually takes care about enabling and disabling the Wake word inside this esp32 S3 so this is the physical button and this is the software button now let's close these two sections and let's look at the light configurations so in under the slide I have two light configurations here one is the internal LED that we have on the es32 S3 chip and this LED strip which is right now connected to our voice assistant right now and this is connected to gpio 9 pin let's close the light section here and I'm going to open the i2s audio section so now in this i2s S Audio section I have one configuration for the microphone and I have one configuration for the speaker now in the comments here I have specified which pin it actually connects to such that it's easier for you all to figure out which pin belongs to which gpio pin on the board itself let's close this section now now in the microphone section we have the gpio 4 being connected to the SD pin and I'm using the Left Channel here now in the Left Channel configuration the ground and the L n r pin are connected to the ground pin of the es32 S3 board now this is the microphone let's look at the speaker so we have the i2s audio been configured and we are using the identifier from this i2s speaker which is belonging to this ID over here let's go to the micro word part so now in the micro word what we have here is that we have some configuration that it does on detecting the micro word so it actually starts the voice assist and then Returns the fake word that it has detected and then it turns on the light for both that is the internal Ed that we have as well as the external strip that we have connected to the voice assistant so now these are the two configurations that I've done for the micro word detection as well as I have set the model as H arvs I like this model and in the future I'll be making a video on how you we can have a custom V word also so I want to reduce it from H Arvest to to just jaras and it detects what we wanted to detect so this is the next video that we'll be coming up so make sure you hit that subscribe button to see the video once it is out now let's look at the voice assistant configuration here I have configured the microphone that is the VA mic and then here also I've configured the speaker now here there are a few configurations being done that is on the speech to text end we turn off this light and on the end we actually turn off this led strip so these are some of the configurations we have done for the voice assistant now you don't have to screen copy all of this configuration I will provide this configuration in the article that I will link into the description below and also make sure you follow me on Twitter because I tend to post a lot of things a lot of updates about my voice assistant whenever I improve things I post it over there so make sure to even follow me on my Twitter handle so now with this configuration being done now what we need to do is we need to install this on our es32 S3 right so if you're doing it for the first time what you will have to do is you'll have to click on install and click on manual download and then after this it will compile this entire configuration that we have given and we will get a binary download file so let's wait for it to complete so right now for me this was pretty quick because I have already compiled this before but because this micro word makes use of tensor flow behind the scene it takes quite some amount of resources as well as some time to actually compile this entire binary so don't expect it to finish within like a 2 minute build or a 3 minute build it may go up to like 5 to 10 minutes build once that's been done then you can select This Modern format option and then after this I'm going to keep this download over here so next what you will have to do is you will have to go to a Chrome browser so this is a Chrome browser right now and I'm going to open web. ESP home. IO and I'm going to connect my voice assistant to it and I'm going to click on connect and I'm going to select the required device over there so right now I have already flashed this binary on my esp32 S3 that is this voice assistant that I have right now and I don't need to do this again so if I want right now I can go back to the ESP home I can click on install and select wirelessly and this will wirelessly put an update back to my voice assistant here so let it right now complete transferring the binary onto the ESP 32 S3 board and let's see once it restarts so right now it has restarted and if you see here a green light came on so on boot it shows that the voice assistant device came on and a green light comes on for that so now if you see in the logs the ESP 32 S3 is now online and live and let's go ahead and connect it to our home assistant so I'm I'm going to stop this here and I'm going to go to home assistant and then here I'm going to go to settings devices and services and it should automatically get detected here now if it does not get detected then what you will have to do is you will have to go to ESP home here and then enter the IP address of this device now if you cannot find the IP address what I usually do is I use this utility called as IP range angry ip scanner so using this I find out which device are connected to my Wi-Fi network and if you see here this one which is the 115 is the IP address that is actually of this device so how I find out this is I actually copy this IP address and I go back here and I paste it here and once you click on submit you should see this thing this is the way I at least find out which is the IP belonging to the device you can find this IP address also from the router config ation page also now let me enter this encryption key so for this I will go back to my configuration here and I'm going to copy this encryption key and with this The Voice Assistant is now configured inside my home assistant itself now let's go to this devices section and I'm going to open this voice assistant so if you see here there are three buttons here so first of all this is the onboard LED so I can turn turn on and turn off the onboard LED and I can also turn on the LED strip so if you see here the LED strip has come on and then also let's see what the mute button does so when I turn on the software mute button this is the effects that it shows okay so right now it remains in a dimly lit red color showing that it is in mute State and when I turn it on it goes back to this green State and obviously the LED strip turns off now if you see from the configuration point of view I have this mute button here right let's actually test this mute button so I'm going to press now the mute button from here so if you see it actually put this device into a mute State I can now turn off the mute State Again by pressing on the physical button again so this is how the mute button works right now now for this voice assistant to work we have to configure the voice assist pipeline right so I have this voice voice assist pipeline that is the on device Pipeline and let's look at the configuration for this so let me go to settings here and I'm going to go to voice assistant and I have this pipeline been configured here now this pipeline requires two components so first of all it needs speech to text component and it needs a text to speech component now for the speech to text component I have this faster whisper and for text to speech I Have This Piper configured and for the Wake word I have nothing being configured so this is is what I have configured right now for my voice assist pipeline now in order to see how you can configure this faster whisper as well as Piper using home assistant add-ons or using Docker containers completely on a separate machine then you can refer to this video that I have linked somewhere here as well as into the description below wherein I have shown you how you can set up this voice assist Pipeline with faster whisper as well as Piper and also with wake word but in this case you don't need the Wake word conf configuration because we are using the on device weord detection on the esp32 S3 chip so with this now we have configured the pipeline we have configured our device also so let's go back to the device again here and this is the final thing that we needed to have our voice assistant up and running so now with all of this being configured let's see if a voice assistant works so for this what I'm going to do is I'm going to go to this overview section and I have this light configured so this is a template switch that I have and let's see if our voice assistant is able to turn on this switch hey Jarvis turn on the light turned on the input Boolean so there we go we have this being turned on by our voice assistant after detecting the Wake word on the device itself so let's try turning this switch off hey Jarvis turn off the light turned off the input Boolean so there we go so now the switch is also turned off using our voice assistant so the main thing that I liked about this on device wake word is that it detects the Wake word really quickly there's not much of delay because the wward detection happens on the device itself it's not being streamed to a machine or home assistant but it's detected on the device however the processing of The Voice still has to go to the machine on which faster whisper is running and then afterwards the output is given by Piper so that's where it takes a little bit of time to do the processing but however the on device wake word is really fast and pretty reliable as per my testing that I have all done now in my next video we will be looking into how we can have custom wake word for this on device wake word detection now if you want to make things smart at home make sure to follow me on Twitter as well as hit that like button and the Subscribe button for more such videos to come till then take care and I will see you in my next one [Music]

Channel: Smart Home Circle

Views: 5,934

Keywords: on-device wake word, micro Wake Word, smart home, home automation, home assistant esp32, wake word detection, home assistant local control, home assistant voice assistant, home assistant local voice control, local voice assistant home assistant, home assistant voice control, home assistant voice, home assistant guide, local voice assistant, voice assistant esp32, home assistant setup, esphome voice control, home assistant wake word, voice assistant, whisper, piper, wake word

Id: aPP2alIU7Gg

Length: 19min 7sec (1147 seconds)

Published: Thu May 23 2024