diy smart camera: ESP32-CAM with TFT

so hey everyone today i'm making a smart camera with a esp32 ai thinker module and a 2.8 inch tft so what i'm going to do i'm going to mount that esp32 onto this piece of perf board and the tft onto another piece and then i'm going to put them opposite each other like that so i can take a photo with with the esp32 and it'll come up on the tft but i'm going to put a um a bit of machine learning on there uh for object for object classification and then i've got a little push button here as well just to take a photo and uh perform object classification so i'll put all that together and we'll see how that goes so hey everyone so now i've connected the camera to the perf board like i've soldered all the all the wires on and i've also um glued this these pin these pin labels onto the perf board it just makes it easier to see what pins are what um and then i put some hot glue on just to stop the wires from bending too much at the connection because that's they can break easily if you bend them there and then this is the tft screen and the push button so what i've done is i've bent the alternate pins in different directions let's see if we can get a better look at that yeah i've just been in opposite directions just um just so when i solder it it doesn't i don't get a short from the solder it just makes it easier to solder everything on and i've also put this these pin labels i glued these here just so i can see what what they all are quite easily and then i'm just going to put these two together like that with a 1850 18650 battery uh in between and this has got both a 5 volt and a 3.3 volt on it which will come in quite useful okay well i'll put all that together and we'll see how that goes hey everyone so i finished the wiring now and this is what it looks like on the back so this is what it's going to look like on the inside of the camera and i've got the battery the 18650 on this wemos 18650 board and then i've broken out these these pins which is the rx or the tx pins and the zero pin just so i've got easy access if i need to reprogram the esp32 cam and then this is the back of the the um the tft screen and these wires here aren't connected yet these wires are for the for the sd card but i'm not sure how i'm going to connect the sd card to the cam given i don't have much in the way of pins available but i'll see if i can do that later so i'm just going to connect these and we'll see how if the camera works and how it works all right so now i'm going to do a photo of myself and i'm going to see if it works so i'm going to take a selfie there we go and that's my photo there it's got me as 59 person so that's pretty good bad 40 yeah bird 40 and 59 as majority person okay thank you hey everyone so i've made a few changes so this is the original and we can see this the tft and the camera uh pins are actually going through the perf board and for this one i've just mounted the tft screen and the camera onto these um onto these female headers here and i've soldered the female headers at the back of the perf board which seems to be better because the connections seem to be a bit more solid and reliable and also the other thing i've done is i've put a bit of a bigger tft screen on this one here so i'll just switch it on and um see it starts up and now i'll take a photo of my bookshelf and i'll see what it thinks that is all right so that's there 98 scene okay so that's pretty good now when i press the button again kind of goes back to streaming mode and i'll take a photo of my 3d printer and i'll see what the thinks that is 84 seen um six percent person and nine percent bird okay so that seems to be working pretty well let's test it on some people and some birds now thank you hey everyone so i've made a bit of a change to what happens when you switch the camera on so when you switch it on the ip address comes up on the screen which just makes it easier to connect so we'll just wait a second and there it is and i'll show you uh now what um how i train the network and how i wrote the code and everything so i'll see you soon hey everyone so this is the interface for the diy smart camera and the first thing i should mention is that i didn't actually write this interface i took it from another project by robot01 and this was a esp32 auto selfie camera with tft um and i just used i basically i just used their code and i inserted my own classification algorithm which i which i developed using the edge impulse website and tensorflow software so what it shows it just shows the images the pictures i've taken which are stored on spiffs on the esp32 and it gives the classification result uh printed over the image so we see uh i mean these are pretty good so we see this is a scene 100 this is a scene 92 percent this is a bird 66 percent uh a bird 90 selfie of me and i'm 94 person and 5 seen another picture of a bird which is um 97 bird so i guess i should mention that um okay and also there's a fruit bowl which is 98 bird um so i should mention that um not all of the images came out this well um so if we look at this where the birds on the on the pavement here if we took a picture of the pavement without the bird it would probably still give us a fairly high probability of being a bird um just because the training images that we used had a lot of pictures of birds like in places like on a pavement and then there were a lot of i guess indoor scenes and outdoor scenes so we've got this is 99 and this is 100 and also if we took like a group of people like far away it would more likely classify that as a scene so what we have to do is once we've trained our image um it actually once we've trained our network it actually works quite well on types of images that's been trained on so once we've trained our network we have to then uh whichever environment we're going to deploy our camera in we'll have to get a lot of photos from them from that environment and then retrain the network um as we go along so it so it adapts to that to whichever environment it's in um okay so uh that the training images i got from this i i downloaded from kaggle and the these were the animal images uh these were the people images and these were the scene images so i used like three different data sets so i did mix it up a bit and this is the edge impulse uh project which i used um so i've made that public so i'll link that in the in the description and hopefully you'll be able to um hopefully you'll be able to get into that but so this is the data here you can see and i've just had it basically in three different folders and i've uploaded each folder with uh with the label um so i've done three labels i've done one label scene one label bird and one label person so if we look at a scene let's have a look here a scene it'll show us so that's just some of them are outdoor some of them are indoor this is a person um okay i've chosen i've chosen um so like if we look at the at the person images from the database they're all pretty um they're all pretty like central like the person is all central in them so if we have like just a person which isn't central to the image it's not actually going to pick that up and then here some birds are uh so i focus because that's so yeah so i focused a lot on um on pigeons just because that's what i want to develop this for i want to develop this to to scare pigeons off off the balcony um so basically there's going to be like a water spray gun which is going to spray water at pigeons which land on the balcony and um i don't want it to spray people so i wanted to differentiate between birds and people so this is this is pretty good and there's a lot of there's a lot of tutorials and stuff online and uh documentation on how to how to create these networks um so you go through um here you create the network and then it's got this thing eon turner which uh which runs a whole bunch of different networks and you can choose which network you want so this is the one i've been using um this is the one in my code and the reason i use that is just because that is the best grayscale network of all of them the one with the one with the best results um i didn't want to go to uh color because it just it was going to take a bit longer to run and it would take a little bit more memory and then once you've done that you can go to deployment and i've downloaded the arduino library and there's three things you need to do basically if you're going to use the arduino library you can use the c plus library for the espi or you can you can um you can use a binary for the espi and i've tried the binary on the espi and it actually works a lot faster than the esp32 cam but of course the espi doesn't have the the gpios that i needed for this project so i used the arduino library just because i was using the arduino ide 1.8 and there's three things you need to do to get it to work with the arduino library the first thing you need to do is you need to make sure you've got board version 1.0.6 installed in your ide and i'll put all these in the description the second thing you need to do is you need to change your user underscore setup.h file in your tft espi library so in here in this library tft espi um you need to go into user setup and change the change the pins and i'll put all that in the description i'll include my user underscore setup on my github repository and the other thing you need to do is you need to edit you need to edit the ei ei porting file in the um just bear with me i'll just get it okay you need to edit the ei classifier porting dot cpp file um to change the calek to ps underscore calic otherwise it's going to run out of memory okay and um okay and then the classification that i've used i the just the classification function that i've used in the code i got from here from the um from the edge impulse github and it says deprecated but that means that you just have to use 1.0.6 board version if you're using uh if you're using arduino ide or i don't think it'll work in platform io or you can use the espi okay so i'll take you to i'll take you through the code now thanks hi everyone i'm just going to go through the code uh really quickly so i've put this on the github so you can have a look at it and if you've got any questions you put them in the comments um so this is just uh just starting out these are just the pin assignments which are in the the pin assignments for the tft which are in the user setup.h file and then this is the library that i've created on edge impulse website this is the pin for the push button pin 4 which is also connected to the flash so when i take a photo you get the light flashes i'm scrolling down the pin assignments because i'm using the the ai thinker so if you're using a different camera you need to change these um so just setting up setting up the tft configuring the camera um so i've set it to to do a vertical flip here just because of the the camera that um i got with my board it's it's been a bit of a luck of a draw some of them are some of them are oriented one way and some of them are oriented upside down so you might need to change that so when i um so when i connect to the network i print out to the tft the local ip and then that just stays on the screen for two seconds to give me a chance to read it these are these are the uh the requests from the web server from the website which i got from robot uh zero ones project um that's just a reading which reads uh integer from the spiffs file so this is the classify function which classifies um which classifies the image so this calls the the edge impulse function and so all i've done here is i've made it print out to the tft screen and this one here ei print that'll print out to the serial monitor um and then this check possibility just checks if the wi-fi network i'm trying to connect to is within range if it's not it just moves on this way i can use the camera when i'm out and about i'm not in my wi-fi and then i can come back home and i can log on to the wi-fi and see the images i've taken this is to delete the image if i if i want to um if i want to delete it from the uh from the web interface this rgb print i got from the camera web server example that comes with the esp32 with with the esp32 board installed on the arduino ide and then here's the loop that goes um that runs the code so stream will display so if i if this this will toggle between streaming to the tft screen and displaying the last image on the tft screen um then it writes the image that that it takes and after it writes it it doesn't classification and then once it classifies it it prints out the results above that image over that image and then it writes it again so i've got two images um in memory one with uh um one with the result printed above it and one without the result printed above it and the reason i did that is because if i need an image to retrain the network i don't want to have the i don't want to have text printed over it uh it's it's just a bit cleaner so if i if i go back to the website and i download this one here so i need to retrain that it'll ask me if i want to download it and this one that i download won't have the writing the text on it and then um and then i just i just so yeah so that's written there and then i just go back to the beginning so um like i said if you've got any questions just leave them in the comments and thanks for watching and i hope to see you in the next video

Channel: Jonathan R

Views: 22,686

Keywords: ESP32, Esp32-cam, arduino, edgeimpules, Edge Impulse, ESP32 AI, TFT, ESP32 Camera, Image Classification, Smart Camera, DIY smart camera

Id: jTZqRG9eZts

Length: 18min 32sec (1112 seconds)

Published: Mon Sep 19 2022