ESP32-CAM Video over Websockets Sensor Visualization MongoDB, Chart.js, node.js: How to

hey thanks for tuning in welcome to nomadic geek are you tired of losing your valuable sensor data and wish you could store it securely over time for future analysis or have you ever dreamed of Designing a stunning dashboard to visualize and make sense of your sensor data through interactive graphs well you're in luck because in this episode of the geeky surveillance system tutorial series we'll guide you step by step on how to store your sensor data from our modules and create captivating visualizations get ready to embark on this journey to unlock the full potential of your iot devices in our previous videos we've set up sensors in esp32 cam modules to stream data to our node.js server via websockets and optimizing the code for maximum efficiency with cluster multi-threading if you haven't checked out those videos I highly recommend you do so as this tutorial builds upon that Foundation although it's not essential for understanding this video watching them afterward will provide valuable context as someone who wants to monitor data like temperature humidity and battery usage patterns in my van and analyze them over time I need an effective way to visualize these changes that's why in this episode we'll dive into the world of data visualization and explore how to create a web browser dashboard with interactive graphs that display our stored sensor data in a user-friendly manner so let's get started on this exciting Adventure if you like this video don't forget to give it a thumbs up and hit the Subscribe button and notification Bell so you know when I post more content if you have any thoughts questions or feedback please don't hesitate to share them in the comments section below stay tuned and let's get started a database is a digital storage system designed to organize manage and retrieve structured data efficiently they come in various types such as relational nosql or graph databases each tailored for specific data models and use cases to interact with the data databases use Query languages like SQL for relational databases or specialized apis for nosql databases the primary purpose of a database is to store information in an organized manner allowing for seamless data retrieval and manipulation in various applications mongodb is a popular nosql database designed for handling large amounts of unstructured data unlike relational databases which use tables and strict schemas mongodb employs a flexible document-based model using json-like objects called Beeson this flexibility allows for easy storage and querying of complex data structures such as nested arrays and documents making it ideal for handling diverse hierarchical or evolving data requirements mongodb is scalable High performing and widely used in applications that require a more Dynamic and versatile approach to data storage and management shark.js is a popular JavaScript library used for creating interactive responsive and customizable data visualizations in web applications it enables developers to easily generate charts and graphs using the library's extensive collection of built-in visualization types such as line bar pi and Scatter Plots by providing a simple API graph.js allows seamless integration with your web application's data empowering users to explore and make sense of their data in a visually appealing and intuitive way first we need to install mongodb on our system go to the website and scroll down ignore the atlas version and continue further down to choose your system then click the download button after downloading mongodb locate the installer and open it to begin the installation process choose the option run Service as network service user select the install mongodb compass while not strictly necessary for this tutorial it can be a useful tool to check that the data is being properly written to the database click install the installation process may take some time but once it's complete we can start using databases in our projects foreign we start by examining the changes in our package.json file to see what we have added as mentioned earlier mongodb is a popular nosql database for managing large amounts of unstructured data Mongoose is an object data modeling library for mongodb and node.js it provides a simple schema-based solution to model your application data making it easier to work with mongodb in the node.js environment by including Mongoose you can use its features for validation querying and data manipulation we've also included the dot in package in our project that end is a useful tool that simplifies managing application configurations by centralizing them in a single file this makes it easy to modify settings without digging through the code base and it provides a convenient way to separate configurations from the main application promoting maintainability and security by using that end you can effortlessly adjust configurations as needed keeping your application flexible and adaptable here we can observe several settings that have been moved from the code base to our DOT in file these include the client HTTP Port the websocket port for client connections the frequency at which sensor data is updated in this case once every 150 milliseconds as well as settings for object detection frequency and score threshold additionally we have mongodb settings such as the database name IP port and maximum pool size first let's take a look at the server.js file in the section where we include libraries not much has changed but two important lines have been added we have the dot end this line enables the use of the dot EnV configuration file and integrates its functionality into our project next we've included the sensor model let's take a quick look at it first we include the mongooseconfig.js file we'll take a closer look at this file after we've reviewed the sensor model file here we Define the Mongoose schema for our sensor data the schema called sensor schema specifies the structure of the data we'll be storing in our mongodb database this makes it easier for us to interact with the data and ensure its consistency across all sensor entries here's a breakdown of the fields in the sensor schema sensor ID this is a required field of type string representing the unique identifier for each sensor timestamp this field is of type date and is set to the current date and time by default using date.now temp hum Co LPG and smoke these fields represent different sensor values such as temperature humidity carbon monoxide LPG gas and smoke levels they are all of type number and you can add more sensor values to the schema as needed with this schema in place we can now manage our sensor data more effectively and ensure it follows a standardized format when stored in the database this function get all sensors data is an asynchronous method defined as a static method on the sensor schema static methods are functions that belong to the schema itself rather than to instances of the schema the method retrieves all sensor data from the database it uses mongoose's find method with an empty object which means it doesn't apply any filters to the search the function then returns an array of all sensor records if an error occurs during the process it logs and rethrows the error for the caller to handle get all sensor IDs is another asynchronous method that retrieves a list of unique sensor IDs from the database it defines an aggregation Pipeline with two stages dollar group to collect all unique sensor IDs and dollar project to remove the underscore ID field and create a new Field named sensor ID the function then executes the aggregation using mongoose's aggregate method and returns an array of unique sensor IDs it handles error in the exact same manner as earlier function get sensor data by ID yet another asynchronous method fetches sensor data for given sensor ID from the database the function uses mongoose's find method to query for data matching the provided sensor ID it then Returns the retrieved sensor data the git sensor data by ID between timestamps function provides a more targeted approach to data retrieval by specifying a sensor ID start time and end time it returns sensor data within the defined time range this function is particularly useful when you want to analyze sensor data from a specific period as always if an error arises it logs the details and rethrows the error for further handling the save sensor data function adds a delightful twist to our sensor model the ability to store sensor data it accepts the sensor's object as input and through the create method stores this data in the database upon successful data storage it logs A celebratory message however if an issue arises it logs the error and rethrows it for proper handling the function then resolves to true as a final touch this part of the code creates a new Mongoose model called sensor using the previously defined sensor schema then the Mongoose instance in the sensor model are exported making them accessible for use in other parts of the application now let's take a look at the mongooseconfig.js file which we included at the top of the file first we import the Mongoose Library which will help us work with mongodb we then include that ends to manage our environment variables making our app more flexible and using the settings we defined in the dot in file shown earlier we sit mongoose's debug mode to false to reduce unnecessary console output next we use the Mongoose connect function to establish a connection to our mongodb server with the specified options taken from the dot inch settings once connected Mongoose logs a confirmation message to the console if an error occurs during the connection Mongoose will log the error details for us when the connection is closed Mongoose will also let us know by logging a message finally we export the configured Mongoose instance to be used throughout our application back in our server.js file we have added commands that can be listened to from web clients foreign we have added two operations sent through websocket messages get sensor IDs when a message with the get sensors operation is received the server queries the database for Unique sensor IDs using the sensor.get All sensor IDs function it then logs the retrieved sensor IDs and sends them back to the client through a websocket message containing an operation field set to send sensors and a sensor IDs field containing the sensor IDs get sensor readings when a message with the get sensor readings operation is received the server extracts sensor ID start time and end time from the data dot command it then queries the database for sensor data between the specified timestamps using the sensor.get sensor data by ID between timestamps function after logging the sensor data the server sends it back to the client in a websocket message containing an operation field set to send sensor readings and a sensor data field with the sensor data all this will be more clear when investigating the client-side code further on in sensors.js we have the same inclusions that end and sensor model all other inclusions are the same as in the last video of this tutorial series then all there is left to do is to utilize the safe sensor data function within our sensor model as previously discussed to persistently store sensor data in our mongodb database [Music] in the client.js file which serves as an interface for every connected web browser client numerous modifications have been implemented specifically client 2 has been designed with the sole purpose of displaying storage sensor data in graphical format as evidenced by our file tree and server.js roots as opposed to client 1 which undertakes a broader range of tasks client 2 has been customized to fulfill the specialized requirements of graphical data presentation first we create a websocket connection to the websocket server next we create several HTML elements these include a drop down menu for listing all sensors that have data in the database two daytime picker inputs for selecting start and end times and a button for getting sensor readings we then create a div element called controls wrapper and append the drop down date time Pickers and button to it following that we create a canvas element for rendering the chart and we append the controls wrapper and the canvas elements to the main wrapper div that is already present in the HTML now we set up an event listener for when the websocket connection is opened when it opens we send a connecting message to the server and request the sensor IDs by sending a git sensors message next we Define an on message event for the websocket when a message is received from the server we check the operation type in the data send sensors or send sensor readings and call the corresponding function to either populate the sensor drop down or update the chart [Music] we had an event listener to the git sensor readings button when the button is clicked we retrieve the selected sensor ID and the specified start and end times we then send to get sensor readings websocket message to the server including this information populate sensor drop down function takes an array of sensor IDs and adds them as options to the sensor drop down allowing the user to select a sensor finally the update chart function is responsible for taking an array of sensor data and using it to create a line chart with multiple y-axis one for each sensor value temperature humidity Co LPG and smoke the chart is created using the chart.js library and if a chart already exists it is destroyed and replaced with a new chart now let's take a look at what it looks like I'm starting the server up we can directly see that a sensor I have connected is sending sensor values to the server and that the server is successfully saving them to the database in the top right corner I have client 2 for the graphs and in the lower right corner I have the standard client which displays all the camera feeds as well as real-time sensor data let's choose some dates to start date and end date so that we can view the stored data for this particular sensor module in the console we can see that it fetches all the data points and in the client 2 browser the graphs are shown we see temperature humidity Co LPG gas and smoke foreign so let's take a look at how it all fits together to make it clearer I will go through the entire flow from drawing the graphs to the stored data in the database and from storing data from the sensors to the database we'll begin with the client.js file the update chart function retrieves sensor data from the server through a websockets called and it's used here to draw the values to the graph these messages are sent from the server.js file so let's take a look at that when it's in turn receives a message from the client it fetches the data from the database and returns it by calling the client and triggering on message function in the client [Music] so let's take a look at the get all sensors IDs function in the sensor model foreign depending on the sensor ID let's open mongodb compass which we installed along with mongodb we connect our database is Test 2 as we stated in the dot in file here we can see all sensor data stored in the database we can display the records in any way we prefer by clicking the buttons foreign back in the code let's open the sensor.js file which represents all of our sensors as you may recall from an earlier episode each sensor has its own instance of this file on the CPU thread cluster if you haven't seen the episode on multi-threading I highly recommend watching it after this video we call the save sensor data function on the sensor object which as usual resides in the sensor model.js file this function creates the record this concludes this episode of the tutorial series if you enjoyed what you saw make sure to hit the Subscribe button and give this video a big like that way you'll never miss a single one of my future uploads and if you have any ideas for topics you'd like to see me cover in the future drop a comment below your feedback and suggestions are always appreciated so until next time farewell and keep on learning [Music] [Music]

Channel: Nomadic Geek

Views: 960

Keywords: Node.js, MongoDB, Chart.js, sensor data visualization, web development, JavaScript, tutorial, IoT, Internet of Things, data visualization, database, Graph

Id: TmZh2nUlgco

Length: 21min 57sec (1317 seconds)

Published: Tue Apr 11 2023