Build a machine learning node for Node-RED using TensorFlow.js

hey everybody welcome to the third video of our introductory AI in nodejs tutorial series i am paul van eck and i'm at the cognitive open that group with IBM this video will be a bit different and will focus on how we can use both tensorflow j/s and something called node-red to make AI enabled event-driven applications now for those unfamiliar node-red is an open source visual programming tool that offers a browser-based flow editor for wiring together devices api's and services running on top of nodejs node-red provides a low code approach for a venture of in programming with the ability to run on your laptop on the cloud and even on low cost hardware like the Raspberry Pi in this video we will show how you can incorporate tensorflow jas into your node-red applications to provide machine learning capabilities enabling a wide variety of interesting AI in IOT use cases like for example a simple safety equipment detector with a Raspberry Pi now there are a number of things you can do combining tensorflow Jas in node red and this video will show you how so let's get started first the easiest way to get started is to simply download and use some of the custom nodes that we have created in order you made publicly available the main in most general nodes that we provide are the node red contribute EF model and node red contribute EF function nodes TF model mode allows you to load and perform inference on any tensorflow GS model within the known bread environment this currently only supports the web friendly json model format but save model support will be added soon the TF function node is just like the core node red function though it also includes the tensorflow JS variables so that you can make any tensorflow jsapi call just a normal function though you'd have to make changes to your node read settings in order to use external packages so this node is more for convenience now with these two custom nodes you are likely set to integrate any arbitrary tensorflow GS model into your flow you just might have to write some code for your pre or post processing steps for additional convenience so we have made some notes for handling some of this pre or post processing for you to demonstrate let's go over some example flows the first flow is an object detection flow running locally on my macbook the idea here is for the flow to recognize objects in an image and annotate the objects with bounding boxes to kick it off an image may be loaded from built-in camera the file system or by injecting the default image the loaded image is passed into the pre processing node as message stock payload where the message object is a JavaScript object used to carry messages between nodes this pre processing node is an example of the TF function though that directly calls the TF know got decode image method with the predefined TF variable the node produces a tester for the image representation as the payload and then passes it to the Coco SSD like node which is an instance of the TF model custom node this loads the cocoa essence the tensor from JS model from an external URL and runs inference on the model the result of the inference is two tensors containing the bounding boxes and the corresponding class predictions however we want a friendly a format for representing our results and not just tensors so we need to post-process this result and for that we use yet another custom node this node will take in these tensors now put a nicely formatted object array containing the top Tom classes their scores and corresponding bounding box coordinates this info can be easily used by the other nodes down the flow for this we just have to be sure the input the path or URL to the labels file the rest of the flow here pertains to just printing out the image into the browser with the bounding boxes and labels also added on we simply have a function known that sits on the input source image and waits for the model results to come in before passing to this bounding box node this custom V box image node takes these inputs and displays an annotated image and that's pretty much it a working object detection map the only real code we wrote at least for the object detection part is a pre-processing and for that it wasn't even that much the rest are just nodes available for download publicly as another example here is a second flow that uses a bird to sentiment model to classify the comments of a YouTube video and then chart the results this also uses the TF function and TF model nodes as well as another custom node called vert tokenizer which converts text into input sensors from the bird model the model returns a soft max output representing the likelihoods of the input being both positive and negative in a tensor array the flow then counts the number of positive and negative comments and outputs the count to a bar chart note this bar chart node is from the node-red dashboard package which allows users to create live data dashboards and widgets to know and read so doing something like this you can easily set up a dashboard the live monitor the sentiment of something like YouTube comments or tweets if you want to check up these example flows yourself be sure to check out the full tutorial link in the description below now the nodes we just went over allow you to quickly bootstrap in the I app however sometimes you might want to do something more sophisticated where you prefer to wrap the processing steps and model prediction into a single node or maybe you want to make a know that there's more user friendly well these are definitely doable as making a custom node grid node is pretty straightforward in node read node is in no js' package consisting of three main files in javascript file defining the nodes behavior in HTML file laying out the knows properties edit dialog and help text and a package.json describing the modules metadata the JavaScript file is where you would wrap your tentacle J's code it would load the tensorflow J's model and run the prediction and once the files are bundled and installed the custom node will be displayed in the editor ready to be wired into a flow and deployed for consistency let's revisit and expand on the cocoa SSD model and code that we covered in the first tutorial of the series using this we can make a dedicated object detection node so starting from scratch the first thing we need to do is make a new node project after initializing the project open the newly created packaged on JSON file and add thgs node as a peer at dependency this needs to be a peer dependency because having it as a normal dependency causes issues when you have multiple custom nodes each relying on thas node multiple tensorflow shared libraries will try to load into the same process prompting an error so yeah just double check it's a peer dependency next just add a node red section specifying the name of the node and its corresponding file now we need to describe the nodes appearance in a node red dashboard so let's create an index.html file where you will add three script tags the first one here will be for defining the nodes edit dialog each property to be passed to the node should have an ID in the format no - input - property name next we add the script tag for registering the node with the editor the category defines which group the place to note on there in the editors palette pane the inputs and outputs describe how many inputs and outputs the node contains defaults set default dies for properties we defined in the previous script tag finally we add the script tag for defining the help text which will provide node information in the nodes info panel with the appearance out of the way we need to now define the custom nodes behavior to do that let's first create in an Excel J's file where we will put this code to register the node function with the node red of runtime and export the node we also have an event listener being registered to the input event which gets called whenever a message arrives at the node our goal here is to take this input message perform inference on it and output a nice payload containing the results to help handle that let's make a new file TF GS tutorial util Dante s and pretty much leverage most of the code written in the first tutorial I won't go into detail on how this code works here but if you'd like to know please check out the first tutorial of the series link in the description below so essentially in this file we have a function for loading the model a function for converting an input image into a tensor and the function for processing the model output into a friendly JSON format we'll reuse some for loops and non maximum suppression we be sure at the bottom to export the functions we want to use in the main index on say s file which are load model process input and process output of course we also need an array of all possible classes for the model so let's make that in a labels ljs file back to index j sv load and save the model using the util file functions we just created then we update the input event listener where we process the input get this and then run the model the output payload will contain the detected objects in a nice JSON response we formulated to test out the node we can use the node package route to make our custom node a local dependency in our node read environment Yauch is a recommended way of working with local packages without publishing anything to a remote registry looks good so we just learned a lot about running and creating nodes using tensorflow js now let's talk about edge devices we're using these technologies would be useful in popular any of the flows we just saw can be run on edge devices where no JSA is supported however there are some caveats for tensorflow Jaso they want to quickly cover as you probably know by now the node back end for tensor village is relies on the tensor flow shared library when you install the thas node package it will install the appropriate tensor flow shared library based on the CPU architecture of your device currently as of this video tends to flow shared library support for arm architectures is a bit spotty depending on the version of Tydeus node you install and the type of ARM architecture you have the corresponding live tensor flow may not be available however tipsiness node does allow users to specify their own custom binary when installing thas node so our team has provided two binaries that can be used one for arm 2d toobut for devices like being raspberry pi and one for arm 64-bit with GPU support for devices like the Jetson nano check out the full tutorial below for more details regarding these and we even show steps for building your own tensor flow shared libraries if you need to go that route in any case with tensor field is enabled on IOT devices we can now make interesting flows the flow here runs on the Raspberry Pi and leverages attached sensors and peripherals similar to the object detection flow this flow detects motion and if motion was detected an image is captured with a USB camera and sent through the TF custom nodes for object detection if the class of interest was detected a specific audio clip is played through the connected speaker we can even change up the model URL to use a model we did some transfer learning on to support other use cases like shown earlier it's all pretty simple and great fun with numerous possibilities so that covers it for this video here you learned how you can combine node read and tends to flow Jas to quickly and easily wire together AI apps for node again be sure to check out the full tutorial linked in the description below there we go more in-depth and cover additional things like deploying these types of apps in cloud environments stay tuned for more videos in this series to learn more about AI and nodejs thanks for watching

Channel: IBM Developer

Views: 35,773

Keywords: Software Development, Application Development, Artificial Intelligence, Machine Learning, Cloud Computing, Data Analytics, Technology, programming, coding, programming languages, learn coding, learn to code, mobile application, IBM, IBM Developer, IBM Code, open source, open technology, Node-RED, TensorFlow, Node.js, Machine Learning Node

Id: bOdlPwWej98

Length: 11min 55sec (715 seconds)

Published: Thu Jun 04 2020