[MUSIC PLAYING] KHANH LEVIET: Hi, everyone. My name is Khanh from
the TensorFlow team. And today I'll show you how to
build smart IoT devices, using Raspberry Pi and TensorFlow. If you are interested
in the IoT space, you probably heard a lot about
Raspberry Pi, the tiny computer that is about the
size of a credit card, but it can be a foundation to
build many custom IoT devices. Raspberry Pi has been used
widely to read and process data from sensors. But now, thanks to the hardware
improvement in recent Raspberry Pi models and the software
improvement in the [INAUDIBLE],, you can now run many complex
AI models on Raspberry Pi to process unstructured
data like audio or video. We use TensorFlow to train and
deploy machine-learning models. However, TensorFlow
was originally built to run on servers with
a lot of computing power and memory. So to deploy TensorFlow
models on Raspberry Pi, you need to use
TensorFlow Lite, which is a framework
built specifically to deploy TensorFlow
models on Edge devices. In this video, I'll
show you how to use TensorFlow to make
your Raspberry Pi hear and see its environment. In particular, we implement
two machine-learning use cases object detection and
audio classification. With object detection,
your Raspberry Pi can recognize objects
from its camera feed. And with audio
classification, it can recognize different
types of sound picked up by its microphone, like
human speech or cat mewing. Let's get started by
preparing your Raspberry Pi. First of all, you need to
make sure that your Raspberry Pi can run TensorFlow Lite. You will need either a
Raspberry Pi 2, 3, or 4, or a Raspberry Pi Zero 2. The Raspberry Pi 1 and the
original Raspberry Pi Zero are not supported
because they are using a CPU architecture that is
not compatible with TensorFlow Lite. Next, you will need to plug-in
a camera and a microphone to your Raspberry Pi. For recording video, you can
use either a Raspberry Pi camera module or a USB camera. And for recording
audio, an easy option would be to use
a USB microphone. Raspberry Pi 3 and 4 also
support Bluetooth microphones, so that's a nice option for you. And this is my own
Raspberry Pi setup. I use a USB microphone
and a Pi camera. Besides those, I have a
few more things connected to my Raspberry Pi. I added a CPU fan because
machine-learning models usually use a lot of computing
power, so I don't want the CPU to be overheated. Then I have a USB-C power
supply and a monitor connected to the mini-HDMI port. And finally, a USB wireless
keyboard and a mouse. This is how the whole
setup looks like. Later in this video, I'll demo
some machine-learning tasks running on Raspberry
Pi using this setup. On the software side,
although, TensorFlow Lite can run on all versions
of Raspberry Pi OS, I recommend you to use the
latest OS Bullseye for the best performance. You can download the
latest Raspberry Pi imager from the Raspberry Pi website to
install the OS on your SD card. Check out the URL in
the video description. And if you are using
Raspberry Pi 3 or 4, make sure to install the
Raspberry Pi OS 64-bit version. TensorFlow Lite runs about
20% faster on the Pi OS 64-bit compared to the 32-bit version. You can choose the 64-bit OS
from the Raspberry Pi imager menu. Once you have installed
the Raspberry Pi OS, the next step is to install
the necessary libraries. We use Python to run TensorFlow
models on Raspberry Pi. So we should start by installing
the tflite-support Python package. This library is used to
run TensorFlow-Lite models. Then you should also
install OpenCV-Python, to capture images
from the camera. Now, your Raspberry
Pi has been setup. Let's write a Python program to
recognize the sound picked up by the microphone. In machine learning, this task
is called audio classification. We use a TensorFlow-Lite
audio-classification model called YAMNet to
write this program. YAMNet can recognize 521
different types of sounds, such as music, speech, vehicle,
cat mewing, and many more. The model takes
one-second-long audio data and returns a list of
probabilities showing how likely the input
audio belongs to each of the 521 predefined classes. Here's a demo of the model
running on a Raspberry Pi. You can see it recognize
speech as I'm speaking. Now let's try some cat mewing. [SCREECHING CAT] And how about some music
playing in the background. [PIANO MUSIC] OK, now let me show you
how to implement this demo. You start by downloading
the YAMNet TensorFlow-Lite model from TensorFlow Hub. For those who don't
know, TensorFlow Hub is a free repo for
TensorFlow models, where you can download many
models created by both Google and the TensorFlow community. You can find the URL
to download the YAMNet model in the video description. The next step is to use the
TensorFlow-Lite task library to run the model. We start by importing
the audio module from the TensorFlow-Lite
[INAUDIBLE] package that we installed earlier. Then we create an audio
classifier using the YAMNet TensorFlow-Lite model. We need to create a tensor audio
to store the input audio data. You can call the tensor
audio from file function to load audio data
from a web file. The second parameter
here, is the length of the input audio required by
the audio classification model. If you have an audio file that
is longer than the model input size, you have to
split the input audio into multiple chunks
and run classification on each of them. And finally, call the classify
function on the tensor audio and get the audio
classification result. The classifier function returns
a ClassificationResult object, which contains a list
of class instances, each representing
a type of audio that the model can recognize. Each class is then has a
name, such as speech or music and the probability score
showing how likely the input audio is of this specific type. Earlier, I showed you how
to do audio classification on an audio file, but like
what I did in the demo, you can also capture audio in
real time from the microphone and continuously classifying it. You need to make some
code changes to do so. The code doesn't change,
has been great till now. So let's focus on what
you need to change. First, instead of creating
a tensor audio from a file, you need to create
an audio recorder to record from the
microphone in real time. Then create a tensor
audio as a placeholder to store the input audio. We'll add data to
the tensor later. Then we start the
audio recording. We create a while loop to keep
the audio recorder running in the background. Then for every half a second,
we load the latest audio data from the audio recorder
into the input tensor. Now, the tensor audio is ready. And we can call the
classifier function to recognize the audio
like we did earlier. You can find a sample code
for audio classification on Raspberry Pi from the
TensorFlow example's GitHub repo. The repo URL is in
the video description. Next, let me show you how to
run this audio classification sample on Raspberry Pi. Go to terminal and first we
will clone the repository with the examples. on Then go to the folder
with the audio classification sample. The next step, is to run
the setup shell script to install the necessary
Python packages and download the TensorFlow-Lite
model from TensorFlow Hub. And finally, run
Python classify.py to start the audio
classification program. And that's all you need to
run audio classification on Raspberry Pi. And next, let me show you
how to recognize objects in images or videos captured
by the Raspberry Pi camera. In machine-learning
terms, this task is called object detection. We use a model called
EfficientDet-Lite for object detection. This model can recognize 70
types of different objects. The model takes an
image as the input and then returns
a list of objects that it recognized
together with the location of the objects in the image. TensorFlow has a
computer-vision task called image classification,
and some of you may wonder how
they are different. So when given this image,
an object-detection model can tell you that there are
three pairs and the location of them in the image. On the other hand, an
image-classification model, can only tell you that
it's an image of pears. Here's a demo of the
efficientdet_lite model running on a Raspberry Pi. You can see that it
can recognize objects like a keyboard, a
remote, our water bottle. Now, let me show you how
to implement this demo. It's very similar to the
audio classification program that we created earlier. We start with downloading
the model from TensorFlow Hub and then use TensorFlow-lite
test library to run the model. Let's start by importing the
vision module from the test library, then initialize
an object detective with the TensorFlow-Lite
model that you just downloaded from TensorFlow Hub. Now, you need to prepare a
TensorImage to fit the input image to the object detector. It can load an image
directly from a file using the from_image function And
then call the detect function with the tensor image to
detect objects from the image. So detect function gives you
a list of detection objects, each representing an object
that it found from the image. Initial detection
object, there's an object name, the confidence
score, and the bounding box indicating the object location. Earlier, I showed you how to
detect objects from an image file, but you can also
run object detection on live video feed captured
from the Raspberry Pi camera. To do so, first, you will
need to import a cv2, which is the OpenCV module, then
create a video-capture object to capture your videos
using the camera. Then create a loop to
continuously capture frames from the camera. Because OpenCV uses
the vcr format, you will need to convert them
to RGB as required by the model. The frame is now a NumPy array,
so we can call the tensor image from_array function to create
an imported TensorImage. And finally, you can
pass this TensorImage to the object detector
like what we did earlier. Here we use OpenCV
for image capturing, but you can also use the Pi
camera library if you want. The both capture video
frames as NumPy arrays. You can find the Raspberry
Pi object-detection demo that I showed you earlier in
the TensorFlow examples GitHub repo. The link to the repo is
in the video description. The object detection module runs
at about six frames per second. If you need your
model to run faster, you can connect a
Coral USB accelerator to your Raspberry Pi. And if you want to
learn more about Coral, check out the iotop code
Coral your next generation IoT for AI. And you can find the
URL to the session in the video description. Now, I've learned how
to do object detection and audio classification. Test library supports many
more machine-learning tasks, such as image segmentation
or audio search. For example, you can feed
a model a piece of music, and it will tell you
which song it is. Besides the TensorFlow
Lite test library, you may also want to
check our MediaPipe. It's a library that we use
on top of TensorFlow Lite and provides easy to use
APIs for many more use cases, such as the combined face
mesh, pose, and hand detection. Please check out the links
in the video description to learn more. Now, you have learned how to do
audio classification and object detection using a
model from TensorFlow. Although the models can
recognize many different types of sound object, there
will be many cases where you want to detect
things that the model don't know about. That's why you need to
train a custom model. TensorFlow-Lite Model
Maker is a Python library that allows you to train custom
models using your own data set, in just a few lines of code. Let me show you how to do that. For example, if you want an
object-detection model that can detect Android figurines, then
the efficientdet_lite model from TensorFlow Hub,
cannot do that for you. You'll need to train a custom
object detection model. There are three steps to do so. First, collect and
label the training data, then train a custom model using
TensorFlow-Lite Model Maker, and finally, deploy
the model by replacing the model you downloaded
from TensorFlow Hub with the custom model. You start by taking a lot of
photos containing the objects that you want to detect. You can start with a dozen
images for each type of object that you want to detect and
train a prototype model. But to get a
production-quality model, you should aim for having more
than 100 images for each class. Then use an image-labeling tool
to label the objects that you want to detect in the images. I prefer a tool
called Labeling It's an open-source tool that you
can download from GitHub. You can find a link
to download the app in the video description. Once you have finished
labeling your images, you can export the label data
to start training the model. To train a
machine-learning model, you need a more powerful
computer than the Raspberry Pi. My recommendation is to
use Google Cola, which is a web-based tool to
run Python programs. It offers you free
CPUs on the Cloud so that you can train custom
machine-learning models faster. Of course, you can also
use your laptop or desktop to train the models if you want. You start by installing
Model Maker on Google Cola or on your computer using pip. Just run pip install
tflite-model-maker. And next, is a Python
code to train a custom model using Model Maker. You start with choosing
the model spec or model architecture for
your custom model. Here we use
efficientdet_lite0, which is the same architecture
with the pre-trained model that we downloaded earlier
from TensorFlow Hub. Then you load your
data set using the object-detector DataLoader. Then you start
training your model. And after training the model,
we use the test data set, which the model
hasn't seen before to evaluate the model accuracy. And finally, you
export the model to deploy on Raspberry Pi. Now, as you have finished
training the custom model, let's switch over to
Raspberry Pi to deploy it. Deploying the custom
model is very, very easy. You can reuse the
code in the demo app, and just replace the pre-trained
model with your custom model. Here's the custom
model in action. You can see that it now detects
Android figurines rather than Zener objects. It is very easy, isn't it. You can also use Model
Maker to train custom audio classification models. And if you want to learn
more about using Model Maker, please check out the tutorials
on the TensorFlow-Lite website. The URLs are in the
video description. So that's all the
content for today. If you have any
questions or if you want to discuss with others
about running TensorFlow on Raspberry Pi, you can
join the TensorFlow forum. It's a place for our
developers and contributors to engage with each other
and with the TensorFlow team. Go to discuss.TensorFlow.org
and create your account today. Thank you for
watching, and don't forget to subscribe to
the TensorFlow YouTube channel for more
machine-learning tutorials. [MUSIC PLAYING]
