How to Use YOLOv8 with ROS2

in this tutorial we will use YOLO V8 with ros2 YOLO V8 is the newest state of the art YOLO model that can be used for object detection image classification and instant segmentation tasks we will integrate YOLO into a raws node create a custom message a publisher and a subscriber YOLO V8 is an anchor-free model it predicts directly the center of an object instead of an offset from a known Anchor Box anchor-free detection reduces number of box predictions which speeds up non-maximum suppression non-maximum suppression is a post-processing technique that is commonly used in object detection tasks to eliminate duplicate detections and to select the most relevant bounding boxes that correspond to detected objects also YOLO V8 uses Mosaic augmentation during training by stitching four Images together the model can learn objects in new locations and against different surrounding pixels this helps to generalize the model but Mosaic augmentation can be detrimental if used during the whole training process so it is recommended to disable it for the last several epics firstly let's see inside the YOLO V8 messages package it consists of a messages folder a cmake lists text file and a package XML file in the messages folder we have definition of our custom messages the inference result message contains class names and coordinates of two points of a boundary box the YOLO V8 inference message contains a header message header message is generally used to add a timestamp data to a message this message also contains sequence and Frame It data we also have a defined array of inference result messages each detected object will be appended to this array as an inference result message now let's see the C make lists text file we have to modify this file to compile our messages successfully since we are using standard messages we need to find their definition the find package command enables the compiler to find a package provided external to the project and load its package specific details raw's idle default generators defines the list of default generators which ensures that they are installed as dependencies to generate a message we are using raw's idle generate interfaces method the Amit export dependencies exports dependencies to Downstream packages this is necessary so that the user of the library does not have to call find package method for those dependencies too raw's idle default runtime provides C make functionality for finding and adding runtime dependencies for raw's idle packages there are some modifications to package XML file too build tool depended Claires a Ross package name for a tool that is executed during the build process dependent declares a raws package name that this package needs exact depend declares a raw's package name that this package needs at execution time now let's see the YOLO recognition package in this package we have a node which publishes inference results of YOLO and a node which subscribes to the published topic and draws an image using opencv we are using the latter node just to ensure that we have published inference results of the former node correctly here is the code of the node which publishes YOLO inference results in the first line we are importing Yolo from ultralytics library here we are importing RCL pi RCL Pi provides the canonical python API for interacting with ros2 in this part we are importing image message type which we will use to publish data we also import CV Bridge Library which enables us to do conversions between raw's image messages and opencv images here we are importing previously defined custom messages in this line we Define YOLO model here we are using the smallest model YOLO v8n in this line we Define instance of YOLO V8 inference message here we are creating a subscriber to camera topic in this line we create a publisher to publish inference results as an array here we create a publisher which will publish inference results as an image in this line raw's messages are converted to CV2 image format here inference task is done in this part we are adding frame it and timestamp to a header of YOLO V8 inference message in this chunk of the code we are putting an array inference results for each object here we extract an annotated image from result and convert it to a raw's message finally the image and inference results are published note that we have to clear an array each time a new topic is received this part of the code is common for every simplest node definition firstly raws communication is initialized then we create an instance of a node spin in the third line means that the node will be executed until shutdown now let's take a look at YOLO subscriber code we will use this code to make sure that we have published boundary box coordinates correctly the code structure is very similar to the code we saw previously but there are some differences here we are using a Threading Library threading allows us to have several nodes run concurrently we using it in main function firstly we create instances of two nodes one node will subscribe to a YOLO inference results topic and another node will subscribe to a camera image topic to execute these two nodes concurrently a multi-threaded executor is used after instantiating a multi-threaded executor each node is added to the thread pool the thread pool maintains multiple threads waiting for tasks to be allocated for concurrent execution by the supervising program then threads are created and started for multi-threaded executor in the camera subscriber node we are subscribing to the camera topic and converting it to the opencv format note that we Define the converted image AS Global variable so that another node can use it in the YOLO subscriber node we are subscribing to a topic we published in the previous script here we are defining a publisher which will publish images with boundary boxes drawn using opencv in this part we are drawing a boundary box around each detected object using rectangle function finally we are publishing the image now we are going to install required libraries install python3pip is a package management system used to install and manage software packages written in Python install the ultralytics package if you are installing all related packages it may take about 40 minutes depending on your internet speed install gazebo install the Python 3 colcon common extensions Culkin is a command line tool to improve the workflow of building testing and using multiple software packages it automates the process handles the ordering and sets up the environment to use the packages next install the raws foxy gazebo Ros packages this package provides interface for using raws with the Gazebo simulator now let's do the simulation please download zip file from the Google Drive and extract it to your home directory move these models to the Gazebo directory move to the yolobot directory and execute the Culkin build command connect your joypad to the computer to launch the simulation execute the source command and launch the yolobot launch Pi script open another terminal execute the source command and then execute the Ros 2 topic list command this is the custom topic we have previously defined by executing raw's 2 topic Echo command we can see that recognized object information is being published as we intended now open our vis2 from the add button select image of inference result topic this image shows objects recognized by YOLO now let's check that we are publishing boundary box information correctly open a new terminal execute Source command and execute YOLO V8 Ros 2 subscriber Pi script using the Ros 2 run command from our viz at menu select the image of inference result CV2 topic we can see that boundary boxes drawn using CV2 are in the same locations with that drawn by YOLO

Channel: robot mania

Views: 7,631

Keywords: ROS, YOLO, robotics, python, deep learning

Id: XqibXP4lwgA

Length: 12min 48sec (768 seconds)

Published: Mon May 01 2023