AWS re:Invent 2020: Use computer vision at the edge to improve operations with AWS Panorama

Video Statistics and Information

Video
Captions Word Cloud
Reddit Comments
Captions
Hi, I'm Mike Miller, Director of AI Devices with AWS, and General Manager of the AI devices team. In 2017, my team launched the AWS DeepLens device, designed to give developers hands on experience with computer vision through a machine learning enabled webcam. Since then, we've talked to hundreds of customers who have used that product to see computer vision in action, and used it to gain intuition about how computer vision can help solve business problems. We're incredibly excited to bring a production ready scalable solution to our customers that took inspiration from that product. With that, I'd like to thank you for joining me as I introduce AWS Panorama, and how companies can use computer vision at the Edge to improve operations. Let's start with a quick look at what we'll cover today. First, an introduction to computer vision, and how it's being used across industries. Then, we'll explore running computer vision at the Edge and how it's interesting, followed by an introduction to AWS Panorama, and how customers are using it to solve problems. We'll have a short demo, where you can see AWS Panorama in action, and I'll close with discussing how you can get started with AWS Panorama. First, computer vision is a term we use when discussing AI and machine learning to process image or video in a human like way. So for instance, we use computer vision to detect or classify objects in an image, like determining if a car or truck is present, or recognizing a whale based on characteristics of its tail fin. Computer vision can also determine the boundaries of where a detected object lies inside an image. It can be used for text and character recognition, pose detection of people, and even identify activities like hand washing. Companies across industries have recognized the value of automating previously manual inspection tasks using computer vision, developing new innovative solutions, and gaining real time insights into their business processes. For instance, in manufacturing, we see computer vision being used to improve industrial processes, by automating product quality inspection, or tracking the movement of goods and inventory around the plant floor. With its ability to detect people, computer vision has become a critical tool for improving public health, such as monitoring social distancing, and assessing density of people within a facility. In retail, computer vision is being used to enhance customer analytics, based on accurate metrics about how many customers visit, what products they interact with, and where there are opportunities to layout, or operations to enhance the shopping experience. Finally, CV can be used to enhance worker safety in a variety of locations, helping to prevent workers from coming into contact with potentially dangerous equipment, detecting falls, or making sure that hard hats are on before entering hazardous zones. Customers are getting more excited about these computer vision use cases. But many are finding that it's difficult to implement and scale these solutions, especially in sites where limitations exist in connecting to the cloud or sending data off site. These customers want to optimize their computer vision for the Edge. Many use cases require real time responses to what's being captured by the video cameras. For instance, a pharmaceutical company may want to inspect vaccine vials in a very fast moving conveyor belt to validate fill levels. They need sub second response time to maintain throughput, where a round trip to the cloud would be infeasible. Some customers operate in environments where there are bandwidth constraints related to cost or infrastructure, or simply intermittent connectivity. And in these instances, it's either very expensive to send high bandwidth video streams to the cloud, or customers simply don't have sufficient bandwidth to do it. They would prefer to be able to process all of that video data at the edge and then only optionally, send select data back to the cloud. Finally, customers in regulated industries may need to process their data at the Edge due to data privacy or governance restrictions. These customers have corporate security policies or regulatory requirements that restrict their ability to send video and image data back to the cloud. As a result of these constraints, customers are looking for a solution that allows them to capture and process image and video data where it resides, at the Edge. So, we developed AWS Panorama to help these customers bring computer vision to the Edge. AWS Panorama is a new machine learning appliance and SDK, both of which allow organizations to bring computer vision to their on premises cameras, to make automated predictions with high accuracy and low latency. With AWS Panorama, companies can use compute power at the Edge without requiring video stream to the cloud to improve their operations, by automating and visual inspection tasks, like evaluating manufacturing quality, finding bottlenecks in industrial processes, and assessing worker safety within their facilities. Let's talk about Panorama components. At the Edge, Panorama supports devices on premises that are optimized to run computer vision applications in real time, including appliances that can connect to existing IP cameras, and run computer vision on those video streams, as well as new smart cameras with onboard processing, which can directly run a machine learning application on the capture video. Customers can use the AWS management console for Device Management, application development, and deployment. Customers use the console to register and manage their Panorama Edge devices, develop the computer vision applications needed using familiar AWS tools such as Amazon SageMaker and AWS Lambda, and deploy these applications to one or many Edge devices. Panorama takes care of automatically optimizing machine learning models for the Panorama Edge hardware, ensuring that applications run fast without additional configuration overhead. Finally, after executing computer vision applications at the Edge, the application results and alerts can be sent to the on premises line of business systems, or AWS services like Amazon S3, Kinesis Video Streams, or CloudWatch, for extra action and analysis. Let's double click a bit on how Panorama works and look at each technical component. Starting at the left, customers who want to improve operational processes begin with the machine learning model, either by supplying one they've trained on Amazon SageMaker, or using a pre-built model from AWS or third party providers. Customers use the management console interface to register and provision their Panorama devices, whether appliances or Panorama enabled cameras. This flow includes specifying the network configuration for the device, downloading that configuration, including the encrypted credentials to an included USB stick. Your Panorama then reads off that USB stick and uses it to get connected, and provision the appliance to your Panorama account. The management console also allows customers to pair their trained ML model with business logic specific to their use case and integration points. So for example, you might use ML to recognize pedestrians who come too close to heavy machinery, and you would use business logic to trigger an alert, like a siren, and log data in a facilities management system. The business logic is created with familiar AWS tools like AWS Lambda, making it easy to quickly build and iterate. For example, you can process ml predictions, like only taking action when the model has a certain confidence threshold, or sending data to a local line of business systems or cloud based AWS services. After pairing your model and the business logic, you deploy this bundle to Panorama devices. Panorama automatically optimizes your ML models for the selected Edge device, removing overhead and complexity in managing multiple Edge devices. The ML model is then run on the Panorama appliance or Panorama enabled camera at the Edge, applying high accuracy and low latency predictions to video. The application results processed by the business logic, then integrate with on premises line of business applications or automation if needed, to route results to familiar AWS services. So let's talk in detail about the Edge devices. As we do at Amazon, we worked backwards from what our customers needed. And as we did this, we recognize that many customers have fleets of existing cameras deployed for manual or reactive monitoring. So we asked ourselves, how could customers add computer vision to those existing cameras without needing to touch them or upgrade the hardware? We developed an Edge appliance optimized for computer vision applications that can automatically discover and connect to those existing IP cameras to run multiple machine learning models. This AWS Panorama appliance sets up in minutes, and includes multiple Ethernet ports for redundancy or to connect to different subnets. Once connected to your local network, it uses the ONVIF industry standard for discovering and connecting to existing IP cameras. With an IP62 rating, it's dustproof and water resistant, meaning it's appropriate for use in harsh environmental conditions. This way, customers can use the appliance to bring computer vision to where it's needed in industrial locations, or they can use the included rack mounting hardware to mount these half rack wide units to a standard rack shelf in a standard server rack. It uses Nvidia's powerful Xavier AGX platform to run multiple machine learning models across multiple streams, analyzing but not storing video from multiple cameras in parallel. Later in 2021, customers will have more options for Panorama enabled devices across a variety of manufacturing partners, for Edge gateways and smart cameras that span a range of form factors, price points and capabilities. We've partnered with the leading silicon vendors, Nvidia and Ambarella, to support the Nvidia Jetson product family as well as the Ambarella CV2X product line. This enables our manufacturing partners to build Panorama enabled Edge gateways and smart cameras that can meet customer's unique Edge computer vision needs. These partners, like ADLINK, Access Communications, Basler AG, Lenovo, Stanley Security, and Vivo Tech, are using the Panorama device SDK, which includes a device software stack for computer vision, sample code, APIs and tools to enable and test their respective devices for the Panorama service. So let's talk about these applications. Currently, 98% of enterprise video recorded is never analyzed, and the 2% that is analyzed is usually via human review. Panorama unlocks the insights in this enterprise video by making it easy to build and deploy computer vision applications to analyze video in real time. Customers can start by using computer vision models, either ones they train themselves, or take advantage of computer vision models from a variety of independent software vendors. Even if a customer starts with a very simple computer vision model, a wide array of use cases can be unlocked. So for example, a machine learning model that has been optimized for detecting faces or people in an image can be used as a basis for an array of use cases, you could simply count the number of people seen, which enables crowd counting and density monitoring applications. You can count people who enter and exit a doorway or path, giving retail businesses insights about foot traffic. You can determine when a person comes too close to another person to monitor physical distancing, or too close to a restricted area, for example, near heavy machinery, to monitor workplace safety. Once a person is detected, you can further determine if they have specific personal protective equipment on, such as safety vests, hard hats, or masks. With Panorama, you then pair business logic with that computer vision model so that you can take action and integrate results like crowd sizes, or the count of people, or the workplace safety observations, into enterprise line of business applications, or other AWS services to trigger alerts, emails, or take other real time actions. So let's talk about a few real customer use cases. Fender Musical Instruments Corporation is the world's foremost manufacturer of guitars, basses, amplifiers, and related equipment. Fender has been using AWS Panorama to improve their guitar assembly process. There are many unique parts that go into each guitar, and Fender relies upon a skilled workforce to craft each part. AWS partnered with Fender to build and train custom machine learning models that can identify various guitar parts, such as the guitar neck and headstock you see on the screen. These models can extract unique identifiers such as serial numbers, regardless of their position in the image, recognize the character, and use those identifiers to track material usage, calculate how long various assembly steps take, and identify bottlenecks in real time. Cargill brings food, agricultural, financial and industrial products to people who need them all around the world. Cargill is excited about how computer vision can help them innovate new processes, and optimize existing ones. First, they're looking to optimize their yard management at their greeneries, by assessing the size of trucks coming into their yard, and determining the optimal loading dock for each truck. They would do this by training a truck identification model, which identifies trucks visually and maybe buckets them by size. They would author business logic that matches these truck sizes to available loading docks by integrating with their yard management systems, and using those systems, can direct trucks to the best dock for them. Cargill also manages large and complex manufacturing and processing plants, and is excited about using computer vision to track the movement of assets in these track plants to remove bottlenecks. With Panorama, they can build solutions which can scale across many sites. For example, if there are identical processes in each place at multiple plants, they can use Panorama to deploy and manage the same application across each of those sites, allowing them to more easily scale their optimizations. Finally, BPX Energy is a division of BP, which oversees onshore continental US oil and gas exploration and production. BP is working closely with AWS to build an IoT and cloud platform that will enable continuous improvement of the efficiency of their operations. One of the key areas that will be part of this effort, is the use of computer vision to help solve issues related to worker safety and security. They're going to use computer vision to automate the entry and exit of trucks to their facilities, and verify that they have completed the correct orders. This could be through automatic recognition of trucks based on their identification numbers, or even license plates. They're also excited about the possibilities for computer vision to help the workers keep safe in a number of ways, from monitoring social distancing, to setting up dynamic exclusion zones, ensuring that pedestrians don't get too close to dangerous machinery, as well as detecting oil leaks. The ability to deliver all of these solutions on a single hardware platform with an intuitive user experience is what has BPX so excited. We're looking forward to what they can do with Panorama. Now, let's have a short demo, where you'll be able to see Panorama in action. In this demo, my colleague Fu is using an AWS Panorama developer kit, which is a version of the AWS Panorama appliance, designed to make development and debugging of your Panorama applications more streamlined. He'll start the demo by registering a new AWS Panorama appliance developer kit, and walk through the creation and deployment of a simple computer vision application. Let's get started. AWS Panorama is a new machine learning service, which gives you the ability to make real time decisions to improve your operations by giving you compute power at the Edge. In this demo, I'll guide you through setting up a Panorama appliance, connecting the appliance to IP cameras on your network, and deploying your first application. I will be using the Panorama appliance developer kit for this demo. Developer kit is not meant for production use, as it allows root access for developers to rapidly build and test [INDISCERNIBLE 00:28:56] applications. Inside the Panorama console, choose Get Started and choose Setup Appliance. First, give your appliance a name. Optionally, you can add description and tags to make managing multiple appliances easier. Second, configure the network settings for the appliance to connect to the AWS Cloud. We recommend using Ethernet for the initial setup. You can configure static IP and DNS settings in advanced network settings. Since this is an appliance developer kit, you can configure SSH access right in the console. Follow the instructions on screen to download the configuration file and transfer it to the appliance using the provided USB flash drive. Your appliance will take a couple of minutes to configure and connect to the AWS Cloud. After your appliance comes online, connect to the IP cameras you want to use with Panorama. The appliance can automatically discover IP cameras that comply to the ONVIF type S protocol on the same subnet. Alternatively, you can manually specify RTSP camera streams on the subnet for the appliance to connect to. After the appliance has found your cameras, you can add the cameras credentials to access their video streams. Your appliance is now ready for use. Let's deploy your first Panorama application. A Panorama application uses a machine learning model to make a prediction about what is in the video, suggest detecting people, identifying objects, or determining positions of items in the video. The machine learning model is paired with business logic code that is customizable to take some action based on the prediction. The code can connect to the cloud to trigger an alert, write data to a database, or integrate with business systems on your network. The machine learning model and code for this demo is available at our Panorama samples GitHub repo. Let's give your application a name. Then, let's add the machine learning model which you would use to generate predictions. If you train your model using SageMaker, you can import your models using the SageMaker training job ID. Panorama supports models trained using TensorFlow, PyTorch, or MXNet. To use a custom model with Panorama, specify the model location, name, and data input parameters. Data input parameters help the machine learning compilers optimize the model to run on the Panorama appliance. You can even use multiple models within one application with Panorama. Now that you have imported your machine learning model, let's work on your business logic code. In this example, you use the AWS Lambda console to create the business logic code to count up the number of people in the frame and display it on the HDMI output. For production use cases, you can integrate the machine learning models results with systems on your local network or in the cloud. After you finish editing your code, publish the version from the Lambda console and return to Panorama to import the code for using your application. You've just created your first machine learning application with AWS Panorama. Now let's deploy the application to your appliance. First, choose the Panorama appliance you want to deploy to. Second, choose the camera streams you want to feed into the application. Panorama applications can process multiple camera streams in parallel. Finally, select deploy. For debugging purposes, you can view the video output on an HDMI monitor connected to the appliance. Here I'm showing a sample video of what you can expect. You can also monitor the status of your deployment in the panel on the console or to logs in CloudWatch. Now that you've deployed your first application, you're ready to start creating your own Panorama applications. Check out our getting started guide, or visit our open source Panorama samples GitHub repo for more sample code and tutorials. So, now that you've seen a little more detail about how AWS Panorama works, let's talk about how our community of partners can accelerate your Panorama usage. To get started with Panorama quickly, you can work with our community of partners to acquire Panorama devices, install them, and build and deploy CV applications that meet your unique business case needs. We have a large community of these businesses who can bring their expertise in computer vision to help you get the most out of AWS Panorama. CV model and application partners can help you build a computer vision models and applications for your specific use case. System integrators and consulting partners can accelerate your CV journey by helping you explore and implement solutions, while distribution and integration partners can assist you with device acquisition and system integration. Here's an example. Parkland is Canada's and the Caribbean's largest, and one of America's fastest growing independent suppliers and marketers of fuel and petroleum products, and a leading convenience store operator. They've partnered with TensorIoT, which was founded on the instinct that the majority of compute is moving to the Edge and all things are going to be coming smarter. TensorIoT and Parkland Fuel are using Panorama to gather retail analytics that will help drive their business, such as counting patrons, and analyzing foot traffic across locations and times, to optimize their staffing, marketing programs, and product promotions. So how do you get started? Well, first, we've launched AWS Panorama in preview. You can sign up for the preview so that you can get started building computer vision applications right away. Just visit aws.amazon.com/panorama and look for the preview signup link. Access to the preview also includes early access to purchase an AWS Panorama appliance developer kit. The developer kit makes it easy for customers to rapidly build, test and debug their computer vision applications. It was designed as a seamless way for customers to transition their applications, from development in the preview to the AWS Panorama appliance when it's available, ensuring that customers can move from proof of concept to production as easily as possible. Well, that wraps up today's presentation on using AWS Panorama to bring computer vision to the Edge. I hope you enjoyed the presentation. I'm excited to see what our customers can do with Panorama. So remember to sign up for the preview at aws.amazon.com/panorama.
Info
Channel: AWS Events
Views: 1,132
Rating: undefined out of 5
Keywords: re:Invent 2020, Amazon, AWS re:Invent, EMB015, Artificial Intelligence, Machine Learning
Id: _qX_oEtE7qI
Channel Id: undefined
Length: 25min 49sec (1549 seconds)
Published: Fri Feb 05 2021
Related Videos
Note
Please note that this website is currently a work in progress! Lots of interesting data and statistics to come.