Use These! Jetson Docker Containers Tutorial

[Music] and just as tropical storm Hillary was making its way into Los Angeles a 5.1 magnitude earthquake shook the nearby town of Ojai yes a hurricane I took it as a message from above you should be using Docker in your Jetson development [Music] sometimes when you are developing on a Jetson you feel like you are building a castle in a swamp you build your project and then new releases of the operating system or libraries make it obsolete undaunted you rebuild again and again so I built a third one that burned down fell over then sunk into the swamp but the fourth one you can use Jets and containers to help you get out of this death of spiral one of the features the Linux kernel offers is containers Jetsons used Docker as their default container manager think of containers as isolated environments for computer programs it's a snapshot of a program and all its dependencies down to the operating system level the kernel can launch multiple containers isolating and protecting each from the rest of the system there are three main components to Docker a Docker file is a script which tells Docker which components should be in the container and how to arrange them the components describe the library scripts and environments of the application a Docker file is built into a Docker image the image represents a template to load into memory includes the actual libraries and code to use you can then run the docker image to create a Docker container the container represents a running process on the computer because the containers are completely independent multiple containers of the same image may be run simultaneously now white use containers as an example let's take the l4t-ml docker image it contains tensorflow pytorch Jupiter lab and other popular machine learning and data science Frameworks this container is available for all Jetsons from the Jetson Nano all the way up to the agx orn you Docker pull the image and then run it you don't have to worry about cobbling together all of the dependencies or the many other gotchas simply pull and run no compiling no fuss no muss the container launches a Jupiter lab server on Startup Point your browser at the jupyter server and you're up and running you can reach the server from a remote machine too once inside the notebook you're ready to get to work a Cuda enabled version of pytorch is only an import away an important resource on Jetson is the Nvidia Jetson Zoo the website contains links to container images and compiled binaries meant for use with jetpack many of these contributions are from Nvidia engineers and professionally maintained they can save you very many hours of development time today we are going to take a look at some of their Docker containers the first containers listed are for machine learning and data science these support tensorflow and pytorch while the ml container contains both along with other supporting packages there are versions available for all Jetsons including the Jetson Nano there are also containers for running Ross and ros2 on all Jetson models when we click on the GitHub link we find the Jetson containers repository on the dusty Dash NV account let's scroll down the page this repository has links to Docker containers along with Docker files and tools to build and combine them there are dozens of different containers to choose from everything is extensively documented there are a couple of ways you can think about how to use this repository the first way is meant for developing your own custom containers it is mostly transparent but you need to do some setup beforehand this method will also run existing containers if they already exist let's scroll down here and go to the system setup page for best results you should be running the latest version of Jetpack 4 or 5 depending on your Jetson model open up a terminal I'll run jtop here I'm running jetpack 5.1.2 which is l4t 35.4.1 we're running the big boy today agx orange then we apt update to make sure that pip is installed then we clone the Repository I'm going to use a python virtual environment let's activate that this is optional now switch over to the Jetson containers directory now install the python module dependencies you can see it's barking at us it's missing the wheel module just to make sure it's happy the installation complete next we set up the docker default runtime add the default runtime Nvidia Association make sure not to forget the comma before the preceding brace then save the file then restart the docker Daemon it's a good idea to confirm your changes make sure to read through the rest of the instructions there are lots of useful tips and tricks if you run the docker program you get a permissions error we can add ourselves to the docker Group which will make us one of the cool kids who doesn't need permission this change does not take effect immediately reboot to have it take effect now that we're all set up let's run a demo stable diffusion we used the Run script to help us this particular Docker image and the model that it runs are about 10 gigabytes in size after the models are pulled down and the image is running it will start a web server on Port 7860. you can also connect from a remote machine using the IP address of the Jetson let's generate an image from a description two robots sitting by a lake by a mountain then hit generate then a minute or two later we have some images this image is a little weak this one is better here's another one one cute golden retriever puppy Masterpiece cinematic lighting detailed fur detailed eyes depth of field let's make this full screen the left eye and the mouth are a little messed up I think this one's pretty good the next one gave me nightmares for a week I'll share it with you you're welcome I like this one the best when we generate these images almost all of the work is done on the GPU one of the nice things about running everything locally is that you get everything back in a constant amount of time this isn't always the case when you have to share the system with other users it looks like it's using about 13 gigabytes of memory the darker image takes up about 15 gigabytes of drive space we can take a look at the container one of the nice things about the Run script is that it will stop and tear down the container when you stop the web server normally you have to manually stop the container another way that you can use the repository is to use the docker pull command to pull images from repository links then you can use Docker to run the image or use the Run shell script with that image let's navigate to the text generation web UI directory then we will scroll down to the containers section let's click on the image we want there's a handy copy button we'll grab that and paste it in the terminal takes a good chunk of time to download and expand it around 27 minutes let's go back to Jets and containers and grab the Run command there's a straight Docker command if you want to use that instead I start up my virtual environment make sure that I'm in the Jetson containers directory let's grab the Run command sync version 35.4.1 takes about 13 seconds to start up it opens a web server on Port 7860. let's head over to the model tab we will load our model select the model I'm going to use a different model loader then hit the load button takes about 15 seconds to load now hit the chat tab to start chatting what types of games do you like to play here's the real-time response I'm just an AI I don't have any personal preferences or the ability to play games and with no tuning we get around two and a half tokens per second I've seen between two and a half to three and a half four tokens per second this is on the 13B model 13 billion parameters with a little work you can get that up to five maybe six tokens per second that's a big difference in feel in summary here's the deal as the environment to run new applications becomes more and more complicated it can feel like an accomplishment just to get the Baseline working make no mistake it is not however the good news is that people have done the heavy lifting for you those results are available in Docker images ripe for the taking even better all of the docker files that build the images are available that makes it much easier to build your own certainly worth checking out let me know in the comments below what you think hey if you like the video give it a thumbs up and if you have not already please subscribe thanks for watching [Music] foreign

Channel: JetsonHacks

Views: 15,104

Keywords: jetson-containers, NVIDIA Jetson, Jetson Nano, Jetson TX1, Jetson TX2, Jetson Xavier NX, Jetson AGX Xavier, Jetson AGX Orin, Jetson Orin Nano, Jetson Development Kit, Jetson Docker, Jetson Development, l4t-ml, stable diffusion, llama2, llama2 GPT, tutorial

Id: HlH3QkS1F5Y

Length: 10min 4sec (604 seconds)

Published: Mon Sep 04 2023