How to run Ollama on Docker

the main way most folks install and run AMA is to start with the installer that you can find on ama.com you can be up and running super quick with that but some like to use Docker Docker is nice because it's fully self-contained for programs that are messy with a lot of dependencies it can be nice to use Docker because when you delete the container and the image there's nothing left olama is a very simple program with one single executable but still it's convenient for some to have everything run in Docker you'll take a little performance hit though on Linux it's pretty minimal but on Mac and windows it'll be a little more significant due to the virtual machine that needs to run underneath also on Mac there is no GPU pass through possible due to the docker platform not supporting it and that's just on Mac once you know you want to run AMA and Docker you still need to put your models in a directory on the base system typically Docker containers and images are designed to be relased small but the models you run with in olama are enormous so that's why they're separate though with Docker it may be a touch easier to put your models anywhere you like now I'm going to assume that Docker is already set up on your machine and is ready to go the command to run is Docker run- g gpus equals all-v colon root. olama publish supports 11434 set a name of AMA and then olama / olama run it and then we can take a look at the command in a bit more detail Docker run is the command to start a new container from an image if you don't have the image it'll be pulled from Docker Hub - D says to run the container as a Damon so it's running in the background -- gpus equals all allows for GPU pass through but this doesn't work on a Mac and on Linux and windows you need to make sure the Nvidia toolkit is installed assuming you have a an Nvidia card I'm running this on a brev dodev instance which already has that installed but you can find the instructions to install it in the readme on the docker Hub page for the Alama image I'll talk more about brev in another video coming very soon the next part of the command is the volume Mount this is saying create a volume named AMA and mount it to/ root. olama inside the container this is a little confusing as there's different ways to use volume outs but basically there are two sides of any command that has a colon the left side is usually what's on the host system and the right side is what's in the container if the left side is a word and not a path then the docker process will create a new directory wherever it's default is for volumes on this system that is /var li/ doer volumes and there you can see anama directory but if you want the models to go somewhere else you need to specify the fully qualified path before the Cod I'm currently in/ home/ obuntu and if I want my models to go into the myama models directory which I've already created I would have to use- vhome Ubuntu myol Lama models colon root. olama next comes ports anytime there's a port that you need to access inside the container and want to use it outside you have to declare it so we want Port 11434 inside the container to be a available outside the container as well after that we set up a name if you don't include a name one will be generated so it's usually best to assign one that you like finally we see the docker image being used amaama which corresponds to the URL on dockerhub which is htps hub. do.com rol let's say you actually want to call the container something else or you want to move the container to a new location or there's a new version and you need to update the olama container well in all those cases if you just run the command again it'll fail because there's a running container already so we need to stop it and remove it luckily we set a good name so this is pretty easy just run Docker stop olama and then Docker RM olama now if you need a new olama image you can run Docker pull amaama to grab the latest then you can run that Docker run command again with the correct parameters one thing that confuses new users is that they want to update the software inside the container instead of pulling a new image images and containers are designed to be immutable you don't want to update the contents of the container if there's updated software you pull the new version of the image and start that up updating the software inside a container will probably make it break much more easily don't do it of course you didn't run olama just to run the server you probably want to use the client to ask a question but olama probably isn't installed on your host system to run the olama client that's inside the container we need to use the docker exact command Docker xa- itama olama run llama 2 so that's Docker exact which will run a command inside a container - it says to run it in an interactive terminal if you just needed to run the command and exit you could probably leave that off the first tells Docker exact which container to use if you hadn't specified a name when running the image then you would have to specify a container ID which is a unique string every time the docker container is run everything after the container name is the command to run on the host so AMA run llama 2 now I've used the terms container and image a lot if you aren't familiar you might think I'm using them interchangeably but they're different all the things on dockerhub are called images ol/ olama is an image it defines how AMA should run a container is a lightweight method to run an image you can think of the container as the instantiation of the image and we can see this on our system we stopped and removed the container just now but the image still existed and we can quickly run it if you run Docker images you'll see a list of all the images you've downloaded to run as container if you want to remove an image then the command is Docker RMI and then the image name but it'll only delete if there's no containers currently using that image okay back to AMA run we were looking at Docker xa- itama olama run Lama 2 if you run that you'll be dropped into the interactive repple to work with olama if you're going to be doing this a lot then it probably makes sense to make an alias on Linux or Mac assuming that you're using bash or the zshell the command and is Alias olama equals Docker exact D itama olama if using fish then just replace that equal sign with a space if you're a Windows user you probably know the right command to create an alias just like that now you can run AMA run llama 2 just like you do on systems with AMA installed if you want to use that Alias after this terminal session is over you'll want to add the command to one of your RC files for your shell that's going to be a file like bash RC or profile or Zell RC or config fish config.sh there's another command you need to know about sometimes you have to look at the logs to see what olama is doing so Docker logs olama is the command to run to see those okay so now you're set up to use AMA in Docker on a host but let's say you've set up AMA on a host like I have on brev and you want to access it from a remote machine something other than the host running Docker well if the host is on the same network as your client machine well this is easy you'll have to stop and remove your Docker container for AMA remember that was Docker stop and Docker RM then run the same command but add olama host equals 0.0.0.0 Port 11434 before Docker run on the same line and then anything that connects to that host on that Port will see your Docker container version of Al llama but what happens if that host isn't on your home network instead it's somewhere else on the internet well then things get a bit tougher somehow you need to give access to your host to your client some will solve this by opening ports but that's an incredibly stupid thing to do without blocking access with some sort of authentication probably the easiest thing to do is to use tail scale which is free for most non- companies for you at home this is braing Ain dead easy and if you're doing this for work then your op scene probably has a solution they like or tail scale is just so ridiculously cheap they can probably expense it and they won't bad an eyelash about it side note I once bought some stuff for the office when I was at data dog it was the only time Finance reached out and they said I needed to describe what I bought you really don't want to know I said but I had to put it in new people new systems don't worry they said okay so I put in what I bought about $200 worth of Nerf guns and those little yellow balls for ammo I thought oh well I'm going to eat that expense no problem expense paid unfortunately the cleaning staff thought the balls were trash and within a week or so all the ammo was gone anyway that's what you need to know to understand how to work with Docker and AMA in most cases you'll want to run olama as a native install but if you insist on running Docker now you're set if you have any questions let me know down in the chat below I love the questions that I see especially when you give me suggestions for what to make a video about so many of these videos including this one come from those suggestions keep them coming thanks so much for being here goodbye inquiring minds want to know this is water I promise it is water you're know going to believe me but it's water

Channel: Matt Williams

Views: 17,881

Keywords: artificial intelligence, local llm, how to install ollama, docker, local chatgpt, large language models tutorial, docker tutorial, docker explained, llama 2, large language models, local llm rag, local llm for coding, local llm model, how to install llama 2, docker tutorial for beginners, artificial intelligence video, artificial intelligence explained, mistral ai, lama, dockr, olama

Id: ZoxJcPkjirs

Length: 10min 37sec (637 seconds)

Published: Tue Feb 27 2024