HOW TO INSTALL AIRFLOW WITH DOCKER ON UBUNTU

[Music] [Applause] [Music] [Applause] [Music] [Applause] [Music] hi and welcome back to quote ups so today i'm going to show you how you can install airflow with docker on ubuntu so i'm going to show you how you can settle your installation and all so as we all know we know apache airflow is an open source workflow management platform that is what we use for handling or data engineering pipelines so before we move into our installation please if you're new user just coming to this channel please click on the subscription button this actually motivates me to do more videos like this please try to subscribe to my channel and please give me a thumbs up so now let's move into our video two things we need before we um go about our installation we know we are making use of docker so you need to have a docker on your machine so as well you need to have the car composed installed in your machine so we need boot.com the docker compose so what we need to do is we first need to install the car engine so for you to install docker engine you could come to your browser i type um install docker engine open sources we're using a button machine all right so just click on it so and for people using mac and photos they're using windows you can as well install just come up to the tab so we have different distribution so just like i type in install docker engine ubuntu just type in install account engine windows or mac so you can use any of this so now let's move to what we want to do which is for the ubuntu machine so this is how we can go about our installation so what you need to do is you need to come up here that is you use this particular repository so you do all these steps that will that were listed here so you move down to the docker engine parts so this is where you perform the main installation so under the docker this docker engine you can see there are different plugins you're installing with it so it's not just the docker engine so you see update the hearts package index and install the latest version of docker engine id and docker compose so after you've done your installation so just come inside your tummy now [Music] so whenever i come inside your terminal to confirm that you installed the car so just type the car then the version just like this so you see the version i installed and all so okay that is how you can install the car engine so just like i said you need both docker and the docker compose on your machine so before you go about installation of the airflow and also i would like to show you something just in case you installed account your machine and you haven't set up your permission properties so what you need to do is because that means you have to start working using sudo every time whenever you want to access docker so what you need to do is you need to use docker without sudo so what you need to do is you look at the following steps so you add the docker group if it doesn't already exist so you send this command you put this command on your um on your terminal then after then you have the connected user that is you follow these steps that i listed below so i'm going to add this under description below so you can use that so after you've done this you've done this first one and you've done the second one now you're going to log out on your machine so you come down here then you log out and click on logouts so after you log out on your machine then you can then you can then restart your docker so you find out this command that is these properties i set is working so you can then be able to use docker without using the sudo so here i have the docker on my own machine i already set up my permission so please follow all the stuff that i mentioned so it can you can just do what you need to do then let's move to the business of the day so you come to your visual studio code because that's what we'll be working with then we hope on our file so this is a folder already opened here so what i need to do is i need to create a new folder inside a particular folder so let's create a folder here let's call the folder my airflow yeah so let's call it my airflow [Music] so now let's move inside the folder so let's type cd and move inside the folder so now okay we're right inside that particular folder so what we need to do is okay we have our docker compose installed already so what we need to do is we need to download a particular docker compose file use docker compose file as all these services that we need for airflow so and that particular docker compose file has been made available for us by helpful community so what we need to do is let's go back to our browser again then inside the new tab let's type install airflow docker okay i click on this so okay so what i need is this particular ym ymf file so you copy it so this is what we're going to use to download a docker compose file this particular command so now let's move to our vs code so let's paste it inside the terminal so it's going to download inside this my flow folder since we already inside it so just press enter so whenever we open this we should have a docker compose file so that's our docker compose file so let's open it let's see what it looks like so you can see all the setup and all you can see this is the dags these are the volumes tags and plugins so we need to create folders for this particular day so let's move down to the services so we have the postgres as a database so we have the postgres user and we have the password so we explain more of this so we also have the redis to the redis we also have the web server so all these particular um services they are looking at each of them they need to they are going to have their own local container so you see them whenever i whenever i start running them so let's move back so just like i told us we need to create folders that is we need to create a dab folder a plugin and a log folder so all this particular folder is where all our information will be stored so for instance the dab folder that folder will contain all the python files that we need to um access with the interface so when we run our web server possibly you want to run your python file it's going to be on that path other particular folder so let me create a folder inside this my hair flow folder so let's add the dags how's the file i used let's delete that flow so let's create another one now [Music] let's create another one we have blocks so those are the folders that we need so let's just look in there like that so the next thing we need to do now is to export our environment variables so the reason why we export our environment variable basically i think we do that on ubuntu and mac os majorly so we do that is because we want the users and good permissions to be the same that is in terms of the folders from our host and the folders in our container because without exporting the experiment variable that means the group permissions aren't going to be the same so now what we need to do now is to export environment variables we're going to use this particular command so you do echo [Music] press enter so we have the environment variables so you can see where you have it so this is [Music] so this is what we have for the uid id grid so ensure you set these group permissions most essentially we are using console and mac os so now we need to run a docker compose file so what we need to do is we need to um ensure like that is running our docker compose file it creates a user airflow and the password air airflow so we're going to create both our user and all so let's run the docker compose file so you run that using docker compose or flow image from enter so you're setting up our user you can see the container is run browser to user update so that's good you can see this is what we have ensure you have this after you've bounded you want that so we already have a user role which is the admin row that's what we set inside a docker compose file so what we need to do now is we want to run all the services that we specified inside this docker compose file you know we have the readys we have the worker we have the web server so we want to run everything so that all our containers will be up and running so for us to run all these services we're going to make use of the command docker compose just press enter so now you can see this is giving me an error okay it's telling me my bind address is already used okay so i'm going to show you how you can resolve this error because i'm very sure many people run into this error so there's a way you can resolve this error so i have um i wrote okay this is it so this is how you can resolve it when you have something like that iron is the same error so what you need to do is you either use either of these two methods so we have the lso for the next stats so using the header sof we are going to try to listen to look at all the services that are using this port 80 so listing all this listing you're going to list all the services that using this particular port 80 so you can know which of these services is using that port so having seen all the services that using the airport you can kill them by using this solution so i'm going to make use of this particular netstats to see any services that's using that but it's a i want to use so let's paste it so now it says java and this thing so all i need to do is i'm going to use the sudo key so then you put the pid so you use this particular value you have here to kill it so you write 17 93 so that's it so let's do it once again for the other one so nine seven so now let's run a docker compose of once again let's see if the port is available let's give it a few times and see if our web server is up it's already running already so the web server is running let's see yeah it's running so we have our progress okay so everything is running so what we need to do is let's build this in browser and see if this is working as what we saw here so what we need to do is let's open a new tab and then you type local posts so just like what i have there you type your calculus they put the colon sign in 8080 then press enter and then ensure to use the same username and password that i said so remember inside a docker compose file just look inside the docker compose file let's move down underneath quest yes you see our username and password so that's what we use so you're sure to use the same design in my password that we use so you click on signing so you can see your home interface so you can see the interface so here you can see the home you can see these example dags that actually came with a flow so now you can run any of this you can any of these any of this come down here on it and you click on trigger attack so this story like this patch operator is running this is complete true so let's click on it you can check that you can use the graph view so this graph view is what i love using a lot so you can see get some information about what um this particular um that is doing so you can see the queue running success field so you see any of attacks being filled so this is success so you can see the results from what you have here you can see the success so you can have different kind of views just look at the calendar so we also have the accounts so you can see these are more informations you have the code view you can see what the codes look like so coming down here you can see the code so we're going to talk more about this in our next tutorial so if we talk about the operators and all talking about the setting attacks and everything so here you can see the audio open these updates logs so you can see these are logs all these information all these analogues information so each of these we're going to talk about it in our next tutorial so i hope this tutorial was helpful and is able to give you a walkthrough on how to set up your system and then you understand how you can install airflow with docker on your ubuntu machine for people that are using mac os and windows machine it's possible i might do a video on that to give you a walkthrough on how to create but so please don't forget to subscribe to my channel and as also give me a thumbs up i would love to see you next time [Applause] [Music]

Channel: CodeTops

Views: 5,063

Keywords: apache airflow, airflow, docker

Id: E-ietxzP_j0

Length: 16min 1sec (961 seconds)

Published: Sun May 22 2022