How to install Apache Airflow on Docker? | Build Custom Airflow Docker Image | Airflow | Docker

hello and welcome one and all today we will cover apache airflow installation we will install airflow on docker during this tutorial we will build a custom docker image of airflow with additional libraries such as sql server provider since we have covered sql server as a source in the previous airflow videos i have covered the airflow and airflow dag in this video here if you like to check it out before you begin make sure you have at least install and configured docker community edition also we will need the docker compose on our workstation i am on a windows machine so i have installed docker desktop and it is up and running we can confirm that the docker is installed and running by issuing a docker dash dash version command in the command prompt if it is installed then it will display the darker version also down in our tray you'll see an icon for docker desktop and if you hover over it it'll tell you that it is running with that out of the way let's begin the installation process we'll launch the airflow documentation site and let's focus on running airflow on docker this page discusses the requirements for airflow which is docker application and docker compose docker desktop includes both we will scroll down and get the docker compose jaml file from here i'll open the file in a new window and copy it in its entirety let's go ahead and paste the file in the text editor of your choice i am using notepad plus plus and i'll go ahead and set the language to yaml for syntax highlighting we will change few settings here first we'll use a custom image so i will go ahead and replace the image name here to the customer image name which we'll call airflow dash sql server second we will update the port for the web server to 909 since i already have an airflow instance running on port 8080 also let's scroll down to the flower and change the port here as well i'll change it to 5 5 5 9 one more change we need to change a line 19 here this has changed we need to remove the s there and with these changes i'll go ahead and save this file in a new folder i'll call the folder airflow dash sql server and in this folder we also need three folders one for dag one for logs and another one for plugins so this is how our installation directory looks at the moment let's create a docker file this will allow us to build a custom airflow image if you are new to docker then feel free to check out the using custom images section on airflows page i will leave the link in the description below whenever we want to add additional libraries then we build a custom image that includes those libraries or dependencies for example in this case we'll add the airflows provider for sql server so for this we'll go ahead and create a custom docker file and we'll save it in the same directory where we have saved our docker-composed yaml file in the docker file we use the official airflow image this comes after the from clause and then we run few pip commands to install airflows odbc and then sql server provider also we installed the pi odbc to establish connection to sql server let's go ahead and save this file remember we saved this file in the same directory so this is how our installation directory looks at the moment so we have three folders one for dags one for logs one for plugins let's open up command prompt and navigate to this installation directory we will go ahead and issue a docker build command with dash t for tag or image name we will call it airflow dash sql server then we supply the docker file with dash f and after that we provide dot for the current directory and we ignore the cache if there was a previous attempt to build a file with the same name this will start building the custom airflow image it can take a while depending on the number of libraries we are installing we'll give this time to finish and come back once the image is complete our image built successfully we can check the docker application and images and our new image is present here so our image built successfully now we can run this image with the docker compose up command so we'll issue this command in the same command prompt window here this will run the image and all the containers under it as the docker is bringing up new containers you'll see the various statuses in the command prompt all of these containers are listed in our docker compose yaml file so if you review that you'll see the web server scheduler trigger flower and so on all of those containers are building once the web server is up we can launch it in the browser with localhost and the port 9099 or whatever port you have specified in the yaml file this will launch the airflow web server so we have successfully installed apache airflow on docker we can log in with default user and password which is set to airflow in the yaml file once we are logged in let's navigate to admin and connections and add a new connection i want to check whether sql server is listed as one of the sources if we install the base airflow image then this provider is not installed by default sql server is listed as one of the providers so let's go ahead and connect to sql server running on the host machine connection id we can specify any name i'll call it sql server uh the connection type will be microsoft sql server you can provide a description if you like and for host we'll use the host ip address schema is the database so in this case it will be adventure works dw 2019 and then we provide the login which is the username and the password and the default port for sql server which is 1433. let's go ahead and test connection and if everything is specified correctly this will be successful and we can go ahead and save this connection so this is how we can connect to sql server or postgres or any other providers that are listed in the connection type drop-down we can connect to those and use them in our tags this is all on airflow installation i hope you enjoyed the session like share and subscribe take care and i'll see you in the next video

Channel: BI Insights Inc

Views: 15,700

Keywords: data pipelines, data pipeline, what is data pipeline, data pipeline vs etl, how to build a data pipeline, how to create data pipeline, batch vs stream processing, big data, data warehouse, ETL, build etl pipeline with Python, build etl pipeline with pyton, Airflow, how to install Apache Airflow, Docker, Build Docker custom image, install Apache Airflow on Docker, Apache Airflow, Apache Airflow install SQL Server provider, SQL Server

Id: t4h4vsULwFE

Length: 7min 24sec (444 seconds)

Published: Mon Sep 05 2022