How to Run Airflow Locally without Docker!

hey yo data guy here back with another video on airflow um so today what I'm gonna be going through is showing you how you can run airflow locally using this super cool Tool uh my colleague caxel developed called airflow CTL without any need for Docker so this will just start up airflow just kind of as a local application um so if you know obviously Docker has a lot of advantages in terms of you know being able to easily access the logs manage it you know and actually have a production light environment but this is also really good you know hey maybe your organization doesn't allow you to download Docker uh maybe your organization doesn't has really strict controls and you know you just want a super lightweight tool so you can just play around with airflow and you're not really worried about production um and before I get into it and show you how this all works if you like my content drop a sub drop a like uh helps me out and I love also helping you out so without further Ado let's get into it and I'll show you how this all works so basically what this is going to do is and you can I'll drop this like a repo in below um it's just open source airflow out of the box nothing fancy it's a pip package you just go pip install airflow CTL and so now I'm going to kick it over to vs code and show you you know the actual process of building this and what it looks like so the first thing you're going to want to do is obviously install pip better flow install airflow CTL already done it so it's going to go real quick for me and then what we're going to do is just initialize a new project called my airflow project and you can see that this airflow project is being initiated just under my user folder it's installing kind of all the necessary packages you can see two point airflow 2.6 the necessary python packages build dependencies just you know all using pip so nothing super fancy here because it's basically creating a airflow virtual environment rather than you know using Docker it's creating its own separate isolated environments that it can you know use whatever python version it wants um and I believe with this as well you can specialize or I don't believe I know you can also uh specify the python version so maybe you know you're using certain packages that only work with certain versions of python you can actually initialize a new project um by specifying just dash dash python version um and you know python version 3.8 let's say and what that will do is create a new file structure um which I'll show you in a second watch this finishes building and so once it's finished building here it's going to give you your admin password and you'll use this link to actually access airflow so let's kick it over to the local airflow environment just fun up and actually before we kick it over to uh the actual airflow UI I do want to go through kind of the file structure here because it is a little bit different from you know what you might be used to so you can see here we have our web server config we have a standalone admin password and here's where you can Define hey maybe I want to even though I want a quick start I want to change the airflow version I want to change the python version I want to pre-define some connections to be built at startup or set some airflow variables those will be set in the settings yaml file so instead of your environment variables file you still have your requirements.txt you have an airflowdb file and it's well your airflow configs file so you can just find hey this is the path to your dags folder hostname you know whatever executor you want to use just all the typical airflow configuration variables you also have your airflow web server PID your dot environment file and you also have your example Dag um so here you can also you know look at the logs just all typical airflow things here and you can also look at your actual python virtual environment to see the configuration details of that so slightly different but still has all the usual airflow features in a local airflow directory just maybe a little bit of a different file structure than you might be used to so just wanted a quick level set you on what's going on there and now we can kick it over into our locally hosted airflow environment and so now what we can see here is a locally hosted non-docker open source airflow environment obviously it's going to give us a classic error messages if don't use sqlite don't use the sequential executor because you know they're both obviously terrible uh you know I would use a sequential executor when you're using airflow for all the things you can do outside of sequential execution um but this just really provides you a super basic easy to set up airflow environment um in like three minutes like I I didn't really cut around a lot around for this video um it is very fast to set up and run um you know you can configure whatever version you want you can configure the uh version of airflow python you can choose to you know set up your install script um and this you know obviously you're going to want to go with something you know like Astro for production because this isn't going to hook into your cicd processes very well this isn't going to hold up when you're actually pushing code from here to production because you're not testing it in a production light environment but if you're just trying to test out dags if you're trying to you know explore airflow and you want a super easy way to get started check out the airflow CTL if you don't have Docker so that's all I have for you today super quick video I just thought it was a really cool feature that you know he just made over the weekend so if you like this video like subscribe and stick around for more great data content data guy out

Channel: The Data Guy

Views: 4,156

Keywords:

Id: IX0uDfCt3P8

Length: 5min 14sec (314 seconds)

Published: Wed Aug 16 2023