From crontab to celery with no regrets by Marco Pavanelli

foreign [Music] thank you um so I guess we can continue and before I introduce our next speaker I have a small announcement we found some earbuds and a credit card so if someone lost it you can find them on the desk please ask our volunteers there uh otherwise uh day two of pycon and our next speaker our second speaker is Marco so let's give him rounded Applause please thank you very much can you hear me yeah okay um we are not talking about something new or super Innovation we are talking about the transition from Legacy stuff to something new and well introduction I'm Marco pavanelli and Ivan tabriel is here with me he helped me with the slides we work in Sasa SPI in bonzano sutirol Italy so this is a tiny piece of Italy where the people speak German I don't but they do because I work there but I live just nearby and it's a funny place in the middle of the Alps and we work for saza which is a company that is running the buses in the in the in the area we have a small developer team inside the company but 80 percent of my colleagues are bus drivers or mechanics and by the way we are hiring if you have a license to drive bus and your dream job is to drive a bus in the middle of the Alps then I can give you the email of our HR so I think that's not the right audience for this but you know okay let's get down to business as I said this is not super well we are not you understood we are not a store top you're not a very new company and we have to struggles like I guess many of you with Legacy stuff and try to bring them to some newer form and so krantab we had when I joined the company about two weeks ago we had like I don't know tones of chronotopes jobs run it at night collected data pushing data send emails whatever and in a complete mess and uh so well we wanted to change this thing a little bit and we managed to do it thanks to celery by the way how many here are already using salary yeah okay because the the speech is a bit well like uh low level so it's for introduction to salary then we will talk a little bit of the our personal experience uh our own experience in the company but okay it's okay it's good that not everyone already uses okay Chrome dub by the way no one step back Crown top is a great piece of software I personally love software that small reliable and makes your life easier in Chrome tab is perfect okay it's a great thing so I'm very sad about what I'm gonna say in the next slide because it's like you know betraying an old friend so okay there are some limits okay Quantum Tasker schedule and run on the same place on the same Hardware they not so easy to scale of course you can do it but difficult patchy problems so complicated running tests staging version deploying devops with ground UPS it's a little bit of a mess and cront tab does not work so well with containers which is our main reason for the transition we are trying to move everything into containers and using devops build automatically deploy automatically and this way we can do it otherwise with salary we can do it otherwise it's a bit of a mess okay so celery sometimes ago about three years ago a friend of mine told me have you heard about celery I said no and so I started reading some dogs my friend is Mario by the way if he was gonna see the video sometimes and some good things is written in Python and we all love python I guess and centralized schedule distribute execution which is what we were looking for and born to scale is made especially for scaling things works great great with containers actually you need them if you really want to do things properly at least we had to use them works with anger which is another very good news and largely adopted there is a community out there but I guess also that if you are sitting here you know the value of a community so requirements what you need for salary you need a server infrastructure if you have one and only server you don't need you can stick to chrome Tab and I will be happy to stay with chrome but if I only had one server but we have many and we have different containers running everywhere and we keep changing them upgrading them so it is something we need some experience with Django and python it you don't have to be a super Guru it's fine if you already know how to run a Django and can manage with it Simon patience of course at the beginning and so just as strongly you need to have some experience with containers if you want to get the best out of it right what we do with that and as I said we do many things at night or during the day with automatic scheduler procedures so basically what we are using salary for is everything that was we were using coronta before like backups of course nightly stuff long running training AI importing that and exporting data this is what we do mostly actually importing exporting stuff from other software that have been used in the company and that they don't talk to each other so we are in between we are very software to make other software tokens which other it's not so nice not the best job to do but this is what we do in saza and check in emails sending email calling apis to get fresh data we do that very often and so it's automatically cleaning this is very important nowadays you probably all have heard about jdpr so sometimes you have to clean data it's because of the law so sometimes people many we do forget a lot about that and with salary we managed to do it a little bit better so but it's not just only for legalism but also for technical reasons you have caches you have sessions you have everything that stores stores and uses spaces so it's a good way then Beyond krantab something that is not scheduled we found that also quite when after we started using salary and we realized the potential we started using for other things like I have an example um I have a colleague in the company she needed a lot of data and she would come to our website our web application and say okay download all the data and the data was always too big and the application would crash because it was too much couldn't download timeout problems so I started I think I said why not switching to salary so she said I want this data that pops up come out and says you're going to get an email when it's ready okay closing the pop-up then salary Works behind okay there is an API in salary through which you can call the API essay launch this job and with his email and send the data to this person so after like half an hour of collecting data um and getting everything ready she gets an email with a link and says get your data here okay this is very good and very comfortable to do and so I wanted to show you this idea sometimes you also use it for something that we may not want to run on the same computer where the web application is okay make it suppose you have a synchronous request from the application but maybe something that your request should do is to CPU intensive so you just want to send it somewhere else and salary does it more or less what I was saying before but you can do it even for shorter stuff not always 30 minutes and okay some Theory these is one of the things you can do with salary one of the architecture this is basically all about scaling okay you have one job to do and that you want to do and you just wanted to replicate many times so with salary you can do that you you can duplicate your worker we're gonna talk about this architecture later and your code can be executed everywhere but run scheduled launch just in one place which is very comfortable so you have a set of cues and you distribute your application and your effort your CPU intensive stuff everywhere you go something more complex which is much more like what we are doing right now is this took me a while let's get this this life together so you have different kinds of tasks yeah I know I'm not exactly a graphic person but the idea was to you know make it simple to understand so okay you get something to do on the left of course then you put something into uh salary there is a broker in salary we'll explain that in a while and then you can say to you can put this task this thing to do in a specific queue and then your workers are connected your specific workers that can do that particular job they are connected to the specific queue so you uh you get the job done by the right worker if you if you have only one queue of course you have to wear workers that can do anything in this way you can make different workers connect to different cues that make job that they only can do okay and finally the results gets collected and end up again in salary and the broker which is the core of the of the architecture of salary and we get to that right away okay architecture as I said salary is made of modules different pieces of software and they are kind of communicating one with each other through a broker broker is not salary broker is another piece of software and by the way here we go models mandatory models are broker and the worker these are necessary otherwise you don't have salary then there are two more which are very useful which are bit which is necessary to replicate the behavior of current up and flower which is nice to have final reports about what's been done during the night or uh while you were looking okay the broker I said this is the core of the of the salary architecture it is not salary itself it's another tool for communication sending messages I said it's all about sending messages from one module to another this is all what seller is all about and that can be you can choose any of those varieties or Arab vitamin Q they have some difference right this is a bitter more lightweight and less memory intensive less uh well it's also a little bit easier to configure rabbitmq is a bit bigger it has a good interface you can check what's going on you can monitor it through the UI of rabbit menu which is very good they book around they both run on Docker which is quite simple to makes it easier to install for this for the examples we're gonna see later uh we to create this by the way yesterday it was a talk from two guys from Ray this is great talk guys very good and it was really funny are they here today um so we said we're using radius as a work simple example this is like the hello world of celery I'm going to show you some pieces of code very little and uh not sure if we can run it here maybe we can run something after so that's the requirements that's it Python and release the version is the latest salary with the square bracket reduce means we're going to use celery with that kind of broker of course you can just change and say rabbit and my Q or imqp I think it's the what you have to say and you use the other broker okay and we run the broker that's how we run a broker and on your single machine of course if you have a complex architecture with containers we have to put it in a inside a Docker compose yaml file which we'll I will show you at the end and okay the broker is up and running the worker code as we said celery is all about running python code somewhere else okay or somewhere where you want to run it and this is an example of hello world worker codes you just have to import salary to find where your broker is that's all you do and Define a worker piece of code with an annotation that says okay this is a piece of code that will be exposed to salary so you can send messages to the worker and execute what you want to execute then how do you do you launch this okay this is how you run it basically what you do is command line salary minus a and you said basic worker is the name of the Python file and then you said worker which is the parameter that says to salary we are running the module worker on this stuff salary makes kind of introspection in your code finds out The annotation and says okay this is the part of code which is exposed and this is the broker to which I'm connected and stays there this becomes a demon and when the message comes from uh from the broker it executes the the sum which we were saying looking at before and Returns the result to the color which here is this this is how you send this is how you launch the application basically you also import salary you define the same broker the same way and then you declare basically that you have a worker with that name in that file that's waiting somewhere and then you just run plus delay and then you get the message back you get the result which will be of course 42 which probably is familiar for someone who what's a serious TV like me second part okay already okay as I said this was the end of the hello world salary as I said two like four lines files a broker and you're up and running you have salary working a worker that's listening and a piece of code oh this one you don't have to do anything special you just launch it you said python and the name of the file and it goes okay and you have and you and you see your message returning so it goes to the broker gets the job done Returns the result and you have it here going ahead okay this is a more complicated much closer to the one we are actually using to replicate cront tab in our company so you need Django of course salary again with radius and two modules of salary that are necessary to make it work with Django and to replicate the scheduler part of of salary especially salary bit is the core of the crown sub-like part of salary okay Django you just make the simplest application the hello world of Django start a project you call it with that name and then you create an application of course you remember to create the super user otherwise you get locked inside and you have your Django application up and running beside that and ah by the way all the code that I will show you here is on a public gitlab which we will find at the end and it's not that big but you know you can use it I test it it was running yesterday last night so that means I'm not sure if it's going to run next week but you know that's life that's how being a programmer is and okay with I said about bit so you create the super simple hello world jungle project you add six lines actually seven lines of code you have to tell also the Django this kind we are talking about settings bi inside the Django application I guess many of you are familiar with this kind of thing and inside that you just have to add some settings which are necessary to tell to Django where celery has his broker and stuff and that you have two application installed again cell rebate will be very important for the next step then you have to add one file so I said junk application super basic junk application six lines of settings one file the basic key is used by salary to get into the junk application and understand where his stuff is okay is more or less the same thing that we've been using we have used yesterday not yesterday sorry it was in the previous example and uh basically you see that salary will go to find his worker that are exposed inside this particular application of Django okay we have a jungle project Django app with the views the euros and whatever you need and inside that there is a worker application that is being exposed and this is why we have to tell find out where the worker is okay this is the actual worker it's a very simple example basically again you import salary then you take one particular piece of code and you declare it as a shared task this will make salary go through the junk application and find what is exposed to the broker and to requests from the outside we took a clear session command clear sessions command is an internal command of Django which is used to clear the session make a cleaning operation which we do normally at night that a lot of people forget to do and then the database gets bigger and your application gets lower and you will understand why and then this is the reason okay so basically what we've done is we took one of the Django internal commands so python man SPI Clear session is already there and it's used by Django we just took that command interna and we expose it to uh to salary to be called from a scheduled tax from another part and of course instead of using call command which has been imported by Django common management you can just write a piece of code and will be the same thing it will be shared and exposed to salary okay and then one so I said basic application Django requirements basic application settings seller API plus pi that's it you are up and running and okay we run the old things we run the broker we run Django the migrate is important and I will show you why then you run beat and you run the worker bit is very important is a bit long but it's very important which is basically what you see is salary manual salary minus a and the Django application the name and then bit which is the command we say we give to salary so make beat module onto this application and this will basically start a demon then we go through your Django database looking for scheduled things to do so you will have Django will have models inside these models there are scheduled tasks it will look for that every minute check if they are there and then send the messages to the broker that's it one of the good things I one of the things I really like about salary is it takes a little bit to understand the old architecture and how the messages go through up and down from one of the other models but once you are understood it's plain simple okay it's just strings that goes to and from the the models the worker is it also how you run of course you need to have two pro different processes this bit becomes a demon and stays there worker becomes a demon of course beat you only need one worker you can have as many as you like okay sometimes we use like 10 altogether they just wait for the next messages whoever is first takes the message runs the job and back the results no Clash they just there is a queue in between the one who takes a message the message is deleted so it goes fine it goes really well so basically they say salary menus hey the Django application worker this will do that introspection work that I was talking before basically goes through the application of Django okay and uh and process the and find the the work the job to expose to salary okay I try to show you this some other way let me see okay well no this one is the worker the demon of the worker running you see very important is the tasks that's been found and exposed you you remember the code clear session a minute ago inside tasks Pi which I show you it's there so when you run salary minus a Django application worker it goes through it goes through the application introspection find this one is the one to be exposed this string is very important because you use it in the interface of of Django schedule the task okay let me see if we can get there oops okay okay that's I wanted to show you the real Django but I have a slide ready anyway so let's stick to this okay basically what you see is we have this I think is familiar for many of you is the admin part of Django you have crown tabs and you have periodic tasks thanks God Chrome tab is back so it's not the data completely thing and so you define a Chrome tab in there it is exactly like the one you used to do with the text file you put a like a star character from minute stays hours like before and and then you connect a periodic task with a name to the to the crown tab so basically you can have many Crown tabs like I have a crowned at four o'clock at night okay and then you you create many periodic tasks which is basically just the string that you have seen uh in exposed by the worker part okay and you link the two things together you say okay this schedule this periodic tax has to be connected to Quantum and gets executed every night at four so with one current that defined at four o'clock you can have like 25 periodic tasks connected to the same ground that they all run at four o'clock of course your workers gets a bit busy if uh if you do that but you can always connect the periodic task to a specific queue so you say four o'clock is the timing but the product task is connected to a specific queue so it's going to go to a specific worker and they will not interfere between each other okay you can run this and it seems quite complicated when I explain it like this but it runs and once you have tried it gets quite easy okay you see this is the basically at 10 o'clock every day every month or every week this is the the ground up and we'll run the periodic task okay this is the worker output the same as I said the one I will show you before from the from the terminal in case my terminal didn't work I was ready with the slide and this is just one little bit is flower which is the presentation of the result is a final module we didn't have time to show you how it works but uh it's really nice because you see all the results of all the schedule tasks that have been going through that particular broker in the morning this is what we do actually in the morning we open this or we have as we are lazy people we have a python file a python script that goes through reads if there are arrows and sends a message so it's even faster and that's it basically I go to the final part well containers just a tiny little what we actually do in real life is this we don't run processes on command line of course we run Docker compose where we have this web which is the Django application the beat and the worker in the same compose file the same image replicated three times run in different way one is run as a web server one is run as a bit and one is but as a worker demon so it's three demons and they all talk to each other together and of course read this on top doing the broker that's it drop and running this is also in the git lab that I will show you recap how many how long do we have okay okay recap super fast Chrome tab is legendary software celery is a good replacement runs well with containers setup is not exactly straightforward but uh with some examples and some efforts you can get it and um and salary is not just for Crown type replacement it's good also for long running tasks that you can launch with other from from other things okay who is the code and that's it thank you [Music] thank you very much Marco for the talk and it's time for a few questions one to three questions maybe yeah I can see one nice well among the other things I forgot to say thanks to the organizers this is a fantastic python we came all the way from Italy and we're very happy for that and it's really nice to be here and thanks for all the other speakers I heard some fantastic talks yesterday really great go ahead hi Marco so I have a question I came here with a question honestly um why did you choose like a salary against like airflow for example against airflow apachea flow to be honest hayflow seems to beat a bit a little too complicated for me as I said as like um Simon was yes Julie Simon yesterday was saying I'm just a developer and an old school developer so airflow I looked at it probably one of my next talks will be from salary to airflow so but it will take a little bit of time and we didn't have time when we started this task okay so the next year thank you one more question Maybe one more no no okay then let's Applause one more time to Marco thank you very much thank you [Music] [Applause]

Channel: PyCon Sweden

Views: 2,476

Keywords:

Id: zvtztKcoUR4

Length: 32min 2sec (1922 seconds)

Published: Wed May 03 2023