Mastering Laravel Queues Basics to Advanced Techniques

hello and welcome to this course where we're going to do a deep dive on laravel Q's from beginner to actually mastering the the subject in this course we're going to do a couple of stuff we're going to build our build up our Q Knowledge from the most basic concepts up until to all the advanced concept that larel offers regarding qes including the testing facilities we're going to build a couple of mini projects image importer and resizer and converter and we're going to build a word to PDF converter and of course we're going to use PHP storm to do that task and we're going to use Advanced larel topics like um larel job chains job batches and more topics that we'll touch and for the final um chapters of the course we're going to use a special package a couple of packages actually we're going to install larvel Orizon and see how that works with monitoring and debugging our cues and we're going to install larvel Venture which is a workflow based um Q jobs programming for laravel that's based on larel Q's actually but it's a m uh much uh richer ecosystem for qes and um I want to uh thank you for taking that course I would also like to offer you a discount in the description you can get a 50% discount when you click on that button and go to my uh udmi course since this is only 2 hours out of 6 hours of the full course so I would like to thank you again and I hope you enjoy and if you have any comments or insights or you would like to share something please WR in the comment section and I will uh do my best and try to answer all of the your questions hi and welcome to this lesson where we're going to do an introduction to larel qes why they are used in the first place how to configure them since larel installations come with this uh Q abilities prepackaged on every new laravel installation besides Lumen but we won't discuss Lumen since Lumen is pretty pretty much uh will be reaching end of life I think because you can for the moment do with laravel everything that U that you do with Lumen you can uh use a strip stript version of laravel with only API functionality uh which you couldn't do in the past but for the last couple of versions of larel you can do that so we're not going to discuss how to use that with Lumen since in order to use that with Lumen by the way you have to enable a couple of stuff on the application bootstrap process but uh that doesn't matter for the course because we are going to focus mainly on larl so um just before we start we're going to set up a new larl project with composer create project after that we're going to do PHP Artis sale install and this will by the way will cause um the package of laravel sale to pre-configure with our application since it also comes pre-installed with that package laravel I mean and sale is a development environment using Docker for laravel so we can uh pretty much achieve the same functionality or same environment that we'll use on our servers in on our computers for example when we go when we'll go over the ability to listen to Q events we will uh initiate a sell uh sell commands to monitor the Q events and operate on them for example we will uh work with supervisor which is which also comes preinstall in the docker version of uh larel sale so all we need to do is just basically attach to to the container and view the worker as it works and we're also going to add another worker because it doesn't come predefined with more workers so we'll add a custom worker which will work on it and we'll see how to do that in production also so after we install save we will use this command which is how to I'm going to show you what what happens when I do it it automatically starts Docker containers for our application and I already in when I clicked PHP Artisan sale install I told him that I want my SQL redis mail server and uh it automatically installed supervisor D and this uh installation uses mail pit mail pit is is a mail catcher for development environments so you can see the mail that you are sending if I'm going to if I'll click it here you see that it automatically waits for emails to be sent which is excellent because previously you had to catch the males in your application and now you can just use it and we will see what are the benefits of using the mail catching in your application since in one of the lessons we are going to do a simulation of of user pressing a button and waiting for the mail to send and what happens in terms of user experience what happens in the server when he presses that button and what is better to do and why while we see that we understand why cues and background jobs are very important in web applications and in any application by the way it any application that you have some kind of a interface between you and the user between your application and the user it's important important that the interface will be responsive if that interface is a web interface like we have here for example it's orted that when I click on buttons that uh the user perception of the application will be uh super fast not not the perception itself but what the user gets when it clicks that button the feedback should be very fast for example if I click uh send email then the operation should complete immediately I should get the feedback that the operation initiated but the operation itself will be in the background and this is why cues are very important for the perception of the user and for many many other use cases for example what happens if that uh email sending is failed you know if you are not saving that operation in uh some kind of a log audit log or a action log then there is no way for you to restore that but if you are using Q jobs and that operation failed there is many ways there are many ways you can restore that operation and uh rerun it so this is another benefit of using the cues so let's get started with our next lesson on U what are the differences in larel between and what is the terminology larel is using in for qes hi and welcome to this lesson where we're going to discuss the basic terminology that larel uses when we when we speak about qes so first of all I opened the Q config file that exists in every larel application besides larel 11 since there it will be basically mostly based on uh environment variables but you can still restore the F files when you need them so let's start with what larel calls connections connections and and drivers okay laravel has a distinction between a connection and a driver and a q job what what I mean by distinction is when we speak about connection and Driver that's basically the same because um connection is the driver and the the real distinction is between the connection SL driver to q and Q is like a a bucket okay it's it's a named bucket where you put your F your jobs that you want to have in that bucket that you want to process for a later time for example and the connection SL driver is the mechanism through which the bucket will be processed for example out of the box you have couple of connections and from this point forward I'm going to refer to them as connections and and won't do the distinctions with between them and drivers so uh the default connection is sync which means synchronous which means that it won't fire a database it won't fire a q job it will it will fire the Q job but the uh process will be synchronous and not asynchronous so that means that your application Fred will will be blocked until that process finishes okay and that is the major distinction between the synchronous connection and the rest of them because the rest of them are asynchronous for example you have the database R this and sqs and Bin install connection out of the box and there there are a couple of differences between them because for example the database connection is free because it's on your local or server database bin stock is also free redis is also free and sqs costs money and if by the way you put null in the database connection that means that the jobs will be discarded so you you will have to pay attention to that also and you do you set it up this environment variable in the EnV file and it's the default is sync and right here in that file on the Q file you have couple of uh parameters that you set up per driver for example the database driver you will have to also run a special command for that table to be migrated to be on your uh list on your tables in the in the database that you use uh for red and rest of cues you don't have to run that job but you will have to run a special Artisan command for fail jobs so you will have a place where you see if some jobs failed and later will when we will install laravel Horizon we will have a nice user interface to see which which uh jobs have succeeded which have failed uh we'll have overview and we can also restart and retree in larel terminology Retreat those jobs and uh we'll continue from that you also have couple of other configurations here for job batching and failed table jobs but we'll see it when we'll get there so let's just sorry let's just do PHP Artisan Q let's see what options we have so basically the options like I said is to create the tables and we have couple of extra options we have uh deleting all the jobs from specific que like if we have a queue for emails we can delete only the emails we have the ability to create the migration for the fail jobs we have ability to list a fail jobs we can Flash specific queue or all the cues we can delete right from here not just clear clear deletes all the jobs even those that haven't started yet we can run the monitor command to monitor them we can prune when we'll get to batches and we'll understand what uh that means we'll use some of those those commands just to to get a feeling of how that works we can also and we should also restart the queue after we do code changes but that also depends if we use laravel Octane and the different versions of octane because when we run octane the files are loaded into memory so we're not just starting the que we also have to restart Octane and yeah this is the command to create a migration for the Q jobs database when we work with the database because if we work with red we don't have to run that because it stores in the r in in red in special key value format for those tables so that was it for this lesson where we did a quick overview of what what are the differences in the terminology of laravel q's and uh we'll continue with the next lesson so thank you hi and welcome to this lesson where we're going to work on creating our first Q job and after we create it we're going to discuss couple of things and couple of gas that we need to look after when we handle when when we deal with Q jobs so first of all let me let me leave that as open the Q connection because I want to show you the difference between the job jobs when I'll dispatch them let me open PHP Artisan help make job now usually I don't run help on the before the command that I run but for for the course I want to show you the options that you have because sometimes the version gets updated and I'm not sure that you always you or other developers read the upgrade and the release notes and new functionality is being added and old functionality is being removed so when you run the helper for example here it show you it shows you what options you have for example you can generate an accompaning pest test or PHP unit test you can force it to be synchronous which basically means that it will remove a trade shoot Q trade from the job okay so that is is important for you to learn to use that command but we're going to run that without the command Now to create a basic job and we'll go over the the class of the structure of the class and the goes that we have to pay attention to so let's create an email I'm going to call it daily emails okay and you see it created them and that directory app jobs previously wasn't there but for if if it doesn't exist for new jobs it creates it and just before that let me initiate a g repository so I I'll can I can push that repository to GitHub and share with you and so you will see the differences between each commit I'm going to do I'm going to do that via PHP but I can do it also via the terminal I can just open the new terminal that I have and I can run git status to see that there is no git here but I do have a git ignore here since every larel installation comes with that by default and I can do get init in it to initialize the repository get add everything to the repository and get commit initiated initialized the repository okay and just come here it should automatically update in PHP storm and here it's it's updated okay and it should also because I committed everything you see if I'll do a change it will automatically show it in the gutter and it will show it in the file browser that something has changed so we're going to use that okay now for the next thing let's go check the job that is created and the structure of the class so the first thing let's let me reformat that because I like this formatting the first thing you should notice is that it implements shoot q and it has the trade of cable so that means this job if if you remember from the previous example I told you that if we we'll do this then the shoot que would not be um not be used it will be regular job let me let me it will be synchronous job let me create another job sync I'm going to add the sync and the syn here and you see it doesn't implement the shoot queue and there is no queueable trade okay this is also useful for uh for the purpose of separation of concerns and uh loose coupling but we're not going to use that because we we can still use the job as sync because right here sorry not that file because right here the connection is syn okay so the jobs will be synchronous no matter what we do let me just close that and close that and let me delete that file because I'm not going to use it I just want to show you so what do we have here we have couple of Trades that are usable for the for the job at hand for example the dispatchable is the trade that makes this Que dispatchable those both trades are the trades that send it to a que tell what what jobs failed what job what is the job status and stuff like that and this trade is very important because when we'll get to the part that will pass in an eloquent model here for example when we do jobs that work operate on users and we dispatch this job with a specific specific user will have to take some stuff into consideration for example uh the models are being serialized which means they're being saved in a special format because this job will this job meta sorry this job metad data will be saved either either into the database either into redis or other Q connection that we'll use but the metad data and you can uh imagine why it's important to serialize it because let me just close that I'm going to always add when you take a big for example if the user object was very big and he had relationships with other tables in laravel and those were were supposed to be eager loaded or lazy loaded no matter what or you had some um restrictions on that user object those stuff need to be serialized because when later when the Q connection fetches the job fetches that job daily emails for example you need to fetch that user also because you are passing it in the when you dispatch that job either via the kernel or via Route or some other mechanism but the important thing here that you have to take that into consideration ation when you dispatch that job because if you put some constraints on your user or if you only need a subset of the data you better use that subset if you won't be careful you can load many many uh tables just for one job so we'll see what what uh what do I mean by that when we'll get there so basically when you dispatch that job somewhere in your code it looks like this daily emails dis dispatch and you pass the user okay this is how you call that job and you can also call that from controller with this with this and this patch okay it's it doesn't exist here because it's not a controller but you can also dispatch it like that but when you do that it serializes that user and when the scheduler or the Q listen call calls that job uh gets that from the database from redis or other connection it needs to unserialize that user and it needs to gets the relationships and this is what that trade does serializes models okay if you want to dig in the code you will understand how it does that but this is the the responsibility of that trait okay let me just remove that and let's go go to the handle method now the handle method is where you do your work basically on that user let's say you need to send a mail okay mail to this user I'm going to save that just for the perp for the example I'm going to initialize and I'm using your shortcuts by PHP Stone which is very nice okay I'm going to mail this that user that email let's pretend that I have I do have actually not pretend because the user is authenticatable I do have the email property and I'm going to do a send some kind of a mailable here okay or I can do an anonymous class I think also new class yeah that implements the mailable let me see it doesn't matter at the moment but let me just remove that I'm going to do mailable okay it won't work of course because this is not ailable you see it requires the mailable contract some kind of implementation of the mailable which means a mail but that doesn't matter what matters is that you can either uh call your class and in this case I'm going calling the male facade but if if you add a service provider for example to image processor right here you can just call that uh let's say that I'm adding an attachment and it's probably here on the mailable itself the attachments because I don't remember at the moment attach yeah probably there let's say if if I'm adding an attachment okay and I can use that uh service because the handle method of the of this class is able to type hint that uh class and the larel container is able to supply that because this is in this example it's a very simple uh class it doesn't have a complex dependency structure so I'm able to use that instead of using for example image processor equals to new image processor sorry processor okay and I'm going to fix my typo here okay instead of using that and initializing the dependencies I can use that here inside of the handle method so this is basically what you need to to know about the first Q about the jobs about the Q jobs that you create and let's continue and discuss what else we need to pay attention to let me just remove all that remove that okay so more gas for example when we we said that we eager load the relationships so when we pass a collection of users for example if this is if this was a collection an eloquent collection ction okay those relationships will not be serialized okay only for a model it will be serialized and this is logical because you don't want all collections with all the relationships to be serialized because it will have to get a lot of data from the database and pass it to this function which is enormous okay and another gotcha is if another actually trick to do if you don't want if you let's say you have a user okay and it's a very big uh data or a very big model object so you can do couple of stuff you can do for example this at user equals user without relations okay and that means basically that it will serialize that user without the relationships if the relationships are very big and it's not are not necessary for the job you know so it's better to do that without relations or if you are using PHP above PHP 8 you can do something like that for example without relations I think and we need to import that class and pretty much what happens is the same like I did before let me see if if I can see that from the class nope can see that but what happens is that larel translated to the code that I show you before it puts this that user user without relations okay and this is just a property notation of PHP 8 and we can use that instead okay some other goas now just like I did on eloquent models that it serialize them just like I told you we have to pay attention when we pass basically any any data to the Constructor because it has to because this is a it will run as a closure it will have to serialize it somehow and for example if you pass any binary data you have to encode that data because it needs to be Json encoded this is how it serialize it serializes it one other F that I wanted to mention is basically if you have um a scenario because we we said that when you dispatch that job it goes to a bucket and that bucket is waiting for more jobs and as soon as the job comes there depends on your Q listener on your connection listener that job is usually either immediately fired to to be processed or waits for some sometime time sometimes you have scenarios where you can only allow one job to run or one job to be in that queue and not to dispatch you have another interface should be unique okay what that means is that only one instance of that job will be on that bucket until it finishes or until you put some kind of a specific time lock or you can either let's say that you have couple of daily emails that you want to send but they depend on the user so you can send them but you can enforce that the user only gets one of that daily emails and there will will be no situation that the user gets two daily emails and you can Implement that you you will need to add a function here that that uh gives a special unique key for the user and to do to give that key you need to implement a function called unique ID let me see if I can use PHP storm for that but no so I'm going to implement that function here public function unique ID and what this will do is will return a special key so for example like I told you let's say that the daily email for a user um you only want one email per user then you can return this for example user ID and what will do what what that will accomplish us is that when that job is added to the bucket then it will make sure that you won't add another job or dispatch another job with that unique ID okay and after the job finishes of course it will be automatically released and I gave you example for daily emails but it can happen on a weekly basis for example or other bases you can also add a property here unique for and I think it's Public public property okay for how many uh millisecs I think yeah so basically let's say we'll have uh six 60 seconds I think it's not milliseconds 60 seconds time 60 so it will you cannot do 60 * 60 but let's say 1 hour it will be unique for 1 hour after that it will be freed from the lock so this basically this is basically how you make sure to restrict that from running from dis patch if you already have a job on the queue one other option that you have here besides that is let's pretend you have a scenario where you have a QA team or um debugging or a you know some kind of other team and the information here on that job is supposed to be private and not available to them but you still want them to debug what happens why why is that not working working so you can use should be encrypted interface and automatically all the serialized data here will be encrypted and not um visible in terms of uh text because it is visible if you look in the database but it's still encrypted okay and after laravel retrieves that it will be decrypted of course but when it's saved it will be encrypted and not visible outside we also have another option for example if we don't want to use the uniqueness of jobs is to rate limit with laravel rate limiter and through the middleware we have to implement a function here which is not available at the moment but we can create it middleware and this method by the way returns an array okay so you will have an array of middle Wares that you want the job to run through okay you can rate limit it you can make another filters or uh make sure that only one job runs without using the should be unique or you can do whatever you want just like route middleware with that job okay you can stop that job from running if it reaches a certain uh condition so basically that that's that's it about this uh the structure of the jobs here in this lesson we discussed a lot about the class structure of the job and how you can modify it to suit your needs we discussed about the gas of using models Collections and other data for like binary data for example and we gave couple of options how to limit the jobs from middleware or through the unique should should be unique interface we are going to do an example with middleware later so that's why I'm not um working on that now but we're going to to use that also so thank you and we'll see you in the next section next lesson hi and welcome to this lesson where we we're going to continue on building our knowledge on larel q's and we we are going to do some basic stuff for uh dispatching the jobs and after we finish that lesson we're are going to start and building our image processing uh application where we have um buttons to upload an image and to process it in different ways via the cues and um we'll go from there and we continue to develop our application to to do more complex jobs on the cues and at one point we're also going to to cover what to do when we have dependencies between the cues and how to work with that but first let's finish the basics and we'll start building our app from here let me just commit the code for now and don't know why it's that commit create the job class we remove that make it like so and let me reformat that but just before that sorry yeah I'm going to reformat the next class I'm going to turn on the auto next commit we format the class so it will be automatically the optimization and the cleanup should should happen automatically also by the way okay so how do we dispatch that job we can go to a controller which is responsible for this view which happens to be in controllers well not here actually since it will be in routes web and it's a closure here that calls that view but what we'll do we'll delete everything from The View and only leave let's say I'm going to delete that I'm going to leave that open and yeah I don't want that ugliness so I'm going to remove couple of stuff let me see documentation okay I think I can remove that that that and that okay so I remove that and I'm going to do the dispatch from right here from the closure and not from the the controller but we can use the controller functions here as well so what can I do and let me go to the job itself to see what what I need for the job I need a user okay and yeah I'm going to I'm going to first show you what happens because of the default laravel Q driver Q connection sorry is synchronous what happens okay I'm going to do U DD here so it will die right here died inside main Fred of execution okay and we'll see how how it works I'm going to go to my web file and I'm going to do a dispatch of daily emails dispatch and and I'm going to do a user I don't have a user actually so I'm going to remove that for now just for the sorry just for the sake of that I'm going to remove the user and I'm going to remove the user here also okay let's see what happens now and as you can see even though we dispatched it sorry about that and it had the interfaces let me remove that interface also by the way it had the shot shoot q and cable interfaces it still depends on the driver and the driver is synchronous so it ran on the main thread of execution which means it ran on the request of the user okay when we for example we'll do redis test with redis we'll see that it doesn't happen or with the database we'll see that it doesn't run on the main F of execution so I'm going to leave that and I'm going to also do another another function that I can use here delay and I'm going to use carbon minutes and minutes I think yeah 10 minutes so I'm going to delay it so not only it's uh it's supposed to be C it's also supposed to be delayable okay with why should I delay it for example if I know that my queue is um if I only want the emails to be sent 10 minutes after the user is uh not a welcome email I'm talking about some kind of a special promotion for the user which I want to be sent uh after 10 minutes he registered I can use that because let's say I don't have a special marketing service that I'm using I only have that so this is one of the solutions other use cases can be can vary you know you can use it for many reasons if you know that your queue is very busy you can just delay the jobs in couple of minutes and that it will cause some kind of a you're only delaying your botton leg basically but uh it's still can help you a lot so let's say I'm delay delaying it in 10 minutes and restart that and still the same happens okay let's say that um I want to use not delay I want there there is another option that I can use but it won't work on all servers it only works on couple of servers and I can do a dispatch after request okay but it won't be delayed as you see I don't have another function here on the on that interface okay this one worked because my service at that at the moment that I'm using which is in larel sale okay here okay it works because this service has a fast CGI and other stuff that allow it but if I if I'd be using PHP Artis and serve it won't work okay so what else can I do and you won't see by the way that it failed but uh this is something you have to be mindful about so I think that that's the basics of how you dispatch a job of course we are in a controller context even though we are in a closure I can do this dispatch and new daily emails like that also okay and it should fail sorry it should fail here route file register dispatch um yeah sorry for mistaking for uh for doing that mistake it won't work here only the dispatch will work here okay this do this patch will work only from the controller so sorry for that and let me do it like this and yeah that works so that that's the basics of the dispatching and uh we're going to start and build our application which uh a user but we won't build the full flow of the user we only focus on the cues and on the jobs here on the Q jobs so the user will be able to upload the picture and we we will do a background job processing for that picture to different sizes and we will see what happens when we use the synchronous driver connection and what happens when we use the redis connection for example or the database connection how it affects the performance or of your application so thank you and I hope you enjoyed the lesson and I'll see you on the next lesson where we start to build our application hi and welcome to this lecture where we're going to preview on what we'll be working on in the next couple of sections and just before that I went ahead and did couple of things let me show you I went ahead and uh issued an npm install and then npm run Dev in order for my v and front end assets to be compiled I also installed larel with alpine and liveware components so we'll have a easier way of working with the JavaScript and the blade input in Blade files and the file input so you will see all that in um I'm going to run that sorry you will see all that of course in the in the git commit let me just come back to the code when you load the commits when you view the git log you see all the files that I changed sorry not that me go to git log okay you will see all the files that I added and I read added the upload component of course after I added Breeze because Breeze removed that component but that doesn't matter the important thing is that this is not the focus of the course so you know I don't want to waste your time on that but I will show you what I did I added right resources here I added the component here in the welcome page which is uh this component okay where you give your email and the upload and the file and it it sends you that uh email with the file okay and this component come from comes from here from Live Wire and and I made the email and the photo are um required and uh sorry validated for example this is this has to be a valid email address and this is maximum 1 Megabyte image and after it finishes it's stored in the storage app photos location under the original name that I gave it and right what actually we don't need that because this is a live wi dispatch for events but we're are going to call this patch like that okay and we're going to call this patch like that and we're going to run besides that we're going to create a new um new job new job class which will process that image before it sends to the email so basically we're going to call it image processor I think it's called like that proc image processor with the name of the image and the email of the user so this is how we'll call it okay let me first create that this is the liare component that I created make job image process so okay I'm going to delete the all job by the way daily emails since this was just for test I don't need that can delete that here and I'm going to go to image processor let me just confirm that yeah okay so what do I need here I said I need the email and the photo location now I could pass the photo as a binary as a base 64 encoded but I rather and it it would work here of course because uh it serializes the closure and deserializes it when it comes to handle the job itself but I rather pass the the path of the photo and work on that instead of that serializing a big image so that's what I'll do and if we'll have the time maybe I'll pass the serialized version I'm going to do a dispatch from here or I'm going to do an um image processor dispatch and here I'm going to pass this that email and this sorry I'm going to need the storage path let see what I have on app path I'm not sure application folder yeah that's not that so I'll need the storage path and it will be under photos app photos and right here oh sorry I don't actually need that since I'm going to do that from the from the job itself so here I only need the file name okay let me do that sorry let me come here and I said photo name let me just fix that so it will be clearer email and photo name and file path this is the photo that will handle this photo name okay and let me just DD that and since we're using the synchronous version I should be able to see that even though it is from Live Wire okay me check that I'm going to do that I'm going to do an image and send file okay so you see that does work it takes the correct P storage app photos mini course which is excellent the next phase is to proceed with that with that dispatch I'm going to I'm going to do a an a processing on that image first of all I'm going to do a simple processing to fre sizes and then I'm going to create a mailable and attach that to the mail that I'm sending to the user first of all I'm going to create the mailable I just un Make mail send user photo okay I'm going to go to send user photos reformat that and let's see send user photos get the message content and attachments okay I'm going to send that as a as an attachment okay so let's say and I do have mail pit here just to catch those emails let's do sorry let's do a temporary check okay I'm going to do mail to this. email and I'm going to do not Q I can do I can do that on on a specific Q bucket by the way but that's not what I'm going to do I'm going to do a send a mailable and here I'm going to do a new send user photos here I'm going to pass the file path or rather yeah I think this is also serialized yeah so I'm going to pass let me try to attach it directly before I'm passing anything to that okay I'm going to make it send user photo make it as a variable and Ma to send okay just make that so it will be easier on the eyes send user photo attach and I can do a file string like that file path okay so let's test that we have mail pit open and let's do that now I'm going to do for example fail to fail in course and send that file and the view is not found let me disable that since I don't need a view and I'm not sure that it created a view by the way for that mail yeah I don't see it those are Breeze and Live Wire views let me check something no views are created here by default so maybe I can just delete that return new content let me see what's content has a view html text markdown with an HTML string well text let me try with text hello [Music] world and let's check that now if that works going to select the file again going to send the file no it it looks for a view okay so let me check on the mailable what can I do well I I can actually create a simple View and that's it that that will close it so I'm going to come to resources views going to create a new view and I'm going to use larel helper make view welcome email okay I'm going to use that welcome email H one hello those are your images that was generated on our server okay let me close that now everything should work so let's test that for now the emails are not sent of course let's see going to send that and it's supposed to redirect me oh I think I didn't do a redirect okay so let's see the mail that has been sent okay you can see that everything worked it attached that email attach that image and that's excellent okay okay let me see if yeah everything is is okay here I supposed to get an attachment yeah so that's that's cool that's very cool let me just add a redirect to refresh that page after it finishes and we'll continue hi and welcome to this lesson where we're going to continue and work on our component and the cues so I went ahead and added another live o component here to display a message after when it uploads when it process the job actually and it displays it immediately since the emails and we have no um TCP TCP IP lag we have no um round trip that it's been is been sent to far distant server so it happens um immediately and that's why you see this um this flash flash message immediately since this is on my local computer it works fast so you don't see the lag but if I if I would be calling Amazon uh email server or Amazon que or using something like that then you would see that it lack for a couple of seconds until for example until it processes that image to different sizes until it sends the email this is why we we are working with the cues from the first place by the way and I know I mentioned it in couple of places so this is the component that I added okay and this one will already so and this one will already so and if we go to this component I I'm using Alpine JS here also by the way so it will be um nicer in terms of JavaScript this is the alert I added and yeah that's basically it we're going to start to work on the background jobs now and uh we'll see on the next couple of lectures what happens when we have couple of dependencies between background jobs what happens when we have a let's say that we'll split this task to free we'll make free image sizes on free different uh background jobs and the the last job will be responsible to send that to an email the last job will also be a background job so uh let's continue on uh with that hi and welcome to this lesson where we're going to install the image manipulation library that we'll work with we don't have to install the extension this this uh package works with the GD library or iMagic and I believe that sale already has GD Library installed pre pre-installed so if we'll need to install it we'll add it I'm I'm going to add it to the docker container and I'm going to uh skip on that part for you because it's not uh totally relevant and the next thing that will need to see is that this job currently is handling the email the image processing job and as I told you I'm going to split that um image process into three jobs three sizes or two sizes and the last one will be dependent on the we're going to do it in parallel so the last one will be dependent on those job to finish before it sends the email to the user with the attach attachments so first of all we're going to do it here on that class and after that we're going to split it let me just install that Library first going to come here I'm going to open another Library okay I'm going to do a com I'm going to compose the requir it and that should be relatively quick but and after yeah it finished let me see what else do I need let me just copy that resize place I don't need the watermark I just need to save it okay oh here it says which librar is by the way needed okay so I'm going to go to my code before I attach it going to copy that reformat the file okay resize don't need the watermark I'm going to need to encode it save it and yeah we'll add it code it I think let me see oh sorry I need to import that class and import that I think I can do new driver like that GD okay let me see one second what happens with this oh it's mailable so that's super nice I can just attach it oh sorry I need to see what happened with this save encoded interface and file okay let me see file okay maybe I can still attach that I'm going to save it as output and I'll need the file name which comes from photo name okay this is that photo name and let me see what that does okay res size okay so I said we're going to do two or three um background jobs and after that we're going to do the fourth one that will be dependent on those so either I'm going to run them from here for example and I'm not going to call it image processor I'm going to call it on by different name uh either I run them directly I'm going to issue those jobs directly from here which will be like the controller of those jobs so I'll see what we'll do but let's start with that let's see that it this works and we'll continue from that okay refresh that I think I need to install something no don't need but for some reason that's stuck let me see what is missing on my end I don't think that something is missing possibly got stuck on my computer yeah let me refresh that that will turn off the the uh Docker containers and yeah I see that it got stuck me force close it and it doesn't close it's closed and that's closed okay let me see what's the issue with that s stop I think it's called it haven't stopped the container yet so that is weird I'm going to have to stop it from here from Docker let me just stop that oh it's waiting for the C workers to stop let me see if I can stop that with four stop or I'm going to have to log into the container and stop that from the container okay probably not so what I'm going to do try to log in with the shell okay I'm going to do up and see if that's working and attach to the [Music] containers I'm going to do shell and see if I can attach to the Shell doesn't look like it that I can attach to the shell and it's possible that I need to kill Docker on my computer in order for that to work so I'll see in a minute yeah so I'm going to kill Docker restart it and continue to the video hi and welcome to this lesson where we continue and work on our jobs I have gone and restarted docker so it works now and I run V again so let us continue and let us check if that works if the first thing that we changed there Let me refresh that and let's see if that works okay something is running and unable to decode input okay I probably forgot to change that let's go here and and going to read let me see if I can read the file pad directly from here let's see can image the path directory doesn't exist okay so I'm going to do something similar to what I did here let's see let's let's say I'm going to do photo path and here I'm going to do also photo paath but with different name and I want to get only the name here not with the extension since this ex this will change I'm going to do instead of photos you know what I'm going to use Photo path and attachment path attachments p and I'll see let me do jump words jump word or expand expand word let me see why it's that okay never mind I'm going to leave that for now I'm going to do it on my free time I'm going to do a directory here output okay okay so come here photos output attachment pad I'm going to put here attachment pads and I'm going to first try that and I can remove um file extension I think it's called no okay I'm going to look for it in a second let me just confirm that this works okay okay let me see why I have what I have here it's a JPEG of the attachment but of the original attachment I think let me [Music] create one attachment save it here and attach the encoded media type to string to file pointer let's not not to file pointer since I'm not sure that the attachment works on pointers so let's do that let's do also let me take that and put it here by the way so I can attach multiple files I'm going to do attach the first [Music] file I'm going to size 300 and attach the second file okay photo name 300 let's see if that will work [Music] 100 okay let's see not sure that works yeah and then issued an [Music] error logs let me see if I have the error here can check that must not conal why it's okay where does it fail well let's see attach [Music] attach let me see if that fails here in this section file name must not contain any null bites I do wonder probably doesn't work with that attachment so what I can do um I can get the path two string I'm not sure that will work let me see what that returns yeah the file pter let me see that nope won't work Pat info I can get the pat info okay yeah so what I'm going to do is get the file pad from here and put it here Let's test that first and then we'll continue yeah that looks like it works but the file is incorrect so it won't work let me fix that let me get the correct attachment here file output path and right here what I do need to do is since this is already to jpeg I think that I'll need to remove the jpeg here so let me confirm that let me get path info this photo name and P in for fining I think so let's see if that will fix it and oh doesn't fix that the file name is correct but something else is missing file output path let me check documentation here oh I see it it's it's also with jpeg okay J let me just add it here and finish with that and okay let's test that again and yeah that works you see that resized the image didn't save aspect ratio but that's okay on 100 the height is 100 I wonder if I have documentation here let me see what can I get from resizing that aspect okay image scale image scale image scale but where is the aspect ratio because what am I I'm doing resize let's let's try scale maybe that will work better and I'm going to do the second option also I'm going to scale to 300 save it to jpeg get the output path file output path 300. jpeg and get that also so I do have redundance here and that's okay because I'm going to work on that I'm going to output and extract that to a different job and this job will run is a function of how many sizes I want so let me close that and let me come here send the file and see what I'm getting here now okay now I'm getting two files and they're much better but the quality is pretty lame but the aspect R is fine and I think I do have some parameters here to control the quality let me see if I'll find something yeah unfortunately they don't have anything to I I have tested the quality parameter here but it it didn't help so I'm going to stay with that and do something else okay I'm going to extract those into and I'm going to extract it outside of the job of course outside of the Constructor of the new job since this also should be serialized and it won't work if I'll use it in the Constructor only the file name and the email will work so this is what I'm going to do I'm going to extract it now and continue after that hi and welcome to this lesson where we continue to work on the image processing and send in the emails via the background jobs and before we do continue I want to show you what happens when you try to add those classes for example or some kind of data to the Constructor and not the handle method and uh you'll see the error I added this image manager and the manager read to this. image in the Constructor of the job and you you will see immediately what happens sometimes it's not clear by Lille errors where that error happens but you will see right now that the exception FR is a serialization exception and it also it points out that in Livewire it it happens here here when I dispatch the job if I open the vendor frames you can see exactly where it happens and you you see that it fails when I'm creating the object payload and uh working on the serialized method okay so that that is usually the error that you receive when you just getting started to work with the cues and you don't know but uh when you remember that the serialization is very important part of the job queue of the que classes sorry then um it will be easier for you since you will put on the Constructor only strings and you know some basic stuff and the the job itself the major part of the job will be on the handle method okay so I just wanted to show you that error before we proceed let me close that and we will proceed this uh on this lesson to take that big image processing job and split it to multiple jobs and I I don't I don't remember if I mention it but qes or background jobs can be used in the kernel file of laravel not only as a dispatchable action but you can use them when you use the task scheduler to do for example daily reports or hourly reports or 5 minute reports so um just I wanted to point it out so I won't forget for later to mention that and uh basically how you do that is you create a job like that one and you come to Kernel to console kernel of course and you let me reformat that and you schedule that command for example schedule job not command New Image processor let's pretend that uh this is a report let me do something else you know what new hourly report okay can schedule that per hour so this is another use case for larel Q jobs and just before we'll continue let me commit that so let me see what I removed there removed the buug I created for testing purposes okay so how do we split that job into multiple jobs now we can use laravel batch jobs and we can use another way to do that and I'm going I'm going to show you both ways because laravel batch jobs and chain jobs are very nice and they the other package that is called laravel Venture came to existence before laravel didn't have that option L Venture didn't have that option to do that but this package has much more functionality than the um basic batch and change jobs that larel Q has I can I can do do dependency tracking here and more stuff so we'll see that when we get there but first let's do let's break that job into multiple jobs and dispatch them so what I am going to do since I cannot put that in a Constructor but the job that I will extract depends on the file name which which I already have and it depends depends on the size the size is the parameter so I either can do it in a 4 in Loop which I don't want since I do want to run that couple of times so you can see what happens when I run that job in parallel or I can um do the other thing which I'm going to do now okay I'm going to create a new job PHP artisan make job I'm going to call it image resize and I'm going to add a new job image resize and send email well send images in email actually because I'm going to extract that also and this will be the controller of the jobs so let me go to my image resite job and let me see what I need here save and attach to the mail well I'm not going to attach it now I'm only going to save it so what I need here is the file path and size reformat that and initialize those properties going to copy that I'm going to read with the file pad from this oh I do need the attachment path and the attachment path I can just put that once here and I I think that's file name actually not file path so let me see attachment path I can take that photo paath I can take that yeah so let me copy that going to put it put it here I'm going to change that parameter sorry file name okay and let's see oh that should be file name also oh only that parameter changed okay file name and what else do I need I have file [Music] name file name here size is this size attachment pad I can use me see file path and attachment path yeah okay so I'm going to do another parameter this file path and this attachment path but I'm I don't need this for the attachment I only need that F name okay let me add those and let me just just remove that for consistency file path and remove that okay so now let's see what else do I need and those are strings by the way so strings in in integer so that's okay I'm going to do this attachment path not attachments attach pad and F yeah that should be good so I'm going to test that before I'm going to split it into more functions okay and send user photo I'm going to remove that I think I can remove all of that let me just confirm that I don't need that also but what do I need to do I need to dispatch for 100 size okay image resize dispatch and I need let me see what I need I need the file name and size okay so this photo name and size is 100 and here since they are all synchronized that should work file output path and file output path should be like that with the file name okay attachment path this photo name and yeah that should be it okay let's see if that works and then we we can continue and Abstract more of that unable to decode input okay you can't find the file name let me see if I have what I'm passing here yeah probably not so what I'm going to do I'm going to DD the file name and see what happens what what is wrong image resize I have the file name here file path attachment path and it fails here let me DD file name this file path and this attachment path let me D that okay file name is okay file p path is okay attachment path it's okay but I'm missing the file here attachment path maybe not okay what I'm reading so this is this is this fails here on file name so what I need to do here is to get the second parameter which is file path f file path okay I'm going to do attachment path get the file name without the extension and add the jpeg here okay that should work that should fix it enable to open path well it it fails on the user photo and let me see see why first of all because the name is wrong but let me see why does it get that name attach file output path probably because of that of course because I didn't do this manipulation like I did there so let me get from file output from yeah there that photo name now that should work yeah that worked and that's okay the size 100 that's okay what else can I do I think I'm going to do H size 600 just to test it yeah that's better I'm going to have 600 500 and four 400 so I'm going to do the first one is 600 500 400 no 500 600 and 700 I'm going to add 500 let's see if I can attach like that yeah I can attach like that so this is what I'm going to do copy that and paste that here remove that of course okay let me test that again 500 doesn't work and why is that not working let me see what what files this it create let's go to storage directory output any course 100 100 300 where is the 500 and 600 files I wonder yeah nowhere to be found that's that's weird that's weird let me see let me disable that going to refresh it take the file send it but it won't be send I just want to see which files it's created when I don't see any files so I [Music] wonder if I'm saving that yeah I am saving that oh this was the problem I'm saving that to the same location so yeah going to do minus this that size okay that should fix it and let me uncomment that let me see that now send it to New basic user send the file and I should be getting yep I'm getting those now 500 600 700 okay excellent that's excellent and you can see already that it took 1 second on my computer and even though this is from the same network so you can imagine how much time it will take to process multiple uh people's files images avatars or video video transcoding you know whatever and then to send the attachment if it's a big attachment we don't uh you know you can it can take hours but unless it's um capped or Limited at the Gmail or other provider but you can understand my point that it takes a lot of time so I want to thank you and Contin and in the next section in this section we learned how to split that job in this lecture we learned how to split that job in the next lecture we are going to do something different we're going to take that and um we only split it to image resizer here but I'm going to do like I told you the um dependency tracking between those files and we are going to run that in parallel get the outputs and um then send the email and I think we we're still going to use that as the controller but we'll see so thank you and I'll see you in the next lecture hi and welcome to this lesson we're going to discuss on job chaining which is relatively new technique in larel qjar but it's very useful and just before we start I do remember one extra use case for encrypted jobs and if we'll take a look at what happens when we use the should be encrypted interface which is a new interface if we take a look here at Q and will follow that function what happens here is that the larel code if the job should be encrypted it checks if I have an encryptor class in the container in the service container Lal uses and then it encrypts after it serializes a a clone of the job basically means that it it's not mutating the job but it serializes the job so why another use case that I remembered is that for some reg regulatory laws and uh process and compliance security compliance mostly um you can use that to your advantage because usually they require for example if you do PCI DSS uh to be a payment uh processor they require you to have separation of database and encryption so you can use that because it will um make your job encrypted and not visible to outside world even if someone um tempers with that so that's another use case that I remembered and okay so how do you use the job chaining and why first of all you use it to sequence jobs okay what that means if you have couple of jobs one after another that should be in some specific order then instead of running instead of dispatching from from your controller let's imagine that this is my controller and you will you will see why why this is a problem at the moment the job here is synchronous which means that if I have another job here and another job here they will run one after the another one after the other but as soon as I'm going to change this one let me search for Q connection to red or database or bin stock whatever you know as soon as I'm going to change it to a sing synchronous skill I'm going to have raise conditions because this job even though it's one after the other in the code this chop could run before this one that depends on which queue for example if this is on Q um let me call that low priority low priority Q okay and this is on another also on low sorry I'm going to switch that high priority okay I switch the the lines let's say this runs before this but the the bucket this bucket is is for high priority cues so I'm going to do some calculation you know just pretend not really but pretend that I'm doing some calculation here that will make that job run before that job so you'll see if you have a sequence if it's important that the jobs will run in order then you have to do something else either you wait and pull for the job to finish which is um wasteful because it wastes your resource your uh your threads your process either you do something else and this is what the job chaining came to solve I'm I'm not sure since which larel version that is but um it's a nice addition to the language and you previously you can mitigate it as I said with couple of stuff for example you can use should be unique and make sure that you have only one job on that queue on specific queue okay with that for let's say that this was I'm going to show you what I mean this was PDF processor and this was word processor okay so all of them would be on the same queue for example okay on Q same and what that what that thing does is basically is to mitigate the the issue of uh the first job didn't run before the second the second job didn't run before the first job okay but as soon as you have more workers for the same queue and we'll get to workers when we go when we we'll have the lesson about the Q workers you will see what happens because we we won't do one worker for que usually that depends depends on how important that queue is or what job is in that queue but let's pretend that this queue is very important so we won't have one worker we can have five six eight workers for that q and that means that if we'll dispatch it even though they are on the same queue this one will be dispatched to this worker this one will be dispatched to this worker and this one will be dispatched to this worker and again we have the problem of race conditions and we don't know which one of them will finish first so this is a problem that the job chaining came to solve and we'll solve that we're going to do we're going to do that let me just delete all of that okay and I'm going to show you how to use that another thing that we need to take a take a take a not mental note to ourself is the how to how do we handle failures in the job job chain because we can we can handle we can put it in try and cash and um I'm not sure that it will catch it but we can do multiple here it won't catch it for sure because this as soon as it dispatches to a worker and not to synchronous job I'm out of context in the PHP I won't be in context so it won't catch it okay so you can imagine the the issue that we'll have so let's let's just start I'm going to do I'm going to remove that also by the way and I'm going to do I'm going to use sorry I'm going to use the dis patch first of all let let me delete that I don't need that okay I'm going to use the dispatch here inside this queue okay so you will see what I mean

Channel: Software Skills Studio

Views: 87

Keywords: online courses, course, programming, full stack, web development, front end, dev ops, laravel, laravel queues, background jobs, asynchronous processing

Id: yJko1GhPNgs

Length: 106min 33sec (6393 seconds)

Published: Sat Mar 16 2024