PHP Generators Explained - Full PHP 8 Tutorial

[Music] we covered iterators and how to iterate over objects back in section 2 in lesson 2.22 if you have not watched that video please consider watching it first because it will help you understand this lesson better i'm going to leave the link to that video in the description so check it out when you have time in this lesson we're going to cover something called generators which in simple terms just let us iterate over large sets of data without consuming too much memory let's check the php documentation and read about the generators so generators provide an easy way to implement simple iterators without the overhead or complexity of implementing a class that implements the iterator interface a generator allows us to write code that uses 4-h to iterate over a set of data without needing to build an array in memory which may cause us to exceed a memory limit or require considerable amount of processing time to generate instead you can write a generator function which is the same as a normal function except that instead of returning once a generator can yield as many times as it needs in order to provide the values to be iterated over so this kind of makes sense but when you read it for the first time you might not understand it right away so to better understand what generators are let's do a couple of examples to clear things up we'll do an example that's provided within the documentation first and then we'll refer back to this definition from time to time to really digest the meaning of it i have created a new route and a controller to demonstrate generator examples so we have this example generator route that is connected to the generator example controller with the index method on it so we'll go into that index method and this is what we're going to use for this example the example from the documentation uses the function called range as you might know the range function creates an array of the provided range or the size and fills its values so we can create a variable that will store array of numbers so we can call the variable numbers equals range and we'll pass 1 and 10 as the start and end range numbers let's print this array and we're going to use print r and we'll also add the pre-tags here so that it's easier to read let's open the browser refresh the page and sure enough we get an array with 10 elements in it and the values of it are just a number starting from 1 all the way to 10. the thing is though this array here is built in memory once the range function is called so it consumes and uses up the memory it needs to build up that array building an array of 10 numbers isn't that bad and not really a memory intensive operation but as you can imagine as you pass in the larger numbers where it will build up a large enough array you will eventually run out of memory so let's test out some numbers let's test out 10 000 and see if this works we're going to refresh the page and sure enough it works let's bump that up to 1 million and seems like it's still working but it's taking a while because it has to print out all those numbers if we keep scrolling all the way to the bottom we see that it is slowly printing all these numbers up so let's bump it up to 3 million and see what happens let's refresh the page and sure enough now it fails and we get the fatal error where it is running out of memory now of course this will depend on few factors one is the memory limit that is set for the environment and another one the memory each element consumes our array is very simple and it only contains numbers so it does not consume a lot of memory which is why it takes a lot of numbers in our array for it to run out of memory but in a real application you could run out of memory with much less data for example a 10 000 fully hydrated models objects or entities that have enough data could very much run out of memory much sooner now there are a few solutions to such memory problems first we can implement pagination so that we're not selecting more than the set limit amount of records we could also add filtering to filter the results which would narrow down the number of records being selected and we could also select only the data that we need these solutions are fine and would work in a lot of cases but in some cases you do need to loop over a large data set that would otherwise run out of memory this is where generators can help so to implement a generator the first thing we need to do is that instead of using the built-in range function here we need to use a custom range function which we're going to implement as a generator function so we can call our custom function something like lazy range in this example i'm going to implement this as a method within this controller but it could be implemented as a helper function or even be part of a separate class and so on but i want to keep this example simple and to the point so i'll let it within this controller so we're going to do this lazy range and we're going to create this method the first parameter is start and the second parameter is end in order to build up an array of numbers with a given range we need some kind of loop so we'll use a for loop so we can do something like for i equals start i less than equal to end i plus plus and then for each iteration we need to kind of return the value so that it can be stored in an array so one option is to have some kind of array that stores all the values and then we can just push the values into that array so we can do something like numbers here equals empty array and then push the number into the numbers array and simply return numbers here but this will have the same memory problems because we're essentially doing the same thing as the range function if we go back and refresh the page we are still getting this fatal error so this doesn't work we need to get rid of this and instead what we need to do is we need to yield the value of i at each iteration now how do we yield the value if we're going to simply try to return i this is not going to work either because if i'm going to go back and refresh we're just going to get 1 because as you know the return statement stops the execution and returns the value so how do we make this into a generator function let's head back to the documentation and read about it we see that the last sentence of the second paragraph states that instead of returning once a generator can yield as many times as it needs in order to provide the values to be iterated over and the keyword here is yield let's click into the yield keyword so we can learn more about it so we see that the heart of a generator function is the yield keyword and in its simplest form a yield statement looks much like a return statement except that instead of stopping execution of the function and returning it yield instead provides a value to the code looping over the generator and pauses the execution of the generator function and the thing to highlight here is this pauses execution part so the yield keyword pauses the execution while the return statement stops the execution so let's head back to the code and change return to yield and what's going to happen here is that instead of stopping the execution like a return statement would it will just yield the value of variable i and it will pause the execution of the generator function until the next time the generator needs to run and this is why i included the word lazy in this method name because we're kind of lazy loading or lazily generating the data this means that we're not loading everything at once into the memory we are kind of doing it on demand so let's test this out and see if this works so i'm going to open the browser and refresh the page and sure enough we're not getting any memory related errors we're not getting any fatal errors but we're also not getting an array printed on the screen instead we're getting a generator object let's open the documentation for the generator class and see what it is because this is an object of a generator class we can see that the generator class implements iterator interface so this means that we can loop over the generator object using the for each loop or we can also call these iterator methods manually so let's call the current method manually and print it out so i'm going to open the code here and instead of doing print our numbers we're simply going to echo out numbers current and because we know that generator functions or methods return the generator instance object we can simply type hint this that this will return generator object so let's open the browser again refresh the page and sure enough we're getting one let's advance to the next one by calling the next method and let's call out the current again let's refresh the page and sure enough we're getting two so the first time we're getting one then we're getting two let's do it one more time i'm going to duplicate it let's refresh the page and now we're getting three so you get the point as you can see the data is now being built up in memory right away but rather the values are yielded one by one so let me reiterate this again and no pun intended here when the lazy range method is called no code within that method actually gets executed right away we can actually easily test this by adding some echo or vardham statement within the lazyrange method i'm going to comment out these things here because we don't want to get the current or the advanced to the next and we're just going to call the lazyrange method we refresh the page and as you can see everything is blank which means that this code right here does not get executed even when we call the method this code within the lazyrange method only gets executed when one of the iterator methods are called or when we loop over it it will then start executing the code until the first yield statement when the yield statement is reached it will pause the execution and the value that was yielded can then be fetched via the current method to resume the execution the next method needs to be called which will resume the execution until the next yield statement and so on so to debug this i'm going to uncomment this and we're going to refresh the page again and see when the hello gets printed we see that hello gets printed first and then we have one two and three so to reiterate this and no pun intended again when we call one of the functions on the generator instance only then it starts executing the code within the generator method which is lazy range in this case it prints out the hello and then continues execution until the yield statement he reaches this yield statement and use the value of the variable i which in this case is number one and then it pauses the execution this value then is accessed by the current method and then it gets printed as one if we were to comment this out and only leave the current here and refresh the page it's not going to print 2 and 3 it only prints one but then we do next here which basically resumes the execution so what's happening is that it picks up where it paused before which was at the yield statement and it resumes the execution and because we're within the loop it will simply go back to the yield statement on the next iteration and it reveals the value 2 because i was incremented right here and it pauses the execution again now if i refresh the page the 2 is not going to be printed because we haven't called the current method it simply got yielded and the execution was paused we can access that by calling the current method again and if we refresh now we have one and two and this goes on and on until there are no more yield statements and the generator function ends so if we uncomment this this is going to resume the execution from where it left off last time which was at the yield statement again and because we're within the loop it's simply going to iterate one more time and then we're echoing out the next yielded value which will be three in this case and if we refresh the page we get one two three so i know i spent a lot more time on this but i want to make sure that this makes sense and it's easier to understand how the generators actually work now in this example we have the loop right that's how the values are being yielded at each iteration but instead of loop we could have multiple yield statements if we wanted to so we can comment this out for example and we can simply yield start and then yield end and i know this is not really the range function i just want to show you that this will work when yielding multiple values it's not going to increment anything it's just going to yield one and then it's going to yield 3 million so what's going to happen is that right here we're going to start the execution of the lazyrange method it's going to print hello and it will keep on executing until it reaches the first yield statement which is here it will yield the start value which is one so it's going to be printed here then we're calling the next method which will resume the execution it will continue the execution from this point on until the next yield statement and to be sure we can add another echo statement here something like world and we'll see that this will get printed only when this next method is called and then we echo it out using the current method so let's see what happens i'm going to comment this out and we'll debug one more time so let's refresh we have hello and one as expected we're going to uncomment next and uncomment the echo statement we'll refresh and we have hello one world and three million which is expected right let's uncomment this code and see what happens next and as you can see nothing gets printed because we don't have any more yield statements or any other code afterwards if we add echo and some kind of exclamation point here if we comment this last part out and we run the code we see that that doesn't get printed it will only get printed once we resume the execution so if we resume the execution and we refresh the page then it gets printed and the get current will still be blank because we don't have any more yield statements so if we go here and refresh we don't get anything after the exclamation point this was one of the confusing topics for me when i learned the generator so i wanted to make sure that i cover these small details and show these examples with the debugging steps and so on to make it easier and clear enough to understand how it actually works also notice that the lazy range method does not have a return statement the generator functions or methods can actually have return statements within them so instead of echoing out the exclamation point here we can simply return the exclamation point so we'll do return exclamation point and notice that the type hint will not change because this is not going to return a string the laser range generator function or method in this case will still return an instance of generator the returned value from the generator method can then be accessed by using one of the methods of the generator object so we can access that by calling getreturn method if we inspect that we see that it returns whatever was passed to the return or null if nothing it will throw an exception if the generator is still valid and this is something to keep in mind because if the generator is still running which means it's still valid it will throw an exception so if i'm going to refresh the page here this is going to work everything is still printed correctly hello world 3 million and exclamation point and we should probably get rid of this exclamation point from the hello so that he reads better so we'll have hello one world three million exclamation point now this doesn't throw any exceptions because we have two yield statements and we're echoing out both yield statements which means that the generator has finished and it yielded all the values and therefore it is ready to return and then we access that return value by using the get return method but what happens if we try to get the return value before generator has finished running so what if we do something like this before we resume execution after the last yield statement if we refresh the page here we see that we get fatal error here saying that we cannot get a return value of a generator that hasn't returned what that means is that this return statement hasn't run yet which makes sense right because if we don't resume the execution the execution is still paused right here and the return statement is never reached the return statement would only get reached once we have resumed the execution and it continues executing and there are no more yield statements otherwise if there is a yield statement somewhere here that gets executed it will again pause the execution and this will not get reached this is why we're getting the exception so this has to run after the generator has finished running through all the yield statements so if we refresh the page now everything is back to normal another thing i want to show you is that we can also yield a key value pair i'm going to comment this out and simply write a simple loop so we're going to do for each numbers as number and we're going to also access the key and we'll simply echo out key colon number and we'll add a break line and we also need to bring back the loop here so that we're building up the array so i'm going to get rid of this hello and this echo statements from here and we'll also remove the multiple yield or the return statements and we'll simply revert back to yielding the variable i so now if we refresh the page we'll see that it's going to be printing values at each iteration all the way up to three million and we're not going to run out of memory and this works right the thing to notice here are the keys the keys as you can see are sequentially numbered starting from zero what i'm going to do is that i'm going to change 3 million to something like 10 so that we don't have to wait for it to print all the way and we're going to yield q value pairs here and to test this out i'm simply going to yield something like i multiplied by 5 let's refresh the page now and see what we get and sure enough we see that the keys are now multiplied by five the first day is five then we have 10 then 15 20 25 and so on so when keys are not specified it will simply be sequentially numbered indexes starting from zero but you could specify that to return custom keys which allows you to do things like associative arrays and so on so as you can see we solved the memory issue where with the regular range function call we were facing the memory issue because it was building up the entire array in memory at the same time but when using generators it does not actually build up the entire array in memory but rather it use a single value at a time alright so we're done with this example let's do another example but now use database records as our data set i have created tickets table and suited it with about 20 000 dummy records and i also created a simple ticket model with the method to fetch all records so i'm going to open the models directory here and i have the ticket model here which has a single method called alt that simply fetches all the records from the tickets table and i'm going to open the database here with the tickets table and simply inspect the tickets table to show you that these are some of the dummy data here with few columns so let's close this up and simply try to call this method from the controller and see if it works as i mentioned we have about 20 000 records in the database and i want to see if this works or not so i'm going to delete this method from here and we're going to get rid of this as well and we're going to inject the ticket model into the constructor here so we'll do private ticket ticket model and we need to import this and we'll simply print the records that we get from the ticket model so we'll do this ticket model all and we'll also echo out some pre-tags to make it look readable let's open the browser refresh the page and we get the fatal layer where it's running out of memory even though it's only 20 000 records as i mentioned before there can be some other solutions to such memory problems and generators are not always the answer you could paginate the results you could select only what you need and not use the select star so in here if we only needed id of the tickets for example along with user id or created at dates we could just select those columns and then if we refresh the page it will work it is not going to run out of memory anymore because we're not selecting that much data we could also add some filters in the form of where clause to only select tickets for specific users and so on basically you should try and solve the problem before reaching for generators but in some cases when you do actually need to iterate over a large data set that has potential memory issues then generators could be a good alternative solution so let's see how we can make this work where we need to get the id title and content and this has potential to run out of memory when a lot of the records are selected at the same time so if we open the browser again and refresh the page we're still running out of memory even though we have narrowed down what we want to select and again you can add the limit here paginate the results and so on but that's not the point of this example the point of this example is that sometimes you get this large data set that you need to loop over and it can run out of memory and that's when you could use generators to solve that problem so the first thing we need to do to fix this is that we cannot use the fetch all method because fetch all loads everything in memory which causes the memory issue as you know the query method returns the pdo statement object right so let's inspect the pdo statement and we can see here that pdo statement implements iterator aggregate which means that we can iterate over the pdo statement object without fetching everything at once or we could prepare the statement and then use the fetch method within the loop so what we'll do is that in here instead of the fetch all we're simply going to iterate the statement and we'll simply yield the ticket and now the all method no longer returns array so we cannot type in this as array so we have to change this to generator and within our controller instead of printing the result of it we can simply iterate over this result and print the id and the content or maybe the first few characters of the content to the screen so that we can see that it's actually working so we're going to do for each loop here as ticket and simply echo out ticket id and the first 15 characters of the content we'll add some line break here and let's test this out to make sure that it works so we're going to open up the browser refresh the page and we're getting this warning that's probably because i'm trying to access it using objects but i have the global fetch mode set to fetch a source which means that these are associative arrays and not objects and to double check this we can open the db class that we are using and yeah we have the default fetch mode set to fetch ourselves here so let's refresh the page here and we can see that it's working now and we're not getting any memory issues and we get the ticket id and we get the first 15 characters of the content now we can improve this just a tiny bit and make it reusable so that we can have ability to use lazy fetching for other queries as well because if we had another model that needed to lazy fetch some of its records we would need to kind of duplicate this part so what we need to do here is that instead of looping over the statement here we can simply call a method something like fetch lazy and pass the statement to that method and return the result of the fetch lazy method and now we can create this fetch lazy method within the base model so we'll go here and we can add a public function here called fetchlazy which will accept a pdo statement as an argument and we're simply going to paste in that block of code with the for each loop and the yield statement and we'll just replace the ticket with the record here in the variable name and we'll add the type hint here that this will return a generator now this should still work so if i refresh the page everything should still work as it worked before and as you can see it does of course this is not going to take into account all the edge cases and other scenarios so please don't use this in production it is just to show you how one could use generators if you worked with laravel or plan to work with laravel framework in future you will come across use of generators within its cursor method and lazy collections generators can also be used when working with large files instead of reading and loading data in memory you could use generators to read and parse on demand speaking of on demand you could also use generators to hydrate models on demand as well and lots of other use cases so now that we know the advantages of generators and what problems he can solve let's also talk about some of its disadvantages or possible downsides for example you can't iterate over a generator more than once let me show you what i mean if we had this assigned to a variable something like tickets equals this and then we iterated over this would of course work however if you wanted to iterate over these tickets again somewhere else in that code and let's say in this case i'm just going to duplicate this for the sake of this example if we refresh the page and scroll all the way down we see that after it finishes iterating the first time before it starts the iteration for the second time we get the fatal error stating that he cannot traverse an already closed generator another thing to know is that you cannot rewind the generator once it has already been started unlike iterators where you could rewind for example if i'm gonna remove this from here and we call rewind right before loop and refresh the page this is going to work without any problems however if i move the rewind within either the forage or after the forage it doesn't matter where as long as we move it after the generator has started yielding values if we refresh the page everything gets printed but if we scroll down we see that we have the fatal error here stating that we cannot rewind the generator that was already run and same would happen if we try to rewind it within the for each if i refresh the page we'll get the same error so this is it for this video i hope this was helpful thank you so much for watching and for all the support i really appreciate it if you enjoy my videos please give them thumbs up share subscribe and hit the notification bell icon and if you have any questions or feedback please feel free to post them down in the comments thanks again and i'll see you next time

Channel: Program With Gio

Views: 2,433

Keywords: php8 tutorial, php course, learn php the right way, object oriented php, full php course, php in 2021, advanced php course, php tutorial for beginners, generators, php generators tutorial, how generators work in php, iterators & generators in php

Id: xH3snMmgDWg

Length: 25min 47sec (1547 seconds)

Published: Wed Nov 03 2021