Python Tutorial: Generators - How to use them and the benefits you receive

hey how's it going everybody in this python video we're going to be going over generators and why you'd want to use them and some of the advantages that they have over lists so in this example i have this function up here called square numbers and what it does is it takes in a list of numbers and then we have this result variable which is set to an empty list and then we loop through all the numbers and from the list that we passed in and we append the square of that number to the result list and then after we're done looping through all the numbers then we return the result and then you can see here i have this by numbers variable set equal to square numbers i'm passing in a list of one two three four five and then i just print down or then i just print out my numbers down here so if i run this code then you can see that our list of one two three four five that was passed in our result is 1 4 9 16 25 so currently our square numbers function is returning a list now how would we convert this to be a generator well to do this we won't need this result variable anymore so we can take that out we don't need the return statement and this result dot append we can take this out and all we have to do is type in this yield keyword and just yield the square number here so this yield keyword is what makes it a generator now if i save this and run it you can see now whenever i print my nums i'm no longer getting the list if you look at the comment here this is what the result used to be i'm no longer getting 1 4 9 16 25 the squares of one through five no longer getting that result i'm getting this generator object here now the reason for this is because generators don't hold the entire result in memory it yields one result at a time so really this is waiting for us to ask for the next result so it has hasn't actually computed anything yet now if i printed out next my nums which asks for the next result then you can see that it's one because we passed in our list of one two three four five and then we're looping through that list and so one is the first value so it's equal to i here and we yielded out one times one and it gave us that result so now if we copy this line here and print out next mynums a few times here and run that then you can see that each time that we run next it goes and gets the next value that's yielded so now we have uh 1 squared 2 squared which is 4 3 squared versus 9 16 25 and so on now 25 is the last value from our result so what if i was to run next one more time well if i do that you can see that i got an error here and the exception that it threw was stop iteration and that means that the entire generator has been exhausted and stop iteration just means that it's out of values now instead of getting these values one at a time we still can use a for loop on these generators and this is personally how i use generators a lot of the time so let me comment out this line and then let me uncomment that one and save it so now we're saying for num and my nums which my nums is our generator print out num so i'll run that and you can see that we get all of our values and we don't get the stop iteration exception because the for loop knows when to stop before that happens so one immediate advantage over a list is that i think that this is much more readable here rather than having the result set to an empty list and then appending to that result and then returning the result this is kind of more readable we're saying okay i'm passing in these numbers for each number and that list of numbers yield the result now for those of you more familiar with python you might have noticed that this entire process here of these lines of code would have been much easier to write as a list comprehension so let me comment it this out and if you don't know what a list comprehension is don't worry about it too much i just want to show the generator example with this as well now this is a list comprehension here and it's going to do exactly what our square numbers function did so what we're doing is we're creating a list and we are taking x times x so the square of x for x in this list of one two three four five so if i save this here and run the code you can see that i still get the same results and i can also print out this list up here at the top now you can create a generator in the same way and it's just as easy as taking out these brackets and instead putting in parentheses so if i take out those brackets put in parentheses now if i run this then you can see that when i printed my nums here i tried to print it all at once i got that generator object and then when i ran my for loop it looped through all the values and gave me that result okay so what if you wanted to actually print out all of the values from the generator well like i said they're not currently all held in memory but you can convert it to a list and it's just as easy as just putting list and then wrapping that and then if i run that you can see that it run it that it printed it out just as if it was a list now when you convert this generator to a list then you do lose the advantages that you gained in terms of performance and i haven't talked about performance yet but i have a better example to show those advantages so a generator is better with performance because like i said it's not holding all the values in memory which isn't a big deal at all whenever you have a small list like this of one two three four five but say that you had tens of thousands or even millions of items to loop through then having that many items in memory will definitely be noticeable but you don't get that with generators so whenever you cast a generator to a list like this if this generator had a lot of values that it needed to convert to that list then you lose that performance in terms of it would put all of those values into memory so let me show you a better example here of this performance difference so i have a file here where um some of this stuff you don't have to worry about like these lines here i'm just printing out the memory and then these names and majors i'm just these are just going to be used to make some random values so i have two different functions here one of these is going to make a list and one of these is going to be a generator and they're both returning the same values so within this list i have my result here and i'm looping through a number of people that i'm going to pass to this function and for each person i'm just going to re make a person dictionary give it a an id and a name that's randomly chosen from the list of names up at top and a major that's randomly chosen from the list of majors and then i'm going to return that result and for the generator it's the exact same thing i'm going to loop through the number of people that i pass in and then i'm going to yield this person dictionary that has the same values as the list function had now really quick just to make these the same i'm going to make that an x range instead so that these are exactly the same okay so right here don't worry about these lines here this time dot clock and this t2 time dot clock all i'm going to do is time how long it takes to run this function which returns a list now i'm going to pass in 1 million values to this function so it should return a list of 1 million results and then down here at the bottom i'm printing out the memory usage and the total time that it took so if i run this then you can see up here at the top of the code so this before here this is before i made anything so my base memory usage was around 15 megabytes and this memory after is after i created that list of 1 million records so you can see here that it jumped up by nearly 300 megabytes and it took um 1.2 seconds now if you're dealing with large amounts of data you know that's not out of the ordinary to have one million records like that so let's see what this looks like if i instead use the generator example so i'm going to going to comment out the uh the function that returned the list and now i'm going to uncomment this function that returns a generator and i'm going to pass in the same number of values i'm going to pass in 1 million values here so if i save that and run it now you can see here after i ran this that the memory is almost exactly the same and that's because the generator hasn't actually done anything yet it's not holding those million values in memory it's waiting for me to grab the next one or to loop through those and it would give me those one at a time now this time that it took here basically it didn't take any time because as soon as it gets to the first yield statement it stops so if i was instead to make this an integer then it would be nearly zero seconds now whenever i said earlier that if you convert this to a list then you lose that performance then let me show you what i mean here so i will convert this result this entire result to a list and now if i run this then you can see basically i got pretty much the same result that i got when i ran the function that returned the list so if i take these back off and just do the generator then you can see that we get our performance back so that's how you use a generator you know i think that it is a little bit more readable and it also gives you big performance boosts not only with execution time but with memory as well and you can still use all of the comprehensions and this generator expression here so you don't lose anything in that area so those are a few reasons why you would use generators and also some of the advantages that come along with that so i hope this video was useful for you guys if you do have any questions about this stuff just ask in the comment section below be sure to subscribe for future python videos and thank you guys for watching

Channel: Corey Schafer

Views: 772,100

Keywords: Python (Programming Language), How-to (Website Category), Python Generators, Generators, Python Tutorials, Python Tutorial, Python, Programming Tutorial, Programming Language (Software Genre), Programming Tutorials, Video Tutorial, Programming, Computer Science, How to, How-to, Software, Software Engineering, Development

Id: bD05uGo_sVI

Length: 11min 14sec (674 seconds)

Published: Mon Aug 17 2015

Reddit Comments

[u/thelindsay]

So once this 10 minute intro has you interested, here is the 3 hour deep dive with David Beazley https://youtu.be/5-qadlG7tWo

[u/JayTh3King]

Good for when you are working with large data sets and don't want to store the whole set in memory.

[u/Kaxitaz]

Good video. Simple, short and straight to the point.

[u/syntaxvorlon]

I only vaguely remember learning about generators, so this is a really great set of examples. I'll probably start using these a lot.

[u/ghillisuit95]

Can generators be recursive? Also: should generators ever be recursive?