A Deep Dive Into Iterators and Itertools in Python

hey there today I'm going to cover iterators in Python and the theater tools standard Library actually a while ago somebody in the audience mentioned that they really like these deeper dives into the python standard Library I think there's a couple of really cool packages that I want to dive into so I thought you know what let's make this a thing so welcome to this new series on the python standard library and the Deep dive into the possibilities without having to install any third-party modules even got a coffee for the occasion this will actually already take seven so I'm like this is a video about it's the tools no just kidding I'm actually I can deal with caffeine pretty efficiently so I normally don't have to worry about that anyway knowledge about the python standard library is actually really useful if you're writing your own code it can often provide you with certain shortcuts to make things easier but it's also helpful if you're reviewing somebody else's code if you want to become better that reviewing somebody else's code and giving more useful feedback and also being able to do that efficiently I've got a free workshop for you you can get access by going to arion.coles slash diagnosis and this Workshop is like 30 40 minutes of me diving into existing code bases and doing a review and show you how to detect issues in code efficiently and effectively it's really practical to the point you can just take those ideas applied immediately to your own code base so ironbot codes slash diagnosis to get access the link is also in description of this video so what is an iterator well it's an object that's used to Traverse through some sequence of items in this process of traversal is called iteration in Python an iterator is an object that implements the iterator protocol and that protocol defines two methods hitter and next both donor methods iterators in Python are basically everywhere you've probably used them already lots of times while not even realizing that you were using iterators they are eloquently implemented within for Loops less comprehensions generators and more basically they're all all of the language but they're hidden in plain sight let's take a look at some examples what I have here is a sequence of countries in this case countries is a tuple of strings and what I can do now there's basically two functions that are important as the inter function and there's the next function so first I can get an iterator as follows like so so this gives me an iterator over the sequence of countries and now what I can do is print the next element in the country sequence by calling next on the iterator and when I run this this is what we get now I can call next a number of times for example there's in total six countries in this sequence so I can call this six times to print out all the countries in the sequence then this is what we get if you do it one more time then next is actually going to raise a stop iteration error this is what you're seeing here so if you want to go over all the elements in the sequence and then stopping at the end you can basically use a while loop for that so let me delete this and then and then we're going to say while through I'm going to try getting a country using next like so and if we catch a stop iteration error I'm going to break out of the while loop else I'm going to print the country so when I run this this simply prints out all the countries in the sequence until we reach the last one which is Greece now of course this isn't normally how you would do it in Python I just want to show you how it works with the stop iteration Arrow catching and so on there's a much shorter way to do it which is using a for Loop and that actually internally more or less does this and then I don't even need to create an iterator the for Loop already of this for me and this gives obviously exactly the same result so the while loop I just showed you before don't use that simply use a for Loop like this you may have heard two terms being used in the python World related to iterators which is iterators and intervals and those are actually not the same thing an iterable is an object that can provide you with an iterator Amazon iterator is an interval that also has a next Dunder method so they are related but they're not the same thing now sequences like lists tuples dictionaries Etc are all examples of iterables and they allow you to get an iterator from them which is what I've just shown you in this example initially I thought the way that iterables and iterators were organized in Python a bit strange I mean why would an iterator have an iter method like the country iterator for example it itself has an iterator method and that's called by the enter function in turn as what it actually does in this case is it returns exactly the same iterator so for example what I can do is I can print next of the country iterator so that's going to print Germany right but then if I create an better copy which is iterator of country iterator and then I print the next one from this copy you'll see that this is going to print France because it's actually the same iterator that's what you see here Germany and France so I could just as well have called here next contributor it would have been the same thing and that's what happens when you call an itter function on an iterator but if you call iter on this sequence which is the iterable the behavior is different so for example I can print here next using this iterator but now I'm creating a new iterator from the sequence and that's actually going to be another separate iterator so now this is going to print Germany twice though I don't find this completely logical the main thing you can do with this is basically restart an iterator or have the possibility to iterate a sequence at the same time in various places in the sequence so in that sense it's kind of useful but it's good to know the difference between iterables which allow you to restart an iterator and iterators which simply just give back themselves when you call iter on them okay a couple of other things you can do with iterates that are actually pretty cool one is that you can call Guitar The Hitter function with a function and a sentinel value and what that does is it going to call the function until the function Returns the Sentinel value and that's very useful for example if you want to go through let's say if a text file and you want to read lines of text until you read the end of the text file so what I have here is I have a file called countries.txt which contains the same countries as I have in my main file but then what I can do is let me replace this I can simply use a for Loop and then I can use width to open the file countries dot I need to type correctly.txt like so and then for line in and now I can use actually an iterator F Dot readline file.readline so I'm going to use the readline function until it reaches the Sentinel value of the empty string which is the end of the file and now I can simply print each of these lines in order to print it nicely I can tell the print statement to end it with the empty string and not with a new line and then this is what we get it simply prints all of the countries like we had before so this is useful if you want to call function a number of times in the sequence until you read some sort of value another nice thing that you can do with iterate is that they provide you a mechanism of abstraction so here I have an example that shows you how this works I have simple class called line item that's a data class it's Frozen so it's basically read only it has price and a quantity and a total price method that simply multiplies the two now I've used this example before in other videos so you may recognize it but it's just to keep it simple there's also a print totals function that basically takes a number of items so here you see I'm using the type iterable so that's basically anything that's iterable and that contains line items and that's going to use a for Loop for item and items and it's going to print the total price and then in my main function I have my line items list which contains a couple of line items it doesn't really matter what the values are and then I'm calling print total when I run this we simply get a number of totals printed here on the screen but what's interesting is because print totals now relies on the type being an iterable I can basically replace this by any type of sequence so instead of a list I could also decide to use a tuple like so and when I run this now obviously I'm going to get exactly the same result because I'm simply iterating through the sequence but print totals doesn't care about the specific type of sequence that items is and that means we've now introduced a level of abstraction the only thing that print totals needs to know is that it's some iterable sequence and it just uses a for Loop to Traverse that sequence we don't really care what the data structure is exactly and that's really nice by the way it is important for the data class to be frozen in this example because some data structures like um like the set for example it relies on being able to find the elements quickly using a hashing function so when I run this this works without issues but if I remove the Frozen indicator here and then I try to run this again you see we're going to get an error that line item is unhashable if you use frozen this problem is solved what you can also do is add your own hash function to the line item and then that's also perfectly fine by the way if you're enjoying this video so far give it a like it helps others on YouTube find the content as well iterators are in principle something that you can build yourself using your own classes with itter and next dollar methods so here you see an example of that's really simple we have a class called infinite number iterator and that has an interdominal method and it has a next donor method it has only one instant variable the number and what it does for next it simply increases the number by one and then returns that number so that just goes through the numbers and it does it infinitely so this also shows that iterators don't have to be finite they can in principle be infinite if you want to have that kind of behavior what's interesting here is that the it's a Thunder method simply Returns the iterator like what happens if you get an iterator from a list and then ask that iterator to provide an iterator is that confusing it's probably getting pretty confusing sorry about that but what's nice is that since I'm using python 3.11 I can use the self type here so another useful case of the self-type here I have a variety of this which I'm simply calling a number iterator and that not only has a number like this one but it also has a maximum and then well it is the same but in the next dollar method I check whether the number is above the maximum if it is I'm going to raise a stop iteration error otherwise I simply call number plus one and return self DOT number and then in my main function here I'm simply calling number on the number iterator and printing that number and of course we see it does that until it reaches the maximum that we indicated here as a parameter so this is in the core how an iterator works on the hood and you can build your own iterators this way and then it integrates nicely with things like for Loops Etc in Python if you want to take iterators to the next level you should take a look at the editor tools package in Python it's included in the standard library and it has some really cool features I'm going to show you a couple of them now basically what it's a tools does is that it provides a sort of algebra of iterators that you can combine in various ways in that sense it builds on functional programming principles like higher order functions I did a video about functional programming in Python a while ago if you want to learn more about that check out this video at the top now in order to get started with get the talks the only thing we need to do is simply import it at the top and we're going to show you a few examples of what you can do with this so This ranges from very basic things to pretty powerful things and you can combine these iterators and all kinds of different ways in order to result in complex behavior and where iterators really shine is by replacing loops with combinations of iterators for example let's say you want to print out totals and you have a list of prices and a list of quantities well you could write a for Loop to do that sort of naively but you can also use the star map function provided with a multiplication function and a tuple of iterables and that's also going to work I'll show you an example of that in a minute but let's start with something a bit more simple I'm going to start with a simple example of using an iterator to do counting so I will use a for Loop and then I'll use it tools dot count and I can provide it with an integer which is a starting value so let's say that's 10. print I and then if I equals 15 I'm going to break out of the for Loop when I run this then this is what we get very basic another thing we can do is repeating and repeat gets an object and that's going to return that object a number of times so for example I can provide it with an object of type integer let's say 10. and I wanted to return that integer four times so this is no longer needed of course and let's add a column so when I run this we get 10 printed four times so simply repeating another thing you can do is accumulate and accumulate you can provide it with a range let's say range from 1 until 11 and then we run this and you see it's going to accumulate these numbers 1 plus 2 plus 3 plus 4 etc etc until 11. and there's way more functions for example let's say we have a list of items a B enhance C very simple what I can do now is use itatools.permutations to get all the possible permutations of these three items so I'm using permutations function and I'm providing it with the items and I want all the permutations of the three items and then I'm using a for loop again to print the permutation when I run this then this is what we get all the possible permutations I can also Supply an extra number here so if I for example Supply number two then these are all the permutations of length two so when I run this then this is what we get as you see it's ordered so a b is considered a different permutation than b a if the order doesn't matter then you don't use permutations for each boost combinations and now when I run this again you see that we get way less we get a B AC and BC and as a variety you also have combinations with replacement and when I run this then this is what you get combinations with replacement work slightly differently because it also allows for example for two A's or two B's or two C's that's by the way simpler way to print this you can also simply print a list made from the iterator like so so I can remove this for Loop and then when I run this it's simply going to print the list of all the elements that the iterator produced other things you can do let's say I have a second list more items and now I can use the chain function to combine them into a single iterator so I can print a list made from it's a two-stop chain and then I'm going to provide it with items and more items and now it's going to print all these elements in the chain and then of course you can combine things for example I could call Anita tools dot combinations on the chain of items and more items like so and then when I print this then I'm going to get all the combinations except I need to provide a number I think yeah so let's provide a number here once all combinations of two and then let's run this again and now I'm going to get all the combinations of two on the chain of items and more items and this is exactly the type of thing that it's a tools likes to do it's an algebra that allows to combine different iterators in different ways by chaining them or by calling other it's a tools functions repeatedly on the iterator you have to be careful though that your code remains readable sometimes if you combine these things in very complex ways then it's really hard to understand in that case the general design principles always apply of keeping things simple and readability is obviously very important so in that case it might make sense to split things up a bit more use sensible variable names or even create a function that groups several ETA tools functions into something new and that gives that function a logical name so a few more things I want to show you so first there is filter false which is also quite useful so what this does it filters out the values for which some predicates so if I do print of a list it's a tools filter false and I'm going to provide it a Lambda function where I'm simply going to check all the elements divisible by two and that column shouldn't be there and then let's say I provide it with a range so that's a sequence of numbers and when I print this you see that it prints only the odd numbers so it's going to filter out the things for which this lamba function returns false and then it's going to put that in this case in the iterator and then put it in list and then our princess this also take while which gets elements from the list until the function is true so in this case if I run this well the first element is zero so it only takes the first element if I turn this into the opposite not equal zero then it gets the empty list because the first element already doesn't agree to that another thing I want to show you is star map which is also a very useful function so in this case I mentioned the example of having an iterable of tuples so that's what you see here and star map applies the function on each element in the Tuple so in this case it takes an X and Y and it's going to multiply the values so what it will do is it goes through this interval takes each Tuple and then applies the function on it so we're going to get 2 times 6 8 times 4 and 5 times 3 when I run this so that's what you see here so if the tools can be pretty powerful you can create pretty complex iterative Behavior by combining these functions in different ways and what's nice is that this is also pretty fast and memory efficient so you don't have to worry about performance too much when you're using these editor tools functions so question for you are you already using these it's a tools functions and what are some of the more complex and uncommon ways in which you use them I'll collect them and then if there are enough examples I might do a follow-up video where I dive into some of those non-conventional uses of itals functions this was a deeper dive into Python's iterators and it's a tools package if you like this you might also like the deeper die if I did into the date time package recently I've put that video right here thanks for watching and see you next week

Channel: ArjanCodes

Views: 47,697

Keywords: itertools python, python itertools, python iterators, itertools tutorial, iterator, itertools library, itertools package, iterators in python, python iterable, python iterator, python iterators and generators, python iterator vs iterable, iterator vs iterable, python iteration, python itertools tutorial, python itertools examples, python itertools install, python itertools repeat, python iterators vs iterables, itertools python install, itertools python github

Id: aumxFs2DO5o

Length: 21min 1sec (1261 seconds)

Published: Fri Jan 13 2023