Function Iterators might just change the way we write loops in Go

go 1.22 is the gift that keeps on giving with the biggest features being the enhanced routing capabilities Loop variables that are now iteration scoped and the new and improved math Rand slv2 package whilst these have been the biggest Headliners of this new release there's actually another that I think is going to have a big impact on some of the common coding patterns found in go this is the new experimental feature called range Funk and after playing around with it for a couple of weeks I think it has the potential to bring some big changes to the language but also a couple of risks this feature allows you to create iterators from functions which you can then use with the range keyword for example here I have an iterator function named backwards which is a function to iterate over a slice of integers in Reverse we can invoke this function similar to how we would typically perform iteration and go by using the range keyword if I run my code with the following command you can see that it prints the elements of the slice in Reverse pretty cool but it's not exactly the most exciting example instead what about a range iterator that we can use to convert values using a function such as converting a slice of ins into their squared values or perhaps just converting them into another type we might also see one that allows us to perform filtering based on a predicate such as whether or not an integer value is odd these are all higher order functions which have typically not made their way into the go programming language however with the new Range Funk feature they're certainly more likely to appear whilst I expect these sorts of iterators to be rather controversial I still see some rather practical uses when it comes to function iterators take for example this iterator function named parallel which was inspired by the rayon crate found in the rust programming language this function allows me to iterate over elements of a slice concurrently which is a rather common pattern when it comes to go before we look at this function iterator in detail let's take a look at how this pattern is normally implemented feel free to skip this section if you already know how to do this to start here I have some simple code this code generates a number of events iterates over them and calls the process function on each one if I go ahead and run this code you can see it takes a bit of time to complete this is because the process function can take up to a full second and we happen to be calling it five times a simple way to speed this up is to call the process functions concurrently we can achieve this pretty easily and go by using the go keyword which will run the process function inside of a go routine however when I go to run this code we'll see that our application exits before any of the events are processed this happens because we reach the end of the main function before any of the go routines are finished or in this case before they've even started to solve this we can use a concurrency primitive founding go called a weit group which lives inside of the sync package this type enables us to wait for a collection of go routines to finish which will allow us to block the main function until all of the events have been processed in order to use a weight group let's first import the sync package followed by creating a new weight group variable called WG then inside of our for Loop we'll call the weight group. add method passing in the value of one the ad method is used to increase the internal counter of the weight group which is used to determine when the weight group should stop blocking then to decrement this counter we can call the weight groups done method which we'll need to do once our process function has completed before we can do that however we need to wrap our process function inside of a closure which will be used for our go routine instead now we can call the done function of the weight group using a defer keyword which will cause it to run when the closure exits the last thing we need to do is call the weight method of the weight group after our for Loop this will cause our main function to block until all of the go routines have completed and called the done method now when I go and run this code you can see that all of our process functions start in parallel and our application waits for them to complete before it exits with that we've managed to implement parallel iteration in go as I mentioned before this is a pretty common pattern when it comes to the programming language however when you look at this code you'll notice that there's quite a bit of boiler plate which makes it harder to see the logic Beyond The Parallel iteration this can be solved by implementing it as a function iterator to do so let's first create a new function called parallel which will accept a slice of events this is the slice we wish to iterate over concurrently next we need to define the return value which is a function that accepts another function as its parameter this parameter function is called the yield function and represents the iterator of the loop with the arguments given to it being the loop variables the first value we're providing is an integer which which will represent the index in the slice that this current iteration is at the second argument is an event which will represent the value in the slice at the current iteration index these two values correlate with the same values you'd normally expect when ranging over a slice lastly the yield function needs to return a Boolean in order for it to conform to what the range Funk feature expects with that we have our function signature defined the next thing to do is to move on to our implementation first add in the following line to return a closure that EX steps our yield function then inside of this closure is where we want to add our logic let's start with adding in the sync. we group similar to our existing implementation then let's go ahead and set the number we want to wait for using the wg. add method rather than calling this method on each iteration of the loop we'll instead only call it once passing in the length of our input slice next we can iterate over the events calling a new go routine inside that does the same thing as our existing implementation performing a deferred call to weit group.one then underneath this we want to call the yield function which will perform the logic inside of the caller's for loop block lastly all we need to do is call the weight method of the weight group that wraps up our function implementation now we can go ahead and use it let's navigate back over to the main function and clear out the existing implementation now we can add in the following code to range over our parallel iterator with our events slice then inside of the for Loop body all we need to do is call the process function when we run this code we can see that everything works in par parallel as it did before with the boiler plate abstracted into the parallel function unfortunately however we have a bug if I change the code to iterate over more events and then add a break into our for Loop when we go to run this code the application will Panic this is because we're still calling the yield function inside of our iterator even after the break has been encountered the documentation states that when you receive a return value of false from your yield function then your range iterator should no longer continue in a single-threaded loop we could easily just check for this value and return early however because we use concurrency then this solution won't work for us instead we need to use another concurrency primitive found in go the context. context type which enables us to handle cancellation across multiple go routines let's go ahead and add this into our parallel iterator first things first we need to import the context package then we can create a new instance of the context type using the context dowi cancel method passing in a background context as the parent this method returns a context and a cancel Funk which is used to notify the context of when we want to cancel closing its internal done Channel let's first use it with the defer keyword so that it's cancelled when the closure exits then inside of our G routine closure we need to check to see if the context has been completed we can do this using a select statement with our first case being the context DOD method if this is the case then we'll return early otherwise we'll set a default case to go ahead and call our yield function as we were before now All That Remains is to check the return value of the yield function and in the case of a false we can cancel our context now when I go ahead and run this code you'll see that we no longer receive a panic and if I go ahead and remove the break statement the existing functionality works as it did before with that our function is now working safely and will handle a break without panicking however currently this iterator is limited to the event type only which means we can't use it with other slic types fortunately we can modify our iterator to work with any type by making it generic we head back on over to our function signature let's first Define a generic type called e which will conform to any type then we just need to make the following changes replacing any references to event with a reference to e instead with that we can now use this parallel iterator with any slice type which means we now have a reusable implementation for performing parallel iteration this gives a glimpse of what potential implementations of function iterators could look like and whilst I think this feature is going to be really powerful I also believe it's going to bring some powerful foot guns when looking at code that makes use of the parallel iterator it does a really good job of hiding the fact that it's running concurrently which means it's going to be easier to make mistakes such as dealing with race conditions or accessing shared State this highlights one of the potential risks with this feature which is the fact that it sometimes obfuscates what is actually happening especially in the case of concurrency as the language currently has no way to specify that a function is concurrent normally this isn't too much of an issue but when it's applied to the range function feature it looks like it's working in a single-threaded context now to be fair the feature is still in an experimental State and so some of this could potentially change with that being said we are starting to see it make its way into the standard library with new packages types and methods being made available one such type is the SE 2 type found inside of the iter package which we can actually use to tidy up the return type of our parallel iterator as well as this type the iter package also provides a really interesting function called iter dop which allows us to pull values from function iterators rather than being pushed to us instead for example here I have a method called zip which takes two function iterators and iterates through them using the pull method as it pulls values from each iterator at the same time it then Zips them up into a single data structure before then sending it to its own yield function we can then use this ZIP function by first passing in two function iterators and then ranging over it printing out the zipped values the new iter package found in the standard Library gives a clue at some of the potential of function iterator compositions and as well as the SD lib we can also expect to see new iterators coming from third party packages one such package is the loop package which I created myself this package provides a number of iterator functions based on patterns I've encountered in the past one of these functions is the batch iterator which takes both a slice of elements and a size allowing you to iterate over the slice in sized chunks so far the loop package has been pretty enjoyable to work on to really push what can be done with this experimental feature and I haven't been the only one kicking the proverbial tires with a number of other packages available on GitHub each trying their own different experiments whilst it's difficult to know whether the rang Funk feature is going to be a good or bad thing for the language I do think it'll bring some standardization to the way we iterate over data structures found in the standard Library which is currently done in a number of different ways either way I'd love to know your thoughts what do you think about this new feature and do you see it impacting the way you currently write go code before you leave to go tell me that I have some good news to share after a lot of requests I've finally started working on my first course which is going to be all about writing command line applications in go if that sounds like something of Interest then there's a link in the description below that you can use to register your email which will allow you to be notified when the course is available but also provide you with a discount code when it goes on sale personally I'm really excited to be building this course and I want to give a big thank you to everyone who supports me in the channel for making that happen otherwise a big thank you for watching and I'll see you on the next one

Channel: Dreams of Code

Views: 49,441

Keywords: golang, rangefunc, experimental, 1.22, iterators, map, filter, reduce

Id: PPn_rgdx220

Length: 11min 35sec (695 seconds)

Published: Mon Apr 15 2024