#21 Golang - Concurrency: Pipeline Pattern

[Music] the pipeline pattern is a powerful design used in concurrent programming particularly in go with its go routines and channels it allows you to set up a series of processing stages where the output of one stage is the input to the next this pattern is beneficial for processing data streams especially when the processing can be broken down into distinct independent steps in this episode we are going to build an image processing pipeline this pipeline will have four stages reading the file reducing the size of the image converting the image to grayscale and finally writing the output to a file these arrows between the stages represent the channels connecting Gore routines let's see how this is going to work the dotted lines here show the the timeline these blue boxes represent part of the stage G routine we first read the first image then the data is passed to the next stage to compress the image simultaneously the second image is read in another goroutine similarly in The Next Step the third stage is executed for the first image compression starts for the second image and the third image is red this is how the pipeline goes on the communication between the stages happens over channels that pass on the data to the next stage without further Ado let's jump to the coding part here we have a bunch of images we will process these images in our pipeline we have a folder called output in which we will store the processed images here we have a package called image processing this package contains the helper functions we will use to build the pipeline let's take a look this function reads the image file and returns an image object as the name suggests this writes the image object to a file this function converts the image object to grayscale similarly this one shrinks the image to size 500 by 500 pixels let's go to main.go here is the list of image file paths we pass this list of file paths to the function load image which reads the files and Returns the first channel next we pass this channel channel one to the function resize this function resizes the images and returns another channel channel 2 in the same way Channel 2 is passed to convert to grayscale Method this results in channel 3 and finally with the save image method we save the processed files these are the four stages of our pipeline these stages need data like the input path of the image the output file path and an image object that is used to perform operations like resizing in and converting to grayscale Let's create a structure for this data we will call it job this structure will have the input file path the image object and the output image file path the channels between the stages will act as a carrier for this structure now we will implement the load image function this function takes a slice of image paths and returns a channel that will output job structs now we create a new channel called out next we will create a go routine in the go routine for each file path P we create a job and add the job to the channel here we iterate over the provided paths next we create a job for each in the job the input path is set to p and the outpath is set to the output folder in the images folder we achieve this with strings replace function now we need to set the image in the job this image is read from the file path P job. image is assigned the image object read by the read image method of the image processing package add this job to the channel out once we are done with the loop we close the channel within the G routine we close this channel as we have no more input file paths and it helps in signaling completion we will discuss this later in the video at the end of the function let's return this channel this method returns a Channel of job structs in the pipeline this return channel is the input for the next stage let's implement the next stage resize this method accepts a Channel of job structs resizes each image using the resize function from the image processing package and sends the modified job to the next stage this function is similar to the previous function let's copy paste it we extract jobs from the input Channel and loop over these in the job we set the image to the resized image this method also returns a Channel of job struct in the pipeline this return channel is again set as the input for the next stage similarly let's implement the convert to grayscale Method we will fast forward it to save time now only the last stage of the pipeline is left let's implement it this stage saves the processed image in the output directory this method takes an input channel of job structs and returns an output channel of Boolean values the Boolean values indicate whether each image of save operation was successful let's copy paste from the previous function the out channel in this case is of type Boolean here call The Right image function from the image processing package passing the outpath and image fields of the current job struct this function is responsible for saving the image data to a file at the specified output path send a Boolean value of true to the out Channel signaling that the image has been successfully said in a more robust implementation this step might include error checking to determine if Right image succeeded or failed and then send true or false accordingly this last stage returns a bullan channel here this channel is assigned to the variable WR results here we initiate a Loop that continuously reads from the right results Channel we print if saving the image was successful or not the loop automatically terminates when the right results channel is closed indicating that all job structs have been processed by the save image function and it has sent out the success status of each save operation in the end let's understand the significance of closing the out channel in each stage for example the output channel of stage one is closed here this channel is the input to stage two the closure of the channel terminates the Range Loop in stage two similarly it propagates through all stages and once all images are processed the program stops let's run the program and all four Images have been processed successfully let's look at the output images this is the first input image of size 1536 by 1536 the output image is of size 500 by 500 and is grayscale as expected similarly other processed images show the expected result and that wraps up our journey through building and understanding the pipeline pattern in go we've seen how leveraging Go's concurrency model with goroutines and channels allows us to efficiently process data in stages making our code not only more efficient but also clean and maintainable remember the key to mastering goes concurrency lies in practice and experimentation so take what you've learned today and start building your pipelines until next time keep rocking

Channel: codeHeim

Views: 15,471

Keywords:

Id: 8Rn8yOQH62k

Length: 13min 0sec (780 seconds)

Published: Thu Feb 08 2024