#18 Golang - Building an Efficient Worker Pool: Mastering Concurrency in Go

[Music] goroutines are a fundamental Concept in go providing a way to perform concurrent operations they are lightweight and more efficient than traditional threads in terms of memory usage and startup time are goroutines the superhero of concurrency or do they sometimes wear the villain's Cape no managing a large number of goru can be challenging this is where worker pools come in a worker pool is a pattern that controls the number of goroutines that are working on a task at any given time it helps in resource management by limiting the number of goroutines and reusing them for multiple tasks in this episode we'll learn how to implement a worker pool for goroutines in go without further Ado let's dive into the implementation I like to keep it in a separate file let's name it workpool Dogo let's keep this in the main package the task is the single most unit of work we will Define a task then a function to process the task next we Define the worker pool and functions to executed tasks in the pool now we will Define a task it is a simple structure let's say the data it has is an identifier ID next we Define a method process on the task struct we will simulate a timec consuming operation by sleeping for a couple of seconds let's simply print the task ID being processed now we Define a workerle struct it should have a field for the tasks next the number of con current workers these many concurrent goroutines will run at a time tasks Chan defines a channel to send tasks to workers and a wait group for synchronizing the completion of tasks this will be used to wait for the tasks to be completed we need a method that receives tasks from the task Channel and processes them let's define a worker method on the worker pool struct let's receive a task from the channel call the process method on the task to process it now we will signal the completion of a task using the worker group Next comes the run method this method initializes the channel sets the concurrency creates the go routines and sends tasks over the channel we will first initialize the tasks Channel with a capacity equal to the number of tasks then start the worker go routines number of workers are set to the concurrency variable add the number of tasks to the weight group counter send tasks to the tasks Chan next close the task Chan after sending all tasks to Signal no more tasks will be sent finally we will wait for all tasks to be processed using the weight method now let's go to the main function and use this worker pool we will initialize 20 tasks let's assign all of them an ID next create a worker pool with the tasks and set the concurrency level to five start the worker pool to process the tasks let's add a print at the end this prints a message once all tasks have been processed let's run the program as we can see it processes five tasks at a time this works as expected this example works for one type of task but what if there are different types of tasks how do we handle them let's convert task to an interface this interface has a method process suppose we have two types of tasks one to send an email and the other one does some image processing let's create an email task struct it has three attributes email ID subject and the message next we implement the process method let's change the print similarly we implement the image processing task generally it is an expensive operation so we add a bigger sleep to this task now our worker pool is ready to take two types of tasks let's go to the main function and make the necessary changes we have created a bunch of tasks here of both types let's run the program yes it works it is evident how five tasks run at a time and image processing takes more time worker pools are useful for controlling resource usage and improving the efficiency of concurrent task processing don't forget to like And subscribe for more goang related videos Happy coding

Channel: codeHeim

Views: 11,748

Keywords:

Id: ZWMiKQXmh9s

Length: 9min 27sec (567 seconds)

Published: Tue Jan 30 2024