Worker Pool Pattern in Go#

Introduction#

The Worker Pool pattern is a fundamental concurrency design in Go that efficiently manages a pool of worker goroutines to process tasks from a shared queue. This pattern excels at handling a large number of independent tasks concurrently while maintaining precise control over system resources and performance.

When to Use#

• Processing a large number of independent tasks that can be parallelized
• Limiting the number of concurrent operations to prevent resource exhaustion
• Balancing workload across multiple processors or cores
• Managing CPU-bound or I/O-bound tasks efficiently
• Handling batch processing operations with controlled parallelism

Why to Use#

• Controls resource utilization by maintaining a fixed number of workers
• Improves performance through efficient parallel processing
• Prevents system overload by limiting concurrent operations
• Enhances application scalability and throughput
• Maintains predictable resource usage patterns

How it Works#

The Worker Pool pattern consists of three essential components:

1. A pool of worker goroutines that process tasks concurrently
2. A job queue (input channel) that holds pending tasks
3. A results queue (output channel) that collects processed results

Workers continuously pull tasks from the job queue, process them independently, and send results to the results queue. This design ensures efficient task distribution and controlled concurrency.

Simple Example#

1
func worker(id int, jobs <-chan int, results chan<- int) {
2
    for job := range jobs {
3
        fmt.Printf("Worker %d processing job %d\n", id, job)
4
        time.Sleep(time.Second) // Simulating work
5
        results <- job * 2
6
    }
7
}
8

9
func main() {
10
    const numJobs = 5
11
    const numWorkers = 3
12

13
    jobs := make(chan int, numJobs)
14
    results := make(chan int, numJobs)
15

16
    // Start worker pool
17
    for w := 1; w <= numWorkers; w++ {
18
        go worker(w, jobs, results)
19
    }
20

21
    // Send jobs to the workers
22
    for j := 1; j <= numJobs; j++ {
23
        jobs <- j
24
    }
25
    close(jobs)
26

27
    // Collect and print results
28
    for a := 1; a <= numJobs; a++ {
29
        result := <-results
30
        fmt.Printf("Job result: %d\n", result)
31
    }
32
}

Real-World Example: Image Processor#

Let’s consider a scenario where we need to process multiple images concurrently:

1
type Job struct {
2
    ID       int
3
    ImageURL string
4
    Size     int
5
}
6

7
type Result struct {
8
    JobID     int
9
    ImageURL  string
10
    NewSize   int
11
    Error     error
12
    TimeSpent time.Duration
13
}
14

15
func imageProcessor(id int, jobs <-chan Job, results chan<- Result) {
16
    for job := range jobs {
17
        startTime := time.Now()
18

19
        fmt.Printf("Worker %d processing image %d from %s\n", id, job.ID, job.ImageURL)
20

21
        result := Result{
22
            JobID:    job.ID,
23
            ImageURL: job.ImageURL,
24
            NewSize:  job.Size,
25
        }
26

27
        // Simulate image processing with realistic steps
28
        err := processImage(job)
29
        if err != nil {
30
            result.Error = err
31
            results <- result
32
            continue
33
        }
34

35
        result.TimeSpent = time.Since(startTime)
36
        results <- result
37
    }
38
}
39

40
func processImage(job Job) error {
41
    // Simulate various image processing steps
42
    time.Sleep(time.Duration(rand.Intn(500)) * time.Millisecond)
43

44
    // Simulate potential errors
45
    if rand.Float32() < 0.1 {
46
        return fmt.Errorf("failed to process image %d: simulation error", job.ID)
47
    }
48

49
    return nil
50
}
51

52
func main() {
53
    numCPU := runtime.NumCPU()
54
    runtime.GOMAXPROCS(numCPU)
55
    numWorkers := numCPU * 2 // Use 2 workers per CPU core
56
    const numJobs = 10
57

58
    jobs := make(chan Job, numJobs)
59
    results := make(chan Result, numJobs)
60

61
    // Initialize worker pool
62
    for w := 1; w <= numWorkers; w++ {
63
        go imageProcessor(w, jobs, results)
64
    }
65

66
    // Send image processing jobs
67
    for j := 1; j <= numJobs; j++ {
68
        jobs <- Job{
69
            ID:       j,
70
            ImageURL: fmt.Sprintf("https://example.com/image%d.jpg", j),
71
            Size:     100 * j, // Varying sizes
72
        }
73
    }
74
    close(jobs)
75

76
    // Collect and handle results
77
    for a := 1; a <= numJobs; a++ {
78
        result := <-results
79
        if result.Error != nil {
80
            fmt.Printf("Error processing image %d: %v\n", result.JobID, result.Error)
81
        } else {
82
            fmt.Printf("Successfully processed image %d to size %dpx in %v\n",
83
                result.JobID, result.NewSize, result.TimeSpent)
84
        }
85
    }
86
}

Best Practices and Pitfalls#

Best Practices:

1. Always close the channel when generation is complete
2. Use buffered channels when appropriate to prevent blocking
3. Include monitoring and logging for production environments
4. Implement graceful shutdown mechanisms
5. Size your worker pool based on available system resources

Pitfalls:

1. Creating too many workers, leading to resource exhaustion
2. Not handling worker failures or panics
3. Forgetting to close channels properly
4. Missing timeout mechanisms for long-running tasks
5. Inefficient job distribution strategies

Summary#

The Worker Pool pattern is a powerful tool in Go’s concurrency toolkit, offering a balanced approach to parallel processing. By maintaining a fixed number of workers, it prevents resource exhaustion while maximizing throughput. The pattern is particularly valuable in real-world scenarios such as image processing, batch operations, and API request handling, where controlled concurrent processing is essential for optimal performance and resource utilization.

Disclaimer#

This article provides an introduction to the Worker Pool pattern in Go. While the pattern is powerful, it’s important to consider the specific needs of your application when implementing it. For production use, additional error handling and optimizations may be necessary.

Testing Worker Pools#

If you test a worker pool, focus on the contract: jobs go in, results come out. The number of goroutines is an implementation detail. I wrote about that in TDD Isn’t About Bugs — It’s Your Permission to Refactor.

Series Navigation#

This article is part of the Go Patterns series:

• Previous: Mastering the Generator Pattern in Go
• Next: Go Pipeline Pattern: Turning Streams into Useful Data
• Series: Go Patterns
• Pillar: Go Concurrency Patterns

For more advanced concurrency patterns and best practices in Go, stay tuned for future articles! 🚀

If you want to experiment with the code examples, you can find them on my GitHub repository.