Advanced Generator Patterns in Go: Test Data and Streams

Introduction#

The introduction to the Generator pattern covered the mechanics: a function returns a receive-only channel, a goroutine sends values, the caller ranges over them. This article is the next step — two applications where generators earn their keep: generating test data for a web service, and composing generators into a streaming pipeline.

Example 1: Test Data for an E-commerce API#

Stress-testing an API needs volume and variety: many users, many orders, varying numbers of products per order. Two generators compose to produce exactly that — one emits a product catalog, the other drains it into orders:

type Product struct {
    ID    int
    Name  string
    Price float64
}

type Order struct {
    ID       int
    UserID   int
    Products []Product
    Total    float64
}

func productGenerator(count int) <-chan Product {
    out := make(chan Product)
    go func() {
        defer close(out)
        for i := 0; i < count; i++ {
            out <- Product{
                ID:    i + 1,
                Name:  fmt.Sprintf("Product-%d", i+1),
                Price: 10.0 + float64(i),
            }
        }
    }()
    return out
}

func orderGenerator(userCount, ordersPerUser int, products <-chan Product) <-chan Order {
    out := make(chan Order)
    go func() {
        defer close(out)
        rng := rand.New(rand.NewSource(42)) // reproducible test data
        var orderID int
        for userID := 1; userID <= userCount; userID++ {
            for i := 0; i < ordersPerUser; i++ {
                orderID++
                var orderProducts []Product
                var total float64
                for j := 0; j < rng.Intn(5)+1; j++ {
                    product, ok := <-products
                    if !ok {
                        break // catalog exhausted; ship the order as-is
                    }
                    orderProducts = append(orderProducts, product)
                    total += product.Price
                }
                out <- Order{
                    ID:       orderID,
                    UserID:   userID,
                    Products: orderProducts,
                    Total:    total,
                }
            }
        }
    }()
    return out
}

func main() {
    products := productGenerator(1000)
    orders := orderGenerator(100, 5, products)

    // Simulate sending orders to an API
    for order := range orders {
        fmt.Printf("Sending order %d for user %d with total $%.2f\n", order.ID, order.UserID, order.Total)
    }
}

Two details matter here:

• Seeded randomness. rand.New(rand.NewSource(42)) makes the generated dataset reproducible — the same run twice produces the same orders, which is what you want when a failing test needs to fail twice. Since Go 1.20 the global rand is auto-seeded, which is fine for demos and wrong for test fixtures.
• The ok check. The order generator can draw up to 2,500 products from a catalog of 1,000. Receiving from a closed channel returns the zero value immediately, forever — without the ok idiom you’d silently ship orders full of Product{}. With it, orders just come out shorter once the catalog is drained.

Example 2: Composing Generators into a Pipeline#

Generators chain naturally: the output channel of one stage is the input of the next. Here a mock stream simulates a slow data source, and a processing stage transforms each item:

type DataItem struct {
    ID   int
    Data string
}

// mockDataStream simulates a data source (file, queue, network stream)
func mockDataStream(count int) <-chan DataItem {
    out := make(chan DataItem)
    go func() {
        defer close(out)
        for i := 0; i < count; i++ {
            time.Sleep(100 * time.Millisecond) // simulate source latency
            out <- DataItem{ID: i + 1, Data: fmt.Sprintf("Data-%d", i+1)}
        }
    }()
    return out
}

// process transforms each item as it arrives
func process(stream <-chan DataItem) <-chan string {
    out := make(chan string)
    go func() {
        defer close(out)
        for item := range stream {
            out <- fmt.Sprintf("Processed: %s (ID: %d)", item.Data, item.ID)
        }
    }()
    return out
}

func main() {
    stream := mockDataStream(10)
    for result := range process(stream) {
        fmt.Println(result)
    }
}

Each stage is a separate generator with its own goroutine, so stages run concurrently and stay independently testable: swap mockDataStream for a real source in production, or swap process for a recording stub in tests. This is the pipeline pattern in miniature — and when the processing stage becomes the bottleneck, you fan it out with a worker pool.

Channels or iter.Seq?#

Both examples above are synchronous at heart — nothing except the artificial Sleep happens concurrently. Written with iter.Seq, they’d need no goroutine, no channel, and no close discipline at all. The honest reason to keep channels here is the shape of the real thing: once the source is a network stream, a message queue, or a pipeline with a worker pool in the middle, values genuinely arrive concurrently, and channels are the right primitive. The comparison in the introduction has the full decision criteria.

Best Practices and Pitfalls#

Best practices:

1. Seed random generators when test data must be reproducible
2. Use buffered channels when a fast producer feeds a slow consumer
3. Accept a context.Context in generators that run long or perform I/O, so callers can cancel
4. Keep stages small and composable — one transformation per generator

Pitfalls:

1. Receiving from a closed channel without the ok idiom — you get zero values, silently, forever
2. Forgetting defer close(out), leaking the consumer’s range loop into an infinite block
3. Generating more data than the test actually exercises
4. Mock streams so clean they hide the failure modes of the real source

Summary#

Generators stop being a toy when they compose: a catalog generator feeding an order generator, a mock stream feeding a processing stage. That composability is what makes them useful for test data and streaming pipelines — and it’s the same shape as the pipeline and worker pool patterns further along in this series.

If you want to experiment with the code examples, you can find them on my GitHub repository.