Corentin GS's Blog

Tokyo in the Browser: City, Cognition, and Japanese Visual Culture

Wed, 03 Jun 2026 00:00:00 GMT

Tokyo in the Browser: City, Cognition, and Japanese Visual Culture

Walk through Shinjuku Station at rush hour and the first impression is almost impossible to separate from the information surrounding you. Platform numbers, exit codes, train lines, arrows, warnings, shop signs, restaurant boards, campaign posters, ticket machines, and convenience-store shelves all compete for attention. And yet the system works. People scan, filter, compare, and move.

Open a Japanese portal website and you'll see the same logic at work. The page behaves like a city compressed into a browser: a field of routes, signals, services, promotions, warnings, and contextual cues.

In the previous essay, I argued that many Japanese websites are better understood as high-density context environments than as simple design failures. This article asks where that logic comes from. The answer lies outside the browser: in the streets, stations, shelves, magazines, and visual habits of Japan.

A Japanese Website Is Not a Poster

Western landing pages often behave like advertisements. One dominant message. One focal object. One call to action. The design assumes the user arrives with a single intent, and the page's job is to guide that intent toward conversion.

Many Japanese portals, service pages, and retail sites behave differently. They behave like maps, directories, counters, or flyers. Their job is to orient, not to persuade. They assume users arrive with questions, comparisons, and multiple possible next steps.

Poster-like page	Map-like page
One dominant message	Many visible destinations
Linear persuasion	Nonlinear navigation
One primary CTA	Multiple task paths
Low visual density	Managed visual density
Brand mood first	Context and service information first

If you judge a map by poster rules, it will look like clutter. But a map follows different rules than a poster.

Japanese web density is a different design logic, shaped by different environmental training.

The City as Interface

Tokyo functions like an interface. The city guides movement through layered navigation systems: color-coded train lines, numbered exits, floor maps, directional arrows, multilingual warnings, and retail prompts stacked at every decision point.

Take Shinjuku Station. It is the world's busiest transport hub, handling over three million passengers daily. The station compresses dozens of train lines, hundreds of exits, shopping malls, offices, and restaurants into a single navigable structure. The density is not random. It is organized through redundancy: the same information appears in multiple formats (signs, maps, digital displays, audio announcements) so that travelers with different needs, languages, and attention states can all find their way.

Japanese urban density is often vertical, layered, and sign-rich. A domestic portal page can feel less strange when read through this spatial logic.

Or walk through Akihabara at night. Vertical signage stacks ten stories high. Storefronts display product categories, price ranges, and promotional campaigns on every surface. The street is not chaotic; it is a browsing environment. People scan, compare, and filter before entering any single store.

Dense does not necessarily mean random. Japanese stations compress routes, exits, platforms, warnings, and services into navigable systems.

The city teaches people to read dense visual fields. Not because they are naturally better at it, but because the environment demands it. Daily commuting, shopping, and wayfinding in Tokyo provide thousands of hours of practice in extracting meaning from layered, overlapping, context-rich surfaces.

Tokyo did not cause Japanese web design — that would be too simple. Tokyo is one visible example of a broader visual environment in which dense, navigable displays of information are common. Japanese web interfaces may reflect this environment because the users who navigate them have been trained by it.

From Street Signs to Screen Modules

The physical-to-digital analogy has limits, but it is useful. Consider how specific urban environments map to interface patterns:

Physical environment	Digital equivalent	Shared logic
Train-station wayfinding	Dense navigation menus	Many routes visible at once
Department-store floor guide	Category mega-menu	Orientation through labels and hierarchy
Konbini shelf	Product grid	Dense comparison, price cues, promotions
Chirashi flyer	Campaign modules	Seasonal offers, urgency, abundance
Restaurant street sign	Promotional banner	Immediate decision cues
Public notice board	Announcements / updates	Completeness, procedural clarity
Ticket machine	Multi-step service UI	Guided task completion
Manga page	Modular visual sequencing	Panel-based scanning, nonlinear reading
Don Quijote shelf	Dense retail discovery	Abundance as browsing mode

A Japanese e-commerce page resembles a digital Don Quijote shelf: hundreds of products, price comparisons, campaign tags, and category cues compressed into a single scannable surface. The shopper is not being guided toward a single purchase. They are being invited to browse, compare, and discover.

A municipal website functions like a public notice board: every procedure, every document, every contact method visible at once, because the institution cannot assume which specific question the citizen arrived with.

These interfaces are digital versions of familiar physical and media environments.

Attention Is Culturally Trained

The science behind this intuition comes from cultural cognition research. In a series of studies, Richard Nisbett and Takahiko Masuda explored differences in visual attention between East Asian and Western participants. Their work contrasts two modes of attention:

Analytic attention: Focus on discrete objects, categories, rules, and focal entities. The background is secondary.
Contextual attention: Focus on fields, relationships, context, and change. The background carries meaning.

In Masuda and Nisbett's 2001 study, Japanese and American participants watched animated underwater scenes and described what they saw. Japanese participants made significantly more statements about background elements and relationships between objects. American participants tended to describe focal objects first. When researchers later tested recognition, Japanese participants' ability to identify focal objects was more disrupted when the original background was changed.

What does this mean for web design? The study did not involve websites, so it does not show that Japanese users prefer clutter. But it does suggest that attention to context and background has been observed experimentally, which makes it plausible that surrounding interface cues carry more meaning than Western minimalist designers assume.

What looks like background noise to one audience may function as contextual information to another.

The Physical Environment Trains the Eye

If attention patterns differ, where do they come from? One answer comes from Miyamoto, Nisbett, and Masuda's 2006 study, which compared Japanese and American physical environments. They found that Japanese scenes tended to contain more elements, more ambiguity, and more overlapping visual relationships than American scenes.

The built environment is not just scenery — it trains perception. If everyday scenes contain more signs, objects, boundaries, and ambiguous relationships, users become practiced at extracting meaning from dense visual fields.

The commuter who navigates Shinjuku Station every morning learns to scan layered signage, filter irrelevant signals, and locate meaningful cues within dense fields. The shopper who browses konbini shelves learns to compare products across multiple dimensions (price, promotion, category, novelty) at a glance. The reader who consumes manga learns to navigate nonlinear visual sequences using hierarchy, rhythm, and framing.

These are skills people learn through repeated exposure to specific environmental conditions. And they may transfer to digital interfaces that present similar information structures.

This shifts the question from "Why do Japanese users like clutter?" to "What environments trained these scanning habits?" — a hypothesis, not a proof.

Manga, Magazines, Flyers, and Dense Surfaces

The visual environment extends beyond urban space into media culture. Japanese print and commercial media have long traditions of dense information display:

Manga pages use panel rhythm, speech bubbles, motion lines, and layered reading paths. The reader does not move linearly through the page but scans across multiple visual entry points, guided by hierarchy and spatial relationships.

Manga and magazine layouts train readers to move through dense visual fields using hierarchy, rhythm, framing, and typographic signals.

Magazine covers often contain dozens of headlines, teasers, and callouts competing for attention. The design assumes readers will scan, not read sequentially.

Chirashi flyers from supermarkets and department stores compress prices, product images, campaign periods, and promotional bursts into single sheets designed for rapid scanning and comparison.

The chirashi flyer is a useful analogue for campaign-heavy websites: dense, comparative, seasonal, price-oriented, and designed for scanning.

Train advertisements must capture attention in seconds, often using dense visual fields with multiple focal points and contextual cues.

Product packaging in Japanese retail frequently displays ingredients, origin, preparation methods, and promotional information simultaneously, designed for the informed comparison shopper.

These surfaces are not meant to be read linearly. They are meant to be scanned, compared, and entered from multiple points. They belong to the same visual environment as the dense web interfaces they resemble.

These are analogies, not causes. Manga does not determine portal layouts, and magazine covers do not dictate web design. They are parallel visual literacies that coexist within the same cultural environment.

What This Explains About Japanese Websites

What does this mean for actual Japanese web interfaces?

A Western minimalist site often assumes: One message, one focal object, one CTA. The user arrives with an intent, and the page's job is to focus that intent and remove distraction.

A Japanese dense site often behaves more like: A map of options, conditions, campaigns, rankings, and service paths. The user arrives with a context, and the page's job is to make that context visible and navigable.

This explains several observable patterns in Japanese web design:

Visible breadth over guided narrative: Category pages show many products. Portal pages show many services. The assumption is that users want to see options before choosing.
Contextual cues over isolated hero messaging: Price comparisons, campaign tags, trust badges, and support links appear alongside primary content, not hidden behind clicks.
Category density over progressive disclosure: Information that Western designers might tuck into dropdowns or secondary pages remains visible, because the cost of clicking may exceed the cost of scanning.
Trust through completeness: Municipal pages, banking sites, and service portals surface every document, procedure, and contact method, because incompleteness creates anxiety.
Navigability through labels and clusters: Dense pages rely on clear typographic hierarchy, color coding, and spatial grouping to make scanning possible.
Multiple user intents on one screen: A single page may serve informational, transactional, comparative, and support needs simultaneously.
Service-counter logic rather than conversion-funnel logic: The interface behaves like a staffed counter where the user can ask multiple questions, not like a billboard that demands a single action.

These patterns are rational responses to different assumptions about user behavior, shaped by different environmental training.

But Density Is Not Automatically Good UX

Cultural plausibility explains tolerance, not efficiency. Familiarity with dense environments may increase tolerance for dense interfaces, but tolerance is not the same as efficiency.

Baughan et al.'s 2021 CHI study tested this directly. Eighty-four U.S. participants and sixty-five Japanese participants searched for information on website screenshots of varying complexity. The researchers hypothesized that Japanese participants might be faster at finding contextual information due to different visual attention patterns.

The results did not support this. Japanese participants took longer overall, and the performance gap increased as website complexity increased. The authors suggest that Japanese participants may be taking in the whole page at once before focusing on the primary task — a broader initial scan, a delay while they orient themselves to the page.

The study recommends: highlight related content areas, use consistent layouts, and reduce unnecessary complexity for both groups.

Japanese users may take in more of the field before acting, but this can slow task completion. Dense interfaces can be culturally familiar and still impose usability costs. Familiarity may increase tolerance for density, but it does not eliminate cognitive load.

Good Density Needs Hierarchy

If density is not automatically good, what makes it usable? The answer is hierarchy.

Tokyo is dense, but it is not random. Shinjuku Station works because redundancy is organized: colors code lines, numbers code exits, maps show spatial relationships, and signs appear at decision points. The density is managed through systems.

Japanese web interfaces that succeed do the same. They use:

Clear typographic hierarchy: Headlines, subheads, and body text create scanning paths.
Color coding: Campaign tags, category labels, and status indicators use consistent color systems.
Spatial grouping: Related information clusters together, separated by whitespace or borders.
Redundant cues: Important information appears in multiple formats (icon, text, color) for different scanning modes.
Consistent layouts: Users learn where to look for specific information types.

The distinction that matters is between useful density and costly density:

Useful density	Costly density
Supports comparison	Hides hierarchy
Shows routes	Forces exhaustive search
Builds reassurance	Creates anxiety
Uses stable clusters	Accumulates randomly
Matches multi-intent pages	Overloads single-task flows

Good Japanese web design should organize density, not just preserve it.

From City to Trust

Japanese web density extends beyond the screen into a wider visual environment: stations, streets, shelves, flyers, magazines, public notices, and commercial signs. That does not make every dense interface good. The evidence suggests that complexity can still slow users down. But it changes the question.

Instead of asking why Japanese websites look like clutter, we can ask why they resemble navigable environments. Instead of assuming Western minimalism is the universal standard, we can recognize that different visual environments produce different interface logics.

The city explains part of the pattern. Repeated exposure to dense physical and media environments shapes scanning habits, expectations, and tolerance for contextual information. But there is another layer. If density feels familiar because it echoes the city, it can also feel reassuring because it answers questions before the user has to ask them.

That trust is the subject of the next essay.

Sources and Further Reading

Baughan, S., et al. "Do Cross-Cultural Differences in Visual Attention Patterns Affect Search Efficiency on Websites?" Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 2021. PDF
Masuda, T., & Nisbett, R. E. "Attending Holistically Versus Analytically: Comparing the Context Sensitivity of Japanese and Americans." Journal of Personality and Social Psychology, 81(5), 922–934, 2001.
Miyamoto, Y., Nisbett, R. E., & Masuda, T. "Culture and the Physical Environment: Holistic Versus Analytic Perceptual Affordances." Psychological Science, 17(2), 113–119, 2006.
Nisbett, R. E., & Masuda, T. "Culture and Point of View." Proceedings of the National Academy of Sciences, 100(19), 11163–11170, 2003.
Würtz, E. "Intercultural Communication on Web Sites: A Cross-Cultural Analysis of Web Sites from High-Context Cultures and Low-Context Cultures." Journal of Computer-Mediated Communication, 11(1), 2005.
Cyr, D., & Trevor-Smith, H. "Localization of Web Design: An Empirical Comparison of German, Japanese, and United States Web Site Characteristics." Journal of the American Society for Information Science and Technology, 55(13), 1199–1208, 2004.
W3C. "Requirements for Japanese Text Layout." JLREQ

Why Japanese Websites Look So Overloaded, and What the Density Is Doing

Sun, 31 May 2026 00:00:00 GMT

Why Japanese Websites Look So Overloaded, and What the Density Is Doing

Open Yahoo! Japan after browsing a Western startup website and you feel the difference immediately. The Western page gives you a hero image, a sentence, and a button. The Japanese page gives you news, weather, shopping, finance, auctions, login, campaigns, rankings, emergency notices, and half a dozen ways to go somewhere else.

To a Western designer, it may look like clutter. But what if the page is not failing at minimalism? What if it is solving a different design problem?

Many Japanese domestic websites use high-density interfaces to show context, reassurance, comparison options, promotions, navigation, and trust cues all at once. To users trained by minimalist design norms, that density reads as overload. But the better UX question is not "why is this bloated?" It is "what job is this density doing?"

A Japanese portal homepage showing high-density UI: news, weather, campaigns, rankings, and navigation modules are all visible at once. To a Western minimalist eye, it may register as clutter — but the density is doing specific jobs.

"Bloated" Is the Wrong Word

Western critiques of Japanese web design often write it all off as bloat. But that word hides more than it reveals. Before we can ask why these pages look the way they do, we need a sharper vocabulary.

Term	What it means	Is it always bad?
Information density	Many facts, explanations, specs, labels, notices, comparisons visible at once	No
Visual density	Many modules, colors, banners, borders, type sizes, icons	Sometimes
Navigational density	Many menus, links, category paths, sidebars, breadcrumbs	Sometimes
Promotional density	Coupons, rankings, campaigns, limited-time banners, points	Depends on business model
Trust density	Security badges, company info, shipping, returns, support, official notices	Often useful
Technical bloat	Heavy JS, slow assets, unoptimized tracking, excessive DOM	Yes

Only technical bloat is inherently bad. The rest may be poorly executed, but it may also serve a function. A page with many trust badges is not the same as a page with unoptimized JavaScript. A page with visible campaign information is not the same as a page with broken visual hierarchy.

Separating density types lets us diagnose what each part of the page is doing, rather than collapsing everything into "clutter." Technical bloat is the only inherently harmful type.

When we call everything "clutter," we stop being able to diagnose it.

Western Minimalism Is a Design Lens, Not a Universal Standard

Western digital design, especially in SaaS, startups, and luxury, tends to value:

Whitespace
One primary call to action
Few choices
Emotional branding
Reduced copy
Progressive disclosure
Polished typography
Simple conversion funnels

This is not neutral. It is a design ideology shaped by specific cultural and commercial assumptions: that the user arrives with a single intent, that persuasion happens through restraint, that confidence is signaled through polish, and that the page's job is to remove friction toward one action.

Many Japanese domestic sites operate on different assumptions. They often value:

Visible categories
Multiple routes
Campaign visibility
Rankings and social proof
Detailed conditions
Support information
Comparison tools
Trust cues
Seasonal updates

This is not a failure of minimalism. It is a different answer to a different question. Where Western minimalism often asks "what can we remove?", Japanese information-rich design often asks "what context does the user need to feel confident?"

Western minimalist design (left) vs Japanese high-density design (right). One strips down to one action. The other shows context for many possible user intents.

The Japanese Homepage as Service Counter

A Japanese homepage often functions as a digital service counter. It must answer many user questions upfront:

What is new?
What campaign is active?
Where do I log in?
What are today's offers?
Where is support?
What are the rankings?
What is popular?
What happens after I click?
What are the conditions?
Can I compare?
Is this official?

In this model, the page is a hybrid of storefront, catalogue, flyer, customer-service desk, trust document, comparison table, and seasonal promotion board. The density serves a function.

A Japanese homepage annotated to show its service-counter zones. Each cluster of elements answers a different user question before it needs to be asked.

This does not mean every dense page is well-designed. A service counter can be chaotic. A flyer can be unreadable. The point is that the density is doing work, and the work is different from what a minimalist landing page is trying to do.

What Research Says About Culture, Attention, and Web Design

The science here is suggestive but not conclusive. It does not prove that Japanese users prefer dense websites. But it does show that attention is not culturally neutral.

Nisbett and Masuda's work on analytic versus holistic attention found that Western participants tend to focus more on focal objects, categories, and formal properties, while East Asian participants tend to attend more to background, relationships, change, and context. In their well-known underwater-scene experiments, Japanese participants reported more contextual and relational information and were more affected by background changes during recognition tasks than American participants.

If a web page is a visual field, then a culture more accustomed to context-sensitive perception may assign more meaning to surrounding cues: rankings, sidebars, labels, badges, notices, seasonal banners, and related options. What one user treats as irrelevant background, another may process as part of the meaningful field.

Context sensitivity does not mean users prefer clutter. Familiarity with dense environments is different from efficiency within them. The research raises the question without settling it.

The Empirical Web-Design Evidence

Cross-cultural web-design research has measured these differences at scale. Nordhoff, August, Oliveira, and Reinecke analyzed 80,901 website designs across 44 countries and found measurable cross-country differences in visual design metrics. Japan, along with China and South Korea, clustered toward higher visual complexity and text density patterns.

Baughan and colleagues studied 84 U.S. American and 65 Japanese participants searching for information on website screenshots of varying complexity. Japanese participants took longer overall, and the negative effect of complexity was stronger with increased website complexity — contrary to the hypothesis that they might perform better on dense sites.

Dense websites may be locally familiar and commercially meaningful, but familiarity is not the same as efficiency. The future of Japanese web design is structured density: context-rich design that does not sacrifice usability.

What Japanese Layout Standards Reveal

There is also a technical layer to this story. Japanese text composition involves unique requirements: mixed writing systems (kanji, hiragana, katakana, Latin characters, numerals), vertical and horizontal composition modes, complex line-breaking rules, ruby annotations, emphasis marks, and specific spacing conventions.

The W3C's Requirements for Japanese Text Layout documents these constraints in detail. Japanese script does not cause clutter, but it changes what compact information display can look like. A kanji label can convey dense semantic information in a compact spatial form in ways that do not map directly to English or French.

Localization is the adaptation of hierarchy, trust, pacing, navigation, density, and proof — not translation plus flags.

So Are Japanese Websites Actually Cluttered?

Sometimes, yes. Density can become noise. Visual hierarchy can collapse. Promotional banners can multiply until they obscure the content they are meant to promote. Technical bloat can slow a page to a crawl. These are real problems, and they deserve real critique.

But sometimes, no. What looks like clutter from one design ideology may be context, reassurance, comparison, and transparency from another. A Western landing page says "here is the action." A Japanese high-density page says "here is the situation."

The UX test is not whether the page looks busy. It is what happens if you remove an element. Does the user lose information they needed? Do they lose confidence? Do they lose a path they wanted to take? If the answer is yes, it was not clutter. It was structure they relied on.

A Better Question for Designers

Instead of asking why Japanese websites are overloaded, ask what that density is doing. Is it clutter, or is it context? Is it noise, or is it reassurance? Is it poor design, or is it a design solving a different trust problem?

Japanese websites look dense because many domestic sites are built as service environments, not statements. They must reassure, compare, announce, explain, promote, and guide the user at the same time.

Some of that density is clutter. Some of it is context. What matters is what work the overload is doing.

Article 2 will take that question outside the browser — into Tokyo's train stations, retail shelves, convenience stores, and urban signage — to see where Japanese visual density comes from.

Sources and Further Reading

Baughan, A., Oliveira, N., August, T., Yamashita, N., & Reinecke, K. (2021). "Do Cross-Cultural Differences in Visual Attention Patterns Affect Search Efficiency on Websites?" Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. PDF
Masuda, T., & Nisbett, R. E. (2001). "Attending Holistically Versus Analytically: Comparing the Context Sensitivity of Japanese and Americans." Journal of Personality and Social Psychology, 81(5), 922–934.
Nisbett, R. E., & Masuda, T. (2003). "Culture and Point of View." Proceedings of the National Academy of Sciences, 100(19), 11163–11170.
Nordhoff, M., August, T., Oliveira, N. A., & Reinecke, K. (2018). "A Case for Design Localization: Diversity of Website Aesthetics in 44 Countries." Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems.
W3C. (2020). "Requirements for Japanese Text Layout." JLREQ

TDD Isn't About Bugs — It's Your Permission to Refactor

Sun, 24 May 2026 00:00:00 GMT

TDD Isn't About Bugs — It's Your Permission to Refactor

The Moment It Clicked

Three years ago, I had a crisis of confidence with test-driven development.

I'd built a user registration system with 80% test coverage, all green, CI passing. Then I decided to refactor the email validation logic: just move it to a separate module. A tiny change.

Thirty minutes later, everything was broken.

My tests tested how the code worked, not what it did. The moment the structure changed, they failed. I spent more time fixing tests than doing the refactoring itself.

I'll show you what I was doing wrong. It might look familiar.

The Trap: Testing Structure

Here's the TypeScript code I had, a User class with inline validation:

class User {
	email: string;
	age: number;

	constructor(email: string, age: number) {
		if (!email.includes("@")) {
			throw new Error("Invalid email");
		}
		if (age < 18) {
			throw new Error("Must be at least 18 years old");
		}
		this.email = email;
		this.age = age;
	}
}

And the test:

describe("User", () => {
	it("should create a user", () => {
		const user = new User("alice@example.com", 25);
		expect(user.email).toBe("alice@example.com");
		expect(user.age).toBe(25);
	});
});

What am I testing? Structure. I'm reaching into the object and checking its properties directly.

Now watch what happens when I hide the internals:

class User {
	private email: string;
	private age: number;

	constructor(email: string, age: number) {
		if (!email.includes("@")) {
			throw new Error("Invalid email");
		}
		if (age < 18) {
			throw new Error("Must be at least 18 years old");
		}
		this.email = email;
		this.age = age;
	}

	get email(): string {
		return this.email;
	}

	get age(): number {
		return this.age;
	}
}

The test breaks. The behavior hasn't changed (the same user is created with the same rules), but I changed the structure.

I'm now rewriting tests for a refactoring that shouldn't require any test changes.

The Fix: Test Behavior

The solution: test what the code does, not how it's organized.

But first, I need to fix another problem. Throwing exceptions makes testing awkward: you need expect(() => fn()).toThrow(), which hides the error behind a callback. The fix is treating errors as first-class values instead of hidden control flow.

Here's the pattern that made this click, the Result type:

type Result<T, E> =
	| { readonly ok: true; readonly value: T }
	| { readonly ok: false; readonly error: E };

function ok<T>(value: T): Result<T, never> {
	return { ok: true, value };
}

function err<E>(error: E): Result<never, E> {
	return { ok: false, error };
}

Instead of throwing, functions return a Result that's either a success or a failure. Both paths are explicit. No try/catch. Errors are data you can inspect, test, and chain. (In production code, you'd use a library like neverthrow which adds map, flatMap, and match for composing results.)

With Result in place, I can rewrite the User class with a static factory method:

class UserCreationError {
	constructor(
		public readonly message: string,
		public readonly field: string
	) {}
}

class User {
	private constructor(
		private readonly _email: string,
		private readonly _age: number
	) {}

	static create(email: string, age: number): Result<User, UserCreationError> {
		if (!email.includes("@")) {
			return err(new UserCreationError("Invalid email format", "email"));
		}
		if (age < 18) {
			return err(new UserCreationError("Must be at least 18 years old", "age"));
		}
		return ok(new User(email, age));
	}

	get email(): string {
		return this._email;
	}

	get age(): number {
		return this._age;
	}
}

Now the tests verify behavior, the observable contract:

describe("User Registration", () => {
	it("should create a user with valid email and age", () => {
		const result = User.create("alice@example.com", 25);

		expect(result.ok).toBe(true);
		if (result.ok) {
			expect(result.value.email).toBe("alice@example.com");
			expect(result.value.age).toBe(25);
		}
	});

	it("should reject invalid emails", () => {
		const result = User.create("not-an-email", 25);

		expect(result.ok).toBe(false);
		if (!result.ok) {
			expect(result.error.field).toBe("email");
			expect(result.error.message).toMatch(/email/i);
		}
	});

	it("should reject users under 18", () => {
		const result = User.create("alice@example.com", 17);

		expect(result.ok).toBe(false);
		if (!result.ok) {
			expect(result.error.message).toMatch(/18/);
		}
	});
});

When I refactor the internals now (make _email private, change the data structure, extract a class), these tests keep passing. They test the contract: valid inputs succeed, invalid inputs fail with the right error. The internal structure is irrelevant.

This is the red-green-refactor cycle that defines test-driven development: write a failing test (red), make it pass with the simplest code (green), then improve the design while keeping tests green (refactor). The tests give you permission to refactor aggressively because they catch real problems, not structural changes.

Going Deeper: Value Objects

Once I started testing behavior, I noticed something. My validation logic was scattered. The same email check appeared in registerUser, changeEmail, and resetPassword. Three places, same rule, same potential for bugs.

The fix is a Value Object: a small, self-validating type that represents a single concept. If you have an Email instance, it's valid by construction:

class InvalidEmailError {
	readonly message = "Invalid email format";
}

class Email {
	private constructor(private readonly _value: string) {}

	static create(raw: string): Result<Email, InvalidEmailError> {
		const value = raw.trim().toLowerCase();
		if (!value.includes("@")) {
			return err(new InvalidEmailError());
		}
		return ok(new Email(value));
	}

	get value(): string {
		return this._value;
	}

	equals(other: Email): boolean {
		return this._value === other._value;
	}
}

Notice the normalization (trim().toLowerCase()) lives inside the Value Object. Every Email instance is guaranteed lowercase and trimmed. The validation rule exists in one place. If the rule changes, you change one class.

Similarly for Age:

class InvalidAgeError {
	readonly message = "Must be at least 18 years old";
}

class Age {
	private constructor(private readonly _value: number) {}

	static create(value: number): Result<Age, InvalidAgeError> {
		if (!Number.isInteger(value) || value < 18) {
			return err(new InvalidAgeError());
		}
		return ok(new Age(value));
	}

	get value(): number {
		return this._value;
	}
}

The User class now composes these Value Objects instead of accepting raw primitives:

class User {
	private constructor(
		private readonly _email: Email,
		private readonly _age: Age
	) {}

	static create(email: string, age: number): Result<User, UserCreationError> {
		const emailResult = Email.create(email);
		if (!emailResult.ok)
			return err(new UserCreationError(emailResult.error.message, "email"));

		const ageResult = Age.create(age);
		if (!ageResult.ok)
			return err(new UserCreationError(ageResult.error.message, "age"));

		return ok(new User(emailResult.value, ageResult.value));
	}

	get email(): string {
		return this._email.value;
	}

	get age(): number {
		return this._age.value;
	}
}

The same tests from before still pass. They test User.create() returning success or failure, which hasn't changed. The internal composition with Value Objects is invisible to the tests.

The Payment System

Last year, I refactored a production payment processing module: about 2,400 lines of conditional logic scattered across processPayment, validatePayment, and handleRefund. Validation rules were duplicated in all three functions.

The old tests looked like this:

it("processes payment", () => {
	const processor = new PaymentProcessor(mockGateway, mockDB);
	processor.validateAmount(100);
	processor.validateCurrency("USD");
	const result = processor.process({ amount: 100, currency: "USD" });
	expect(result.status).toBe("success");
	expect(mockGateway.charge).toHaveBeenCalledTimes(1);
});

Those tests called internal methods directly. When I extracted validation into a Payment Value Object, every test broke because validateAmount and validateCurrency no longer existed as separate methods.

I rewrote the tests to focus on the contract:

it("accepts valid payments", () => {
	const result = Payment.create({ amount: 100, currency: "USD" });
	expect(result.ok).toBe(true);
});

it("rejects negative amounts", () => {
	const result = Payment.create({ amount: -50, currency: "USD" });
	expect(result.ok).toBe(false);
});

Then I refactored aggressively. Extracted Currency as its own Value Object. Moved validation into the Payment factory. Deleted 800 lines of duplicated logic. Tests never broke once, because they tested what happened, not how.

The whole refactoring took a day. With the old tests, it would have taken a week: three days of refactoring, two days of fixing tests.

The same idea shows up in concurrency code too. When you refactor a pipeline or a worker pool, the contract stays the same while the internals change. The tests that survive are the ones that verify behavior, not goroutine counts.

The "TDD Is Slow" Myth

I used to think test-driven development slowed me down. Studies tell a more nuanced story: TDD has a modest upfront cost of 10–30% more development time, but reduces defects by 40–90% (Nagappan et al., Microsoft/IBM, 2008). George & Williams (2003) found TDD pairs took 16% more time but passed 18% more black-box tests.

The upfront cost is real. The payoff is also real. And it compounds: every safe refactoring makes the next one easier.

When I couldn't refactor safely, I coded defensively. I avoided changes. I left bad code in place because changing it was risky. Refactoring without tests is what's actually slow. It compounds with every month.

When TDD Doesn't Help

Test-driven development isn't always the right tool. Spikes and prototypes (code you'll throw away) don't need tests. Exploratory work where you don't know the shape of the solution yet. UI code where the behavior is visual, not logical. One-off scripts. Code with genuinely unstable requirements that change every sprint.

TDD pays off when you're building something you'll maintain and refactor over time. If the code won't survive the week, write the minimum.

Monday Morning

Take one function you wrote recently. Rewrite its tests to verify behavior (what it does) instead of implementation (how it's organized).

Watch what happens to your code.

Three years ago, I was afraid to refactor. Now I do it aggressively, because my tests give me permission. Test-driven development gave me a permission slip I didn't know I needed.

Next time, I'll show you how to build Value Objects that make invalid states impossible, and how testing them first makes the design emerge naturally.

If you found this useful, the next article in the series shows how to build Value Objects that make invalid states impossible. It comes out in a couple of weeks. You can subscribe via RSS or Atom to catch it when it lands.

— Corentin

Docker 29 Broke Traefik — Here's the Fix (and Why It Happened)

Mon, 04 May 2026 00:00:00 GMT

Docker 29 Broke Traefik — Here's the Fix (and Why It Happened)

The "Everything Is Down" Moment

You ran apt upgrade on a Sunday evening on your Linux server. Seemed harmless. Docker updated to v29.0.0. You rebooted because why not.

Now every service behind your Traefik reverse proxy returns a 404. Or a 502. Or just… nothing. Your monitoring is screaming. Your personal projects are unreachable. Your CI pipeline is stuck. And if you're running Coolify, Dokploy, or any platform that bundles Traefik, you can't even access the management UI to fix it.

You check the Traefik logs and see this, repeating every few seconds:

ERR Failed to retrieve information of the docker client and server host
  error="Error response from daemon: client version 1.24 is too old.
  Minimum supported API version is 1.44, please upgrade your client
  to a newer version" providerName=docker

Your first thought: "I didn't change anything in Traefik."

You're right. You didn't. Docker changed something under you.

Let me save you the debugging. Here's the fix.

The 30-Second Fix

This applies to Linux hosts running Docker Engine directly. If you're using Docker Desktop (macOS or Windows), you're not affected — updates are bundled automatically.

If your services are down right now and you need them back immediately, do this:

Edit (or create) /etc/docker/daemon.json:

sudo nano /etc/docker/daemon.json

Add this:

{
	"min-api-version": "1.24"
}

If you already have content in daemon.json (storage driver, log driver, etc.), merge the key into the existing object. Don't overwrite your whole config:

{
	"log-driver": "json-file",
	"log-opts": {
		"max-size": "10m",
		"max-file": "3"
	},
	"min-api-version": "1.24"
}

The most common mistake is a missing or trailing comma. JSON is strict about this.

Then restart Docker:

sudo systemctl restart docker

Important: Restarting Docker stops all running containers. If your containers have a restart policy (restart: unless-stopped or restart: always), they'll come back on their own. If not, you'll need to start them manually:

docker compose up -d

Docker Swarm: This fix works the same on Swarm nodes. Apply it to all managers and workers. Swarm services will reschedule automatically after the daemon restarts.

Rootless Docker? The config file lives at ~/.config/docker/daemon.json and you restart with systemctl --user restart docker.

That's it. Your services should come back within seconds. Traefik will reconnect to the Docker daemon, discover your containers, and routes will light up again.

Verifying it worked

Check Traefik's logs:

docker logs traefik --tail 20

You should no longer see the client version 1.24 is too old error. If you see Configuration received from provider docker, that's a positive confirmation (this message only appears at INFO log level, which may not be your default).

The strongest verification is the API version check below.

Confirm your Docker API versions:

docker version --format '{{.Server.MinAPIVersion}}'

This should now return 1.24 instead of 1.44.

Why this works: The min-api-version setting overrides Docker's built-in floor. It tells the daemon to accept connections from clients using API version 1.24 and above, restoring the backward compatibility that Docker 29 removed. This config change keeps Docker 29 fully operational while re-enabling communication with older API clients. You're still running Docker 29 with all its features — you've just told it to accept a wider range of client versions.

Is this safe? For now, yes. This setting only changes how Docker negotiates API versions with clients. The API version is a negotiation protocol that controls feature detection between client and server. It doesn't touch encryption or authentication. The risk is that older API clients might miss newer fields or features, but Traefik only needs to read container labels and inspect networks. It doesn't need API 1.44 for that.

The Real Fix: Update Traefik

The daemon.json workaround is a bandage. The actual fix is updating Traefik to v3.6.1 or later.

Traefik v3.6.1, released shortly after this incident, added automatic Docker API version negotiation. Instead of hardcoded v1.24, Traefik now queries the daemon for its supported version range and picks the highest mutually compatible version. This is how the Docker SDK is supposed to work.

If you manage Traefik directly (docker-compose, Dockerfile, etc.), change your image tag:

services:
  traefik:
    image: traefik:v3.6.1

Or pull the latest:

docker pull traefik:v3.6.1

Then restart:

docker compose up -d traefik

If you're on the older standalone docker-compose, the command is docker-compose up -d traefik.

Once Traefik is on 3.6.1+, you can remove the "min-api-version" workaround from daemon.json and restart Docker again. Everything will keep working because Traefik now negotiates the API version correctly on its own.

If you can't update Traefik directly

If you're running a platform that bundles Traefik (Coolify, Dokploy, Appwrite, CapRover), you might not control the Traefik image version. In that case:

Check if your platform has an update. Most of them shipped patches within days.
Use the daemon.json workaround until the platform updates its bundled Traefik.
Check the platform's GitHub issues. This broke so many deployments that every affected platform has a thread about it.

What Actually Happened

Let me explain the technical chain of events, because understanding it helps you prevent similar breakages.

Docker's API version floor

Docker Engine exposes a versioned API. Clients (CLI, SDK, third-party tools) negotiate which version to use during the initial handshake. Before Docker 29, the daemon supported API versions going all the way back to v1.24, which was introduced with Docker 1.12 in 2016. Nearly a decade of backward compatibility.

With Docker Engine v29.0.0 (released November 10, 2025), Docker raised the minimum supported API version to v1.44. API v1.44 corresponds to Docker Engine v25, released in late 2023.

This was deliberate and documented. Docker's deprecation policy states that API versions are supported for a limited number of major releases. API 1.24 had been deprecated for years. Docker 29 finally enforced the deprecation.

The result: any tool compiled against an older Docker SDK that defaults to API v1.24 as its negotiation floor gets rejected immediately. No fallback. No graceful degradation. A hard error: client version 1.24 is too old.

Why Traefik specifically exploded

Traefik uses Docker's official Go SDK (github.com/docker/docker/client) for container discovery. It reads container labels, figures out routing rules, and dynamically updates its configuration.

The problem is in how the Go Docker SDK initializes. When you create a new client with client.NewClientWithOpts(), the SDK defaults to negotiating from API v1.24. This has been the SDK's default since it was written. It was never a problem because Docker always accepted v1.24.

Docker 29 stopped accepting it. The negotiation fails before Traefik can even list containers. No containers discovered means no routes configured means every request hits a 404 or 502.

This affected all Traefik versions through v3.6.0, including v2.x. Traefik was using an SDK default (API v1.24) that Docker 29 no longer accepts.

It's Not Just Traefik

Traefik got the most attention because a broken reverse proxy takes down everything behind it. But Docker 29's API floor broke a wide range of tools:

Tool	What Broke	Fix
Traefik (v3.6.0 and earlier)	Container discovery, all routing	Update to v3.6.1+
Portainer (before v2.33.5)	Can't connect to Docker environments	Update to v2.33.5+ (LTS) or v2.36.0+ (STS)
Watchtower	Can't query containers for updates	Switch to `nickfedor/watchtower` (original `containrrr/watchtower` is unmaintained)
LazyDocker	Docker connection fails	Update to v0.24.2+
Dokploy	Bundled Traefik fails	Update Traefik image to v3.6.1
Coolify	Bundled Traefik fails	Check for Coolify update
Appwrite	Bundled Traefik v2.11 fails	Update to Appwrite version using Traefik v3
JetBrains IDEs	Docker plugin throws 400 errors	Update IDE / Docker plugin
Spring Boot (Testcontainers)	docker-java defaults to API v1.32	Update to Spring Boot 3.5.8+ (includes workaround), or set `api.version=1.44` in `docker-java.properties`
Swarmpit	Docker API calls fail	No fix: project is archived
CapRover	Docker API connectivity	Check for update
CasaOS	Docker API connectivity	Check for update
cAdvisor	Metrics collection stops	Update to v0.53.0+

The pattern repeats across every entry: the tool uses a Docker SDK with a hardcoded minimum version below 1.44, Docker 29 rejects the connection, and the tool can't communicate with the daemon.

Swarmpit is the saddest entry on that list. The project is archived. No fix is coming. Docker 29 finished what abandonment started.

Timeline

For the record, here's how this played out:

Nov 10, 2025 — Docker Engine v29.0.0 released. Minimum API version raised to v1.44. Traefik immediately breaks for everyone who auto-updates.
Nov 10-11, 2025 — Reports flood in. Traefik GitHub issue #12253 is filed. The Docker Community Forums thread explodes. The Traefik community forum fills up. Coolify, Dokploy, and Appwrite all see the same issue.
Nov 11, 2025 — Community identifies the daemon.json workaround and the DOCKER_MIN_API_VERSION environment variable as immediate workarounds.
Nov 12, 2025 — Traefik maintainer PR #12256 adds automatic API version negotiation. Community test builds appear on Docker Hub.
Nov 13-14, 2025 — Traefik v3.6.1 released with the fix. Portainer ships v2.33.5 and v2.36.0 with updated API support. LazyDocker ships v0.24.2.
Nov 2025 onward — The rest of the tooling world catches up. Spring Boot, JetBrains, and others ship fixes over the following weeks.

From "everything is broken" to "official fix available": roughly 48 hours. Solid open-source incident response. But if you were one of the people whose services went down on a Sunday night, it probably felt a lot longer.

Takeaways

For your infrastructure

Pin your Docker version in production. Docker 29's release notes documented the API version change. But if Docker auto-updates via your package manager (as it does on Ubuntu with the official repository), you get the breaking change whether you read the notes or not.

On Ubuntu/Debian:

sudo apt-mark hold docker-ce docker-ce-cli containerd.io

When you're ready to upgrade, unpin:

sudo apt-mark unhold docker-ce docker-ce-cli containerd.io

Pin your Traefik version too. Using traefik:latest means you get updates automatically, which is usually fine, but it also means you can't predict when behavior changes. Use explicit version tags like traefik:v3.6.1.

Keep a daemon.json backup. This file is tiny but critical. If you have it backed up or version-controlled, you can apply the workaround in under a minute instead of googling for it.

For the broader picture

A huge number of Docker-adjacent tools depended on an API version Docker deprecated years ago. The SDK default of v1.24 was never updated because Docker was always backward-compatible. Tools that were actively maintained shipped fixes within days. Tools that weren't (Swarmpit, older CasaOS setups) are now permanently broken on Docker 29+.

The min-api-version escape hatch is Docker acknowledging the pain with a temporary lever. Eventually, even that setting may go away.

If you maintain a tool that talks to Docker, check your SDK's default API version negotiation. If it's hardcoded to a minimum, make it dynamic. The Traefik PR that fixed this was a few dozen lines of code.

Update Traefik. Pin your Docker version. The order is up to you.

Why I Built ZaString

Thu, 30 Apr 2026 00:00:00 GMT

ZaString started with a frustration. I was working on a high-throughput system in C#, and every profiler run showed the same thing: strings everywhere. Allocations piling up. The garbage collector working overtime. Not because the code was wrong, but because strings in C# are immutable and every operation creates a new one.

The Problem With Strings

Here's what I mean. Take something innocent-looking:

var label = $"User {name} ({age}) logged in at {DateTime.Now:t}";

One line. Clean. Readable. And underneath, it allocates. The interpolated string creates a string on the heap. The DateTime.Now:t formatting creates another. The name substring reference adds overhead. Even something as simple as str1 + str2 doesn't modify in place. It allocates a brand new string with the combined content.

In C#, strings are immutable. That's not a flaw. It's a deliberate design choice that makes them thread-safe and predictable. But immutability has a cost: every operation that "changes" a string actually creates a new one. Concatenation, formatting, trimming, splitting, they all allocate.

For most code, this is fine. The garbage collector handles it. You never notice. But in hot paths like game loops, UI rendering, or request handlers processing thousands per second, those allocations add up. I was working with ImGui, where every frame builds strings for labels, tooltips, and debug output. Frame after frame, sixty times a second. The profiler wasn't subtle about it: strings dominated the allocation graph.

I tried the usual fixes. StringBuilder for concatenation. string.Create() for pre-sized allocations. ArrayPool<char> for reuse. They helped, but they all still allocated on the heap eventually. The fundamental issue was that the output was always a string, and strings live on the heap.

I didn't want to optimize allocation. I wanted to eliminate it.

Enter Span<T>

.NET 2.1 introduced Span<T>, and it changed what was possible.

A Span<T> is a view over a contiguous region of memory. It can wrap a managed array, a stack-allocated buffer, or unmanaged memory. Crucially, it's a ref struct. That means it lives on the stack. It can't be boxed, can't be stored in a field, can't be used across async boundaries. The runtime enforces this at compile time.

What you get in exchange is zero-allocation memory access. Slicing a span doesn't copy. It just creates a new view with an adjusted offset and length. Writing into a span modifies the underlying memory directly. No GC pressure. No hidden allocations.

Span<char> buffer = stackalloc char[64];
var written = "Hello".AsSpan().CopyTo(buffer);
var slice = buffer[..5]; // zero-copy view

This was the insight: instead of building strings and letting the GC clean up, write directly into a buffer you already own. The buffer can be on the stack for small strings, or pooled for larger ones. When you're done, you have a ReadOnlySpan<char>, which is just a view over the result with no heap allocation at all.

The catch is that working with raw spans is verbose. There's no fluent API, no formatting helpers, no interpolation. You're manually tracking offsets and calling TryFormat on every value. It works, but it doesn't feel like writing C#. It feels like assembly with extra steps.

I wanted the ergonomics of StringBuilder with the allocation profile of Span<T>. That's where ZaString came from.

The Design Philosophy

The core idea behind ZaString is simple: zero-allocation should feel normal.

Compare the two:

// StringBuilder — 146 ns, 480 B allocated
var sb = new StringBuilder();
sb.Append("Name: ").Append("John").Append(", Age: ").Append(25);
var result = sb.ToString();

// ZaSpanStringBuilder — 115 ns, 0 B allocated
Span<char> buffer = stackalloc char[50];
var builder = ZaSpanStringBuilder.Create(buffer);
builder.Append("Name: ").Append("John").Append(", Age: ").Append(25);
var result = builder.AsSpan();

Same shape. Same chaining. Same mental model. But the second version never touches the heap. The buffer is stack-allocated, the builder is a ref struct, and AsSpan() returns a view, not a copy.

The fluent API was non-negotiable. Every Append returns ref this, so you can chain calls without creating intermediate references. It supports all the types you'd expect: strings, chars, numbers, booleans, dates, with formatting and culture providers:

builder.Append("Pi: ").Append(Math.PI, "F2")
       .Append(", Date: ").Append(DateTime.Now, "yyyy-MM-dd");

C# 10's interpolated string handlers make it even cleaner:

builder.Append($"User: {name}, Age: {age}, Pi: {Math.PI:F2}");

The handler intercepts the interpolation at compile time and routes each placeholder directly into the span buffer. No intermediate string. No Format() call. Just writes.

I also wanted safety. Raw spans are dangerous. Write past the end and you corrupt memory. ZaString provides TryAppend variants that return false instead of throwing when the buffer is full. TryAppendLine is atomic: if there isn't room for both the content and the newline, nothing gets written. No partial state. No corruption.

The library grew from there. Escape helpers for JSON, HTML, URLs, CSV. Path and query parameter builders. A pooled builder for cases where the stack isn't enough. A UTF-8 writer for scenarios that need bytes instead of chars. Each piece follows the same principle: write into a buffer you control, return a view, allocate nothing.

What I Learned

The biggest lesson was about tradeoffs. Zero-allocation is not free. It's a constraint, and constraints shape design. ref struct limitations mean you can't store a builder in a field, pass it to async methods, or return it from a function that escapes the stack. You have to think about buffer sizes upfront. You lose the convenience of string being everywhere in the .NET ecosystem.

But those constraints also force clarity. When you have to declare your buffer size, you think about how much data you actually handle. When you can't allocate, you design around reuse. The code becomes more intentional.

I also learned that zero-allocation isn't always the answer. For a one-off log message, string interpolation is fine. The GC won't notice. ZaString is for the hot paths, the code that runs thousands of times per frame, per request, per second. The profiler tells you where those paths are. Trust the profiler.

On API design, I found that the best abstraction is the one you forget is there. If someone can read ZaString code and not realize it's zero-allocation, that's a win. The ergonomics should be invisible. The performance should be the default, not something you opt into with ceremony.

And honestly? There's a specific kind of joy in running a benchmark and seeing the allocation column read 0 B. Not "reduced." Not "acceptable." Zero. The garbage collector has nothing to do. The memory doesn't move. The result is just... there, in the buffer, where you put it.

Sometimes the best code is the code that doesn't happen.

ZaString is available on GitHub and NuGet.

Generic Methods Coming to Go

Mon, 27 Apr 2026 00:00:00 GMT

Generic Methods Coming to Go

"We do not anticipate that Go will ever add generic methods."

— Go FAQ, for roughly a decade.

That line sat in the FAQ like a monument. It wasn't a "not yet." It was a "never." And for years, every time someone opened an issue asking for type parameters on methods, that quote was the mic drop that ended the conversation.

Then, in January 2026, Robert Griesemer opened proposal #77273: "Generic methods for concrete types." The label was added: Proposal-Accepted.

Before you get too excited: this is an accepted proposal, not a shipped feature. You cannot use this in any Go release today. That said, the path forward is clear, and understanding what's coming, and what isn't, is worth your time now.

The Status Quo: The Anti-Pattern We All Live With

Since Go 1.18 gave us generics, there's been a frustrating gap. You can write generic functions and generic types, but you can't combine them into generic methods. The method set of a generic type can only contain non-generic methods.

This means many post-1.18 Go codebases have functions like these scattered around:

type Cache[K comparable, V any] struct {
    items map[K]V
}

// This is fine: non-generic method on a generic type.
func (c *Cache[K, V]) Get(key K) (V, bool) {
    v, ok := c.items[key]
    return v, ok
}

// But you can't do this:
// func (c *Cache[K, V]) Transform[T any](fn func(V) T) []T { ... }

// So you end up with this:
func Transform[K comparable, V any, T any](c *Cache[K, V], fn func(V) T) []T {
    result := make([]T, 0, len(c.items))
    for _, v := range c.items {
        result = append(result, fn(v))
    }
    return result
}

That last function takes the receiver as its first argument. It's not a method. It doesn't show up on autocomplete, can't be chained, and breaks the left-to-right reading flow that methods provide. cache.Transform(...) reads naturally; Transform(cache, ...) reads like inside-out code. This isn't just aesthetics: it makes APIs harder to discover, document, and compose.

Why This Became Possible

The reason generic methods were blocked for so long wasn't technical laziness. It was a genuine design problem.

The original Type Parameters proposal discussed generic methods and rejected them. The reasoning went like this: if methods can have type parameters, then interface methods should too. And generic interface methods are genuinely hard. They break type erasure, create dynamic dispatch problems, and don't play well with reflection. So the whole idea was shelved.

What changed? Griesemer's insight, captured in the proposal, is disarmingly simple:

"Concrete methods are a language feature that is useful in itself, irrespective of interfaces."

Methods do two things: they implement interfaces, and they organize code on a type. The Go team had been conflating these two roles. Once you separate them, the problem dissolves:

// This is what's being added (concrete type, generic method):
func (s *MySlice[E]) Map[T any](fn func(E) T) []T

// This is NOT being added (interface method with type parameters):
type Reader[E any] interface {
    Read[E]()  // Still impossible
}

The first one is straightforward. The compiler knows the concrete type at compile time, so it can statically resolve the method using the existing type argument mechanism. No dynamic dispatch, no reflection, no runtime complexity.

The second one remains unsolved because Reader[E] doesn't name a single method. It names a family of methods, and Go's interface satisfaction model can't handle that.

This decoupling is the key insight. Generic methods on concrete types were always implementable. They were just held hostage to the harder problem of generic interface methods.

The history matters here. Issue #49085, opened in October 2021 by the community, accumulated over 900 positive reactions. It was the primary pressure point. Issue #50981 followed in February 2022 with simpler motivating examples. Both were effectively deferred. Until now.

The Syntax

The grammar change is minimal. The method declaration production gains an optional type parameter list:

MethodDecl = "func" Receiver MethodName [ TypeParameters ] Signature [ FunctionBody ]

In practice:

type Stack[T any] struct {
    items []T
}

// Regular method on a generic type (valid since Go 1.18):
func (s *Stack[T]) Push(items ...T) {
    s.items = append(s.items, items...)
}

// Generic method with its own type parameter: new!
func (s *Stack[T]) Map[U any](fn func(T) U) *Stack[U] {
    result := &Stack[U]{}
    for _, item := range s.items {
        result.Push(fn(item))
    }
    return result
}

// Calling it:
nums := &Stack[int]{}
nums.Push(1, 2, 3)
strs := nums.Map[string](func(n int) string { return strconv.Itoa(n) })

// Type inference works:
strs = nums.Map(strconv.Itoa)

There's a subtler grammar change too: TypeArgs moves from Operand to PrimaryExpr. This is what makes expr.Method[T](args) parseable. The type arguments attach to the method call expression, not just to identifiers.

Method expressions and method values work as you'd expect, with one catch:

// Method expression: Stack[int].Map produces a generic function
// with signature [U any](*Stack[int], func(int) U) *Stack[U]

// Method value: s.Map produces [U any] func(func(int) U) *Stack[U]

// But this is INVALID: you must instantiate the type first:
// Stack.Map[U any]  // ERROR: Stack is not instantiated

The Boundary: What This Does NOT Do

After seeing the syntax, the natural next question is: "Can I use this in interfaces?"

No.

Generic interface methods are explicitly out of scope. This is a feature that solves the right problem first. The interface problem is a different, harder one, and leaving it unsolved here doesn't preclude a future proposal.

The Case Against

I'd be dishonest if I didn't engage with the real objections.

"Go said never. Why should we trust them now?"

The FAQ reversal is uncomfortable. When language designers draw a line, reversing it risks eroding trust. Some developers chose Go because of its restraint. Adding features incrementally is one thing; reversing explicit "never" statements is another.

My read: the original resistance was principled but overly conservative. The reasoning was "generic methods require generic interface methods," and that premise turned out to be false. Changing course when you discover a false premise isn't flip-flopping. It's good engineering. The Go team has a track record of shipping minimal, well-thought-out language changes. I'd rather they correct a mistake than defend it out of pride.

"This adds complexity for marginal benefit."

This is the strongest objection. Go's complexity budget is real. Every new feature makes the language harder to learn, harder to tool, and harder to reason about. Is the convenience of cache.Transform(...) worth the cognitive cost of another grammar production?

I think so, for two reasons. First, the implementation cost is surprisingly low. The compiler can statically resolve generic methods on concrete types without any runtime changes. This isn't adding a new dispatch mechanism; it's removing a parsing restriction. Second, the ergonomic benefit is disproportionate. I keep running into the (receiver, typeParam) pattern in post-1.18 Go codebases. Fixing it at the language level eliminates a class of API awkwardness that I encounter daily.

"This opens the door to generic interface methods, and then we're Java."

Slippery slope arguments are easy to make. But the proposal explicitly addresses this: "It also doesn't preclude the implementation of generic interface methods at some point, should we find an acceptable implementation solution." This is honest. It doesn't promise never, and it doesn't promise soon. It leaves the door open for a future proposal that makes its own case.

I'd argue the opposite slippery slope: if Go doesn't ship generic methods on concrete types, the pressure for full generic interface methods only grows. Shipping this targeted feature might actually reduce demand for the more complex one by solving the practical pain point.

Practical Examples

Let me show you what this actually looks like in code you might write.

Type-Safe Builders

type Query[T any] struct {
    filters []func(T) bool
    limit   int
}

func NewQuery[T any]() *Query[T] {
    return &Query[T]{limit: -1}
}

func (q *Query[T]) Where(pred func(T) bool) *Query[T] {
    q.filters = append(q.filters, pred)
    return q
}

func (q *Query[T]) Limit(n int) *Query[T] {
    q.limit = n
    return q
}

func (q *Query[T]) Run(items []T) []T {
    // Applies all filters and returns matching items (implementation omitted)
}

// Usage: notice how the type parameter flows through the chain:
type User struct {
    Name string
    Age  int
}

users := []User{{"Alice", 30}, {"Bob", 25}, {"Charlie", 30}}
activeAdults := NewQuery[User]().
    Where(func(u User) bool { return u.Age >= 18 }).
    Where(func(u User) bool { return u.Name != "" }).
    Limit(10).
    Run(users)

Today, you'd need to make Where a package-level function that takes the query as its first argument. The chaining breaks. The type parameter doesn't flow naturally. With generic methods, the builder pattern becomes first-class.

GroupBy: A Method-Level Type Parameter

Here's a case where the method genuinely needs its own type parameter, unrelated to the receiver's:

type Table[R any] struct {
    rows []R
}

func (t *Table[R]) GroupBy[K comparable](keyFn func(R) K) map[K][]R {
    groups := make(map[K][]R)
    for _, row := range t.rows {
        k := keyFn(row)
        groups[k] = append(groups[k], row)
    }
    return groups
}

// Usage:
orders := &Table[Order]{rows: allOrders}
byCustomer := orders.GroupBy[string](func(o Order) string { return o.CustomerID })

The type parameter K belongs to GroupBy, not to Table. This is the kind of thing you simply cannot express as a method today. With generic methods, it becomes natural.

The `List[E].Format[F]` Pattern

This is the motivating example from the proposal itself: a method that introduces its own type parameter independent of the type's parameter:

type List[E any] struct {
    elements []E
}

func (l *List[E]) Format[F any](formatter func(E) F) []F {
    result := make([]F, len(l.elements))
    for i, e := range l.elements {
        result[i] = formatter(e)
    }
    return result
}

// E is string, F is int: two independent type parameters
names := &List[string]{elements: []string{"hello", "world"}}
lengths := names.Format[int](func(s string) int { return len(s) })
// lengths == []int{5, 5}

This is the case that's truly impossible to express cleanly today. The method needs a type parameter (F) that's unrelated to the type's parameter (E). A package-level function can do it, but then you lose the method call syntax entirely.

How to Prepare Today

You can't use generic methods yet, but you can write code that migrates trivially when they land.

1. Wrap your (receiver, typeParam) functions on types.

If you have:

func Transform[K comparable, V any, T any](c *Cache[K, V], fn func(V) T) []T {
    // ...
}

Keep the type as the first argument. When generic methods ship, the migration is mechanical: move the function onto the type, drop the receiver parameter, done. The function body doesn't change.

2. Stop reaching for interface{} workarounds.

If you're using empty interfaces to work around the inability to write generic methods, stop. Design your types with proper type parameters now. The code will work today with package-level functions and become cleaner tomorrow with methods.

3. Name your helper functions after the future method.

If you plan to add a Transform method to your Cache type, name the package-level function Transform, not TransformCache or ApplyToCache. Mechanical migration depends on name alignment.

When Will It Land?

Here's where things stand:

Parser: Already handles type parameters on methods (parses them, currently rejects with an error). The change to accept them is trivial.
Type checker: Needs modifications to remove the current restriction and handle method expressions/values with type parameters. In progress.
Compiler backend: Generic method calls on concrete types can be statically resolved and rewritten as generic function calls. This is well-understood: no new dispatch mechanism needed.
Export/import format: This is the hardest part. The serialized format for compiled packages needs to represent generic methods, and changing it affects tooling across the ecosystem (gopls, vulncheck, build cache compatibility). This is what determines the timeline.

The x/tools repository already has a tracking issue (#77549) for generic method support across the toolchain. The Go team typically allows one or two release cycles for tooling to catch up after a language change.

Go 1.27 or 1.28 feels realistic. This is a well-scoped change with a clear implementation path, not a years-long research project like the original generics design.

My Take

I'm genuinely excited about this, and not just because I get to delete a category of awkward functions from my codebases.

What I appreciate about this proposal is the discipline. The Go team could have swung for the fences and tried to solve generic interface methods too. Instead, they identified the part that's well-understood, implementable, and high-value, and they scoped just that. That's the kind of restraint that made Go worth using in the first place.

What I'd warn against: don't use generic methods as a hammer. If a package-level generic function reads clearly, keep it as a function. Methods aren't inherently better. They're better when they improve API discoverability and composability. Use the feature where it earns its complexity budget, not everywhere it's syntactically legal.

Go's relationship with generics has been cautious, sometimes frustratingly so. But this proposal feels like the right next step: small, principled, and immediately useful. The FAQ was wrong. That's okay. Recognizing a mistake and correcting it is better than defending it indefinitely.

I Write, I Code, I Explore — Why Verbs, Not Nouns

Sun, 26 Apr 2026 00:00:00 GMT

I was listening to one of my favorite episodes of the ChessMood Podcast from my friend GM Avetik, How you can perform at your peak like top athletes do | Todd Herman on ChessMood Podcast, and around 01h07 one small idea from it kept following me around.

He said he prefers verbs to nouns when he talks about himself.

Not "I'm a coach," but "I coach." Not "I'm an entrepreneur," but "I build." Not "I'm an author," but "I write."

The more I sat with that, the more it explained a tension I've felt for years.

There are days when "developer" feels too heavy a word for me. I can ship something useful in the morning, get stuck on a dumb bug in the afternoon, and end the day feeling like I should hand back the badge. The same thing happens with writing. If I tell myself "I am a writer," every weak paragraph feels like evidence against me. But if I say "I write," the whole thing softens. I wrote today. Some of it was good. Some of it wasn't. Both can be true.

That's what I like about verbs: they leave room for motion.

Nouns can be useful. They help other people understand roughly where you live in the world. But they also freeze you at strange moments. They can turn yesterday's result into today's identity. "I am this kind of person." "I am this level." "I am this role." And once that sentence hardens, you start protecting it. You hesitate to be bad at something new because it might threaten the name you've chosen for yourself.

Verbs don't ask for that kind of defense. Verbs ask for practice.

"I write" means I can write something sharp one day and clumsy the next. "I code" means I can build a feature, break something obvious, learn, fix it, and keep going. The verb doesn't collapse because the performance wasn't perfect. In a quiet way, it's a more merciful grammar.

What stayed with me most from Todd Herman's framing was that this changes across contexts. The verb that describes how I work is not always the same one that describes how I love people. In one part of life, maybe I build. In another, I listen. In another, I encourage. The noun tries to gather everything into one static identity. The verb asks a better question: what am I actually doing here?

That question feels closer to how life is really lived.

I don't want to be trapped by a title that only describes me on my best days. I'd rather use words that still fit on ordinary days. I wrote bad code today. I rewrote a paragraph five times. I explored an idea without being sure it would go anywhere. None of that cancels the practice. It is the practice.

And there is something quietly reassuring in that. If a title disappears, the verb often remains. If one day I stop calling myself a developer, I can still build. If I stop calling myself a writer, I can still write. The label may change with the season. The work can keep moving.

So I find myself trusting verbs more.

Not because nouns are evil, and not because language alone can save us, but because verbs keep me closer to the living part of things. They remind me that I am not a finished category. I am a person in motion, practicing.

Go Pipeline Pattern: Turning Streams into Useful Data

Fri, 24 Apr 2026 00:00:00 GMT

Pipeline Pattern in Go

Introduction

Sometimes, the hard part of concurrent programming is not making things run in parallel. The hard part is keeping the flow of data understandable.

A pipeline is a simple way to do that. Instead of putting every responsibility inside one large loop, you split the work into small stages. Each stage receives values from a channel, does one transformation, and sends the result to the next stage.

source -> parse -> filter -> enrich -> sink

In Go, this pattern feels natural because goroutines and channels already give us the building blocks. A generator can create the initial stream, each pipeline stage can transform it, and a final consumer can collect or print the result.

This article continues the Go Patterns series after Producer-Consumer, Generator, and Worker Pool. The goal is not to build a framework. The goal is to learn how to structure data processing without turning one function into a pile of responsibilities.

When to Use

Use the Pipeline Pattern when data moves through multiple steps:

Processing log streams
Reading, validating, and transforming CSV rows
Cleaning API responses before storing them
Building small ETL-like flows
Splitting parsing, filtering, enrichment, and reporting into separate responsibilities

The pattern is especially useful when each step can be described as:

take a stream of values, transform or filter it, and return another stream of values.

Why Use It

Pipelines are useful because they keep each stage focused.

Separation of concerns: parsing, filtering, and reporting do not live in the same loop
Composability: stages can be reused and reordered
Natural backpressure: channels slow down upstream stages when downstream stages cannot keep up
Testability: each stage can be tested with a small input channel
Readable concurrency: the data flow is visible from the stage composition

The pattern is not magic. It is mostly discipline: one stage, one responsibility.

How It Works

A pipeline stage usually has this shape:

func stage(in <-chan Input) <-chan Output {
    out := make(chan Output)

    go func() {
        defer close(out)

        for value := range in {
            out <- transform(value)
        }
    }()

    return out
}

The stage receives a read-only channel, creates its own output channel, starts a goroutine, then closes the output channel when the input channel is exhausted.

This gives us a chain:

raw := source()
parsed := parse(raw)
filtered := filter(parsed)
enriched := enrich(filtered)
sink(enriched)

Simple Example

Before looking at logs, let's start with a tiny pipeline:

numbers -> square -> keep even -> print

package main

import "fmt"

func numbers(max int) <-chan int {
    out := make(chan int)

    go func() {
        defer close(out)

        for i := 1; i <= max; i++ {
            out <- i
        }
    }()

    return out
}

func square(in <-chan int) <-chan int {
    out := make(chan int)

    go func() {
        defer close(out)

        for n := range in {
            out <- n * n
        }
    }()

    return out
}

func keepEven(in <-chan int) <-chan int {
    out := make(chan int)

    go func() {
        defer close(out)

        for n := range in {
            if n%2 == 0 {
                out <- n
            }
        }
    }()

    return out
}

func main() {
    values := numbers(10)
    squared := square(values)
    evenSquares := keepEven(squared)

    for n := range evenSquares {
        fmt.Println(n)
    }
}

Each function owns one small part of the work. numbers produces values, square transforms them, keepEven filters them, and main consumes the final stream.

That is the pipeline pattern in its smallest useful form.

Real-World Example: Log Processing Pipeline

Now let's use a more realistic example.

Imagine we receive raw log lines and want to turn them into useful information. We can model that as a pipeline:

raw log lines -> parse logs -> filter errors -> enrich logs -> print report

For a complete runnable version, this example needs:

import (
    "fmt"
    "strings"
    "time"
)

First, define the data we want to pass between stages:

type RawLog string

type LogEntry struct {
    Timestamp time.Time
    Level     string
    Service   string
    Message   string
    Alert     bool
}

The source stage sends raw log lines:

func logSource(lines []string) <-chan RawLog {
    out := make(chan RawLog)

    go func() {
        defer close(out)

        for _, line := range lines {
            out <- RawLog(line)
        }
    }()

    return out
}

The parser turns each raw line into a structured LogEntry:

func parseLogs(in <-chan RawLog) <-chan LogEntry {
    out := make(chan LogEntry)

    go func() {
        defer close(out)

        for raw := range in {
            parts := strings.SplitN(string(raw), "|", 4)
            if len(parts) != 4 {
                continue
            }

            timestamp, err := time.Parse(time.RFC3339, parts[0])
            if err != nil {
                continue
            }

            out <- LogEntry{
                Timestamp: timestamp,
                Level:     parts[1],
                Service:   parts[2],
                Message:   parts[3],
            }
        }
    }()

    return out
}

The filter stage keeps only errors:

func filterErrors(in <-chan LogEntry) <-chan LogEntry {
    out := make(chan LogEntry)

    go func() {
        defer close(out)

        for entry := range in {
            if entry.Level == "ERROR" {
                out <- entry
            }
        }
    }()

    return out
}

The enrichment stage adds a small piece of derived information:

func enrichLogs(in <-chan LogEntry) <-chan LogEntry {
    out := make(chan LogEntry)

    go func() {
        defer close(out)

        for entry := range in {
            entry.Alert = entry.Service == "payment" || entry.Service == "auth"
            out <- entry
        }
    }()

    return out
}

Finally, the sink consumes the enriched entries:

func printReport(in <-chan LogEntry) {
    for entry := range in {
        alert := ""
        if entry.Alert {
            alert = " [ALERT]"
        }

        fmt.Printf("%s %s: %s%s\n", entry.Service, entry.Level, entry.Message, alert)
    }
}

The full pipeline becomes very readable:

func main() {
    lines := []string{
        "2026-04-24T10:00:00Z|INFO|api|request completed",
        "2026-04-24T10:00:01Z|ERROR|payment|card authorization failed",
        "2026-04-24T10:00:02Z|ERROR|worker|job timeout",
        "2026-04-24T10:00:03Z|ERROR|auth|invalid token",
    }

    raw := logSource(lines)
    parsed := parseLogs(raw)
    errors := filterErrors(parsed)
    enriched := enrichLogs(errors)

    printReport(enriched)
}

The important part is not the log format. The important part is that the flow is explicit.

Each stage can be read, tested, and replaced independently.

Error Handling

The log parser above silently skips invalid lines. That keeps the example small, but it is not always what you want in production.

Two common approaches are:

Send errors to a separate error channel
Pass a result type through the pipeline

For example:

type LogResult struct {
    Entry LogEntry
    Err   error
}

This makes failures explicit without panicking inside a goroutine. It also lets the final consumer decide whether to log, count, retry, or ignore invalid records.

Cancellation

The examples above work for finite inputs. For long-running pipelines, use context.Context so every stage can stop when the caller is done.

The shape usually looks like this:

func parseLogs(ctx context.Context, in <-chan RawLog) <-chan LogEntry {
    out := make(chan LogEntry)

    go func() {
        defer close(out)

        for {
            select {
            case <-ctx.Done():
                return
            case raw, ok := <-in:
                if !ok {
                    return
                }

                entry, ok := parseLog(raw)
                if ok {
                    out <- entry
                }
            }
        }
    }()

    return out
}

Without cancellation, a pipeline that reads from a never-ending source can leak goroutines when the consumer stops early.

Best Practices and Pitfalls

Best Practices:

Keep each stage focused on one responsibility
Return receive-only channels (<-chan T) from stages
Close the output channel from the goroutine that writes to it
Use context.Context for long-running or cancellable pipelines
Test each stage independently with small input channels

Pitfalls:

Forgetting to close output channels
Stopping early without cancelling upstream stages
Creating too many tiny stages that hide simple logic
Mixing parsing, filtering, enrichment, and reporting in one function
Assuming ordering will stay the same if you later parallelize a stage

Related Patterns

Generator Pattern: Creates the initial stream of values
Producer-Consumer Pattern: Separates production from consumption
Worker Pool Pattern: Parallelizes expensive stages
Fan-Out/Fan-In Pattern: Distributes one stage across multiple workers and merges the results

Summary

The Pipeline Pattern is one of the most readable ways to structure data processing in Go. It lets you split a flow into small stages, connect them with channels, and keep each responsibility isolated.

It works well when data naturally moves through a sequence: read, parse, filter, enrich, report.

The pattern is also a bridge to more advanced concurrency designs. Once one stage becomes too slow, you can combine a pipeline with a Worker Pool or Fan-Out/Fan-In to parallelize only that part of the flow.

Testing Pipelines

The best pipeline tests treat each stage as a pure function: send a channel in, get a channel out. The goroutines are implementation details. I wrote about that approach in TDD Isn't About Bugs — It's Your Permission to Refactor.

Series Navigation

This article is part of the Go Patterns series:

Previous: Mastering the Worker Pool Pattern in Go
Series: Go Patterns

If you want to experiment with the code examples, you can find them on my GitHub repository.

Flexible Approaches to Worker Pools in Go

Thu, 12 Dec 2024 00:00:00 GMT

Flexible Approaches to Worker Pools in Go

Introduction

While the standard Worker Pool pattern is powerful, Go's flexibility allows for alternative approaches to concurrent task processing. This article explores the Shared Semaphore method and discusses the use of third-party libraries for managing concurrency in Go applications.

Shared Semaphore Method

The Shared Semaphore method uses a buffered channel as a semaphore to limit concurrency across various parts of an application.

package main

import (
	"fmt"
	"sync"
	"time"
)

func processWithSemaphore(tasks []int, maxConcurrency int) {
	sem := make(chan struct{}, maxConcurrency)
	var wg sync.WaitGroup

	for _, task := range tasks {
		wg.Add(1)
		sem <- struct{}{} // Acquire semaphore
		go func(task int) {
			defer wg.Done()
			defer func() { <-sem }() // Release semaphore
			processTask(task)
		}(task)
	}

	wg.Wait()
}

func processTask(task int) {
	fmt.Printf("Processing task %d\n", task)
	time.Sleep(time.Second) // Simulate work
	fmt.Printf("Completed task %d\n", task)
}

func main() {
	tasks := []int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
	processWithSemaphore(tasks, 3)
}

Advantages:

Simple and lightweight implementation
Scales to zero when not in use
Can be used in multiple places without increasing complexity

Use Case:

Ideal for scenarios where you need to limit concurrency across various parts of your application without maintaining a persistent worker pool.

Third-Party Libraries

While implementing your own worker pool is often the best approach, there are some third-party libraries that can be useful in certain situations:

Ants: A high-performance goroutine pool in Go, providing features like automatic scaling and reuse of goroutines.
sourcegraph/conc: A package for structured concurrency in Go, offering higher-level abstractions for common concurrency patterns.

These libraries can be beneficial when you need advanced features or when working on complex concurrent systems. However, it's important to note that in approximately 90% of cases, maintaining your own worker pool implementation is preferable. This approach gives you more control, better understanding of your concurrency model, and avoids unnecessary dependencies.

Choosing the Right Approach

Consider the following factors when deciding on your concurrency approach:

Assess your concurrency needs:

Do you need to limit concurrency across multiple parts of your application?
Is your workload consistent or variable?

Evaluate your task characteristics:

Are tasks short-lived or long-running?
Do you need fine-grained control over task execution?

Consider your application architecture:

Is simplicity a priority?
Do you need to scale workers dynamically?

Decision tree:

If (need to limit concurrency in specific sections) AND (variable workload): Use Shared Semaphore
Else if (consistent workload) AND (need fine-grained control): Use Standard Worker Pool
Else if (need advanced features) AND (complexity is justified): Consider third-party libraries like Ants or sourcegraph/conc
Else: Implement and maintain your own worker pool

This decision process helps guide your choice based on your specific context and requirements, ensuring you choose the most appropriate concurrency pattern for your needs while prioritizing simplicity and control in most cases.

Summary

While the standard Worker Pool pattern is versatile, Go's concurrency model allows for flexible approaches like the Shared Semaphore method. This alternative can be particularly useful for applications with varying concurrency needs across different components. Third-party libraries offer advanced features but should be used judiciously, as maintaining your own worker pool often provides the best balance of control and simplicity.

Disclaimer

This article provides an overview of flexible approaches to worker pools in Go. While these patterns are powerful, it's important to consider the specific needs of your application when implementing them. For production use, additional error handling and optimizations may be necessary.

For more advanced concurrency patterns and best practices in Go, stay tuned for future articles! 🚀

If you want to experiment with the code examples, you can find them on my GitHub repository.

Mastering the Worker Pool Pattern in Go

Tue, 10 Dec 2024 00:00:00 GMT

Worker Pool Pattern in Go

Introduction

The Worker Pool pattern is a fundamental concurrency design in Go that efficiently manages a pool of worker goroutines to process tasks from a shared queue. This pattern excels at handling a large number of independent tasks concurrently while maintaining precise control over system resources and performance.

When to Use

Processing a large number of independent tasks that can be parallelized
Limiting the number of concurrent operations to prevent resource exhaustion
Balancing workload across multiple processors or cores
Managing CPU-bound or I/O-bound tasks efficiently
Handling batch processing operations with controlled parallelism

Why to Use

Controls resource utilization by maintaining a fixed number of workers
Improves performance through efficient parallel processing
Prevents system overload by limiting concurrent operations
Enhances application scalability and throughput
Maintains predictable resource usage patterns

How it Works

The Worker Pool pattern consists of three essential components:

A pool of worker goroutines that process tasks concurrently
A job queue (input channel) that holds pending tasks
A results queue (output channel) that collects processed results

Workers continuously pull tasks from the job queue, process them independently, and send results to the results queue. This design ensures efficient task distribution and controlled concurrency.

Simple Example

func worker(id int, jobs <-chan int, results chan<- int) {
    for job := range jobs {
        fmt.Printf("Worker %d processing job %d\n", id, job)
        time.Sleep(time.Second) // Simulating work
        results <- job * 2
    }
}

func main() {
    const numJobs = 5
    const numWorkers = 3

    jobs := make(chan int, numJobs)
    results := make(chan int, numJobs)

    // Start worker pool
    for w := 1; w <= numWorkers; w++ {
        go worker(w, jobs, results)
    }

    // Send jobs to the workers
    for j := 1; j <= numJobs; j++ {
        jobs <- j
    }
    close(jobs)

    // Collect and print results
    for a := 1; a <= numJobs; a++ {
        result := <-results
        fmt.Printf("Job result: %d\n", result)
    }
}

Real-World Example: Image Processor

Let's consider a scenario where we need to process multiple images concurrently:

type Job struct {
    ID       int
    ImageURL string
    Size     int
}

type Result struct {
    JobID     int
    ImageURL  string
    NewSize   int
    Error     error
    TimeSpent time.Duration
}

func imageProcessor(id int, jobs <-chan Job, results chan<- Result) {
    for job := range jobs {
        startTime := time.Now()

        fmt.Printf("Worker %d processing image %d from %s\n", id, job.ID, job.ImageURL)

        result := Result{
            JobID:    job.ID,
            ImageURL: job.ImageURL,
            NewSize:  job.Size,
        }

        // Simulate image processing with realistic steps
        err := processImage(job)
        if err != nil {
            result.Error = err
            results <- result
            continue
        }

        result.TimeSpent = time.Since(startTime)
        results <- result
    }
}

func processImage(job Job) error {
    // Simulate various image processing steps
    time.Sleep(time.Duration(rand.Intn(500)) * time.Millisecond)

    // Simulate potential errors
    if rand.Float32() < 0.1 {
        return fmt.Errorf("failed to process image %d: simulation error", job.ID)
    }

    return nil
}

func main() {
    numCPU := runtime.NumCPU()
    runtime.GOMAXPROCS(numCPU)
    numWorkers := numCPU * 2 // Use 2 workers per CPU core
    const numJobs = 10

    jobs := make(chan Job, numJobs)
    results := make(chan Result, numJobs)

    // Initialize worker pool
    for w := 1; w <= numWorkers; w++ {
        go imageProcessor(w, jobs, results)
    }

    // Send image processing jobs
    for j := 1; j <= numJobs; j++ {
        jobs <- Job{
            ID:       j,
            ImageURL: fmt.Sprintf("https://example.com/image%d.jpg", j),
            Size:     100 * j, // Varying sizes
        }
    }
    close(jobs)

    // Collect and handle results
    for a := 1; a <= numJobs; a++ {
        result := <-results
        if result.Error != nil {
            fmt.Printf("Error processing image %d: %v\n", result.JobID, result.Error)
        } else {
            fmt.Printf("Successfully processed image %d to size %dpx in %v\n",
                result.JobID, result.NewSize, result.TimeSpent)
        }
    }
}

Best Practices and Pitfalls

Best Practices:

Always close the channel when generation is complete
Use buffered channels when appropriate to prevent blocking
Include monitoring and logging for production environments
Implement graceful shutdown mechanisms
Size your worker pool based on available system resources

Pitfalls:

Creating too many workers, leading to resource exhaustion
Not handling worker failures or panics
Forgetting to close channels properly
Missing timeout mechanisms for long-running tasks
Inefficient job distribution strategies

Summary

The Worker Pool pattern is a powerful tool in Go's concurrency toolkit, offering a balanced approach to parallel processing. By maintaining a fixed number of workers, it prevents resource exhaustion while maximizing throughput. The pattern is particularly valuable in real-world scenarios such as image processing, batch operations, and API request handling, where controlled concurrent processing is essential for optimal performance and resource utilization.

Disclaimer

This article provides an introduction to the Worker Pool pattern in Go. While the pattern is powerful, it's important to consider the specific needs of your application when implementing it. For production use, additional error handling and optimizations may be necessary.

Testing Worker Pools

If you test a worker pool, focus on the contract: jobs go in, results come out. The number of goroutines is an implementation detail. I wrote about that in TDD Isn't About Bugs — It's Your Permission to Refactor.

Series Navigation

This article is part of the Go Patterns series:

Previous: Mastering the Generator Pattern in Go
Next: Go Pipeline Pattern: Turning Streams into Useful Data
Series: Go Patterns

For more advanced concurrency patterns and best practices in Go, stay tuned for future articles! 🚀

If you want to experiment with the code examples, you can find them on my GitHub repository.

Advanced Generator Pattern: Consuming and Testing Data Streams

Sun, 08 Dec 2024 00:00:00 GMT

Advanced Generator Pattern: Consuming and Testing Streams

:::warning[Difficulty Level] Advanced :::

Introduction

Expanding on our previous discussions of the Generator pattern, we'll explore two advanced applications: consuming large datasets lazily and simulating data streams for testing. These techniques are crucial for efficient data processing and robust application testing.

When to Use

Processing large datasets that don't fit in memory
Simulating data sources for testing
Implementing ETL (Extract, Transform, Load) processes
Creating reproducible test scenarios for data processing pipelines

Why to Use

Memory Efficiency: Process large datasets without loading everything into memory
Testability: Create controlled environments for testing data processing logic
Flexibility: Easily switch between real and simulated data sources
Reproducibility: Generate consistent test cases for data processing scenarios

How it Works

Create generator functions that yield data items one at a time
Use channels to stream data from the source to the consumer
Implement lazy loading for large datasets
Create mock data generators for testing scenarios

Example 1: Lazy Loading of Large Datasets

type DataItem struct {
    ID   int
    Data string
}

// lazyDataLoader simulates loading a large dataset lazily
func lazyDataLoader(filePath string) <-chan DataItem {
    out := make(chan DataItem)
    go func() {
        defer close(out)
        // Simulate opening a large file
        fmt.Printf("Opening file: %s\n", filePath)

        // Simulate reading the file line by line
        for i := 0; i < 1000000; i++ {
            // Simulate processing delay for each item
            time.Sleep(1 * time.Millisecond)
            out <- DataItem{
                ID:   i + 1,
                Data: fmt.Sprintf("Data from line %d", i+1),
            }
            if i%100000 == 0 {
                fmt.Printf("Processed %d items\n", i)
            }
        }
    }()
    return out
}

func processData(data <-chan DataItem) {
    for item := range data {
        // Simulate data processing
        processedData := fmt.Sprintf("Processed: %s (ID: %d)", item.Data, item.ID)
        fmt.Println(processedData)
    }
}

func main() {
    dataStream := lazyDataLoader("large_dataset.txt")
    processData(dataStream)
}

This example demonstrates lazy loading of a large dataset, processing items one at a time without loading the entire dataset into memory.

Example 2: Simulating Data Streams for Testing

type DataItem struct {
    ID   int
    Data string
}

// mockDataStream simulates a data source (e.g., a file, queue, or network stream)
func mockDataStream(count int) <-chan DataItem {
    out := make(chan DataItem)
    go func() {
       defer close(out)
       for i := 0; i < count; i++ {
          // Simulate reading from a data source
          time.Sleep(100 * time.Millisecond)
          out <- DataItem{
             ID:   i + 1,
             Data: fmt.Sprintf("Data-%d", i+1),
          }
       }
    }()
    return out
}

// dataGenerator consumes the mock stream and yields processed data
func dataGenerator(stream <-chan DataItem) <-chan string {
    out := make(chan string)
    go func() {
       defer close(out)
       for item := range stream {
          // Process the data item
          processedData := fmt.Sprintf("Processed: %s (ID: %d)", item.Data, item.ID)
          out <- processedData
       }
    }()
    return out
}

type StreamGenerator struct{}

func (g StreamGenerator) Execute() {
    // Create a mock data stream
    dataStream := mockDataStream(10)

    // Create a generator to process the stream
    processedDataGen := dataGenerator(dataStream)

    // Consume and print the processed data
    for data := range processedDataGen {
       fmt.Println(data)
    }
}

This example demonstrates a more structured approach to using the Generator pattern for testing data processing pipelines:

mockDataStream simulates a data source by generating items with controlled timing
dataGenerator shows how to process a stream of data items and transform them
The StreamGenerator type provides a clean interface for executing the pipeline and can be replaced with real data sources in production using DI (Dependency Injection)
Each stage of the pipeline is clearly separated and testable

Best Practices and Pitfalls

Best Practices:

Use buffered channels for improved performance when processing large streams
Implement timeout mechanisms for long-running operations
Use the context package for cancellation in long-running generators
Create configurable mock generators for diverse test scenarios

Pitfalls:

Not handling errors or edge cases in data generation
Overlooking resource cleanup in generators (e.g., closing file handles)
Creating overly complex mock generators that don't reflect real-world scenarios
Ignoring performance implications in lazy loading implementations

Summary

The Generator pattern proves invaluable for both consuming large datasets efficiently and creating robust test environments for data processing logic. By leveraging Go's concurrency features, we can create flexible, memory-efficient, and testable data processing pipelines that can handle real-world scenarios and simulated test cases alike.

Disclaimer

While these examples demonstrate the power of the Generator pattern for data processing and testing, real-world implementations may require additional error handling, resource management, and optimizations. Always consider the specific requirements and constraints of your application when applying these patterns.

For more advanced concurrency patterns and best practices in Go, stay tuned for future articles! 🚀

If you want to experiment with the code examples, you can find them on my GitHub repository.

<div style="font-size: 0.875rem; color: color-mix(in oklch, var(--text-base) 75%, transparent);"> <span property="dct:title">Educational Go Patterns </span> by <a rel="cc:attributionURL dct:creator" property="cc:attributionName" href="https://corentings.dev">Corentin Giaufer Saubert</a> is licensed under <a href="https://creativecommons.org/licenses/by-nc-nd/4.0/?ref=chooser-v1" target="_blank" rel="license noopener noreferrer">Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International </a><br/> The code examples are licensed under the <a href="https://opensource.org/licenses/MIT" target="_blank" rel="license noopener noreferrer">MIT License</a>. The banner image has been created by (DALL·E) and is licensed under the same license as the article and other graphics. </div>

Advanced Generator Pattern in Go: Test Data Generation

Fri, 06 Dec 2024 00:00:00 GMT

Advanced Generator Pattern: Test Data for Web Services

Introduction

Building upon our previous exploration of the Generator pattern, let's dive into a more advanced real-world application: generating test data for a web service. This pattern is particularly useful for creating large datasets to stress test APIs or simulate high-load scenarios.

When to Use

Stress testing web services
Simulating high-load scenarios for databases
Creating diverse datasets for QA environments
Benchmarking system performance

Why to Use

Scalability: Easily generate large volumes of test data
Customization: Tailor data generation to specific test scenarios
Realism: Create data that closely mimics production patterns
Efficiency: Generate data on-the-fly, reducing storage needs

How it Works

Define structures representing your API's data models
Create generator functions for each data type
Combine generators to create complex, interrelated data sets
Use channels to stream generated data to consumers (e.g., API clients)

Advanced Example: E-commerce API Test Data Generator

type Product struct {
    ID    int
    Name  string
    Price float64
}

type Order struct {
    ID       int
    UserID   int
    Products []Product
    Total    float64
}

func productGenerator(count int) <-chan Product {
    out := make(chan Product)
    go func() {
        defer close(out)
        for i := 0; i < count; i++ {
            out <- Product{
                ID:    i + 1,
                Name:  fmt.Sprintf("Product-%d", i+1),
                Price: 10.0 + float64(i),
            }
        }
    }()
    return out
}

func orderGenerator(userCount, orderPerUser int, products <-chan Product) <-chan Order {
    out := make(chan Order)
    go func() {
        defer close(out)
        var orderID int
        for userID := 1; userID <= userCount; userID++ {
            for i := 0; i < orderPerUser; i++ {
                orderID++
                var orderProducts []Product
                var total float64
                for j := 0; j < rand.Intn(5)+1; j++ {
                    product := <-products
                    orderProducts = append(orderProducts, product)
                    total += product.Price
                }
                out <- Order{
                    ID:       orderID,
                    UserID:   userID,
                    Products: orderProducts,
                    Total:    total,
                }
            }
        }
    }()
    return out
}

func main() {
    productChan := productGenerator(1000)
    orderChan := orderGenerator(100, 5, productChan)

    // Simulate sending orders to an API
    for order := range orderChan {
        // In a real scenario, you'd send this to your API
        fmt.Printf("Sending order %d for user %d with total $%.2f\n", order.ID, order.UserID, order.Total)
    }
}

This example demonstrates a more complex use of the Generator pattern to create realistic test data for an e-commerce API. It generates products and orders, simulating a scenario where multiple users are placing orders with varying numbers of products.

Best Practices and Pitfalls

Best Practices:

Use buffered channels for improved performance when generating large datasets
Implement cancellation mechanisms for long-running generators
Consider using worker pools for parallel data generation in complex scenarios
Seed random number generators for reproducible test data

Pitfalls:

Generating more data than necessary, leading to increased test times
Not closing channels properly, causing goroutine leaks
Overlooking edge cases in data generation, leading to incomplete test coverage
Generating unrealistic data that doesn't reflect real-world scenarios

Summary

The advanced application of the Generator pattern for test data generation showcases its power in creating scalable, customizable, and efficient solutions for testing web services. By leveraging Go's concurrency features, we can create sophisticated data generation pipelines that closely mimic real-world scenarios, enabling thorough and realistic testing of our systems.

Disclaimer

This article expands on the Generator pattern with a focus on test data generation. While the example provided is more complex, it's still simplified for educational purposes. In real-world applications, additional considerations such as data variety, error handling, and integration with actual API endpoints would be necessary.

For more advanced concurrency patterns and best practices in Go, stay tuned for future articles! 🚀

If you want to experiment with the code examples, you can find them on my GitHub repository.

Mastering the Generator Pattern in Go

Tue, 03 Dec 2024 00:00:00 GMT

Generator Pattern in Go

Introduction

The Generator pattern in Go is a powerful concurrency pattern used to create functions that produce a sequence of values. It leverages Go's goroutines and channels to generate data asynchronously, providing an elegant way to work with streams of data or implement iterators.

When to Use

Creating sequences of numbers or data
Implementing iterators
Processing streams of data
Generating test data
Simulating real-time data sources

Why to Use

Lazy Evaluation: Values are generated on-demand, saving memory
Encapsulation: Hides the complexity of data generation
Concurrency: Allows for asynchronous data production
Flexibility: Can generate infinite or finite sequences
Composability: Generators can be chained or combined easily

How it Works

A function creates and returns a channel
The function starts a goroutine that sends values through the channel
The caller receives values from the channel, typically using a for-range loop

Simple Example

func evenGenerator(max int) <-chan int {
    out := make(chan int)
    go func() {
        for i := 0; i <= max; i += 2 {
            out <- i
        }
        close(out)
    }()
    return out
}

func main() {
    for num := range evenGenerator(10) {
        fmt.Println(num)
    }
}

This example creates a generator that produces even numbers up to a specified maximum. The evenGenerator function returns a receive-only channel (<-chan int). It starts a goroutine that sends even numbers through the channel and closes it when done.

Real-World Example: Log Line Generator

Let's consider a scenario where we need to generate sample log lines for testing a log analysis system.

type LogEntry struct {
    Timestamp time.Time
    Level     string
    Message   string
}

func logGenerator(count int) <-chan LogEntry {
    out := make(chan LogEntry)
    go func() {
        levels := []string{"INFO", "WARNING", "ERROR"}
        messages := []string{
            "User logged in",
            "Failed login attempt",
            "Database connection lost",
            "API request received",
            "Cache miss",
        }
        for i := 0; i < count; i++ {
            out <- LogEntry{
                Timestamp: time.Now().Add(time.Duration(i) * time.Second),
                Level:     levels[rand.Intn(len(levels))],
                Message:   messages[rand.Intn(len(messages))],
            }
            time.Sleep(100 * time.Millisecond) // Simulate delay between log entries
        }
        close(out)
    }()
    return out
}

func main() {
    for entry := range logGenerator(5) {
        fmt.Printf("[%s] %s: %s\n", entry.Timestamp.Format(time.RFC3339), entry.Level, entry.Message)
    }
}

This generator creates a stream of log entries, simulating real-world log generation. It's useful for testing log processing systems, allowing developers to generate a controlled stream of diverse log entries.

Best Practices and Pitfalls

Best Practices:

Always close the channel when generation is complete
Use receive-only channels (<-chan) as return types
Consider using context for cancellation in long-running generators
Implement error handling for robust generators

Pitfalls:

Forgetting to close channels, leading to goroutine leaks
Creating infinite generators without proper termination conditions
Blocking indefinitely on channel operations without timeout mechanisms
Overusing generators for simple, finite sequences where slices might suffice

Related Patterns

Pipeline Pattern: Often used in conjunction with generators to process data streams
Fan-Out/Fan-In Pattern: Can distribute generator output to multiple consumers
Iterator Pattern: Generators can be seen as concurrent iterators

Summary

The Generator pattern in Go provides a powerful way to create sequences or streams of data asynchronously. By leveraging goroutines and channels, it offers lazy evaluation, encapsulation of complex logic, and seamless integration with Go's concurrency model. Whether you're working with infinite sequences, simulating data sources, or implementing iterators, the Generator pattern offers a flexible and efficient solution.

Disclaimer

This article provides an introduction to the Generator pattern in Go. While the pattern is powerful, it's important to consider the specific needs of your application when implementing it. For production use, additional error handling and optimizations may be necessary.

Testing Generators

Generators are easy to test when you treat them as contracts: send input, receive output. The goroutine is an implementation detail. I wrote about why that matters in TDD Isn't About Bugs — It's Your Permission to Refactor.

Series Navigation

This article is part of the Go Patterns series:

Previous: Understanding the Producer-Consumer Pattern in Go
Next: Mastering the Worker Pool Pattern in Go
Series: Go Patterns

For more advanced concurrency patterns and best practices in Go, stay tuned for future articles! 🚀

If you want to experiment with the code examples, you can find them on my GitHub repository.

Producer-Consumer in Go: Beyond the Basics

Mon, 02 Dec 2024 00:00:00 GMT

Producer-Consumer in Go: Beyond the Basics

In our previous article, we explored the fundamentals of the Producer-Consumer pattern in Go. Today, we'll take it a step further by examining more practical scenarios and introducing buffered channels - a powerful feature that can significantly improve your concurrent applications.

Buffered Channels: A Game Changer

While regular channels provide immediate synchronization between producers and consumers, sometimes we need more flexibility. Buffered channels act like a small warehouse, temporarily storing items when producers are faster than consumers or when we want to batch process items.

Let's see how they work:

// Unbuffered channel - synchronous
ch := make(chan int)

// Buffered channel - can hold up to 5 items
bufferedCh := make(chan int, 5)

The key difference? With a buffered channel, producers can send up to 5 items without waiting for a consumer to receive them. This can significantly improve performance in certain scenarios.

Real-World Example: Log Processing System

Imagine building a log processing system for a busy web application. Logs come in rapidly during peak hours, but we want to process them in batches for efficiency. This is a perfect use case for buffered channels.

type LogEntry struct {
    Timestamp time.Time
    Level     string
    Message   string
}

func logGenerator(logs chan<- LogEntry) {
    // Simulate incoming logs
    for i := 0; i < 10 ; i++{
        log := LogEntry{
            Timestamp: time.Now(),
            Level:     []string{"INFO", "WARNING", "ERROR"}[rand.Intn(3)],
            Message:   fmt.Sprintf("Event #%d occurred", i),
        }
        logs <- log
        time.Sleep(100 * time.Millisecond) // Simulate varying log frequencies
    }
    close(logs)
}

func logProcessor(logs <-chan LogEntry) {
    batch := make([]LogEntry, 0, 3) // Process logs in batches of 3

    for log := range logs {
        batch = append(batch, log)

        if len(batch) == 3 {
            // Process batch
            processLogBatch(batch)
            batch = batch[:0] // Clear the batch
        }
    }

    // Process remaining logs
    if len(batch) > 0 {
        processLogBatch(batch)
    }
}

func processLogBatch(batch []LogEntry) {
    fmt.Println("Processing batch of logs:")
    for _, log := range batch {
        fmt.Printf("[%s] %s: %s\n",
            log.Timestamp.Format("15:04:05"),
            log.Level,
            log.Message)
    }
    fmt.Println("Batch processing complete\n")
}

func main() {
    // Buffer size of 5 to handle burst of logs
    logs := make(chan LogEntry, 5)

    go logGenerator(logs)
    logProcessor(logs)
}

When you run this program, you'll see output similar to this:

Processing batch of logs:
[14:23:45] INFO: Event #0 occurred
[14:23:45] WARNING: Event #1 occurred
[14:23:45] ERROR: Event #2 occurred
Batch processing complete

Processing batch of logs:
[14:23:46] INFO: Event #3 occurred
[14:23:46] WARNING: Event #4 occurred
[14:23:46] INFO: Event #5 occurred
Batch processing complete

Processing batch of logs:
[14:23:47] ERROR: Event #6 occurred
[14:23:47] INFO: Event #7 occurred
[14:23:47] WARNING: Event #8 occurred
Batch processing complete

Processing remaining logs:
[14:23:48] INFO: Event #9 occurred
Batch processing complete

Understanding the Benefits

This implementation showcases several advanced concepts:

Burst Handling: The buffered channel (size 5) can handle bursts of logs without blocking the producer, even if the consumer is busy processing a batch.
Batch Processing: Instead of processing each log immediately, we batch them for more efficient processing. This is common in real-world scenarios where batching can reduce database writes or API calls.
Graceful Shutdown: The system handles remaining logs before shutting down, ensuring no data is lost.
Flexible Processing: The batch size (3) is independent of the channel buffer size (5), allowing us to optimize both independently.

When to Use Buffered Channels

Buffered channels are particularly useful when:

Your producers and consumers work at different speeds
You want to batch process items for efficiency
You need to handle burst of data without blocking producers
You want to improve performance by reducing synchronization overhead

However, remember that buffered channels don't solve all problems. They can hide deadlocks and make it harder to reason about your program's behavior if used incorrectly.

Tips for Success

Choose buffer sizes carefully—too small negates the benefits, too large can hide problems
Monitor channel capacity in production using len(ch) and cap(ch)
Consider using contexts for cancellation and timeouts
Always handle the remaining items when shutting down

What's Next?

Now that you understand buffered channels and batch processing, you're ready to explore more advanced patterns like Fan-Out/Fan-In or the Pipeline pattern. Stay tuned for our next article where we'll dive into those topics!

Conclusion

The Producer-Consumer pattern becomes even more powerful when combined with buffered channels and batch processing. While staying true to Go's simplicity, these features allow us to build more efficient and resilient systems.

Remember: The key to mastering these patterns is practice. Try modifying the log processing example to handle different batch sizes or add error handling. The possibilities are endless!

Understanding the Producer-Consumer Pattern in Go

Sat, 30 Nov 2024 00:00:00 GMT

Producer-Consumer Pattern in Go

Introduction

The Producer-Consumer pattern is a fundamental concurrency pattern in Go that elegantly separates the production of data from its consumption. By using channels as intermediaries, this pattern creates a robust foundation for concurrent data processing.

When to Use

When you need to separate data generation from its processing
In scenarios where production and consumption rates differ
When scaling producers and consumers independently is desired
For managing workload distribution in concurrent systems

Why Use It

This pattern offers several compelling advantages:

Modularity: Clear separation between data production and consumption logic
Flexible Scaling: Easy to add more producers or consumers as needed
Buffer Management: Handles different processing rates naturally
Resource Control: Better management of system resources through controlled data flow

How it Works

The pattern consists of three main components:

Producers: Goroutines that generate data
Queue: A channel that acts as a buffer between producers and consumers
Consumers: Goroutines that process the data

Simple Example

func producer(ch chan<- int) {
    for i := 0; i < 5; i++ {
        ch <- i
        fmt.Printf("Produced: %d\n", i)
    }
    close(ch)
}

func consumer(ch <-chan int) {
    for num := range ch {
        fmt.Printf("Consumed: %d\n", num)
    }
}

func main() {
    ch := make(chan int)
    go producer(ch)
    consumer(ch)
}

Real-World Example: Web Scraper

type Page struct {
    URL     string
    Content string
}

func scraper(urls []string, pages chan<- Page) {
    for _, url := range urls {
        // Simulate web scraping
        time.Sleep(100 * time.Millisecond)
        pages <- Page{
            URL:     url,
            Content: fmt.Sprintf("Content from %s", url),
        }
    }
    close(pages)
}

func processor(pages <-chan Page, results chan<- string) {
    for page := range pages {
        // Simulate content processing
        time.Sleep(200 * time.Millisecond)
        results <- fmt.Sprintf("Processed %s: %s", page.URL, page.Content)
    }
    close(results)
}

func main() {
    urls := []string{"https://example1.com", "https://example2.com"}
    pages := make(chan Page)
    results := make(chan string)

    go scraper(urls, pages)
    go processor(pages, results)

    for result := range results {
        fmt.Println(result)
    }
}

Best Practices and Pitfalls

Best Practices:

Always use directional channel types (chan<- for send-only, <-chan for receive-only)
Consider buffered channels when producers and consumers work at different speeds
Implement proper error handling and propagation
Use context for cancellation when needed

Common Pitfalls:

Forgetting to close channels
Not handling backpressure when producers are faster than consumers
Creating deadlocks by improper channel management
Memory leaks from unclosed goroutines

Related Patterns

Pipeline Pattern: For multi-stage data processing
Worker Pool Pattern: For parallel task processing
Fan-Out/Fan-In Pattern: For distributed workload processing

Summary

The Producer-Consumer pattern is a cornerstone of concurrent programming in Go. Its simplicity and effectiveness make it an excellent starting point for learning concurrency patterns. By separating concerns and managing data flow through channels, it provides a clean and efficient way to handle concurrent data processing tasks.

Series Navigation

This article is part of the Go Patterns series:

Next: Mastering the Generator Pattern in Go
Series: Go Patterns

Disclaimer

This article is a simple introduction to the Producer-Consumer pattern in Go. For more advanced use cases and optimizations, consider exploring additional resources and best practices in concurrent programming. I may write more articles on this topic in the future, so stay tuned! 🚀 <br/> If you want to have a look at the code examples, you can find them on my GitHub repository.

My Experience at the Karen Asrian Memorial Tournament

Fri, 17 May 2024 00:00:00 GMT

Introduction

My recent trip to Armenia wasn't just a vacation—it was a chess pilgrimage! I participated in the prestigious Karen Asrian Memorial, a chess tournament that pushed me to my limits and left me with unforgettable memories.

The Tournament

The Karen Asrian Memorial is a tournament dedicated to a chess legend. GM Avetik Grigoryan wrote a fantastic article about him, which I highly advise you to read: <a rel='noopener noreferrer' aria-labal="In Memory of GM Karen Asrian" href="https://chessmood.com/blog/karen-asrian">In Memory of GM Karen Asrian</a>. One of the tournament's highlights was meeting Levon Aronian, a true chess superstar. I was fortunate to meet such a player in person.

The tournament format consisted of nine rounds spread over nine days. Each game was a marathon, lasting between 3 and 4.5 hours.

My performance

Overall, this was my most robust performance yet. I beat a player rated over 2150 and even faced a Grandmaster in a classical chess game for the first time. While I achieved a dominant position, time pressure ultimately led to a bittersweet loss.

At the beginning of the tournament, I was rated 1803 Elo and was the 67th player over 73 according to the initial ranking. I finished with 3.5 points over 9, half a point more than my initial goal. I earned 34 fide elo points and a performance of 2000 elo! My final rank is 57th, meaning I had an excellent performance.

Chessmood

Speaking of improvement, a big part of my chess journey wouldn't have been possible without the Chessmood team. Their training platform helped me climb over 300 points in online blitz and rapid chess ratings. This translated into real-world results, allowing me to consistently defeat players rated around 2000 Elo. I also had the opportunity to meet them in person. I played in the tournament with IM David Shahinyan, who helped me improve recently. Furthermore, I could visit Chessmood's office, discover their waiting room, and have a great barbecue with them!

Conclusion

This Armenian adventure tested my skills and reminded me of the power of dedicated practice. It left me eager to return and continue my chess journey!

If you want to know more about what I discovered in Armenia besides chess, stay tuned for my next article! I would like to share some insights about the country and its culture.

Solving the Sum of Squares Problem: Optimizing Performance

Wed, 09 Aug 2023 00:00:00 GMT

Disclaimer: Enhancing Algorithm Discussions

Before delving into the main topic, I want to express my respect for the original post's intent. My response aims to provide slight corrections that can benefit readers seeking accurate information. In the spirit of shared learning, I intend to contribute constructively rather than criticize. <br />Clarifications and alternative viewpoints can foster a deeper understanding of complex concepts. Let's continue engaging in open discussions, embracing diverse insights as we collectively refine our knowledge.

Introduction

Greetings, fellow Gophers! Are you ready to explore the intriguing world of Goroutines and unravel the mysteries of the sum of squares problem? Recently, a friend learning Go shared a <a href={'https://medium.com/@ShivamSouravJha/golang-only-things-i-know-for-the-interview-4322d29d67a3'} style="font-weight: 700; text-decoration: underline; color: var(--primary);" target="_blank" rel="noopener">medium article</a> with me. It discussed Goroutines and how they can be used to solve a simple problem: calculating the sum of square numbers in a given array.

In this blog post, we'll delve into how Goroutines, channels, and data structures in Golang can be used to tackle the sum of squares problem. We'll also discuss common pitfalls and inefficient practices that can hinder performance. Fear not, my friends, for we shall also unveil secrets to optimizing your code and achieving exceptional performance. So, let's embark on this quest together!

The Sum of Squares Problem: A simple problem

The medium article of my friend addressed the following problem: "6. Write a program that returns the sum of the squares of each element of an array in Golang."

Easy enough. We need to iterate through every element in the array and add the square of each one.

func simpleSumSquare(items []int) int {
	total := 0 // total sum
	for i := 0; i < len(items); i++ {
		total += items[i] * items[i] // square the item and add it to the total
	}
	return total // return the total sum
}

Can we simplify it further? The code is reasonably straightforward. However, can we make it faster?

Goroutines: Harnessing the Power of Concurrency

Goroutines, the superheroes of Golang, are lightweight threads that enable concurrent task execution. By using Goroutines, we can execute multiple functions simultaneously, making our code more efficient and faster. But with great power comes great responsibility.<br /> One common mistake in solving the sum of squares problem is the improper use of Goroutines. Some developers create too many Goroutines without proper synchronization, leading to chaos and incorrect results. Remember, coordination is crucial!

Goroutines: Common pitfall

When solving the sum of squares problem, the choice of algorithm and data structures significantly impacts performance. The aforementioned medium blog post highlights the use of Goroutines and their potential to substantially improve performance.

func sumSquare(items []int) int {
	number := make(chan int)   // channel for sending numbers
	response := make(chan int) // channel for receiving responses

	var wg sync.WaitGroup // wait group for waiting for all goroutines to finish

	total := 0 // total sum

	// Create a goroutine for each item in the slice
	for _, item := range items {
		wg.Add(1)           // increment the wait group counter
		go func(item int) { // create a goroutine
			defer wg.Done()    // decrement the wait group counter when the goroutine finishes
			sum1 := <-number   // receive a number from the number channel
			sum1 = sum1 * sum1 // square the number
			response <- sum1   // send the result to the response channel
		}(item) // pass the item to the goroutine
		number <- item      // send the item to the number channel
		total += <-response // receive the result from the response channel
	}

	defer close(number)   // close the number channel
	defer close(response) // close the response channel

	wg.Wait() // wait for all goroutines to finish

	return total // return the total sum
}

This code creates a new goroutine for each item in the array and then performs the squaring operation. It uses channels to pass values between the goroutine and the main thread.

However, is this code correct? As discussed in <a href={'../mergesort-parallel'} style="font-weight: 700; text-decoration: underline; color: var(--primary);" target="_blank" rel="prefetch noopener">my previous blog post about the mergesort algorithm</a>, there are better practices than spawning numerous goroutines. More than being lightweight is needed to justify using many goroutines; it still uses memory and time, and we must be cautious.

As always, let's benchmark the code!

Benchmarking: The Quest for Ultimate Performance

Benchmarking lets us measure code performance and compare implementations to identify the best.

With Golang's built-in benchmarking tools, we can easily measure the execution time of our code and identify bottlenecks. By tweaking our implementation and experimenting with different approaches, we can achieve optimal performance and revel in our achievements.

Hence, I've created a small function to benchmark our algorithms using randomly generated arrays of varying sizes to observe their performance at different scales:

func benchmarkFramework(b *testing.B, sumFunction func([]int) int) {
	sizes := [][]int{RandomArray(100, 0, 100),
		RandomArray(1000, 0, 1000),
		RandomArray(10000, 0, 10000),
		RandomArray(100000, 0, 100000),
		RandomArray(1000000, 0, 1000000),
	}
	b.ResetTimer()
	for _, size := range sizes {
		b.Run(fmt.Sprintf("%d", len(size)), func(b *testing.B) {
			for i := 0; i < b.N; i++ {
				sumFunction(size)
			}
		})
	}
}

Here are the results:

name	time/op
SimpleSumSquare/100	49.5ns
SimpleSumSquare/1000	450ns
SimpleSumSquare/10000	4.41µs
SimpleSumSquare/100000	43.8µs
SimpleSumSquare/1000000	437µs
.
SumSquare/100	75.2µs
SumSquare/1000	730µs
SumSquare/10000	7.24ms
SumSquare/100000	72.4ms
SumSquare/1000000	739ms

As we can see, the simple algorithm outperforms the complex one! Why? Because spawning too many Goroutines slows down the program and consumes a significant amount of memory.

Data Structures & Algorithms: Choose Wisely

When solving the sum of squares problem, the choice of data structures and algorithms significantly impacts performance.

I decided to develop a more efficient function. Consequently, I used chunks and a Goroutine pool.

A goroutines pool manages a set number of reusable goroutines, reducing overhead in concurrent programs. Chunks break data into segments for parallel processing, optimizing resource use, and enhancing efficiency. Combining these techniques streamlines parallelism, maximizing concurrency benefits.

Here is my code:

func simpleParallelSumSquare(items []int) int {
	const chunkSize = 10000

	if len(items) <= 10000 { // Threshold for small slices
		return simpleSumSquare(items) // Use the simpleSumSquare function
	}

	// Divide the items into chunks
	chunks := make([][]int, 0)
	for i := 0; i < len(items); i += chunkSize {
		end := i + chunkSize // end index for the chunk
		if end > len(items) {
			end = len(items) // last chunk may be smaller than chunkSize
		}
		chunks = append(chunks, items[i:end]) // append the chunk to the chunks slice
	}

	// Create a goroutine for each chunk
	wg := sync.WaitGroup{}
	resultChan := make(chan int, len(chunks)) // channel for receiving results

	for _, chunk := range chunks { // iterate over the chunks
		wg.Add(1)              // increment the wait group counter
		go func(chunk []int) { // create a goroutine
			resultChan <- simpleSumSquare(chunk) // send the result to the result channel
			wg.Done()                            // decrement the wait group counter when the goroutine finishes
		}(chunk) // pass the chunk to the goroutine
	}

	wg.Wait()         // Wait for all goroutines to finish
	close(resultChan) // close the result channel

	// Sum the results
	total := 0
	for partialSum := range resultChan {
		total += partialSum
	}

	return total // return the total sum
}

Can this code be improved further? I could reduce the number of allocations and enhance performance. However, I opted for simplicity to better illustrate pitfalls and good practices related to Goroutines.

Is it genuinely efficient? Let's benchmark it, as always!

name	time/op
SimpleSumSquare/100	49.5ns
SimpleSumSquare/1000	450ns
SimpleSumSquare/10000	4.41µs
SimpleSumSquare/100000	43.8µs
SimpleSumSquare/1000000	437µs
.
SumSquare/100	75.2µs
SumSquare/1000	730µs
SumSquare/10000	7.24ms
SumSquare/100000	72.4ms
SumSquare/1000000	739ms
.
SimpleParallelSumSquare/100	49.5ns
SimpleParallelSumSquare/1000	450ns
SimpleParallelSumSquare/10000	4.41µs
SimpleParallelSumSquare/100000	18.0µs
SimpleParallelSumSquare/1000000	67.2µs

As we can observe, our new code is significantly faster!

Pitfalls and Bad Practices: The Road to Destruction

Ah, the treacherous road of pitfalls and bad practices. One common mistake is the excessive creation and destruction of Goroutines. Creating Goroutines carries a cost; making too many can slow your program and consume unnecessary resources.

Furthermore, we can use unsafe features to improve our code's performance but must not sacrifice maintainability and safety for speed if it's not really required. If you want to learn more about unsafe features available on Golang, you can look at the code published <a href='https://github.com/CorentinGS/go-teaching/tree/main/goroutines_sum_square' style="font-weight: 700; text-decoration: underline; color: var(--primary);" target="_blank" rel="noopener nofollow"> here on my Github </a>.

Conclusion: The Sum of Success

Congratulations, my friends! We've journeyed through Goroutines, channels, and data structures, conquering the sum of squares problem. We've learned from our mistakes, optimized our code, and achieved top-notch performance.

But remember, the pursuit of knowledge is everlasting. Keep exploring, experimenting, and pushing the boundaries of what's possible. And always remember, with great power comes great responsibility. So go forth, my fellow Gophers, may your code be swift, your bugs be few, and your adventures be legendary!

References

<a href="https://medium.com/@ShivamSouravJha/golang-only-things-i-know-for-the-interview-4322d29d67a3" style="font-weight: 700; text-decoration: underline; color: var(--primary);" target="_blank" rel="noopener nofollow">Medium article</a>
<a href="https://github.com/CorentinGS/go-teaching/tree/main/goroutines_sum_square" style="font-weight: 700; text-decoration: underline; color: var(--primary);" target="_blank" rel="noopener">Code snippets</a>

Upgrading to dnf5: A step-by-step guide for Fedora users

Fri, 28 Apr 2023 00:00:00 GMT

Upgrading to dnf5: A guide for Fedora users

:::warning This guide is old and may contain outdated information. :::

Introduction

DNF5 is Fedora's new, faster, and more powerful package manager. Although it is still in development and won't be the default package manager until Fedora 39, you can install it now and start using it. This blog post will show you how to replace DNF with DNF5 on your Fedora system.

What's the difference between dnf5 and dnf ?

DNF is an old, single-threaded package manager with much legacy code. It's written in Python and is usually described as slow by users. DNF5, on the other hand, is a complete rewrite of DNF written in C++. It's multi-threaded, has a better user experience, should be easier to maintain, and is faster.

Why should I upgrade to dnf5?

Upgrading to DNF5 offers several benefits, including improved speed and efficiency. DNF5 is designed to be faster and more efficient than DNF, which can help speed up your system's package installation and update process. Additionally, as it will be the default package manager in Fedora39, starting to use it now and reporting any bugs you encounter will help the developers fix them before the release.

How to upgrade to dnf5?

Step 1: Install dnf5

To install DNF5 from the unstable repository, run the following command:

dnf copr enable rpmsoftwaremanagement/dnf5-unstable ;
dnf install dnf5 dnf5-plugins

If you're using sudo, use this command instead:

sudo dnf copr enable rpmsoftwaremanagement/dnf5-unstable ;
sudo dnf install dnf5 dnf5-plugins

Step 2: Create an alias for dnf5 (optional)

You can create an alias if you want to use DNF5 instead of DNF. Run the following command:

alias dnf="dnf5"

To make this alias permanent, add it to your ~/.bashrc file:

echo "alias dnf=\"dnf5\"" >> ~/.bashrc

Or add it to your ~/.zshrc file if you use zsh:

echo "alias dnf=\"dnf5\"" >> ~/.zshrc

Disclaimer

Remember that DNF5 is still in development and not ready for production use. It may contain bugs and should not be used on production systems. Use it at your own risk. I am not responsible for any damage caused by using DNF5.

Conclusion

DNF5 may still have some bugs, so it's essential to experiment with it and keep this blog post up-to-date with the latest changes. This guide helped you upgrade to DNF5. If you have any questions or suggestions, feel free to contact me on <a href="https://twitter.com/GSCorentinDev" target="_blank" rel="noopener" style="font-weight: 700; text-decoration: underline; color: var(--primary);">Twitter</a> or <a href="https://corentings.dev/discord" target="_blank" rel="noopener" style="font-weight: 700; text-decoration: underline; color: var(--primary);">Discord</a>.

References

<a href="https://fedoraproject.org/wiki/Changes/ReplaceDnfWithDnf5" style="font-weight: 700; text-decoration: underline; color: var(--primary);" target="_blank" rel="noopener">Fedora Project</a>
<a href="https://www.reddit.com/r/Fedora/comments/12jv7uc/what_is_the_state_of_affairs_with_fedora_38_and/" target="_blank" rel="noopener" style="font-weight: 700; text-decoration: underline; color: var(--primary);">r/fedora</a>
<a href="https://github.com/rpm-software-management/dnf5/issues/411" style="font-weight: 700; text-decoration: underline; color: var(--primary);" target="_blank" rel="noopener">Github issue</a>

Merge Sort using Goroutines

Wed, 11 Jan 2023 00:00:00 GMT

Parallel Merge Sort vs Simple Merge Sort

This is a simple example of how to use goroutines to parallelize a merge sort algorithm. We compare the performance of a simple merge sort algorithm with a parallel merge sort algorithm that uses goroutines.

The Merge Sort Algorithm

The merge sort algorithm is a divide and conquer algorithm that recursively splits the input array into two halves, sorts each half, and then merges the two sorted halves into a single sorted array.

To speed up the merge sort algorithm, we use insertion sort for small subarrays (less than 12 elements).

The implementation of the algorithm uses generics to allow sorting of any type of numbers.

func MergeSort[T Number](items []T) []T {
	size := len(items)
	if size < 2 {
		return items
	}

	if size < K {
		return Insertionsort(items)
	}

	middle := size / 2
	var a = MergeSort(items[:middle])
	var b = MergeSort(items[middle:])

	return merge(a, b)
}

The Merge Sort Algorithm with Goroutines

The parallel merge sort algorithm uses goroutines to sort the two halves of the input array in parallel.

To prevent the creation of too many goroutines, we use a threshold to determine when to use goroutines. If the size of the input array is less than the threshold, we use a simple merge sort algorithm instead of a parallel one.

Here we use a threshold of 512 elements. We can benchmark the performance of the algorithm with different thresholds to find the optimal threshold, but we will not do that in this example.

// ParallelMerge Perform merge sort on a slice using goroutines
func ParallelMerge[T Number](items []T) []T {
	if len(items) < 2 {
		return items
	}

	// Use a simple merge sort algorithm if the size of the input array is less than the threshold
	if len(items) < 512 {
		return MergeSort(items)
	}

	// Create the wait group to wait for the goroutines to finish
	var wg sync.WaitGroup
	wg.Add(1)

	var middle = len(items) / 2  // Find the middle index of the input array
	var a []T                   // Create a slice to hold the first half of the input array
	go func() {                // Create a goroutine to sort the first half of the input array
		defer wg.Done()       // Decrement the wait group counter when the goroutine finishes
		a = ParallelMerge(items[:middle]) // Sort the first half of the input array
	}()
	var b = ParallelMerge(items[middle:]) // Sort the second half of the input array

	wg.Wait() // Wait for the goroutine to finish
	return merge(a, b) // Merge the two sorted halves
}

Benchmarking the Merge Sort Algorithms

Now we can benchmark our merge sort algorithms to compare their performance.

To benchmark the algorithms, we use arrays of different sizes and measure the time it takes to sort the arrays.

To run the benchmark we use the following command:

go test -bench=. -benchtime 5s > benchmark.txt && benchstat benchmark.txt

Benchstat is a tool that can be used to compare the results of benchmarks.

The results of the benchmark are as follows:

name                                   time/op
Mergesort/1000                     14.6µs ± 0%
Mergesort/10000                     473µs ± 0%
Mergesort/100000                   6.33ms ± 0%
Mergesort/1000000                  87.4ms ± 0%

MergesortWithGoroutines/1000       18.7µs ± 0%
MergesortWithGoroutines/10000       217µs ± 0%
MergesortWithGoroutines/100000     2.71ms ± 0%
MergesortWithGoroutines/1000000    29.0ms ± 0%

As we can see, the parallel merge sort algorithm is much faster than the simple merge sort algorithm for large arrays.

Why is the Parallel Merge Sort Algorithm Faster?

The parallel merge sort algorithm is faster because it uses goroutines to sort the two halves of the input array in parallel. The simple merge sort algorithm sorts the two halves of the input array sequentially.

As discussed in the previous article, the cost of creating a goroutine can be high. So we should use a parallel merge sort algorithm only when the size of the input array is large enough to justify the cost of creating goroutines.

Conclusion

In this example, we have seen how to use goroutines to parallelize a merge sort algorithm. We have also benchmarked the performance of the merge sort algorithms to compare their performance. Goroutines are a powerful tool that should be used only when the cost of creating goroutines is justified by the performance improvement.

In this example, the code isn't more complex when using goroutines therefore it's worth using them.

Mastering the Worker Pool Pattern in Go

Code

The code for this article can be found on <a href="https://github.com/CorentinGS/go-teaching/tree/main/goroutines_merge_sort" style="font-weight: 700; text-decoration: underline; color: var(--primary);"> GitHub </a>

Goroutine vs Simple function

Wed, 11 Jan 2023 00:00:00 GMT

Goroutine vs Simple function

This is a simple example of why goroutines might be overkill for some tasks and less efficient than a simple function.

Structures

We got a simple structure that contains sensitive information that we don't want to be exposed to the outside world. Therefore, we created a second structure that hides the sensitive information and only exposes the information we want to be public.

// Pineapple is a struct that represents a database object with sensitive data that should be hidden
type Pineapple struct {
	Paro       string `faker:"name"`
	Turkey     string `faker:"name"`
	Banana     string `faker:"name"`
	Age        int    `faker:"number"`
	Size       int    `faker:"number"`
	IsAlive    bool
	ID         uint
	SecretCode []byte
	Created    time.Time
	Updated    time.Time
}

// SafePineApple is a struct that represents a Pineapple object without sensitive data
type SafePineApple struct {
    Paro    string
    Turkey  string
    Banana  string
    IsAlive bool
    Age     int
    ID      uint
}

Conversion

We need to convert our Pineapple object to a SafePineApple object. We can do this by creating a method on the Pineapple struct that returns a SafePineApple object.

// ToSafePineApple converts a Pineapple object to a SafePineApple object
func (p *Pineapple) ToSafePineApple() SafePineApple {
	return SafePineApple{
		Paro:    p.Paro,
		Turkey:  p.Turkey,
		Banana:  p.Banana,
		IsAlive: p.IsAlive,
		Age:     p.Age,
		ID:      p.ID,
	}
}

Use case

In our use case we have an array of Pineapple objects coming from our database that we want to convert to SafePineApple objects and store them in a new array. The order of the objects in the array should be the same as the original array as it has already been sorted by a sql query.

Simple function

We can do this by creating a simple function that takes an array of Pineapple objects and returns an array of SafePineApple objects.

// SimpleConvertPineApplesToSafety converts an array of Pineapple objects to an array of SafePineApple objects
func SimpleConvertPineApplesToSafety(pineapples []Pineapple) []SafePineApple {
	safePineApples := make([]SafePineApple, len(pineapples))

	for idx, pineapple := range pineapples {
		safePineApples[idx] = pineapple.ToSafePineApple()
	}

	return safePineApples
}

This function is very simple and easy to understand. We loop through the array of Pineapple objects and convert them to SafePineApple objects.

Goroutine without mutex

We can do this by using goroutines to work on the array concurrently and store the results in a new array.

func GoroutinesNoMutexConvertPineApplesToSafety(pineapples []Pineapple) []SafePineApple {
	// Create a slice to store the SafePineApples
	safePineApples := make([]SafePineApple, len(pineapples)/2, len(pineapples))
	safePineApples2 := make([]SafePineApple, len(pineapples)/2)

	var wg sync.WaitGroup // Create a WaitGroup to wait for all goroutines to finish
	wg.Add(1)            // Add 1 to the WaitGroup

	// Create a goroutine to convert the first half of the Pineapple objects
	go func(chunk []Pineapple) {
		defer wg.Done() // Decrement the WaitGroup when the goroutine is done
		for idx, pineapple := range chunk { // Loop through the chunk of Pineapple objects
			safePineApples[idx] = pineapple.ToSafePineApple() // Convert the Pineapple object to a SafePineApple object
		}
	}(pineapples[:len(pineapples)/2]) // Pass the first half of the Pineapple objects to the goroutine

	// Convert the second half of the Pineapple objects in the main thread
	for idx, pineapple := range pineapples[len(pineapples)/2:] {
		safePineApples2[idx] = pineapple.ToSafePineApple()
	}

	// Wait for all goroutines to finish
	wg.Wait()

	// Group both pineapples
	safePineApples = append(safePineApples, safePineApples2...)

	// Return the SafePineApples
	return safePineApples
}

This function is a bit more complex than the simple function. We use a goroutine to convert the first half while the other half is handled by the main thread. We use a WaitGroup to wait for the goroutine to finish before returning the results.

Goroutine with mutex

We can also add mutexes to the goroutine to make it thread safe.

func GoroutinesConvertPineApplesToSafety(pineapples []Pineapple) []SafePineApple {
	// Create a slice to store the SafePineApples
	safePineApples := make([]SafePineApple, len(pineapples))

	// Split the offers into chunks
	chunks := [][]Pineapple{pineapples[:len(pineapples)/2], pineapples[len(pineapples)/2:]}

	mutex := sync.Mutex{} // Create a mutex to lock the slice when writing to it

	var wg sync.WaitGroup // Create a WaitGroup to wait for all goroutines to finish
	wg.Add(1)           // Add 1 to the WaitGroup

	// Create a goroutine to convert the first half of the Pineapple objects
	go func(chunk []Pineapple) {
		defer wg.Done() // Decrement the WaitGroup when the goroutine is done
		for idx, pineapple := range chunk { // Loop through the chunk of Pineapple objects
			mutex.Lock() // Lock the mutex
			safePineApples[idx] = pineapple.ToSafePineApple() // Convert the Pineapple object to a SafePineApple object
			mutex.Unlock()  // Unlock the mutex
		}
	}(chunks[0]) // Pass the first half of the Pineapple objects to the goroutine

	// Convert the second half of the Pineapple objects in the main thread
	for idx, pineapple := range chunks[1] {
		mutex.Lock() // Lock the mutex
		safePineApples[idx+len(chunks[0])] = pineapple.ToSafePineApple() // Convert the Pineapple object to a SafePineApple object
		mutex.Unlock() // Unlock the mutex
	}

	// Wait for all goroutines to finish
	wg.Wait()

	return safePineApples
}

This function make use of the mutex to lock the slice when writing to it. This makes sure that the goroutine and the main thread don't write to the same index at the same time but instead wait for the other to finish.

Benchmark

Now that we have our functions we can benchmark them to see which one is the fastest. To benchmark our functions we run them with arrays of different sizes.

Our benchmark function looks like this:

func Benchmark_SimpleConvertPineApplesToSafety(b *testing.B) {
	for _, n := range []int{500, 1000, 2000, 5000, 10000} {
		b.Run(fmt.Sprintf("Benchmark_SimpleConvertPineApplesToSafety-%d", n), func(b *testing.B) {
			pineApples := make([]Pineapple, n)
			var pine Pineapple
			for i := 0; i < n; i++ {
				_ = faker.FakeData(&pine)
				pine.Created = time.Now().AddDate(0, 0, -i)
				pine.ID = uint(i)
				pine.IsAlive = true
				pineApples[i] = pine
			}
			for i := 0; i < b.N; i++ {
				SimpleConvertPineApplesToSafety(pineApples)
			}
		})
	}
}

To run the benchmark we use the following command:

go test -bench=. -benchtime 5s > benchmark.txt && benchstat benchmark.txt

Benchstat is a tool that can be used to compare the results of benchmarks.

The results of the benchmark are as follows:

name                                                                        time/op
Benchmark_SimpleConvertPineApplesToSafety-500-32                          15.8µs ± 0%
Benchmark_SimpleConvertPineApplesToSafety-1000-32                         32.0µs ± 0%
Benchmark_SimpleConvertPineApplesToSafety-2000-32                         66.5µs ± 0%
Benchmark_SimpleConvertPineApplesToSafety-5000-32                          193µs ± 0%
Benchmark_SimpleConvertPineApplesToSafety-10000-32                         465µs ± 0%
Benchmark_GoroutinesConvertPineApplesToSafety-500-32                      23.5µs ± 0%
Benchmark_GoroutinesConvertPineApplesToSafety-1000-32                     46.2µs ± 0%
Benchmark_GoroutinesConvertPineApplesToSafety-2000-32                     87.7µs ± 0%
Benchmark_GoroutinesConvertPineApplesToSafety-5000-32                      242µs ± 0%
Benchmark_GoroutinesConvertPineApplesToSafety-10000-32                     507µs ± 0%
Benchmark_NoMutexGoroutinesConvertPineApplesToSafety-500-32               28.3µs ± 0%
Benchmark_NoMutexGoroutinesConvertPineApplesToSafety-1000-32              48.7µs ± 0%
Benchmark_NoMutexGoroutinesConvertPineApplesToSafety-2000-32               105µs ± 0%
Benchmark_NoMutexGoroutinesConvertPineApplesToSafety-5000-32               257µs ± 0%
Benchmark_NoMutexGoroutinesConvertPineApplesToSafety-10000-32              533µs ± 0%

As you can see the simple function is the fastest.

Why ?

The reason why the simple function is the fastest is that the process of converting the Pineapple objects to SafePineApple objects is very fast. The time it takes to create the goroutines and wait for them to finish is longer than the time it takes to convert the Pineapple objects to SafePineApple objects.

Furthermore, in our goroutines implementation we have to convert the Pineapple objects then lock the mutex, write to the slice and unlock the mutex. This is a lot of overhead for a very simple task.

Conclusion

Don't use goroutines when you don't need them. That may seem obvious, but it's easy to forget when you're trying to optimize your code. In this example the overhead of creating the goroutines and waiting for them to finish is longer than the time it takes to convert the Pineapple objects to SafePineApple objects.

Goroutines are great for tasks that take a long time to complete and can be done in parallel. I would suggest to write the simplest code possible and then benchmark it to see if you can improve it using goroutines instead.

Moreover, simple code is easier to read and maintain than complex code, that's why writing complex code might not be necessary if performance is not an issue. I'd prefer to have a simple function that takes a few milliseconds longer to complete than a complex function.

Code

The code for this article can be found on <a href="https://github.com/CorentinGS/go-teaching/tree/main/goroutines_simple_vs_complex" style="font-weight: 700; text-decoration: underline; color: var(--primary);"> GitHub </a>

How to optimize a Go deployment with Docker

Tue, 08 Nov 2022 00:00:00 GMT

How to optimize a Go deployment with Docker ?

In this article, I will present the different steps that led me to optimize the deployment of services in golang using docker.

A simple Dockerfile

When I started Go and deployed a rest api service for my Memnix application, I used a basic Dockerfile that I had found on the internet.

FROM golang:1.19
RUN mkdir -p /go/src/myapp
WORKDIR /go/src/myapp
COPY archived /go/src/myapp

RUN go get -d -v
RUN go install -v

EXPOSE 8080
CMD ["/go/bin/myapp"]

The problem is that I ended up with a 2GB image while my project compiled locally without docker and optimized was only 10MB! So I decided to try to optimize the size of my Docker image as much as possible.

The first step: using a multi-stage build

It is possible to separate our Dockerfile in several parts and to use several images:

One image to build the project
A minimalist image to deploy it

This method allows us not to keep the sources and all the development tools provided with the golang image. Thus, we only keep the binaries. I decided to use the golang:1.19-alpine image as the builder.

FROM golang:1.19-alpine as builder

LABEL stage=gobuilder
ENV CGO_ENABLED=0
ENV GOOS linux
WORKDIR /build

COPY go.mod go.sum .
RUN go mod download

COPY . .
RUN go get -d -v
RUN go build -o /app/myapp .

This first part of the Dockerfile allows us to copy the sources, to synchronize the packages and to launch the command go build to generate the binary. Once the build is finished, we use a minimalist alpine image that will be used for deployment. We just need to copy the binary from our builder and run it.

FROM alpine:latest

WORKDIR /app

COPY --from=builder /app/myapp .

EXPOSE 8080

CMD ["/app/myapp"]

After testing this new Dockerfile, the final image was 34MB. It's far from the original 10MB but it's still much better than 2GB.

Improve the build process

It is possible in Golang to add flags to the build command in order to slightly optimize the binary size. After a little research on internet, I discovered the existence of "-s -w" flags. After testing them locally, I noticed an improvement of a few MB so I decided to add them to my Dockerfile. I also discovered Upx style="font-weight: 700; text-decoration: underline; color: var(--primary);"></a> which is a small program that allows to compress binary files to reduce their size. I tested this little software in a terminal and I noticed an improvement of almost <b>50%</b> on the size of the Memnix api. So I also added this step to my Dockerfile.

FROM golang:1.19-alpine as builder

LABEL stage=gobuilder
ENV CGO_ENABLED=0
ENV GOOS linux
RUN apk update --no-cache && apk add upx
WORKDIR /build

COPY go.mod go.sum .
RUN go mod download

COPY . .
RUN go get -d -v
RUN go build -ldflags="-s -w"  -o /app/myapp .
RUN upx /app/myapp

FROM alpine:3.14

WORKDIR /app

COPY --from=builder /app/myapp .

EXPOSE 8080

CMD ["/app/myapp"]

And here is my new Dockerfile! It may seem much longer and more complex than the first version but in the end, the result is very interesting. After testing this new Dockerfile, the final image is 16Mb so only 6Mb more than the version without Docker which is almost negligible.

Conclusion

Docker is a great tool to simplify application development and deployment, but it can be problematic if used incorrectly. This experience helped me understand how to better use Docker and how to optimize my images.

Solving the Sum of Squares Problem: Optimizing Performance This article is inspired by a <a href="https://twitter.com/GSCorentinDev/status/1564536030795075585?s=20&t=l2hgvKBPDMVs4pS6um2HeA" style="font-weight: 700; text-decoration: underline; color: var(--primary);">twitter thread</a> and the complete code of my Dockerfile is available on my <a href="https://github.com/memnix/memnix-rest/blob/e8e52b3d10731df5b2767f0d24bcdcb5d13d72e3/Dockerfile" style="font-weight: 700; text-decoration: underline; color: var(--primary);">github</a>. I hope this first article will interest you, don't hesitate to give me feedback so I can improve for the next articles and share your tips on Docker or Golang!

Corentin GS's Blog

Tokyo in the Browser: City, Cognition, and Japanese Visual Culture

Tokyo in the Browser: City, Cognition, and Japanese Visual Culture

A Japanese Website Is Not a Poster

The City as Interface

From Street Signs to Screen Modules

Attention Is Culturally Trained

The Physical Environment Trains the Eye

Manga, Magazines, Flyers, and Dense Surfaces

What This Explains About Japanese Websites

But Density Is Not Automatically Good UX

Good Density Needs Hierarchy

From City to Trust

Sources and Further Reading

Why Japanese Websites Look So Overloaded, and What the Density Is Doing

Why Japanese Websites Look So Overloaded, and What the Density Is Doing

"Bloated" Is the Wrong Word

Western Minimalism Is a Design Lens, Not a Universal Standard

The Japanese Homepage as Service Counter

What Research Says About Culture, Attention, and Web Design

The Empirical Web-Design Evidence

What Japanese Layout Standards Reveal

So Are Japanese Websites Actually Cluttered?

A Better Question for Designers

Sources and Further Reading

TDD Isn't About Bugs — It's Your Permission to Refactor

TDD Isn't About Bugs — It's Your Permission to Refactor

The Moment It Clicked

The Trap: Testing Structure

The Fix: Test Behavior

Going Deeper: Value Objects

The Payment System

The "TDD Is Slow" Myth

When TDD Doesn't Help

Monday Morning

Docker 29 Broke Traefik — Here's the Fix (and Why It Happened)

Docker 29 Broke Traefik — Here's the Fix (and Why It Happened)

The "Everything Is Down" Moment

The 30-Second Fix

Verifying it worked

The Real Fix: Update Traefik

If you can't update Traefik directly

What Actually Happened

Docker's API version floor

Why Traefik specifically exploded

It's Not Just Traefik

Timeline

Takeaways

For your infrastructure

For the broader picture

Why I Built ZaString

The Problem With Strings

Enter Span<T>

The Design Philosophy

What I Learned

Generic Methods Coming to Go

Generic Methods Coming to Go

The Status Quo: The Anti-Pattern We All Live With

Why This Became Possible

The Syntax

The Boundary: What This Does NOT Do

The Case Against

Practical Examples

Type-Safe Builders

GroupBy: A Method-Level Type Parameter

The List[E].Format[F] Pattern

How to Prepare Today

When Will It Land?

My Take

I Write, I Code, I Explore — Why Verbs, Not Nouns

Go Pipeline Pattern: Turning Streams into Useful Data

Pipeline Pattern in Go

Introduction

When to Use

Why Use It

How It Works

Simple Example

Real-World Example: Log Processing Pipeline

Error Handling

Cancellation

The `List[E].Format[F]` Pattern