Key Technical Lessons from Production Systems

Reflections on hard-won lessons from building and maintaining production software.

This post was originally published on Medium.

The Gap Between Learning and Shipping

The first time you push code to a real production system — with real users depending on it — you realize how different it is from building side projects. The lessons that follow are not from documentation. They come from things that broke at the wrong time, in ways you didn't expect.

1. Observability Is Not Optional

The most expensive bugs are the ones you don't know about. Before shipping any feature, ask: how will I know if this is broken?

Logging at the right level, emitting meaningful metrics, and adding tracing to critical paths are not "nice to haves" — they're what lets you sleep at night. A bug you can diagnose in ten minutes is ten times less costly than one that takes two days to reproduce.

2. Boring Code Ships Faster

There's a temptation, especially early in a career, to reach for the most clever or elegant solution. Production systems are not the place for it. The code that has worked for years is the code that was easy to understand, easy to debug, and easy to change.

Write code for the engineer who will debug it at 2am. Often, that engineer is you.

3. The Spec Is Always Incomplete

Requirements will be missing edge cases. Business rules will contradict each other. Stakeholders will discover what they actually want only after seeing what they asked for.

Your job is not to implement the spec literally — it's to understand the intent well enough to ask the right questions before writing code. The five minutes you spend clarifying a requirement saves hours of rework.

4. Tests Are Documentation

A well-written test suite tells the next engineer (and future-you) exactly what the system is supposed to do. This is more valuable than comments, more reliable than docs, and more durable than the memory of the original developer.

Write tests that read like specifications. Name them after the behavior, not the implementation. Make failures informative.

5. Performance Is a Feature

Users don't articulate "this page is slow" — they just leave. Performance is invisible when it's good and catastrophic when it's bad. The time to care about performance is before you have 100,000 users, not after.

This doesn't mean premature optimization. It means understanding where your performance budget is going, measuring before you optimize, and not shipping code you know is quadratic.

6. Communication Is an Engineering Skill

The engineer who can clearly explain a technical trade-off to a non-technical stakeholder is more valuable than the one who can't. The engineer who writes a clear PR description saves reviewers hours. The engineer who documents decisions saves future engineers days.

Code is communication between humans. Take it seriously.

These lessons took years and real mistakes to internalize. None of them are original — experienced engineers will nod at all of them. But there's a difference between knowing something and having it burned into your intuition. Production does the burning.