If you’ve never been on-call before, it’s easy to imagine production incidents as scenes from the movies.
A few engineers typing furiously.
Some dashboards flashing red.
A quick fix.
Everyone goes back to sleep.
Reality is… considerably less glamorous.
My first real production incident happened in the middle of the night.
And it taught me more about engineering than an entire semester of university ever did.
The Phone Call Nobody Wants
It was around 3 AM when my phone rang.
You don’t get many calls at that hour.
Certainly not from work.
Half asleep, I answered.
“Can you join the Emergency Incident Management bridge? We think one of your pipelines is causing a production issue.”
Nothing wakes you up faster than hearing your name attached to a production incident.
Within a couple of minutes, I had my laptop open and joined the call.
There were already around 30 people on it.
Engineers.
Managers.
Site Reliability.
Operations.
Representatives from other teams.
Somebody from incident management trying to coordinate everything.
Everyone was talking at once.
That was the moment it hit me.
This wasn’t just my problem anymore.
My Jobs Had Brought the Cluster to a Standstill
The issue wasn’t that my jobs had failed.
In some ways, that would’ve been easier.
The problem was that they weren’t finishing at all.
They entered the processing state…
…and simply stayed there.
Forever.
These were scheduled jobs running periodically through cron.
Every few minutes, a fresh batch would start.
The previous batch was still running.
Then another one started.
Then another.
Soon dozens of my jobs were sitting there consuming CPU, memory, and executors on a shared compute cluster.
Eventually, the inevitable happened.
Other teams’ workloads couldn’t get resources anymore.
Their jobs started queueing.
Then timing out.
Then failing.
One pipeline had quietly become everyone else’s problem.
”Can Someone Just Kill Them?”
That was the obvious question on the call.
And surprisingly…
The answer was no.
Those jobs processed financial data.
Stopping them blindly could leave transactions half-processed or create inconsistencies that would take days to reconcile.
Only the owning engineering team—that is, me—could decide which jobs were safe to terminate.
I remember staring at dozens of running jobs and realizing that nobody else in the room could make that decision for me.
For the first time in my career, I wasn’t just responsible for writing the software.
I was responsible for deciding what happened when the software misbehaved.
That’s a very different feeling.
The Textbook Ended There
Up until that point, everything I’d learned about disaster recovery came from documentation.
Articles.
Architecture talks.
Internal playbooks.
They all made perfect sense when you were reading them with a cup of coffee.
They feel very different when thirty people are waiting for you to say something.
Step One: Stop Making the Problem Bigger
The first decision was surprisingly simple.
Don’t think about fixing the existing jobs yet.
First…
Stop creating new ones.
We immediately disabled the scheduling process so that no new cron jobs could start.
That sounds obvious in hindsight.
But during an incident, it’s incredibly tempting to jump straight into debugging.
Instead, the priority is to stabilize the system.
If the bathtub is overflowing…
…you turn off the tap before you start mopping the floor.
Step Two: Buy Everyone Else Some Breathing Room
Once new jobs had stopped entering the system, we classified the existing ones.
Not every workload had the same business impact.
Some jobs were mission critical.
Others could safely be rerun later.
We grouped them into tiers.
The non-critical ones were terminated to free cluster resources.
Critical workloads were either migrated elsewhere or executed manually in interactive mode where possible.
Slowly…
The queue began to clear.
Other teams’ jobs started running again.
One by one, the dashboards turned green.
It wasn’t over.
But at least the fire had stopped spreading.
Step Three: Tell People What Was Happening
One thing that surprised me was how much of incident management had nothing to do with code.
While part of the team investigated the issue, another responsibility was communication.
Downstream teams needed to know:
- Which systems were affected.
- Which datasets might be delayed.
- Whether they should pause their own processes.
- When to expect updates.
Silence creates confusion.
Clear communication creates trust.
I hadn’t appreciated that before.
The Real Investigation Began Afterwards
Once the platform was stable, we finally had time to answer the most important question.
Why did perfectly healthy jobs suddenly stop finishing?
Nothing had changed in my code.
Nothing had been deployed.
Nothing obvious had happened.
Which made the incident even stranger.
Eventually, we traced the issue to an upstream dependency.
Another team had modified one of their services and, as part of that change, removed the timeout on an endpoint my jobs depended on.
My pipeline assumed that endpoint would eventually time out if something went wrong.
Instead…
It waited forever.
Every new job inherited the same fate.
And because my scheduler didn’t know previous executions were still running…
It happily kept launching more.
The Incident Was Over. The Engineering Had Just Begun.
Fixing the immediate problem wasn’t enough.
We asked ourselves a different question.
“How do we make sure this entire class of failure never happens again?”
That led to several changes.
We introduced independent timeouts instead of relying on external systems.
We prevented new scheduled executions from starting if a previous run was still active.
We made the jobs idempotent so retries became significantly safer.
And finally…
We built alerting and notification systems so we’d know something was wrong long before another engineer had to call me at 3 AM.
None of those changes fixed that incident.
They prevented the next one.
Engineering Looks Different in Production
College teaches you how to build software.
Production teaches you how to operate software.
Those are related skills.
They are not the same skill.
Writing a pipeline is engineering.
Recovering a failing production system while dozens of people wait for decisions is also engineering.
Designing for failure is engineering.
Communicating during uncertainty is engineering.
Learning from incidents is engineering.
Nobody had told me that before I joined the industry.
Looking Back
People often ask what the biggest difference is between college and industry.
I don’t think it’s the technology.
It’s the responsibility.
In college, if your program crashes, you lose a few marks.
In production, if your system stalls, entire teams can be blocked from doing their work.
That’s intimidating.
But it’s also what makes this profession so rewarding.
Because over time, you stop measuring your growth by the number of programming languages you know.
You start measuring it by something else.
How calmly you can think when everything around you is on fire.
Every engineer remembers their first production incident.
Not because it proves they can write code—but because it proves people are depending on that code.
The day you’re responsible for keeping a system alive is the day you truly begin learning what software engineering is.