When the Light at the End of the Tunnel is an Oncoming Train: Notes from an Applied Scientist

Hi! I've been working as an Applied Scientist in the Last Mile team at Amazon for the past two years. I'm passionate about building algorithms that drive real-world impact and equally bullish on not breaking stuff.

A lot of my work has been in complex multi-stage, multi-stakeholder systems. The problems are high-impact, messy and ambiguous. At times, it has felt like flying but through a fog with just 5 metres of visibility.

In this post, I want to share some of the lessons I've learned along the way, the kind of things I wish I'd known when I was just starting out.

Disclaimer: All opinions are my own and not my employer's.

Anchoring to the Truth

Early in my career, if something failed, my first instinct was panic. I would scramble to fix it before informing my manager or stakeholders. Sometimes it was my fault (like the cardinal sin of writing '=' instead of '==' in an if-else). Other times, there were problems in other dependent systems which were completely out of my control.

This worked for a while, I thought I was maintaining my reputation of always doing faultless work. But after 8 weekends in a row spent firefighting, I realized something was off (really took me that long, please don't judge). With each passing weekend, my productivity and my mental peace took a hit. In my quest for perfection, two things were happening:

1. I was setting unrealistic time estimates
2. Systemic problems were flying under the radar because my "brilliance" was covering them up.

Of course, my manager had no visibility of my weekend shenanigans and from their perspective, work that actually took 7 days, was getting wrapped up in 5.

Eventually, I started being transparent and upfront about everything to my manager — what was breaking, where the dependencies were and what my real timelines looked like. We were able to re-calibrate my work, adjust timelines and even push back with stakeholders as needed. (At that point, I didn't even know pushing back was an option.)

I was back to working only on weekdays and happier. It made me realize the only way to stay sane is by anchoring to the truth, it really does set you free.

Ask the Oracle

When you have bandwidth for only N tasks at a time and there are N+10 things to get done, hyper-prioritization is key. As an individual contributor, it's easy to get tunnel vision because most of your focus is on implementation. Your manager, however, has more context and better visibility into what truly matters for the business.

Treat your manager as an oracle and ask them to rank stuff.

That way, even if something fails or timelines slip, you're aligned on what truly matters. It takes the pressure off trying to do everything at once and helps you avoid spreading yourself too thin. Much like a CPU, while you can learn how to "pipeline" tasks, you can focus on only one task at a time. During production season, it'll keep you sane and stakeholders happy. As a scientist, I also feel it's important to work with your manager to stitch together a few weeks where no one bothers you, so you can explore deeply. I'm still learning how to do this well.

Think Systems, Not Just Models

A model is never just a model, it's part of an ecosystem.

Improving precision-recall is nice, but if it doesn't move metrics on-ground, the model adds no real value. It helps to understand not just the algorithm but the system it lives in.

Take timelines for example. If training your ML model takes 2 days, but production deployment takes 7 due to dependencies you don't control, the true estimate is 9 days, not 2. Unless you account for the entire system, you'll always be chasing unrealistic deadlines.

I've also learnt not to spend weeks polishing every edge case before launching even once in production. In one project, we discovered 30% of the issues surfaced only after we did our first production ingestion. Some had nothing to do with ML at all, just downstream systems not being ready.

Beta testing reduced this, but given the complexity of systems, it's often too expensive and time-consuming to replicate everything exactly. So Beta and Prod were never identical. Testing end-to-end system behaviour early on realistic samples has helped avoid surprises. Launch and iterate.

Finally, focus on heavy hitters. On the eve of a code freeze deadline, we discovered that our model was failing for certain inputs in our offline testing. We obviously did not have time to re-train. Instead, we blocked those cases to ensure no regressions and deployed in production. Not perfect, but it worked and unblocked the business.

When you can't solve everything, solve what matters most.

Write Less Code

Every line of code that you write needs to be reviewed, tested, and waiting to betray you in production on some obscure edge case. The less you write, the less there is to maintain, debug, or break.

Writing less doesn't mean cutting corners, it often means more work. It means reusing existing code, building good abstractions, and resisting the urge to duplicate blocks just because refactoring feels like a hassle. You won't always have the luxury of perfect cleanup while a project is moving fast, but make it a habit to revisit and tidy things up. Your future self (and your teammates) will thank you when the next project builds on top of your code.

At scale, the abstraction-inheritance mindset becomes even more important. Strive for model parity across regions, but accept that ML models don't always capture the long tail. Sometimes you'll need geography-specific tweaks. Planning with reuse and expansion in mind from the start makes those exceptions easier to handle.

Ugly Code > Missed Deadlines

I love writing code. I love writing code for the sake of writing code. But a tough pill to swallow for nerds like me: no one cares about how elegant your code is if it takes weeks to deliver when the business needs it yesterday. Slightly messy code that works is infinitely better than beautiful code that delays timelines. I'm not suggesting that we abandon principles, but it is important to remain aware of what keeps the system afloat. Now, if years of spaghetti code is dragging things down, call it out and make sure the cost of fixing it is accounted for.

Early on, when I gave timelines, I'd be hyper-optimistic and leave no room for things going wrong. My "buffer" was basically the weekend. Unsurprisingly, that wasn't sustainable. What's helped is building in realistic estimates and asking myself: what's the absolute minimum version of success here? Sometimes that's a working prototype instead of a polished system. That shift alone has saved me a lot of stress and cut down on unnecessary perfectionism.

Communicating in a Multi-Stakeholder World

Write It Down, Once

You simply cannot verbally explain the same thing over and over. Like a good stock investment, your work should keep adding value even while you sleep and the way to do that is through documentation. A clear doc with examples goes a long way. Examples, in particular, hit like a truck.

Writing things down takes time you could spend improving the solution, but in a multi-stakeholder world where ambiguity is high, transparency comes first. A solid doc helps everyone up and down the chain focus on what really matters. I've lost count of the times a well-chosen example turned a heated debate into alignment. There will still be disagreements, but grounding them in the end goal clears the fog fast.

Small Rooms, Big Wins

In multi-stakeholder projects, people are stretched across multiple threads. They don't always have time to dive into all the details, and percentages or summary stats can mislead when ambiguity is high. I've found it helps to first align in smaller groups before taking things to a broader forum. It avoids miscommunication, reduces delays, and makes conversations more objective.

And through all of this, I've learned that business problems will come and go, but the relationships you build with your co-workers are permanent. No one wants to work with someone unpleasant, so be nice!

---

Acknowledgments

Grateful to Piyush Joshi and Minaxi Goel for helping me when I was spiralling. Shoutouts to Sai Srinivas, Hari Chandana, Pranoy, and Nitin for keeping me sane. And of course, my family, without their support, none of this would have been possible.

---

Originally published on Medium