Deploying to Prod

November 20, 2018

This is mostly a post for myself, to mark a small milestone.

The work I do at Grubhub involves supporting our Care team. Our cadence for deployment aims to have a new version go out to prod once every week. While we’re developing features and fixing bugs, we build to our preprod environment. But when we go out to prod, we update the tool that all of our Care Agents (in multiple cities, in multiple countries) use.

This week, I did my first prod deploy. The act itself was fairly basic, as our SRE Sina made things super easy for us. All I really did was trigger the job, entering in a new version and old teardown version.

But all the work leading up to the deploy was the challenge – ensuring all the work that was going in was verified, trying to pin down a good version to go with. Some work is deemed critical (with other teams relying on the functionality). And some other work may be higher risk (changing important, existing functionality).

Up until this week, I served primarily as a contributor. But being the guy who pushed the button, there was a definite shift in the way I looked at things.

There’s such a thing as a “rollback” if a deployment goes bad (if our users discover errors, or functionality breaks to where they can’t do their jobs). So in a worst case scenario, we roll back to a prior, stable version. But doing so indicates a failure in our process: we didn’t test enough, or we didn’t verify enough, to have caught the issue in preprod before it rolled out to prod.

While rollbacks are a possibility, you always want to avoid one if at all possible. It’s not necessarily evidence of failure, but to me it’s close – it’s an indicator that more could and should have been done, but did not.

For most of my time at Grubhub, Sajit’s been the primary person doing our prod deploys. And this week, I’m proud to have joined that club and participated. And I look forward to doing this more, as I’m joining a few others in taking on this weekly responsibility.

It’s daunting and stressful, but I’m proud of our team and the work we do. And take a lot of pride in the work we sent out into the world.

Again, this is mostly a blog post for me, marking the occasion. In an ideal scenario, when we do a deployment to prod… I’m hoping the Care Agents barely notice. If we’ve done our job right, they just keep on working without missing a beat.

grubhub

« Battery Security at Walgreen’s

Prepping the Ceiling for Drywall »