Facebook introduced me to a new role at its London DevOps Meetup earlier this month, described as “Production Engineers” in their view this is making DevOps possible.”
This wasn’t all Facebook taught us. The way they do devOps is incredibly unique in the industry, just by sheer scale. When they talk the number of servers required to do something, they don’t talk about hundreds or even thousands. They talk in terms of hundreds of thousands.
They left us with five key recommendations for optimizing DevOps, which I will leave for the end of this post.
Getting bits from developers
Mark Drayton, product engineer for Facebook’s web foundation team, presented for Facebook.
Facebook practices true continuous delivery, weekly shipping code three times in a single day!
Mark said the engineers on Facebook focus on the PHP code they write, not on the hardware or other peripherals. Here’s how they build and deploy a release:
There is a single codebase. According to the developer workflow, developers check out code locally and each engineer has a way to test fully on a development server. They can iterate until they are happy, and then they can make a proposal for code changes.
Others review the updated code, then they sync a number of tests. They use Performance Lab to test the performance. Only then do they commit code.
Committing a trunk where multiple developers commit changes, they cut a new branch once a week, and test for two days, relying on employee testing and automation. Once it passes here, they merge more code into that branch. Then the code moves to release.
Having a single branch represents a commitment that “this can ship.”
The fact that engineers have to be responsible for their code is really key to the success and reliability of what they do. By not just passing code to QA, Facebook embodies the true ethos of DevOps.
Oh yes, I did promise to deliver five recommendations. So here are the five takeaways Mark left with us:
- Release often: Frequent pushes are often great for users.
- Get new features out: Frequent releases often get new features out into production. This is fantastic for customers, and for engineers.
- Standardize a single code base: Keep the process simple.
- Automate for scaling: Rely on testing and automation.
- Put engineers on the hook: Make engineers responsible for code push.