We’ve all heard about 10x engineers. Whether real or mythical, 10x engineers are supposedly 10 times more productive than their regular peers. The internet is full of advice on how to become a 10x engineer and most of them include tips for individuals on how to improve their expertise or mindset.
Brandon Bayer from Fligtcontrol offered a somewhat different perspective at the Shift conference in Miami in his talk on 10 ways to improve how you ship software. His thesis is that you can become a 10x developer by making 10 other people 10 % more productive.
Bayer shared 7 general principles and 10 tactics on how to achieve that.
The principles are:
Principle 1: Speed
Developers are speed junkies. If they find a tool that’s 10ms faster, they would almost want to rewrite the entire application in it. But when it comes to how often they deploy code, they tend to be much more cautious. They would always rather deploy tomorrow, or next week…so they don’t have to stay late today to fix whatever problems might happen.
When it comes to deploys, speed is safety and small is speed.
Speed of shipping has massive compounding effects. If your deploy takes hours, you’re going to want to make dang sure there’s no bugs, because if you find a critical bug right after a deploy goes live, then you have wait another couple of hours for your fix to go live.
Aim for speed in every part of the software lifecycle:
• Fast feedback loops
• Fast builds
• Fast tests
• Fast deploys
• Fast fixes
• Fast rollbacks
Principle 2: Reliability
Your systems must be dependable. Flaky CI/CD issues are one of the worst possible things in the universe. Avoid flakiness at all costs.
Principle 3: Reproducibility
In order to build and maintain complex systems, we need to know that if we do X, we get Y, every time. If one time you do X, you get a Y, but then another time you get Z, you will lose all confidence in your systems.
Use scripts & code to control everything you do instead of manual clicking, manual commands, etc.
Principle 4: Calm On-Call Rotations
Calm on-call rotations are important because people can usually put up with stress during the workday, but if creeps up in the rest of their life…that’s when they start updating thei resumes!
Principle 5: Easy to debug
Everyone causes bugs. Even seniors, even when there is 100% test coverage. You will ship bugs to production, so you must be prepared to fix production bugs. If it’s hard to debug and slow to deploy fixes, it’s going to slow you down because you are going to add lengthy QA processes and before you know it, you’ll be deploying code only once every few weeks.
But even with lots of QA, you’ll still have production bugs. Production is the only tried and true way to find all the bugs. So our systems must be easy to debug.
Principle 6: Self-serve infrastructure & deployments
The old way of doing infrastructure and code deployments was you had two separate teams, Devs and Ops. If devs needed a new database, they would fill a ticket with Ops.
If they needed a new service, file a ticket with Ops.
Need to deploy your code? File a ticket with ops
This way led to devs often being blocked and waiting on ops to get to some ticket, devs not monitoring infra stuff, and not getting feedback if their code was fast or slow, efficient or inefficient.
As an industry, we figured out it’s much better for devs to deploy and run their own code.
This is more efficient, they get stuck less, there is less communication overhead, and devs get more feedback from the real world on how their code is performing.
Principle 7: Ship on Fridays
Healthy engineering organizations deploy multiple times per day, including on Fridays!
If you are afraid to deploy on Fridays, it’s likely because:
• You don’t have reliable deploys
• You have slow deploys
• You don’t have good monitoring and alerting
• Your app is hard to debug
• You don’t have tests
Those are all things you must fix.
Brandon then shared 10 practical tactics to use to become a 10x developer:
1. Decouple deploys from releases
Fundamentally there are two possible actions for changing code in production: Deploys and Releases. Releasing refers to the process of changing user experience in a meaningful way. Deploying refers to the process of building, testing, and rolling out changes to production software. The traditional process is that you change the code on a branch, and when it’s ready, you merge and deploy it. Once it’s deployed, users will see the new code.
But today’s modern engineering organizations use feature flags. What is a feature flag?
It allows you to write new code alongside the old code, and then based on a runtime feature flag your app decides to run the old or new code. This is HUGE.
Remember, speed is one of the key principles! Separating deploys from releases speeds up both. Because engineers can deploy their code as soon as it’s ready without waiting on product management to be ready to release it.
It also speeds up releases, because now you can release a feature instantly. And rollback a feature instantly, without needing a deploy. Product managers can do releases on their own schedule.
Then even more advanced things are possible, like progressive rollouts, where you enable the feature flag for a percentage of your users. So, you could start out with 5 % and if everything is working, then increase to 10 %, 20%, etc. This is the best way to roll out changes because if there are problems, only a subset of your users will experience them, instead of all of them.
Feature flags are far superior to long-lived feature branches. You have less merge conflicts, less stale code, more team momentum, better team morale, and more flexibility for testing things with real users.
2. Continuous deployment (CD)
Deployments are the heartbeat of an engineering organization. And just like a human heart, you want a steady rhythm of simple heartbeats. With this steady rhythm of deployments, you get a momentum. And a momentum is life for organizations.
And along with momentum, you get efficiency of being able to deliver code to production as quickly as possible after it’s written.
You want to deploy as often as possible, lmultiple times per day. Every commit should be deployed. Small, understandable changes mean you have debuggable and understandable deploys.
The engineer who writes the code should merge their own PR and then monitor production for any unexpected issues Feature flags, are a prerequisite for this, because they allow your team to ship small changes continuously instead of having long-lived feature branches.
3. Set up notifications
Make sure you set up notifications for the following
• Failed builds
• Failed deploys
• Service downtime
• Unhealthy servers
• Unexpected errors
• Unusual amount of traffic
• Statuses of third-party services
⁃ Many have a status page that you can subscribe to in Slack
The benefit is relatively obvious, but it does take at least a bit of effort to set them up, so this is something many teams can add as a quick win.
And if you already have notifications & alerts, you should audit them and ask these 3 questions:
• Are they useful?
• Are you getting too many false alerts so that you ignore them?
• Are there things you should add new alerts for?
4. Optimize deploy speed
Your target for a happy team is 15 minute deploys or less. Significantly longer than this is significantly painful. But significantly faster than this could take more work than it’s worth, depending on your circumstances.
There are several ways to improve your speed of deploys:
a. Track and measure your current deploy times
You want to understand your starting point, and which parts of the process are taking a long time. Those will be where you want to focus on improving
b. Speed up dependency installation
This step is often one of the longest segments of a deploy, so it can be a good first step
1. Switch to PNPM
If using JavaScript, switch to pnpm which is a drop-in replacement for npm & yarn that has significant performance & caching improvements
2. Cache the entire install step
Use your build system to totally cache this step, so this step doesn’t even run if your dependencies haven’t changed.
c. Improve your build speed
The main way to do this is switch to a faster bundler.
If you are using create-react-app which uses webpack, you can switch to Vite which is way faster.
If using nextjs, upgrade to the latest and make sure you don’t have a .babelrc file, this will use SWC instead of babel which is a lot faster
If you like living on the edge you can use turbopack by passing the –turbo flag to next dev but it’s only available for dev, not production builds yet
d. Set up build caching
If you are using Docker, there are several things you can do.
Optimize your build layers. Use multi-stage builds. Set up remote caching.
Locally docker caches by default and is fast, but in CI you don’t have a permanent machine. So you have to set up caching yourself. Refer to the docker docs.
e. Caching from the future
If you want to really live with your hair on fire, in a good or a bad way, there is another style of caching that can give you some big wins, if you have a javascript project. Caching from the future works by having a single shared cache between your entire team and also CI.
This means if you build a commit locally and then push that commit to CI, the CI build will just download the cached build that you made and be done in mere seconds.
Or if someone else on your team already build a commit, and then you run the build locally, it will again download the cache and be done in mere seconds instead of building from scratch again for you.
5. Replace staging with preview environments
Preview environments are temporary environments, typically tied to the lifecycle of a pull request. When you open a pull request, infra can be automatically provisioned for that PR.
This enables stakeholders to easily see the changes in a production-like environment. And then when the pull request is merged or closed, it’s environment will be automatically cleaned up.
They are a companion to feature flags. Larger changes should use feature flags and will often have many PRs for that feature. But small changes are often easier to re‐ view with a preview environment instead of going the hassle of managing a feature flag for it.
Preview environments are usually a better replacement for long-lived staging environments because staging is running all the time whether it needs to or not. And you have to merge a PR before anyone can verify the change.
6. Infrastructure-as-code
Hopefully you already have automated deployments, but you might not have automated infrastructure. Without infrastructure-as-code, you typically define your infrastructure config by clicking through a dashboard.
Bringing automation to your infra config in the form of infrastructure-as-code has a huge number of benefits
• Repeatability / reproducibility
• Consistency & standardization – Predictability
• Version controlled
• Collaboration & review
7. Platform Engineering
Back in the very old days, you had to rack your own servers in a data center. The abstraction level of infrastructure was extremely low. Then came AWS which introduced cloud and raised the abstraction level. But it was still far too low for the average developer.
This led to the creation of Heroku, to the glee of developers all around the world. But that excitement was not to last, because Ops were not happy. And it turns out ab‐stractions like Heroku do not scale for large companies.
So Ops takes over and tries creates a new thing between AWS & Heroku, infrastructure-as-code and Terraform. This works very well, but developers were not happy again.
Additionally, company leadership wants to increase operational efficiency, and one way to do that is provide devs with self-serve infrastructure, and so we started creating internal platforms that provide a better developer experience. Through that, the industry found a way for both ops & devs to be happy. And it has become known as platform engineering.
Now many companies are building some type of internal platform, sometimes known as internal developer platform.
This platform can look more like an internal Heroku or more like just Terraform.
The 3 key concepts are:
1. Deploys to your own AWS/GCP account
2. Self-serve infra with good developer experience
3. Ops can dictate and customize the underlying primitives
Three of the main tools are Backstage, Humanitec and Flightcontrol.
8. Work as a TEAM
Small team (2-6 people, 4 is ideal)
• Entire team works on 1 project at a time, together
• Kickoff meeting (deep discussions on how to build)
• Break project into small tasks (usually in kickoff)
• Work on subtasks in parallel
• Small PRs, at least 1 per day
• Quick reviews / don’t be a blocker
• Unblock yourself – merge your own safe PR if no one is available to review and you are blocked
9. Trunk based development
Get rid of pull requests, commit straight to the main branch. It’s called trunk based development.
Think of it like a cafeteria. Your tray is the main branch. When burger is done, it goes on the tray. When fries are ready, on the tray. When milk shake is poured, tray. Once the tray is full, someone calls your number.
That’s how trunk-based development works. Every feature goes straight to main when it’s ready. Subtask or entire project doesn’t matter because they’re all fully working, independent, and deployable.
If committing straight to main is too radical for you, then do the next best thing which is short-lived branches. PRs shouldn’t last more than a day maximum. For a project, you should have many short PRs that are quick for others to review and are easy to merge. That’s how you get momentum.
10. Eat healthy & exercise
We all need this reminder, being hunched over computers all day. Make sure you take care of yourself.
