Accelerate
Troubleshoot
Observe
This is a cautionary tale about the consequences of not investing enough in Developer Experience, what to do about it if you find yourself having to address those consequences, and how to avoid getting into this situation in the first place.
In a recent article (September 2022), Eugene Kim of Insider reports that Amazon CEO Andy Jassy is trying to fix a crumbling engineering culture with a new unit called Amazon Software Builder Experience (ASBX). The ASBX mission is to tackle “foundational pain points” raised by the company’s frustrated developers.
Despite attempts to market the new team as a proactive step to “help Amazon become Earth’s best employer for software builders,” its launch comes after internal complaints about an apathetic engineering culture and bureaucratic inefficiencies.
According to an Insider source, since launching, the ASBX team has grown to more than 400 employees, working on code automation, improved developer tools, and enhanced tutorials and safety infrastructure. Further, its charter is to improve the experience of software builders across Amazon. And its initial focus will be on some of the foundational pain points reported by developers, such as how code is managed and deployed daily and the usability of operational tools.
Developer Pain at Amazon
Recent internal surveys, viewed by Insider, quantify some developer pain and process waste:
- 34% of the engineers said they spent 4 to 8 hours on undifferentiated (i.e., non-value-added) efforts weekly.
- 10-20% of engineering time is spent doing things unrelated to building new products.
- 30% of engineering time is spent performing “repetitive tasks.”
Additionally, sources within Amazon have expressed directly to Gradle their frustration with developer tooling. Developers shared that:
- Internal tools often are not compatible with open-source tools like the Gradle Build Tool and that these tools are slowing down developers. Even worse, this has been tied directly to attrition.
- Many tools are antiquated. Development teams want to use state-of-the-art technologies like Develocity, but often can’t because engineering teams are too removed from the management team that is exclusively empowered to buy software.
- There is no observability tooling that can directly measure developer pains (e.g., feedback cycle time), especially with respect to local builds.
- Trend data on build performance and failure rates is not tracked consistently, so there is no baseline for measuring pain and no way of knowing if key productivity and developer experience metrics are improving or worsening.
The Cost of Inaction
So, what are the reported consequences of a poor developer experience that have precipitated Amazon’s response?
- Many longtime executives responsible for the early company growth have moved on to smaller startups or competitors.
- The Amazon culture known for speed and intrapreneurship is being eroded.
- A significant remedial investment, including the formation of the new 400-person organizational unit to solve the problem, could have been avoided if DevX was identified earlier as a foundational priority.
- Time spent performing mundane, manual tasks is crowding out core productive coding time, including time spent being creative and innovative.
- The CoI (Cost of Inaction) for Amazon is in the tens of millions of dollars, if not hundreds.
The Road to DevX Redemption
Amazon looks well on its way to addressing their challenge. They took the first step, which as always is to admit there is a problem. But Amazon also clearly sees the opportunity to turn a current weakness into a strength. With that goal in mind, they took the next step which is to secure management support and sponsorship at the highest level, as well as the necessary resources. Jassy has personally seen to all of this. The last step is to have a plan and execute it. The ASBX team’s plan includes a unit dedicated to “Build Tools.” This is important because the build process is commonly a primary source of developer pain and frustration.
The best advice we can give Andy Jassy then is to make the practice of Developer Productivity Engineering (DPE) a more central part of the plan to improve “Build Tools.” Why? This will provide a hard ROI and a quick win to address the developer productivity pain points and process waste quantified in the surveys cited above. We would also encourage Andy Jassy to push tooling decisions and buying power closer to the teams directly feeling the pain of inadequate tooling daily.
DPE tools improve the developer experience by minimizing what Amazon refers to as “undifferentiated work,” and maximizing time spent on differentiated work like innovating and writing useful code. By speeding up builds and tests using DPE acceleration technologies, developers can spend more time doing productive work because they have less idle time caused by unnecessarily slow feedback cycles. Further, DPE processes and tools for failure analytics make troubleshooting more efficient by delivering actionable insights for determining problem root causes quickly. This minimizes the tedious and sometimes manual work associated with incident root cause analysis and debugging avoidable build failures and flaky tests.
Also, DPE tools can make build and test telemetry and trend data more observable. This enables developers and build engineers to take proactive steps to prevent the build process incidents that drive undifferentiated work from happening in the first place.
The No Brainer ROI
The acceleration technologies alone typically give developers at least one day back per week in lost productivity waiting for build and test feedback cycles to complete.
And the ROI calculation for this DPE use case is straightforward and compelling. Simply take the minutes saved slashing build times (e.g., from a ten-minute baseline to a new average build time of two minutes or an 8-minute savings) and multiply that by the average number of combined local and CI builds per year per developer. Then take that total time saved per year for each developer and multiply that by the total number of developers to get time saved per year in minutes for your entire development team. Assuming not 100% of that time is time wasted (our customers usually assume conservatively about 80%), adjust total time saved by that factor and multiply that result by the cost of an engineering minute. This is your total annual savings from speeding up build and test feedback cycles.
Even for modest-sized development organizations not nearly the size of Amazon’s, operational savings can quickly measure in the tens of millions of dollars. Layer on top of that the operational savings from minimizing the mean-time-to-resolution to troubleshoot build and test failures to pad your ROI further.
Conclusion
We encourage Andy Jassy and his team to learn more about the role Developer Productivity Engineering can play in addressing Amazon’s considerable challenge to solve their DevX crises and transform the Amazon developer experience into a competitive advantage. Amazon is not alone (See also Advice for Sundar Pichai: How to Solve Google’s Billion-Dollar Productivity Problem . If you face similar challenges and want to learn more about improving developer productivity and the developer experience at scale, check out the Developer Productivity Engineering Handbook. It’s also an excellent resource for development teams that want to proactively embed developer productivity and experience practices and tools into their existing processes. By investing in developer productivity processes and tools starting now, organizations can avoid paying a much higher Cost of Inaction later.