Improving Developer Productivity With DORA, SPACE, and DevEx

Editor's Notes

• #6: The problem is that most organizations lack observability into the full value stream, and even organizations who are measuring parts of the value stream are often measuring the wrong things, or they don’t know how to take action on what they’ve measured.
• #7: The myth of the 10x engineer is pervasive, but, as we will explore throughout this workshop, it is just that, a myth. We’ve all heard of the concept, that engineer that is somehow able to produce at 10x the rate of their peers, but in study after study of organizational productivity, engineers within a single organization tend not to significantly outperform one another, with a few exceptions.
• #8: Every Myth has an Origin Story… In 1977, Tom DeMarco and Timothy Lister devised a study called the Coding War Games From 1984 – 1986, more than 600 developers from 92 companies participated The purpose was to discover the ”best” and “worst” traits of developers The competition unit was a pair of programmers from the same organization These were not peer programmers, in fact they competed against each other A program specification was fixed, and participants logged their time in completing it Participants used their own workspace using familiar tooling and languages This study is the foundation for an excellent book on building highly productive organizations, though part of why we are here today is that this book was last published in 2013!
• #9: Unpaired programmers from the same organizations performed at roughly the same level The average difference was only 21% between participants at the same company They didn’t work together on the task, but they came from the same organization The best organization performed 11.1x better than the worst
• #10: So, we don’t have 10x developers, *but we do have 10x organizations!* (11x, really)
• #11: So, we don’t have 10x developers, *but we do have 10x organizations!* (11x, really)
• #12: So, we don’t have 10x developers, *but we do have 10x organizations!* (11x, really)
• #13: Look at what actually makes these top performing companies better. 10x organizations prioritize the experience of their developers, with huge productivity gains as a result. NOTES Tie these data back to flow state / cognitive load / here’s SPACE framework You’ve beent alkinga bout and know about these things, we have different language to label them “What we see here is a survey from 2000, a predecessor of seeing DevEx and SPACE at work here, same story different language now Light a fire to care about this, too slow and you’ll lose otherwise” Is this a talk about 10x engineers, how I win, how I improve developer experience? 10x developers don’t exist, but 10x organizations do Heres how you know I only trust DX to do this because they taught me
• #14: Non-factors: Language - With the exception of assembly Salary - Only 10% diff in salary between top and bottom performers Experience - 2 - 10 years largely the same (dip at 6 months) Number of Defects - Top performers provided defectless code
• #15: Do this as a Zoom poll or have answers in Zoom chat
• #16: Despite knowing this, even the rise of DevOps didn’t fully include the experience of Developers. There’s a whole timeline, going back to the JIT Manufacturing in the 70s where we’ve tried to figure out ways to measure and optimize productivity, each phase adapting to new cultural norms, and iterating on and adapting what was learned in the past. NOTES Could we end with Developer Experience Agile Transform, DevOps Transform, DevEx Transform
• #17: Let’s drill into these origins and Introduce The Theory of Constraints, talk about its modernization into The Phoenix Project
• #18: Double down on the importance of this book
• #19: Talk about Theory of Constraints as it forces us to the think about the *purpose* of the machine (throughput) as opposed to its subcomponents of Cost and Inventory, and map those concepts to a Software Organization
• #20: The idea here is that physics is consistent across all complex machinery, even the complex machinery of a business, and it has formed much of the foundation for what we think of as DevOps practice and philosophy
• #21: So we have tried to quantify and optimize everything in our processes to make it run something resembling a factory. Make fun of SAFe a little: “Agile Waterfall, the neverending Daily Standup, Agility through more process” - The Cathedral of SAFe
• #22: Kidding aside, there are good ideas to take away from W Edwards Deming’s work on Systems Thinking, Variability, and other aspects of Profound Knowledge
• #23: But as much similarity as there is here, opinionated knowledge workers as part of the value stream add a whole new dimension of complexity
• #25: Now, just because these concepts form the underlying philosophy of DevOps, this is not to say that DevOps is bad, dead, or any of those existing hyperboles. The DevOps mindset must remain part of our thinking as we optimize for developer experience and productivity. And what was the origin of this? It was probably earlier than Patrick DeBois and DevOpsDays Ghent in 2009. It probably started with a mindset.
• #26: And its this mindset, and it's a good mindset, but Speed isn’t the whole picture.
• #27: DORA was really the first on the scene to try and measure DevOps outcomes. not only do the four key DORA metrics specifically apply to software engineering, but it gives a direction of what to do to get better. So this is where a lot of engineering leaders started, or do start today.
• #28: DORA is over a decade old, an in addition to these four key metrics there is a lot more to the organization. These metrics were popularized in Accelerate. This is a book about software delivery and devops, not about developer productivity. and since then, our understanding of it has changed a lot. What were orgs struggling with in 2015? We live in a different world now. Would you rely on a webapp book from 10 years ago?
• #32: And so we have witnessed the rise of the platform engineer, a role that includes disciplines learned from DevOps, and combines them with optimizing the experience of the platform user, i.e. the developer, and delivering these platforms in a way a product manager would deliver to customers. Developers become internal customers.
• #33: And with this new focus on Developer Experience, we started to see the evolution of DORA. The industry got closer to a better definition of productivity in 2021 when the SPACE framework was released. SPACE was sort of a sleeper hit. Notice Nicole Forsgren who led the DORA initiative, and see that she is joined by a number of other researchers, Margaret-Anne (Peggy) Storey, and a number of researchers from Microsoft.
• #34: A pervasive challenge for platform engineers is properly and effectively measuring the impact of their platform on things like developer experience and productivity. We’ve learned that no single metric can tell us exactly how effective our platform is, so we know we need to measure against multiple metrics. But should we measure the quantitative qualities of our platform, or should we rely on qualitative metrics such as survey data?
• #35: And in considering that, the organization should seek to explore multiple measures and avoid Goodhart’s Law, which says that any measure which becomes a target ceases to be a good measure.
• #36: SPACE expands upon DORA, learning that a single metric set will not be enough to gain a full understanding of developer experience and productivity. Rather, we must look at “a constellation of metrics in tension with one another”
• #40: Developer experience is all about going to the developers for the answers. So we can start an improvement cycle by asking devs about current conditions, setting targets, using other metrics to track progress, and then evaluate whether we’re reaching our goals.
• #43: Emphasize Cost of Qual metrics is high, time spent filling out surveys, etc, have to be repeated to get a trend
• #47: Now we are getting somewhere!! We have some great sets of metrics with precedent, validated by real engineering teams… But could we streamline this even further? Each of these metrics can influence other metrics, we learned that from SPACE, could we reduce these based on those correlations?
• #48: YES! Presenting: the most modern developer experience and productivity metric framework available, the DX Core 4. The most effective metrics have been pulled by these three major metric sets, and move into four new dimensions of Speed, Effectiveness, Quality, and Impact DX Core 4 is the diagnostic tool One ring tool
• #50: Zooming in, we can see that our Key Metric for Effectiveness is something called the DXI
• #53: The DXI is a formative measure of how, and to what extent, the right conditions exist for developers to be able to deliver effectively. The DXI measures 14 dimensions (Figure 2) found to be actionable (and changeable) at both the manager and organizational level, such as deep work, local iteration speed, release process, and confidence in making changes. DX uses linear regression analysis to assess the strength of the relationship between the DXI and time waste, and for modeling the predictive relationship between them. The most recent analysis of DXI was conducted in early 2024, based on a sample of 50,000 responses from 200 organizations. The DXI is calculated by taking the mean 16 items corresponding to key dimensions of developer experience. Efficiency is calculated through a self-reported outcome measure of time loss due to inefficiencies in the work environment. Past research by DX identified a comprehensive set of factors underlying developer experience. In designing the 14 dimensions included in the DXI, our researchers took into account both the actionability of issues for management and predictive power against outcomes such as engineering velocity, quality, and efficiency.
• #54: But this is essential to building an 10x organization by way of optimal developer experience
• #56: To use the survey template, load the questions into your preferred tool. If you’d like to use Google Forms, you can save time by making a copy of this ready-to-use template. Next, decide whether to make your survey anonymous or not. Either way can work, but anonymous surveys generally get better participation. Before sending out your survey, be sure to send out clear announcements over email or Slack to make sure that developers know (1) what the purpose of the survey is and (2) how the data will be used. To officially launch your survey, a short message like this works well: After you’ve collected responses, score the multiple choice questions using either mean or top box scoring. Mean scores are calculated by assigning each option a value between 1 and 5 and taking the average. Top box scores are calculated by the percentages of responses that choose one of the top two most favorable options. DX’s industry benchmarks use top-box scoring, if you’re interested in comparing your scores. Also be sure to carefully review open text responses, which can contain the most actionable bits of information. If you’ve collected a large number of comments, LLM tools such as ChatGPT can be useful for extracting core themes and suggestions. When you’ve finished analyzing results, be sure to publish your findings back to your organization so people feel that their time filling out the survey was worth while. I
• #57: Reviewing the recommended frequency and scope of these surveys Google clip on rolling out metrics “An ally to developers”
• #58: So what’s next? What can you start doing today? We have a ton of free resources available including survey templates and guides which can get you on your way to calculating and presenting metrics right now!
• #60: NOTES A Decade of DORA talk - Nathen Equipped them to feel more confident in decision-making Before: I don’t know if what I’m doing is right After: I feel confident, well-researched, well-read Not seeing pain reflected in beginning part Getting questions about how to measure, whats platform, whats devops, how are they different? Know from the beginning about what their after picture looks like Set up pain at the beginning to bring them in, find empathy, know why they should listen Heavy on background theory at the beginning In spirit of WUB, spread more context/value throughout They are in a DORA (possibly SPACE) world Background lets us know that this is not unprecedented Works well as keynote, but workshop audience will want something more practical What decisions do I have to make