Measuring Technical Lag in Software Deployments (CHAOSScon 2020)
Download as PPTX, PDF0 likes120 views
Presentation at CHAOSSCon Europe 2020 about the generic technical lag software measurement framework. Technical lag measures the increasing difference between deployed software components and the ideal upstream software components. For more information, see https://doi.org/10.1002/smr.2157
Ad
More Related Content
Similar to Measuring Technical Lag in Software Deployments (CHAOSScon 2020) (20)
More from Tom Mens (20)
Ad
Recently uploaded (20)
Ad
Measuring Technical Lag in Software Deployments (CHAOSScon 2020)
- 1. chaoss.community @tom_mens Dr. Tom Mens Dr. Ahmed Zerouali Software Engineering Lab University of Mons [email protected]
- 2. chaoss.community secoassist.github.io @secoassist "Excellence of Science" Research Project
- 3. chaoss.community Focus Which measures can help software developers and deployers to decide when and why they should update? 3
- 5. chaoss.community Online survey What would be the most appropriate (i.e., ideal) version of a software library to depend on? • 17 respondents Highly educated with an average of 3 years of development experience • Responses: 5 ★ Most stable (14) ★ Latest available (9) ★ Most documented (7) ★ Most secure (5)
- 6. chaoss.community Idea: Technical Lag “The increasing difference between deployed software packages and the ideal available upstream packages.” 6 Ideal • stability, security, functionality, recency, etc. Difference • time, version updates, bugs, vulnerabilities, features, … J. Gonzalez-Barahona, P. Sherwood, G. Robles, D. Izquierdo (2017) "Technical lag in software compilations: Measuring how outdated a software deployment is.” IFIP International Conference on Open Source Systems. Springer
- 7. chaoss.community Importance of Technical Lag Semi-structured interviews: 2019 5 highly educated software practitioners with an average of 10 years of experience 7 Technical Lag is important, especially if we mix between the benefits of updating and the effort required to do that.
- 8. chaoss.community Measuring Technical Lag ∆ version ∆ time ∆ bugs ∆ vulnerabilities
- 9. chaoss.community Measuring Technical Lag A technical lag framework F is a tuple (C, L, ideal, delta, agg) with • C a set of component releases • L a set of possible lag values • ideal: C → C computes the “ideal” (upstream) component release for a given (deployed) release • delta: C x C → L computes the difference between two component releases • agg: 2L → L aggregates the results of a set of lags A formal framework for measuring technical lag in component repositories – and its application to npm. A. Zerouali, T. Mens, J. Gonzalez-Barahona, A. Decan, E. Constantinou, G. Robles. Wiley Journal on Software Evolution and Process, 2019
- 10. chaoss.community Measuring Technical Lag Given a technical lag framework F, we define techlagF(c) = delta(c,ideal(c)) for any deployed component c aggLagF(D) = agg( {techlagF(c) | c in D} ) for any set of deployed components D A formal framework for measuring technical lag in component repositories – and its application to npm. A. Zerouali, T. Mens, J. Gonzalez-Barahona, A. Decan, E. Constantinou, G. Robles. Wiley Journal on Software Evolution and Process, 2019
- 11. chaoss.community Technical Lag - Example Time-based measurement of technical lag (ideal = most recent release; delta = time difference) 1.0.1 1.1.0 2.0.01.2.0 2.0.1 deployed package upstream package Time lag date(2.0.1) - date(1.1.0)
- 12. chaoss.community Technical Lag - Example Version-based measurement of technical lag (ideal = highest release; delta = version difference) 1.0.1 1.1.0 2.0.12.0.0 1.2.0 deployed package 1 major upstream package 1 patch Version lag 1 major + 1 patch
- 13. chaoss.community Technical Lag - Example Vulnerability-based measurement of technical lag (ideal = least vulnerable release; delta = #vulnerabilities) 1.0.1 1.1.0 2.0.01.2.0 2.0.1 deployed package upstream package Security lag 1 vulnerability fix behind
- 14. chaoss.community Technical Lag - Example Bug-based measurement of technical lag (ideal = least known bugs; delta = #known bugs) 1.0.1 1.1.0 2.0.0 deployed package upstream package 1.2.0 2.0.1 Dependency needs to be downgraded to be able to use most stable version… Bug lag 1 more bug than most stable version
- 15. chaoss.community Case study 1: Technical lag in npm distribution of JavaScript packages Credits: https://exploring-data.com/vis/npm-packages-dependencies/ A. Decan, T. Mens, E. Constantinou (2018) On the evolution of technical lag in the npm package dependency network. IEEE Int’l Conf. Software Maintenance and Evolution +20M dependencies
- 16. chaoss.community package.json Technical Lag – Example 16
- 17. chaoss.community Technical Lag – Example 17
- 18. chaoss.community Technical Lag – Example 18 Time-based technical lag for deployed release debug 2.6.9 ideal (debug 2.6.9) = debug 3.1.0 timeLag(debug 2.6.9) = 26-09-2017 - 22-09-2017 = 4 days versionLag(debug 2.6.9) = 1 major + 1 minor + 1 patch
- 19. chaoss.community Technical Lag – Example 19 Time-based technical lag for deployed release ms 2.0.0 ideal (ms 2.0.0) = ms 2.1.1 timeLag(ms 2.0.0) = 30-11-2017 - 16-05-2017 = 198 days versionLag(ms 2.0.0) = 1 minor + 1 patch
- 20. chaoss.community Technical Lag – Example 20 Aggregated transitive time lag for deployed release youtube-player 5.5.0 agglag({debug 2.6.9, ms 2.0.0}) = max(4 days, 198 days) = 198 days
- 22. chaoss.community Case study 2: Technical lag in Debian-based Docker containers A. Zerouali, T. Mens, G. Robles, J. Gonzalez-Barahona (2019). On the relation between outdated Docker containers, security vulnerabilities, and bugs. IEEE In’tl Conf. SANER
- 23. chaoss.community Case study 2: Technical lag in Debian-based Docker containers Important issues faced when deploying Docker containers: • Security vulnerabilities • Dependence on external software packages • Presence of bugs in third-party software • Outdated third-party software
- 24. chaoss.community 1.0.1 1.1.0 2.0.01.2.1 2.1.0 Docker container C Technical lag ∆ versions (freshness) ∆ vulnerabilities (security) ∆ bugs (stability) « ideal » release deployed container included Debian package release upstream releases of the Debian package Technical Lag in Debian-based Docker containers 26
- 25. chaoss.community Tool support Example: snyk container
- 26. chaoss.community Summary Technical Lag is a very useful generic measure for assessing to which extent deployed software is outdated w.r.t. upstream releases. • Different ways to measure (time, version, bugs, vulnerabilities, …) and aggregate (max, sum, …) technical lag • It can be operationalized in different contexts (package dependency management, container deployment, …) Suggestion: • Include this measure as part of the CHAOSS Metrics and Tooling Open Challenges: • How to measure effort required to update? • How to combine multiple dimensions of technical lag? • How to assess whether updates do not cause breaking changes? 30
- 27. chaoss.community New proposed CHAOSS project metrics • Dependencies • Number of / List of; Direct or transitive • Dependency depth • Outdated dependencies • List of / Number of / Ratio of • Vulnerable dependencies • List of / Number of / Ratio of • Dependents (i.e. reverse dependencies) • Number of / List of; Direct or transitive • Dependency lag • aggregated dependency-based technical lag of a project • Deployment lag • Aggregated lag of set of deployed components w.r.t. upstream
- 28. chaoss.community SoHeal, May 2020 http://soheal.github.io 3rd Int’l ICSE Workshop on Software Health What? • Focus on the health of software projects, communities and ecosystems • Discuss about technical, social, legal and business aspects related to project effectiveness, success, longevity, growth, resilience, survival, diversity, sustainability, popularity, inclusiveness, ... Who? • Open Source Community Members, Industry and Academia Why? • Raise awareness on software health • Present tools, methods, practical experiences • Advance body of knowledge on software health Seoul, South Korea – May 2020 @iw_soheal
- 29. chaoss.community SoHeal 2020 http://soheal.github.io/cft.html Extended call for submissions Are you a involved in software projects or ecosystems, and have something to say about software health? Submit a short paper or talk proposal on Open source and industrial experiences from individual, team or community level Relation between software health and social, technical, legal, process and business aspects Tools, dashboards and models to enable, assess, predict and recommend software health Guidelines and lessons learned Submission deadline: Friday, February 7, 2019
Editor's Notes
- #7: So the goal and the focus of this thesis was to answer the following question: [read slide]
- #8: we performed semi structured interviews with 5 software practitioners during the event of FOSDEM in Brussels, In these interviews, we explained the technical lag and asked the interviewees about their opinion regarding the technical lag. We found that throughout the interviews, the importance of technical lag was stressed. However, participants acknowledged that a mix between what is missing (i.e., benefits) and the effort (i.e., cost) needed to update would be even better than only knowing what is missing. The benefits here can refer to many things, like a new feauture, a bug fix, or a vulnerability fix, etc. So the notion of ideal is related to what we want to have as benefits and since there are many benefits, there should be many ideals also
- #9: So, since the ideal of a version is not a specific version with some specific characteristic and since the difference between the ideal and the deployed software isnt also a specific metric, we needed to create an extensible framework that supports all these variants. Moreover, we needed a framework that should not only support one deployed software component, but it should support a set of deployed components. Since in most of the cases, a deployed software will come with its dependencies.
- #10: so indeed, we presented a technical lag framework as a tuple of : [ Read slide]
- #11: so indeed, we presented a technical lag framework as a tuple of : [ Read slide]
- #12: In other words, to compute the technical lag or the time lag for the time-based instatiation, we use the difference in days between the releases dates of the deployed version and the ideal version.
- #13: and then to compute the technical lag in terms of missing updates, we compute how many versions are between the deployed and the ideal versions while checking their version type. For example here, from here to here the minor version number changes,
- #14: and then to compute the technical lag in terms of missing updates, we compute how many versions are between the deployed and the ideal versions while checking their version type. For example here, from here to here the minor version number changes,
- #15: and then to compute the technical lag in terms of missing updates, we compute how many versions are between the deployed and the ideal versions while checking their version type. For example here, from here to here the minor version number changes,
- #17: For example, in the case of npm packages, to specify a dependency, developers need to put a dependency constraint that refers to the range of allowed versions to be installed. Let’s take a concrete example of the youtube-player npm package, In the figure, we can see that this package makes use of three dependencies. However, to specify which versions of these packages to use, the package youtube-player make use of dependency constraints. So In many cases, the use of these constraints can lead to the use of outdated or not ideal package version.
- #18: Concrete example: here. the version 5.5.0 of the youtube player package makes use of three direct dependencies (...). Using different dependency constraints. While the debug package version makes use of the ms package using the constraint 2.0.0 this dependency here is considered as a transitive dependency for the youtube player package version. Because of the use of these constraints, the debug and ms dependencies could not be installed with the latest available version at the release date of the youtube player package and thus they were inducing a technical lag So while this situation is not necessarily a problem, it can sometimes come with issues. For example, if the outdated package version here was suffering from a bug or vulnerability, then this bug will propagate and debug will suffer from it and also youtube player will suffer from it.
- #19: Concrete example: here. the version 5.5.0 of the youtube player package makes use of three direct dependencies (...). Using different dependency constraints. While the debug package version makes use of the ms package using the constraint 2.0.0 this dependency here is considered as a transitive dependency for the youtube player package version. Because of the use of these constraints, the debug and ms dependencies could not be installed with the latest available version at the release date of the youtube player package and thus they were inducing a technical lag So while this situation is not necessarily a problem, it can sometimes come with issues. For example, if the outdated package version here was suffering from a bug or vulnerability, then this bug will propagate and debug will suffer from it and also youtube player will suffer from it.
- #20: Concrete example: here. the version 5.5.0 of the youtube player package makes use of three direct dependencies (...). Using different dependency constraints. While the debug package version makes use of the ms package using the constraint 2.0.0 this dependency here is considered as a transitive dependency for the youtube player package version. Because of the use of these constraints, the debug and ms dependencies could not be installed with the latest available version at the release date of the youtube player package and thus they were inducing a technical lag So while this situation is not necessarily a problem, it can sometimes come with issues. For example, if the outdated package version here was suffering from a bug or vulnerability, then this bug will propagate and debug will suffer from it and also youtube player will suffer from it.
- #21: Concrete example: here. the version 5.5.0 of the youtube player package makes use of three direct dependencies (...). Using different dependency constraints. While the debug package version makes use of the ms package using the constraint 2.0.0 this dependency here is considered as a transitive dependency for the youtube player package version. Because of the use of these constraints, the debug and ms dependencies could not be installed with the latest available version at the release date of the youtube player package and thus they were inducing a technical lag So while this situation is not necessarily a problem, it can sometimes come with issues. For example, if the outdated package version here was suffering from a bug or vulnerability, then this bug will propagate and debug will suffer from it and also youtube player will suffer from it.
- #22: so, as a conclusion statement : The technical lag framework, the actionable results and the tools developed in this thesis can be used to help open source software developers and deployers to keep their software in a healthy shape.
- #23: So, starting with the first case, we focused on images that are based on Debian. and to compute the technical lag, we relied on the same standards, which is the ideal version and then the difference between the used and ideal version. In this case analysis, we considered three different ideal versions. For developers interested in freshness we considered the latest available version as ideal version. For developers interested in security, we considered, the most secure software version as the idea version, and for For developers interested in stability, we considered, the most stable software version as the idea version, and so to compute the technical lag, we relied respectively on three measurement units
- #24: So, starting with the first case, we focused on images that are based on Debian. and to compute the technical lag, we relied on the same standards, which is the ideal version and then the difference between the used and ideal version. In this case analysis, we considered three different ideal versions. For developers interested in freshness we considered the latest available version as ideal version. For developers interested in security, we considered, the most secure software version as the idea version, and for For developers interested in stability, we considered, the most stable software version as the idea version, and so to compute the technical lag, we relied respectively on three measurement units
- #27: So, starting with the first case, we focused on images that are based on Debian. and to compute the technical lag, we relied on the same standards, which is the ideal version and then the difference between the used and ideal version. In this case analysis, we considered three different ideal versions. For developers interested in freshness we considered the latest available version as ideal version. For developers interested in security, we considered, the most secure software version as the idea version, and for For developers interested in stability, we considered, the most stable software version as the idea version, and so to compute the technical lag, we relied respectively on three measurement units
- #28: So, starting with the first case, we focused on images that are based on Debian. and to compute the technical lag, we relied on the same standards, which is the ideal version and then the difference between the used and ideal version. In this case analysis, we considered three different ideal versions. For developers interested in freshness we considered the latest available version as ideal version. For developers interested in security, we considered, the most secure software version as the idea version, and for For developers interested in stability, we considered, the most stable software version as the idea version, and so to compute the technical lag, we relied respectively on three measurement units
- #29: So, starting with the first case, we focused on images that are based on Debian. and to compute the technical lag, we relied on the same standards, which is the ideal version and then the difference between the used and ideal version. In this case analysis, we considered three different ideal versions. For developers interested in freshness we considered the latest available version as ideal version. For developers interested in security, we considered, the most secure software version as the idea version, and for For developers interested in stability, we considered, the most stable software version as the idea version, and so to compute the technical lag, we relied respectively on three measurement units
- #31: so, as a conclusion statement : The technical lag framework, the actionable results and the tools developed in this thesis can be used to help open source software developers and deployers to keep their software in a healthy shape.