What Do Contribution Guidelines Say About Software Testing? (MSR 2025)

Apr 29, 20250 likes33 views

Software testing plays a crucial role in the contribution process of open-source projects. For example, contributions introducing new features are expected to include tests, and contributions with tests are more likely to be accepted. Although most real-world projects require contributors to write tests, the specific testing practices communicated to contributors remain unclear. In this paper, we present an empirical study to understand better how software testing is approached in contribution guidelines. We analyze the guidelines of 200 Python and JavaScript open-source software projects. We find that 78% of the projects include some form of test documentation for contributors. Test documentation is located in multiple sources, including CONTRIBUTING files (58%), external documentation (24%), and README files (8%). Furthermore, test documentation commonly explains how to run tests (83.5%), but less often provides guidance on how to write tests (37%). It frequently covers unit tests (71%), but rarely addresses integration (20.5%) and end-to-end tests (15.5%). Other key testing aspects are also less frequently discussed: test coverage (25.5%) and mocking (9.5%). We conclude by discussing implications and future research.

What Do Contribution Guidelines
Say About Software Testing?
Bruna Falcucci, Felipe Gomide, Andre Hora
DCC/UFMG
andrehora@dcc.ufmg.br
MSR 2025
1

Motivation & Problem
Software testing plays a crucial role in the contribution
process of open-source projects
Contributions introducing new features are expected to
include tests
Contributions with tests are more likely to be accepted
2

“Tests help contributors feel con
fi
dent that they
won’t break anything. They also make it easier for
you to review and accept contributions quickly”
3

Despite software testing’s critical role in contribution guidelines, we are not
yet aware of how software testing is explained to contributors
Maintainers may focus on various topics, including run/write tests, test
types, coverage, mocking, and other software testing practices
8

Case Study
200 popular software repositories (100 Python and 100 JS)
200
Repositories
10

Case Study
200 popular software repositories (100 Python and 100 JS)
200
Repositories
184
With contribution
guidelines
11

Case Study
200 popular software repositories (100 Python and 100 JS)
200
Repositories
156
With test doc.
for contributors
184
With contribution
guidelines
12

Research Questions
RQ1: Frequency of test documentation for contributors
RQ2: Content of test documentation for contributors
13

RQ1: Frequency of test documentation for contributors
15

RQ1: Frequency of test documentation for contributors
• 78% (156 in 200) of the projects include some form of test
documentation for contributors
• Test documentation is mostly located in CONTRIBUTING
fi
les
(58%) and external documentation (24%)
16

RQ2: Content of test documentation for contributors
17

RQ2: Content of test documentation for contributors
• Test documentation commonly explains how
to run tests (83.5%), but less often provides
guidance on how to write tests (37%)
• Test documentation frequently covers unit
tests (71%), but rarely addresses integration
(20.5%) and e2e tests (15.5%)
• Other key testing aspects are less frequently
discussed: test coverage (25.5%), mocking
(9.5%), and best practices (9%)
18

Examples
• LangChain: “Unit tests: run on every PR, so they should be fast and reliable”
• CPython: “Increase test coverage [...] Ideally we would like to have 100% coverage,
but any increase is a good one”
• scikit-learn: “We expect code coverage of new features to be at least around 90%”
• Pandas: “A unit test should not access a public data set over the internet due to
fl
akiness of network connections. To mock this interaction, use the httpserver
fi
xture”
• Next.js: “Best Practices: When applying a
fi
x, ensure the test fails without the
fi
x.
This makes sure the test will properly catch regressions”
19

Conclusion
78% of the projects have test documentation for contributors
More focus on how-to-run than than how-to-write
Unit tests are commonly covered, while e2e tests receive less attention
Discussions on test coverage and mocking are rarer
• Future Work: qualitatively explore test documentation to catalog best
practices and guidelines for contributors
20

What Do Contribution Guidelines
Say About Software Testing?
Bruna Falcucci, Felipe Gomide, Andre Hora
DCC/UFMG
andrehora@dcc.ufmg.br
MSR 2025
21

The IEEE 1633 provides practical guidance for developing reliable software and making key decisions that include reliability. There are qualitative and quantitative tasks starting from the beginning of the program until deployment. These methods are applicable for agile and incremental development environments. In fact, they work better in an agile environment. This document has practical step by step instructions for how to identify failure modes and root cause, identify risks that are often overlooked, predict defects before the code is even written, plan staffing levels for testing and support, evaluate reliability during testing, and make a release decision. Examples of the techniques are provided. This document was written by people who have real world experience in making software more reliable while still on time and within budget. It covers software failure modes effects analysis, software fault trees, software defect root cause analysis, reliability predictions, defect density predictions, software reliability benchmarking, software reliability growth estimation, developing a reliability driven test suite, allocating reliability to software, evaluating the portion of the total system failures that will be caused by the software, and managing software for reliability. The working group is chaired by Ann Marie Neufelder who is the global leader in reliable software. The document will be updated in 2023 for the Common Defect Enumeration and relationship with DevSecOps.

Software Testing BasicsBelal Raslan

The document outlines topics related to quality control engineering and software testing. It discusses key concepts like the software development lifecycle (SDLC), common SDLC models, software quality control, verification and validation, software bugs, and qualifications for testers. It also covers the quality control lifecycle, test planning, requirements verification techniques, and test design techniques like equivalence partitioning and boundary value analysis.

Chapter -5 Agile Testing types and its examples.pptxManishaPatil932723

Unlike the Waterfall method, Agile Testing can begin at the start of the project with continuous integration between development and testing. Agile Testing methodology is not sequential (in the sense it’s executed only after coding phase) but continuous. Agile is an iterative development methodology, where requirements evolve through collaboration between the customer and self-organizing teams and agile aligns development with customer needs. The word Agile signifies something that can be performed quickly and immediately ,in the area o f software development

Software Quality Management.pptxAbhishek Prasoon

The guest lecture covered various topics related to software quality management including why it is important, the software testing life cycle, predictive versus agile testing approaches, different types of testing, emerging trends like test automation and artificial intelligence, and career opportunities in the field. The speaker discussed concepts like unit testing, integration testing, security testing and performance testing.

Essential Test Management and PlanningTechWell

The key to successful testing is effective and timely planning. Rick Craig introduces proven test planning methods and techniques, including the Master Test Plan and level-specific test plans for acceptance, system, integration, and unit testing. Rick explains how to customize an IEEE-829-style test plan and test summary report to fit your organization’s needs. Learn how to manage test activities, estimate test efforts, and achieve buy-in. Discover a practical risk analysis technique to prioritize your testing and become more effective with limited resources. Rick offers test measurement and reporting recommendations for monitoring the testing process. Discover new methods and develop renewed energy for taking your organization’s test management to the next level.

Essential Test Management and PlanningTechWell

Get strategic with qa in dev opsApplause

DevOps processes have become critical for organizations to release software as quickly as necessary in today’s fast-paced economy. The very name of “DevOps” calls out developers and operations. But where does QA fit into that mix? The answer is that QA plays an equal role in DevOps, and a shift to DevOps actually presents an opportunity for QA teams to play a more strategic role than previously alongside development and operations.

Agile Testing: Best Practices and Methodology Zoe Gilbert

Agile testing focuses on delivering value to customers through frequent testing and feedback. It differs from the traditional waterfall model which separates development and testing. The document discusses four main agile testing methodologies: behavior driven development, acceptance test driven development, exploratory testing, and session based testing. It also covers the agile testing quadrants framework and how companies can implement best practices for agile testing.

An introduction to Software Testing and Test ManagementAnuraj S.L

The document provides an introduction to software testing and test management. It discusses key concepts like quality, software testing definitions, why testing is important, who does testing, what needs to be tested, when testing is done, and testing standards. It also covers testing methodologies like black box and white box testing and different levels of testing like unit testing, integration testing, and system testing. The document is intended to give a basic overview of software testing and related topics.

Software Engineering 2 lecture slideAdil Mehmoood

The document discusses software engineering metrics and quality assurance. It covers: - Why measurement is important in software engineering for objective evaluation, estimation, quality control, and improvement. - Types of software metrics including direct metrics like lines of code and indirect metrics like functionality. - Frameworks for measuring software quality attributes like correctness, maintainability, integrity, and usability. - The importance of software quality assurance in reducing costs and improving time-to-market through defining quality, identifying assurance activities, and using metrics for process improvement.

Methodology Selection Strategy Ajeng Savitri

This document discusses methodology selection strategy for systems analysis and design. It explains that there is no single framework appropriate for every software project, and the methodology needs to be adapted to the specific team and product. It outlines important factors to consider like clarity of requirements, technology familiarity, system complexity, schedule constraints. The methodology should guide the team's process but allow flexibility. The success of the methodology is measured by timely delivery of quality product increments that satisfy stakeholders.

Agile testing guide_2021QMetry

DevOps - Right Tool for Right PhaseDalibor Blazevic

DevOps combines software development and IT operations to shorten the systems development life cycle. It promotes a culture of collaboration between developers and operations staff. Implementing DevOps culture requires trust, time, education, and well-defined processes. The DevOps pillars of continuous integration, continuous delivery, and release management automate the software development lifecycle. Continuous integration involves regularly integrating code changes. Continuous delivery aims to make code releases frequent and easy. Release management oversees the development, testing, and deployment of software releases. Choosing the right DevOps tools requires considering an organization's needs, existing tools, and culture.

Too many filesnikhilawareness

This document provides an overview of software testing techniques and the history of research in this area. It discusses: 1) The goals and types of software testing, including functional vs structural techniques. 2) Major milestones in the evolution of testing concepts from the 1950s to present day, shifting from debugging to prevention. 3) Key theoretical and methodological contributions to testing techniques from the 1970s onward, including work on path coverage, data flow testing, and model-based approaches. 4) How research in testing techniques has matured over time based on a technology maturation model, moving from ad hoc practices to a systematic discipline grounded in theory.

Software testing techniques - www.testersforum.comwww.testersforum.com

An Ultimate Guide to Continuous Testing in Agile Projects.pdfKMSSolutionsMarketin

Software testing pptPoonkodi Jayakumar

This document discusses software testing principles and concepts. It defines key terms like validation, verification, defects, failures, and metrics. It outlines 11 testing principles like testing being a creative task and test results needing meticulous inspection. The roles of testers are discussed in collaborating with other teams. Defect classes are defined at different stages and types of defects are provided. Quality factors, process maturity models, and defect prevention strategies are also summarized.

A Research Study on importance of Testing and Quality Assurance in Software D...Sehrish Asif

Mt s10 stlc&test_planTestingGeeks

The document outlines the software testing life cycle (STLC) which includes 10 phases: 1) requirement analysis, 2) prepare test plan, 3) test case designing, 4) design review, 5) code review, 6) test case execution, 7) bug reporting and regression testing, 8) inspection and release, 9) client acceptance, and 10) test summary analysis. It then provides details on requirement analysis, preparing the test plan, roles and responsibilities, and guidelines for developing an effective test plan. The test plan is a strategic document that defines the testing objectives, scope, approach, resources, and schedules to ensure software quality.

Continuous Testing in Agile and DevOps.docxjohn221769

QM in Software ProjectsAbhishek Prasoon

The presentation provided an overview of software quality management and emerging trends in software testing. It discussed fundamentals like the software testing lifecycle, principles of testing, and types of testing. Emerging trends that were covered include increased test automation, use of testing tools, and the evolving role of artificial intelligence in quality assurance. The presentation also discussed how testing roles differ based on software development lifecycles like agile and DevOps. It concluded with an overview of career opportunities in software testing and the skills required to be an effective tester.

Agile testingYogita patil

This document provides an overview of agile testing. It discusses what agile testing is, common agile testing strategies and stages, principles of agile testing, advantages such as reduced time and money and regular feedback, challenges like compressed testing cycles and minimal time for planning, and concludes that communication between teams is key to agile testing success. The agile testing life cycle involves four stages: iteration 0 for initial setup, construction iterations for ongoing testing, release for deployment, and production for maintenance. Principles include testing moving the project forward, testing as a continuous activity, everyone on the team participating in testing, and reducing feedback loops.

Agile Methods: Fact or FictionMatt Ganis

The document discusses Agile software development methods and provides evidence that Agile approaches are effective. It defines Agile development as iterative and incremental with close collaboration. Case studies show organizations achieving better results with Agile, including increased productivity, quality, and customer satisfaction. Adopting Agile practices like Scrum and test-driven development enables organizations to adapt to changing priorities and deliver working software more frequently.

Essential Test Management and PlanningTechWell

Code Reviewsphildenoncourt

This document discusses establishing code review processes for a team. It outlines the benefits of code reviews, including finding bugs earlier, increased maintainability, and coaching opportunities. It provides suggestions for an effective code review process, such as using checklists, keeping reviews small in scope, and measuring the results. While code reviews require commitment and can be challenging to implement, the document argues they are important for improving code quality and the team.

Better Software East 2016: Evolving Automated to ContinuousParasoft

Evolving from Automated to Continuous Testing Testing issues can be a significant barrier to taking full advantage of agile approaches to software development and the emerging DevOps movement. To leverage these development and delivery strategies to their fullest, you need to evolve beyond automated testing to continuous testing. Arthur Hicken discusses the testing and development processes and technology that enable continuous testing. He shares insights on how to close the gap between business expectations and development activities by encapsulating clearly defining development policies for software releases. Arthur describes how to prevent defects in code and prioritize defect remediation before a release candidate goes live. Explore ways to realistic test environments and simulations—critical features of the dev/test infrastructure—that enable continuous testing. Learn how to create a feedback loop that exposes defect patterns while highlighting opportunities to improve application design. Take back a comprehensive to do list for processes and infrastructure that must be in place for your organization to implement continuous testing and accelerate the SDLC.

Dev ops lpi-701Radhouen Assakra

DevOps LPI 701 Module 1 : Modern software development . Module 2 : Component , platforms and cloud deployment . Module 3 : Source code management Module 4 : System image creation and VM Deployment . Module 5 : Container usage . Module 6 : Container infrastructure . Module 7 : Container deployment and orchestration . Module 8 : CI / CD . Module 9 : Ansible and configuration management tools. Module 10 : IT monitoring . Module 11 : Log management and analysis.

Exceptional Behaviors: How Frequently Are They Tested? (AST 2025)Andre Hora

Exceptions allow developers to handle error cases expected to occur infrequently. Ideally, good test suites should test both normal and exceptional behaviors to catch more bugs and avoid regressions. While current research analyzes exceptions that propagate to tests, it does not explore other exceptions that do not reach the tests. In this paper, we provide an empirical study to explore how frequently exceptional behaviors are tested in real-world systems. We consider both exceptions that propagate to tests and the ones that do not reach the tests. For this purpose, we run an instrumented version of test suites, monitor their execution, and collect information about the exceptions raised at runtime. We analyze the test suites of 25 Python systems, covering 5,372 executed methods, 17.9M calls, and 1.4M raised exceptions. We find that 21.4% of the executed methods do raise exceptions at runtime. In methods that raise exceptions, on the median, 1 in 10 calls exercise exceptional behaviors. Close to 80% of the methods that raise exceptions do so infrequently, but about 20% raise exceptions more frequently. Finally, we provide implications for researchers and practitioners. We suggest developing novel tools to support exercising exceptional behaviors and refactoring expensive try/except blocks. We also call attention to the fact that exception-raising behaviors are not necessarily “abnormal” or rare.

TestMigrationsInPy: A Dataset of Test Migrations from Unittest to Pytest (MSR...Andre Hora

Unittest and pytest are the most popular testing frameworks in Python. Overall, pytest provides some advantages, including simpler assertion, reuse of fixtures, and interoperability. Due to such benefits, multiple projects in the Python ecosystem have migrated from unittest to pytest. To facilitate the migration, pytest can also run unittest tests, thus, the migration can happen gradually over time. However, the migration can be timeconsuming and take a long time to conclude. In this context, projects would benefit from automated solutions to support the migration process. In this paper, we propose TestMigrationsInPy, a dataset of test migrations from unittest to pytest. TestMigrationsInPy contains 923 real-world migrations performed by developers. Future research proposing novel solutions to migrate frameworks in Python can rely on TestMigrationsInPy as a ground truth. Moreover, as TestMigrationsInPy includes information about the migration type (e.g., changes in assertions or fixtures), our dataset enables novel solutions to be verified effectively, for instance, from simpler assertion migrations to more complex fixture migrations. TestMigrationsInPy is publicly available at: https://github.com/altinoalvesjunior/TestMigrationsInPy.

More Related Content

Similar to What Do Contribution Guidelines Say About Software Testing? (MSR 2025) (20)

Agile Testing: Best Practices and Methodology Zoe Gilbert

An introduction to Software Testing and Test ManagementAnuraj S.L

Software Engineering 2 lecture slideAdil Mehmoood

Methodology Selection Strategy Ajeng Savitri

Agile testing guide_2021QMetry

DevOps - Right Tool for Right PhaseDalibor Blazevic

Too many filesnikhilawareness

Software testing techniques - www.testersforum.comwww.testersforum.com

An Ultimate Guide to Continuous Testing in Agile Projects.pdfKMSSolutionsMarketin

Software testing pptPoonkodi Jayakumar

A Research Study on importance of Testing and Quality Assurance in Software D...Sehrish Asif

Mt s10 stlc&test_planTestingGeeks

Continuous Testing in Agile and DevOps.docxjohn221769

QM in Software ProjectsAbhishek Prasoon

Agile testingYogita patil

Agile Methods: Fact or FictionMatt Ganis

Essential Test Management and PlanningTechWell

Code Reviewsphildenoncourt

Better Software East 2016: Evolving Automated to ContinuousParasoft

Dev ops lpi-701Radhouen Assakra

Agile Testing: Best Practices and Methodology Zoe Gilbert

An introduction to Software Testing and Test ManagementAnuraj S.L

Software Engineering 2 lecture slideAdil Mehmoood

Methodology Selection Strategy Ajeng Savitri

Agile testing guide_2021QMetry

DevOps - Right Tool for Right PhaseDalibor Blazevic

Too many filesnikhilawareness

Software testing techniques - www.testersforum.comwww.testersforum.com

An Ultimate Guide to Continuous Testing in Agile Projects.pdfKMSSolutionsMarketin

Software testing pptPoonkodi Jayakumar

A Research Study on importance of Testing and Quality Assurance in Software D...Sehrish Asif

Mt s10 stlc&test_planTestingGeeks

Continuous Testing in Agile and DevOps.docxjohn221769

QM in Software ProjectsAbhishek Prasoon

Agile testingYogita patil

Agile Methods: Fact or FictionMatt Ganis

Essential Test Management and PlanningTechWell

Code Reviewsphildenoncourt

Better Software East 2016: Evolving Automated to ContinuousParasoft

Dev ops lpi-701Radhouen Assakra

More from Andre Hora (15)

Exceptional Behaviors: How Frequently Are They Tested? (AST 2025)Andre Hora

TestMigrationsInPy: A Dataset of Test Migrations from Unittest to Pytest (MSR...Andre Hora

When should internal interfaces be promoted to public? (FSE 2016)Andre Hora

Commonly, software systems have public (and stable) interfaces, and internal (and possibly unstable) interfaces. Despite being discouraged, client developers often use internal interfaces, which may cause their systems to fail when they evolve. To overcome this problem, API producers may promote internal interfaces to public. In practice, however, API producers have no assistance to identify public interface candidates. In this paper, we study the transition from internal to public interfaces. We aim to help API producers to deliver a better product and API clients to benefit sooner from public interfaces. Our empirical investigation on five widely adopted Java systems present the following observations. First, we identified 195 promotions from 2,722 internal interfaces. Second, we found that promoted internal interfaces have more clients. Third, we predicted internal interface promotion with precision between 50%-80%, recall 26%-82%, and AUC 74%-85%. Finally, by applying our predictor on the last version of the analyzed systems, we automatically detected 382 public interface candidates.

Assessing the Threat of Untracked Changes in Software Evolution (ICSE 2018)Andre Hora

While refactoring is extensively performed by practitioners, many Mining Software Repositories (MSR) approaches do not detect nor keep track of refactorings when performing source code evolution analysis. In the best case, keeping track of refactorings could be unnecessary work; in the worst case, these untracked changes could significantly affect the performance of MSR approaches. Since the extent of the threat is unknown, the goal of this paper is to assess whether it is significant. Based on an extensive empirical study, we answer positively: we found that between 10 and 21% of changes at the method level in 15 large Java systems are untracked. This results in a large proportion (25%) of entities that may have their histories split by these changes, and a measurable effect on at least two MSR approaches. We conclude that handling untracked changes should be systematically considered by MSR studies.

JavaScript API Deprecation in the Wild: A First Assessment (SANER 2020)Andre Hora

Building an application using third-party libraries is a common practice in software development. As any other software system, code libraries and their APIs evolve over time. In order to help version migration and ensure backward compatibility, a recommended practice during development is to deprecate API. Although studies have been conducted to investigate deprecation in some programming languages, such as Java and C#, there are no detailed studies on API deprecation in the JavaScript ecosystem. This paper provides an initial assessment of API deprecation in JavaScript by analyzing 50 popular software projects. Initial results suggest that the use of deprecation mechanisms in JavaScript packages is low. However, we find five different ways that developers use to deprecate API in the studied projects. Among these solutions, deprecation utility (i.e., any sort of function specially written to aid deprecation) and code comments are the most common practices in JavaScript. Finally, we find that the rate of helpful message is high: 67% of the deprecations have replacement messages to support developers when migrating APIs.

Assessing Mock Classes: An Empirical Study (ICSME 2020)Andre Hora

During testing activities, developers frequently rely on dependencies (e.g., web services, etc) that make the test harder to be implemented. In this scenario, they can use mock objects to emulate the dependencies' behavior, which contributes to make the test fast and isolated. In practice, the emulated dependency can be dynamically created with the support of mocking frameworks or manually hand-coded in mock classes. While the former is well-explored by the research literature, the latter has not yet been studied. Assessing mock classes would provide the basis to better understand how those mocks are created and consumed by developers and to detect novel practices and challenges. In this paper, we provide the first empirical study to assess mock classes. We analyze 12 popular software projects, detect 604 mock classes, and assess their content, design, and usage. We find that mock classes: often emulate domain objects, external dependencies, and web services; are typically part of a hierarchy; are mostly public, but 1/3 are private; and are largely consumed by client projects, particularly to support web testing. Finally, based on our results, we provide implications and insights to researchers and practitioners working with mock classes.

What Code Is Deliberately Excluded from Test Coverage and Why? (MSR 2021)Andre Hora

Test coverage is largely used to assess test effectiveness. In practice, not all code is equally important for coverage analysis, for instance, code that will not be executed during tests is irrelevant and can actually harm the analysis. Some coverage tools provide support for code exclusion from coverage reports, however, we are not yet aware of what code tends to be excluded nor the reasons behind it. This can support the creation of more accurate coverage reports and reveal novel and harmful usage cases. In this paper, we provide the first empirical study to understand code exclusion practices in test coverage. We mine 55 Python projects and assess commit messages and code comments to detect rationales for exclusions. We find that (1) over 1/3 of the projects perform deliberate coverage exclusion; (2) 75% of the code are already created using the exclusion feature, while 25% add it over time; (3) developers exclude non-runnable, debug-only, and defensive code, but also platform-specific and conditional importing; and (4) most code is excluded because it is already untested, low-level, or complex. Finally, we discuss implications to improve coverage analysis and shed light on the existence of biased coverage reports.

Googling for Software Development: What Developers Search For and What They F...Andre Hora

Developers often search for software resources on the web. In practice, instead of going directly to websites (e.g., Stack Overflow), they rely on search engines (e.g., Google). Despite this being a common activity, we are not yet aware of what developers search from the perspective of popular software development websites and what search results are returned. With this knowledge, we can understand real-world queries, developers’ needs, and the query impact on the search results. In this paper, we provide an empirical study to understand what developers search on the web and what they find. We assess 1.3M queries to popular programming websites and we perform thousands of queries on Google to explore search results. We find that (i) developers’ queries typically start with keywords (e.g., Python, Android, etc.), are short (3 words), tend to omit functional words, and are similar among each other; (ii) minor changes to queries do not largely affect the Google search results, however, some cosmetic changes may have a non-negligible impact; and (iii) search results are dominated by Stack Overflow, but YouTube is also a relevant source nowadays. We conclude by presenting detailed implications for researchers and developers.

Availability and Usage of Platform-Specific APIs: A First Empirical Study (MS...Andre Hora

A platform-specific API is an API implemented for a particular platform (e.g., operating system), therefore, it may not work on other platforms than the target one. In this paper, we propose a first empirical study to assess the availability and usage of platform-specific APIs. We analyze the platform-specific APIs provided by the Python Standard Library and mine their usage in 100 popular systems. We find that 21% of the Python Standard Library APIs are platform-specific and that 15% of the modules contain at least one. The platforms with the most availability restrictions are WASI (43.69%), Emscripten (43.64%), Unix (6,76%), and Windows (2.12%). Moreover, we find that platform-specific APIs are largely used in Python. We detect over 19K API usages in all 100 projects, in both production (52.6%) and test code (47.4%). We conclude by discussing practical implications for practitioners and researchers.

How and Why Developers Migrate Python Tests (SANER 2022)Andre Hora

Nowadays, Python developers can rely on two major testing frameworks: unittest and pytest. Due to the benefits of pytest (e.g., fixture reuse), several relevant projects in the Python ecosystem have migrated from unittest to pytest. Despite being performed by the Python community, we are not yet aware of how systems are migrated from unittest to pytest nor the major reasons behind the migration. In this paper, we provide the first empirical study to assess testing framework migration. We analyze how and why developers migrate from unittest to pytest. We mine 100 popular Python systems and assess their migration status. We find that 34% of the systems rely on both testing frameworks and that Python projects are moving to pytest. While some systems have fully migrated, others are still migrating after a long period, suggesting that the migration is not always straightforward. Overall, the migrated test code is smaller than the original one. Furthermore, developers migrate to pytest due to several reasons, such as the easier syntax, interoperability, easier maintenance, and fixture flexibility/reuse, however, the implicit mechanics of pytest is a concern. We conclude by discussing practical implications for practitioners and researchers.

Test Polarity: Detecting Positive and Negative Tests (FSE 2024)Andre Hora

Positive tests (aka, happy path tests) cover the expected behavior of the program, while negative tests (aka, unhappy path tests) check the unexpected behavior. Ideally, test suites should have both positive and negative tests to better protect against regressions. In practice, unfortunately, we cannot easily identify whether a test is positive or negative. A better understanding of whether a test suite is more positive or negative is fundamental to assessing the overall test suite capability in testing expected and unexpected behaviors. In this paper, we propose test polarity, an automated approach to detect positive and negative tests. Our approach runs/monitors the test suite and collects runtime data about the application execution to classify the test methods as positive or negative. In a first evaluation, test polarity correctly classified 117 tests as as positive or negative. Finally, we provide a preliminary empirical study to analyze the test polarity of 2,054 test methods from 12 real-world test suites of the Python Standard Library. We find that most of the analyzed test methods are negative (88%) and a minority is positive (12%). However, there is a large variation per project: while some libraries have an equivalent number of positive and negative tests, others have mostly negative ones.

Monitoring the Execution of 14K Tests: Methods Tend to Have One Path that Is ...Andre Hora

The literature has provided evidence that developers are likely to test some behaviors of the program and avoid other ones. Despite this observation, we still lack empirical evidence from real-world systems. In this paper, we propose to automatically identify the tested paths of a method as a way to detect the method’s behaviors. Then, we provide an empirical study to assess the tested paths quantitatively. We monitor the execution of 14,177 tests from 25 real-world Python systems and assess 11,425 tested paths from 2,357 methods. Overall, our empirical study shows that one tested path is prevalent and receives most of the calls, while others are significantly less executed. We find that the most frequently executed tested path of a method has 4x more calls than the second one. Based on these findings, we discuss practical implications for practitioners and researchers and future research directions.

PathSpotter: Exploring Tested Paths to Discover Missing Tests (FSE 2024)Andre Hora

When creating test cases, ideally, developers should test both the expected and unexpected behaviors of the program to catch more bugs and avoid regressions. However, the literature has provided evidence that developers are more likely to test expected behaviors than unexpected ones. In this paper, we propose PathSpotter, a tool to automatically identify tested paths and support the detection of missing tests. Based on PathSpotter, we provide an approach to guide us in detecting missing tests. To evaluate it, we submitted pull requests with test improvements to open-source projects. As a result, 6 out of 8 pull requests were accepted and merged in relevant systems, including CPython, Pylint, and Jupyter Client. These pull requests created/updated 32 tests and added 80 novel assertions covering untested cases. This indicates that our test improvement solution is well received by open-source projects.

Predicting Test Results without Execution (FSE 2024)Andre Hora

As software systems grow, test suites may become complex, making it challenging to run the tests frequently and locally. Recently, Large Language Models (LLMs) have been adopted in multiple software engineering tasks. It has demonstrated great results in code generation, however, it is not yet clear whether these models understand code execution. Particularly, it is unclear whether LLMs can be used to predict test results, and, potentially, overcome the issues of running real-world tests. To shed some light on this problem, in this paper, we explore the capability of LLMs to predict test results without execution. We evaluate the performance of the state-of-the-art GPT-4 in predicting the execution of 200 test cases of the Python Standard Library. Among these 200 test cases, 100 are passing and 100 are failing ones. Overall, we find that GPT-4 has a precision of 88.8%, recall of 71%, and accuracy of 81% in the test result prediction. However, the results vary depending on the test complexity: GPT-4 presented better precision and recall when predicting simpler tests (93.2% and 82%) than complex ones (83.3% and 60%). We also find differences among the analyzed test suites, with the precision ranging from 77.8% to 94.7% and recall between 60% and 90%. Our findings suggest that GPT-4 still needs significant progress in predicting test results.

SpotFlow: Tracking Method Calls and States at Runtime (ICSE 2024)Andre Hora

SpotFlow is a tool that eases runtime analysis in Python. It executes and monitors Python programs, collecting detailed information on method calls and states at the method level, such as executed lines, argument values, return values, and thrown exceptions. SpotFlow supports novel empirical studies on runtime analysis, the development of novel software testing tools, and the creation of novel datasets with runtime metrics.