White Paper

The Economics of Software Testing: Service Virtualization Return on Investment Analysis

By:

WinterGreen Research, Inc.

COPYRIGHT 2013, WINTERGREEN RESEARCH, INC. www.wintergreenresearch.com tel 781-863-5078 781 863 1235 Lexington, Massachusetts

1 www.wintergreenresearch.com/blog email: [email protected]

Executive Summary: The Driving Forces For Service Virtualization

Software testing and service virtualization have become core driving forces to improving the efficiency of business operations. Software testing comes to the forefront as a way to improve automated process implementation because services virtualization can be used directly to grow the business by improving project quality. Service virtualization is used to catch programming errors early in the test cycle, prevent shipping software that has errors in it, support mobile application development, respond to complexity, reduce cost, mitigate risk, and improve time to market. Service virtualization is a vital part of the business automation process, freeing people for other tasks. As part of automated process, businesses need to achieve better, faster testing of software, thereby reducing time to market and freeing development and testing people resources. Service virtualization is good because with it, it takes less time and fewer people to do project software testing. Since software testing is done faster and better, service virtualization frees people resource for other project tasks.

Service Virtualization Executive Summary:

Provides The Equivalent Of Several New People Hired Deals With Complexity Using Automated Testing Reduces IT Cost Manages Risk in the Business Improves Time to Market

Complexity
The increasing complexity of IT systems is a fact of life. Addressing complexity to hide it and thereby contribute to software development efficiency is addressed with component architectures including SOA. Yet, for an organization to realize the full potential of SOA depends on the ability to test these composite applications early in the development lifecycle, before all of the services/components are even available.

Cost

Testing applications that are based on composite architectures can require a large number of assets and resources, which in todays economy are just not feasible as the business continues to be faced with pressures to do more with less. Constraints in budgets mean that management of software testing costs and quality is a necessity. The ability to virtualize applications, sometimes even before they are ready is more critical than ever, because it has the dual effect of improving quality while lowering cost.

Risk
The quantity of new code and existing code changes are continually an issue. As applications become more complex, software testing is a risk management concern, contributing to the overall success of the business by meeting customer demands while adhering to, security, and regulatory compliance requirements.

COPYRIGHT 2013, WINTERGREEN RESEARCH, INC. www.wintergreenresearch.com tel 781-863-5078 781 863 1235 Lexington, Massachusetts

2 www.wintergreenresearch.com/blog email: [email protected]

Time To Market
Businesses have an ongoing need to grow the business and to provide flexible response to changing market conditions. Speeding new solutions to market is a priority. This creates software development challenges with the move to shorten product delivery cycles. Pressures on software testers result from the need to leverage the Internet, implement mobile apps, and manage a large and growing number of customers. The testing process cannot wait for completed systems, we need to test early and with increased automation.

Software Testing and Service Virtualization Research Approach / ROI Calculator

Software testing savings of 30% can be achieved by implementing a services virtualization solution. This white paper analysis is backed by an online live ROI calculator that identifies the metrics that have the biggest impact on costs and savings for software testing and shows how savings accrue under various scenarios. Assumptions about service virtualization include # Projects Per Year, Average Project Cycle, Number of Q/A technicians per Project, and the time spent on testing with and without service virtualization automation in the testing process. In the WinterGreen Research testing software calculator there are 48 metrics that can be locked down or adjusted as variables to determine a probable ROI for the automation scenarios considered for various situations. The resultant ROI is not a black box, it is a set of calculations that can be examined and customized to match your particular situation. It can be used by to get the basis of potential savings accrued by changing 3 or 4 variables to customize the calculations, or it can be used thoughtfully and carefully changing as many variables as desired.

ROI Calculator
Savings of 30% Gist of Savings or In Depth Calculations

Service Virtualization Driving Forces

The driving forces for service virtualization are core to running the business: using computers and software to grow the business, catching programming errors early in the test cycle, and supporting mobile application development. Service virtualization supports responding to complexity. It is used for reducing cost, mitigating risk, and improving time to market as a way to grow the business.

Need To Do More With The Same Amount Of Resources

COPYRIGHT 2013, WINTERGREEN RESEARCH, INC. www.wintergreenresearch.com tel 781-863-5078 781 863 1235 Lexington, Massachusetts

3 www.wintergreenresearch.com/blog email: [email protected]

Leveraging Service Virtualization for Application Testing

Addressing the challenges of software development means that software needs to run with fewer bugs and to do so, bugs need to be found earlier in the development / testing cycle. Installed Application testing is More capacity needs to be tested rapidly and brought up and running in a timely manner. The need to test everywhere that new code Efficient if It Catches touches existing code is daunting; the need to test new code Programming Errors Early in with a range of different the Test Cycle types of applications and API modules can be time consuming. Existing Mobile Application application components Development Brings Special may not be readily available for Complexities To Application interoperability testing. Testing. When they are available it Rising costs may be weeks later than when the developer is Quality issues ready - significantly delaying the move of newly developed business capabilities or entire applications into production. Complex applications

Growing the Business Is a Huge Priority

Service Virtualization Driving Forces:

The elements of application testing that can be managed efficiently using software virtualization solve all the issues listed above: software virtualization supports testing faster, finding errors faster, Fast development iterations Addressing fixing errors faster, and managing testing across different platforms. Managers challenges Rapid response to changing contending with slow revenue growth need Testing rapidly efficient ongoing operations so they can implement projects that have hope of growing the business. There is a continuing need to do more with the same amount of resources and flat or decreasing budgets. Creating better testing efficiency can expand the amount of available resources, sometimes exponentially. Freeing testers to test may mean getting to market sooner or increasing the testing coverage to ensure higher quality releases.

Modular / composite applications (including SOA)

CIOs are wrestling with how best to function in an environment where all new hiring is frozen. This lack of new labor resource stymies innovation and stymies any but the most compelling upgrades of the data center. To free up testers provides at least a cadre of technical resource possibly able to handle the

COPYRIGHT 2013, WINTERGREEN RESEARCH, INC. www.wintergreenresearch.com tel 781-863-5078 781 863 1235 Lexington, Massachusetts

4 www.wintergreenresearch.com/blog email: [email protected]

processing and service demands not now being met, while helping IT to continue to support the enterprise to remain a viable and competitive business entity. Service virtualization has emerged as a way to expand existing technician resources and mange more effectively. Services virtualization supports simulation of data and application integration diminishing volume impact on software testing, supporting integration of the business and IT. Application testing bottlenecks are becoming a key focus area for many managers. The interest is in improving the efficiency of project management in any way possible. Through service virtualization, testers can simulate the data and application integration points making it possible to produce more testing volume, improving testing efficiency. Software developers and software testers have limited access to some code. A key pain point is to achieve access to code that is not yet available. That part of a packaged application that the tester does not have non-production access to is a problem. The cost of leveraging pay per use services for testing is a problem. Services virtualization helps address the issues of code availability. Thus, service virtualization is much appreciated when the tester is under the gun to produce high quality software quickly. Software virtualization overcomes testing problems and software delay problems not previously possible to overcome. Services virtualization does support the implementation of new code and the adjustment to existing code in a far more efficient manner than has previously been possible. Service virtualization provides a way to manage available options and quantify the value of speeding the software testing process to accelerate the delivery of new software applications into production.

Testing Solutions

Do more with the same amount of resources

Rational automates business processes with systems through Further integrate Business application development, testing, and production delivery. This is a and IT complex task. The aim is to pull together servers and applications from various production groups. Moving testing earlier in the process means bugs are easier to fix, errors have less impact on the software architecture enabling developers to perform tasks more effectively and efficiently.

Discovering Bugs Early in the Process Improves Testing Effectiveness

COPYRIGHT 2013, WINTERGREEN RESEARCH, INC. www.wintergreenresearch.com tel 781-863-5078 781 863 1235 Lexington, Massachusetts

5 www.wintergreenresearch.com/blog email: [email protected]

IBM Rational Is Automating Test Environments with Service Virtualization

IBM Rational Test Workbench is automating test environments through service virtualization. Service virtualization is being used to improve the economics of software testing, freeing people. In an environment with a hiring freeze, freeing up testers is able to support more resource for projects. People can be retrained using new just in time training techniques, and those people utilized in ways that may not have been thought possible before.

IBM Rational Test Workbench Tools Are Same Tools Developers Are Using

Service virtualization is efficient. It is effective. It saves customers millions of dollars per year. It is easy to use. It frees people resources to be available to work on projects that Agile Development Demands Complex Software Testing may grow the business. IBM Rational service virtualization capabilities powered by Green Hat replace the need to manually implement, configure and administer test environments which can be extremely time consuming, may require specialized skills, and are very expensive. The elimination of No one ever really manual process enables mission critical application testing to thought it possible to be automated and managed in an agile manner. Multiple iterations, each having different conditions to be validated recreate all the different are supported easily with the IBM solutions.

server and application configurations virtually

Testing From The Users Perspective

Developers and testers who are testing from the user perspective are able to find and fix problems much earlier in the development and test cycle, creating a lower cost development. Every error testers find improves the brand. User interface testing is part of the testing challenge that is sped up by leveraging service virtualization. By having a seamless and automated developer and tester tool environment, bug identification and fixes become more efficient. The development cycle benefits from test automation that is able to keep pace with agile and iterative cycles.

Now IBM has made this possible with the Green Hat acquisition and introduction of Rational Test Workbench

COPYRIGHT 2013, WINTERGREEN RESEARCH, INC. www.wintergreenresearch.com tel 781-863-5078 781 863 1235 Lexington, Massachusetts

6 www.wintergreenresearch.com/blog email: [email protected]

The Benefits of IBM Rational Service Virtualization:

The cost savings are achieved by automating the process of recreating every existing configuration of the production environment for complete software testing. Using service virtualization saved over $6 million in three years according to the above example. Until recently, no one ever really thought it possible to recreate all the different server and application configurations virtually.

Table 2 IBM Rational Service Virtualization Application Project Testing

Full remote troubleshooting capabilities Network loopback Test access Test signal generation Testing integrated into the platform Ensure new application meets all operational requirements via rigorous testing Source: WinterGreen Research, Inc.

This means that it is possible to simulate a real production environment as soon as the developer is ready to test their code even if dependencies, services, components and data structures are not ready or available. The developer is able to catch errors and fix them earlier, even before the formal testing cycle begins.

COPYRIGHT 2013, WINTERGREEN RESEARCH, INC. www.wintergreenresearch.com tel 781-863-5078 781 863 1235 Lexington, Massachusetts

7 www.wintergreenresearch.com/blog email: [email protected]

Dramatic Savings Brought by IBM Rational Automated Test Solution

The ROI analysis above demonstrates dramatic savings from use of IBM Rational Test Workbench. Quality of software code can be dramatically improved by pushing defect detection close to the point of introduction. Service virtualization directly relates to increasing collaboration between development and test teams. Discovering major issues needs to occur early in the code development process. Defects found later in the process take more effort, more time and cost more to fix. Positive ROI is achieved through the ability to minimize the cost and time to identify and fix errors, and diminish subsequent impacts of the defect.

Use of integrated simulation solutions on the cloud provides significantly more attractive ROI. Online sliders are included as part of the web site ROI that accompany this paper. WinterGreen Research endorses the IBM Service Virtualization offering as the best testing approach, particularly for software testing organizations using their own physical infrastructure to create unique test environments for each new application. It is as critical when leveraging third party applications, especially when a system integrates with a partner, to have service virtualization available. Because, even though a tester may have access to the internal corporate piece of the software solution, that tester may not have access to code so that testing against the third party piece can proceed apace. This is a very common and important example. Testing approaches using manual process have become particularly onerous as agile development is implemented, leading to the need for testing modernization.

Simple Automation Of Testing Software Can Yield Significant Benefits In Keeping The Process Balanced To Detect Errors Earlier In The Testing Process, Resulting In Better Overall Utilization of Newly Developed Or Changed Software

WinterGreen Research endorses IBM Service Virtualization

In conclusion, automation of testing using service virtualization software yields significant cost savings benefits. It keeps the software development process balanced. The detection of errors earlier in the testing process results in better overall systems deployment, faster software utilization, and overcomes testing bottlenecks.

IBM Rational Automated Test Solution, Powered by Green Hat Technology Reduces The Testing By 30% And The Q/A New Functionality Testing Is Reduced by 25%. Virtualizations Savings Average 30%, Range from 20% to 500%

COPYRIGHT 2013, WINTERGREEN RESEARCH, INC. www.wintergreenresearch.com tel 781-863-5078 781 863 1235 Lexington, Massachusetts

8 www.wintergreenresearch.com/blog email: [email protected]

RAL14071-USEN-01