CompuwareCorporation

Continuous Integrated Testing

Delivering Reliable, High-Performing Applications
Irvin Tan Automated Software Quality

Page 2

Compuware
Powerful solutions for enterprise IT

IT Management

Application Development

IT Operations

Page 3

Compuware Solutions Support the Application Life Cycle

Support Abend-AID Vantage File-AID STROBE XPEDITER Development XPEDITER/DevEnterprise File-AID/CS OptimalJ OptimalView

TrackRecord Reconcile

QACenter Performance Edition STROBE Manage

DriverStudio File-AID Abend-AID QACenter DevPartner STROBE XPEDITER Studio Quality Assurance

CompuwareCorporation

Page 5

The IT Business Value Curve

The Typical Application Life Cycle

Development

Production
Planned

Benefit

Time

Cost

Planned Outcome:

Application is deployed on schedule and within budget Functionality meets business expectations and drives expected business value Application is high quality, secure and optimized for performance

Page 6

The IT Business Value Curve

ITs Challenge: Delivering on the Expectations of the Business

Development

Production
Planned

Benefit

Time
Actual

Cost

Actual Outcome:

Application is deployed behind schedule and costs significantly more than planned Application fails to meet quality or performance requirements Ultimately, the business fails to realize immediate or sustained value

Page 7

The IT Business Value Curve

ITs Challenge: Delivering on the Expectations of the Business

Development

Production
Planned

Benefit

Time
Actual

Cost

Application Delivery Success Rates

Actual Outcome:

Application is deployed behind 29% Succeeded schedule and costs significantly more than planned 53% Challenged Application fails to meet quality or 18% Outright Failures performance requirements Ultimately, Source: Standish Group the business fails to realize immediate or sustained value

Page 8

Application Delivery Management

The Ideal Solution: Building Quality In to Ensure Consistent Delivery

A life cycle approach to quality assurance:

Builds quality into the application through clear alignment with the business needs and a focus on testing in development Implements an agile testing approach in development that results in higher quality code

Page 9

Quality Assurance
IT Challenge: What Happens Today

Defects are not found until late in the development life cycle or worse, in production Code is sent back to development causing delays which reduces the available testing time This cuts into new development time for further business innovation

End of Life Cycle Testing Approach

QA

Plan

Develop

Test

Deploy

Page 10

Quality Assurance
IT Challenge: What Happens Today

Risk of failure in production is significantly increased due to insufficient time to test.

Loss of Revenue to the Business Cancellation of Project Loss of faith in the ability of the project team and its management

End of Life Cycle Testing Approach

QA

Plan

Develop

Test

Deploy

Page 11

Quality Assurance
Problem: Finding and Fixing Bugs Late in the Life Cycle is Costly
Errors found during testing may be 15 to 75 times more time consuming to fix than those that are discovered and fixed during the coding phase.
Software Engineering Economics, Barry W. Boehm

There can be as many as 20 to 30 bugs per 1,000 lines of software code.

Sustainable Computing Consortium

Page 12

Quality Assurance
There are More Cost Effective Ways to Solve These Problems

Formal design and code inspections average about 65% in defect removal efficiency.
Software Quality: Analysis and Guidelines for Success Caper Jones

Peer reviews of software will catch 60% of defects.

Institute of Electrical and Electronics Engineers

Continuous integrated testing improves defect removal rates in development by 20% or more.
Compuware Corporation

Page 13

Continuous Integrated Testing

Is not a product but a methodology

Begin testing at the same time as coding

Application testing needs to be started with development
Employ an integrated development and testing approach Introduce quality with initial coding

Increase number of test cycles per project

Test continually
Test whatever is available everyday

Page 14

Compuwares CIT Solution

Repeatable, Consistent, Iterative Process to Improve Application Quality

Building quality in from day one

An increased number of test cycles provides more opportunities to fix defects Tests are fully automated and can be run unattended, offering minimal distraction to developers

Test Early, Test Often, Test with Confidence

Test Cases

QA

Defects are found shortly after they are created

Plan Develop Test Deploy

Root cause analysis reduces the time spent on problem diagnosis by QA and development Higher quality code is sent to QA

Continuous Integrated Testing

Integration

Page 15

CIT Process

Page 16

Test Execution
Automated functional testing tool

Scripting language VBA

Visual Basic like editor

Supports Client/server, 4GL and web application

Page 17

Test Execution
User actions generate object-based scripts

Record and replay function

Checks verify application functionalities

Events synchronization
Data-driven testing GUI-less testing

Results log

Page 18

Comprehensive Developer Toolset

Microsoft & Java editions Integration with IDEs / App server Automatic error detection Memory analysis:
Memory Leaks Temporary Objects

RAM Footprint

Performance analysis Coverage analysis Code review

Page 19

Visual Studio Integration

Page 20

Source Code Review

Automate the manual code review process A teams 24x7

Mentor Teacher Enforcer

Page 21

Memory Analysis at Runtime

RAM usage Over time

Real-Time
trace of memory usage

System memory Your code memory

Page 22

Performance Analysis
Useful data views, pre-filtered and sorted

Performance detail by line of code

Call graph overview

Call graph detail with performance data

Page 23

Performance Comparisons

Quickly visualize change, positive or negative

In-line highlights show timing differences

Page 24

Code Coverage Analysis

Visually assess overall application coverage

Useful data views, pre-filtered and sorted

Color-coded source code: What is or is not tested?

Page 25

Automatic Error Detection

Page 26

Java Edition
Memory Analysis:
Temporary Objects Ram Footprint Memory Leaks Code Support

Page 27

Java Edition
Application performance profiler automatically locates Java performance bottlenecks Increases Application usability

Page 28

Java Edition
Code coverage analysis automatically locates untested code Ensure test coverage during unit and system testing Produce reliable components and applications Helps developers locate runtime multi-threading problems

Page 29

Java Edition

A set of over 200 rules in three categories

correctness performance

design

With severity (High, Medium, Low) Ability to filter rules Comprehensive explanation of each rule View offending code Generate reports

Page 30

Continuous Integrated Testing

System Walkthrough
1. Use the record option to automatically generate test scripts 2. Run nightly build that includes execution of the automated test scripts 3. Static code analysis identifies and recommends coding improvements 4. Potential memory and performance improvements are identified 5. Improvements can be made and the results viewed the next day 6. Provides simple way to validate and improve testing coverage

Page 31

Capabilities and Benefits of CIT

Deliver Highly Reliable, High-Performing Applications

Automates daily code inspections to uncover poor coding practices and complexity metrics prior to new code being written

Uncovers memory and performance issues to eliminate catastrophic events down the road

Automates application code security inspections to detect and remediate security vulnerabilities

Analyses for code coverage during unit testing to ensure that all code has been thoroughly tested prior to moving to QA

Page 32

CIT in Action
Improving Quality with Every Build
Plan Test Cases Test Cases Test Cases Test Cases

QA

Cycle 1

Cycle 2

Cycle N-1

Cycle N

New code is written

Problems are corrected

Problems are corrected

Problems are corrected

Test

Deploy

Unit tests are created

New code is written

New code is written

Automated build runs and tests are executed

Unit tests are created

Unit tests are created

Run session files identify coding, memory, security and performance issues

Automated build runs and tests are executed

Automated build runs and tests are executed

Run session files are created

Run session files are created

Page 33

The IT Business Value Curve

CIT: Building Applications with Excellence

Application Delivery Management

Application Service Management

Stretch Planned

Benefit

Time
Actual

Cost

Short-term Results:

Detect and remediate defects early in the development cycle

Long-term Results:

Implement a process that allows you to consistently deliver code that is high quality, secure and optimized for performance