Test Automation Success: Choosing the Right People & Process

Kiran Pyneni, Automation Manager Aetna, Inc.

Agenda
Company Background Automation Team Automation Objectives Our Journey Strategy Team Delivery Team Benefits of Teams Strategy Automation Lifecycle Benefits of Process Integration

Metrics Of Our Success

Automation Coverage, Efforts Automated, Utilization & Cost Avoidance Questions

Aetna Background
Aetna is one of the nation's leading diversified health care benefits companies, serving members with information and resources to help them make better informed decisions about their health care.
Products and management Programs:
Aetna offers a broad range of insurance and employee benefits products. The first national, full-service health insurer to offer a consumer-directed health plan, Aetna continues to lead the way with its Aetna HealthFund2 line of products, including HSA, HRA and RRA options. Aetna offers a wide array of programs and services that help control rising employee benefits costs while striving to improve the quality of health care, such as case management; disease and patient safety programs; integrated medical, dental, pharmaceutical, behavioral health and disability information. Aetna provides members with access to convenient tools and easy-to-understand information that can help them make better-informed decisions about their health and financial wellbeing.

Automation Team
Improves efficiencies and quality at our organization by automating consistent and repeatable manual tasks using industry standard technologies / tools / frameworks.
Standards & Best Practices

Reports & Metrics

Automation Team

Assessments & Designs

Automation Development & Maintenance

Automation Objectives

Automation Objectives

Automating Right Tasks

Utilization of existing automation Sustaining and Growing Automation Suite

Our Journey
Decentralized Approach Centralized Approach

Benefits
Easy identification of automation opportunities Higher collaboration between application & automation teams Automation development has become expensive Usage of different tools & solutions Standards & Best practices are not shared across applications

Benefits
Standardization of automation tool set at enterprise level Sharing of coding best practices, templates and standards Cost of automation development is lowered Slower automation coverage progression Lower automated scripts utilization Funding Challenges

Challenges

Challenges

Our Journey Contd. Challenges & Solutions

Challenges
Identification of automation opportunities Prioritization of automation projects based on ROI Cost of automation Implementation of optimized designs

Challenges

Early completion of maintenance for automation scripts

Funding for automation projects Collaboration between automation & business teams

Solution
Structuring automation as strategy & delivery team

Solution
Create automation life cycle process and integrate with company's overall software development life cycle

Automation Team Structure

Automation Team

Automation Strategy Team

Automation Delivery Team

Automation Strategy Team

Definition:
To provide automation strategic direction and identify potential new automation opportunities

Characteristics:
Project management experience Various automation frameworks/tools knowledge Ability to manage multiple projects in parallel Very strong written and verbal communication skills Very strong leadership and analytical skills

Activities:
Process & Best Practices Assessment & Strategies Design Solutions & Techniques Estimations & Planning ROI Cost/Benefit Analysis Reports & Metrics Tool Technical Support & Analysis

Automation Delivery Team

Definition:
To develop, deliver, and maintain automation solutions

Characteristics:
Understanding of programming principles and best practices Knowledge of various automation tools Knowledge of multiple programming languages Strong communication skills

Activities:
Scripts Development Scripts Maintenance Coding Standards Scripts Planning & Execution Day to Day interactions w/End User

Benefits of Teams Strategy

Benefits

Easy identification of automation opportunities Prioritization of automation projects based on ROI Improved automation design Optimized the cost of automation Clearly defined roles & responsibilities Better communication process Optimized automation project planning

Automation Lifecycle
Phases
Automation Evaluation Automation Preparation
Identify the resources needed. Understand the requirements with the help application demos. Create automation framework / design Identify timelines and milestones Training required for automation tools Tool installation if required Access & Environment Setup

Automation Generation
Create Business Process Test (BPT) components Parameterize test data in BPT components Review components for quality and correctness String the BPT components together to create a test script Develop automation scripts for BPT components Identify test data to conduct unit & acceptance testing

Execution & Acceptance

Understand functionality of the project / application Understand testing processes

Unit testing of automation Deliver automation to application team Execute automation by application team Validation of ROI Retrospection Sign off

Activities

Understand target environments Understand test data details Verify automation tool compatibility Identify Automation opportunities

G1

G2
Automation Project Plan Framework documentation Re-usable components, Portability, test data grid design, etc. Business Process Test (BPT) components Automation Scripts Test Data

G3
Tested automation scripts Automated test execution results Lessons Learned Sign Off

Deliverables

Preliminary Estimations Automation Test Strategy Business Case - ROI Selected automation tool Detailed Estimations Go, No Go

Team Collaboration

Automation Strategy Team

Automation Delivery Team

Automation Evaluation

Automation Preparation

Automation Generation

Execution & Acceptance

Automation ROI Calculator

Automation ROI Calculator

Integration of Automation Lifecycle & Software Development Lifecycle

Automation Evaluation Automation Preparation Automation Generation Execution & Acceptance

Feasibility/Analysis

Design

Implementation

Testing & Maintenance

Benefits of Process Integration

Benefits

Improved understanding on automation process across the entire organization Better collaboration between business and automation teams Funding challenges are removed Improved on-time delivery of automation scripts Helped to promote the automation across entire organization Formal escalation process developed Initialized the creation of automation reports & metrics

Automation Coverage Regression Testing

Year-2008 Q1-2009 Q2-2009 Q3-2009 Q4-2009

35% 42% 45% 47% 52%

Manual Testing Efforts Automated

13,803 hrs 11,750 hrs 10,200 hrs 8,570 hrs 7,047 hrs

Year-2008 Year

Q1-2009 As of 2008 7047 hrs

Q2-2009 As of 2009 13803 hrs

Q3-2009

Q4-2009 2009 only efforts 6756 hrs

Total Efforts Automated

Utilization & Cost Avoidance

Utilization
150% 87% 45% 134% 130%

Year-2008

Q1-2009

Q2-2009

Q3-2009

Q4-2009

Cost Avoidance
10,810 hrs 9,618 hrs 9,867 hrs

7,200 hrs

Q1-2009

Q2-2009

Q3-2009

Q4-2009

Savings Vs. Cost Avoidance

Cost Savings
Efforts difference between manual and automation testing Manual (Not automation) execution frequency Produced by automation strategy team after assessment Automation ROI Calculator Test coverage difference between manual & automation testing Efforts difference between manual and automation testing Automation execution frequency Generated from Quality Center after release implementation

Cost Avoidance

Helped to improve the credibility of automation metrics significantly

Automation Metrics Best Practices

Automation is a way of Testing and process followed for manual testing can be applied

Best Practices followed in Test Management Tool Quality Center Define automation work flow
Regression Testing identifier Automation Assessment status Automation Development status

Store automation inventory Manage automated test scripts execution

Questions