CSE 403

System Testing and Test Plans

These lecture slides are copyright (C) Marty Stepp, 2007. They may not be rehosted,
sold, or modified without expressed permission from the author. All rights reserved.
1
 Outline
 extreme programming

 some kinds of system testing

 UI testing
 build/check-in tests
 load testing
 usability testing
 security testing
 monitoring
 performance profiling

 components of a test plan

2
 eXtreme Programming
 eXtreme Programming (XP): A software
development approach for small teams on risk-prone
projects with unstable requirements.
 created by Kent Beck: Extreme Programming Explained
 Testing is a core aspect of Extreme Programming

write unit test code first - before writing the application code

acceptance tests are preferrably automated, and are modified and
rerun for each of the frequent development iterations

QA and test personnel are an integral part of the project team
 detailed requirements documentation is not used; frequent re-
scheduling, re-estimating, and re-prioritizing is expected

3
 Build tests
 build tests (AKA check-in tests)
 as code is checked in, these tests (often unit tests) are run
 helps find bug regressions

 tools
 BuildBot: http://buildbot.sourceforge.net/
 SVNAnt: http://subclipse.tigris.org/svnant.html
 Apache Harmony:
http://harmony.apache.org/subcomponents/buildtest/index.html

 (some companies roll their own check-in testing systems)

4
 UI testing ("acceptance")
 automated UI testing ("automation")
 scripts and such that use your app and look for failures
 a black-box system test

 Selenium: http://www.openqa.org/selenium/

 manual tests
 human beings click through predetermined paths
 need to write down the specific tests each time

 ad-hoc tests
 human beings are "turned loose" on the app to see if they can
break it

5
 Load testing
 load testing
 How many hits/requests should the system be able to handle?
 What should be its performance under these circumstances?

 tools
 JMeter: http://jakarta.apache.org/jmeter/
 curl-loader: http://sourceforge.net/projects/curl-loader
 Database Open-source Test Suite (DOTS):
http://ltp.sourceforge.net/

 others:
http://www.opensourcetesting.org/performance.php

6
 Usability testing
 usability testing: making sure the user can successfully
utilize the system
 usability studies
 field tests

 Some web usability guidelines to consider:

 Pages should be 3-5 screens max unless content is tightly
focused on a single topic. If larger, provide internal links.
 The page layouts and design elements should be consistent
throughout a site.
 Pages should be as browser-independent as possible, or pages
should be provided or generated based on the browser-type.
 All pages should have links external to the page; there should
be no dead-end pages.
 The page owner, revision date, and a link to a contact person or
organization should be included on each page.
7
 Monitoring
 monitoring
 making sure the site is still up, resources are reachable, etc.

 tools
 ExternalTest: http://www.externaltest.com/
 StillUp: http://www.stillup.co.uk/
 DotCom-Monitor: http://www.dotcom-monitor.com/
 Internet Vista: http://www.internetvista.com/

 others:
http://www.softwareqatest.com/
qatweb1.html#MONITORING

8
 Web app compatibility
testing
 motivation
 ensure that your web app is compatible with various browsers,
platforms, etc.
 ensure that your app's HTML code complies with web standards
 ensure that you have no broken links or other HTML errors

 W3C HTML validator: http://validator.w3.org/

 W3C link checker: http://validator.w3.org/checklink

9
 Testing web applications 1
Questions to ask when testing web apps:
 What are the expected loads on the server (e.g., hits/time)?

 What kind of performance is required under such loads (server

response time, database query times)?

 What kinds of tools will be needed for performance testing (such

as web load testing tools, other tools already in house that can be
adapted, web robot downloading tools, etc.)?
 Who is the target audience? What kind of browsers will they be

using? What kind of connection speeds will they by using? Are they
intra- organization (high connection speeds and similar browsers)
or Internet-wide (thus with a wide variety of connection speeds
and browser types)?
 What kind of performance is expected on the client side (e.g., how

fast should pages appear, how fast should animations, applets,

etc. load and run)?
 Will down time for server and content maintenance/upgrades be

allowed? how much?

 What kinds of security (firewalls, encryptions, passwords, etc.) will

be required and what is it expected to do? How can it be tested? 10

 Testing web applications 2
 How reliable are the site's Internet connections required to be?
And how does that affect backup system or redundant connection
requirements and testing?
 What processes will be required to manage updates to the web
site's content, and what are the requirements for maintaining,
tracking, and controlling page content, graphics, links, etc.?
 Which HTML specification will be adhered to? How strictly? What
browsers will be allowed?
 Will there be any standards or requirements for page appearance
and/or graphics throughout a site or parts of a site?
 How will internal and external links be validated and updated? how
often?
 Can testing be done on the production system, or will a separate
test system be required? How are browser caching, variations in
browser option settings, dial-up connection variabilities, and real-
world internet 'traffic congestion' problems to be accounted for in
testing?
 How extensive or customized are the server logging and reporting
requirements; are they considered an integral part of the system
and do they require testing?
11
 Test plans
 Outline how your app will be tested in detail

 more information:
 http://en.wikipedia.org/wiki/Test_plan

 example test plan templates:

 http://members.tripod.com/~bazman/testplancontents.html
 http://www.sqatester.com/documentation/testplansmpl.htm
 http://www.coleyconsulting.co.uk/testplan.htm

 example test plan for linux kernel:

 http://ltp.sourceforge.net/25_testplan.php

 testing plans and checklists:

 http://members.tripod.com/~bazman/
12
 Elements of a test plan 1
 Title
 Identification of software (incl. version/release #s)
 Revision history of document (incl. authors, dates)
 Table of Contents
 Purpose of document, intended audience
 Objective of testing effort
 Software product overview
 Relevant related document list, such as requirements,
design documents, other test plans, etc.
 Relevant standards or legal requirements
 Traceability requirements

13
 Elements of a test plan 2
 Relevant naming conventions and identifier
conventions
 Overall software project organization and
personnel/contact-info/responsibilties
 Test organization and
personnel/contact-info/responsibilities
 Assumptions and dependencies
 Project risk analysis
 Testing priorities and focus
 Scope and limitations of testing
 Test outline - a decomposition of the test approach by
test type, feature, functionality, process, system,
module, etc. as applicable
 Outline of data input equivalence classes, boundary
value analysis, error classes 14
 Elements of a test plan 3
 Test environment - hardware, operating systems, other
required software, data configurations, interfaces to
other systems
 Test environment validity analysis - differences
between the test and production systems and their
impact on test validity.
 Test environment setup and configuration issues
 Software migration processes
 Software CM processes
 Test data setup requirements
 Database setup requirements
 Outline of system-logging/error-logging/other
capabilities, and tools such as screen capture software,
that will be used to help describe and report bugs
15
 Elements of a test plan 4
 Discussion of any specialized software or hardware
tools that will be used by testers to help track the cause
or source of bugs
 Test automation - justification and overview
 Test tools to be used, including versions, patches, etc.
 Test script/test code maintenance processes and
version control
 Problem tracking and resolution - tools and processes
 Project test metrics to be used
 Reporting requirements and testing deliverables
 Software entrance and exit criteria
 Initial sanity testing period and criteria
 Test suspension and restart criteria 16
 Elements of a test plan 5
 Personnel allocation
 Personnel pre-training needs
 Test site/location
 Outside test organizations to be utilized and their
purpose, responsibilties, deliverables, contact persons,
and coordination issues
 Relevant proprietary, classified, security, and licensing
issues.
 Open issues
 Appendix - glossary, acronyms, etc.

17