STQA Notes ch 1
STQA Notes ch 1
CSE
PUNYASHLOK AHILYADEVI HOLKAR SOLAPUR UNIVERSITY, SOLAPUR
COURSE OBJECTIVES
1) To learn the principles, techniques and tools of software testing in order to improve the quality of software
product.
2) To gain knowledge of the software testing process, various methods of testing, different levels of testing,
software quality concepts, assurance & standards
4) To learn manual and automatic software testing & various kinds of testing tools.
COURSE OUTCOMES
1) Identify what a software bug is, how serious they can be, and why they occur.
2) Test software to meet quality objectives & requirements.
3) Apply testing skills to common testing tasks.
4) Perform the planning and documentation of test efforts.
5) Describe software quality concepts, assurance & standards.
6) Use testing tools to test software in order to improve test efficiency with automation.
Syllabus:
SECTION I
Life Cycle, Essential of Software Testing, Features of Testing, Misconceptions about Testing,
Principles of Software Testing, Test Policy, Strategy, Planning, Process, Challenges in Testing,
Test Team Approach, Methods, Defect Classification, Defect, Error, Mistake in Software,
Defect Life Cycle, Defect Management Process, Developing Test Strategy, Developing Testing
Software Verification and Validation, Black-Box and White-Box Testing, Static and Dynamic
Other Black Box Test Techniques. White-Box Testing Techniques-Data Coverage, Code
Levels of Testing, Proposal Testing, Requirement Testing, Design Testing, Code Review, Unit
GUI Testing, Compatibility Testing, Security Testing, Performance Testing, Volume Testing,
Stress Testing, Load Testing, Installation Testing, Regression Testing, Smoke Testing, Sanity
Testing, Ad hoc Testing, Usability Testing, Acceptance Testing-Alpha Testing, Beta Testing,
Gamma Testing.
Cases-The Goal of Test Case Planning, Test Case Planning Overview, Test Case Organization
And Tracking, Reporting Bugs- Getting Your Bugs Fixed, Isolating and Reproducing Bugs
Software Quality Assurance, SQA Processes and Product Characteristics, SQA Tasks, Goals and
Metrics
Statistical SQA, Software Reliability, The ISO 9000 Quality Standards, CMM, The SQA Plan
Random Testing, Realities of Using Test Tools and Automation, Open Source Testing Tools,
Internal Continuous Assessment (ICA) : Minimum 8-10 assignments on the above topics.
Text books:
1. Software Testing Principles, Techniques and Tools by M G Limaye, Published by Tata
2. Software Testing, Second Edition By: Ron Patton, Published by SAMS, ISBN-13: 978-
Reference books:
5. Lessons Learned in Software Testing by Cem Kaner , James Bach , Bret Pettichord,
Publisher Wiley
6. Testing Computer Software Cem Kaner, Jack Falk, Hung Q. Nguyen, Publisher
Wiley
978-1- 84951-574-0 8. Dr. K.V.K.K. Prasad, “Software Testing Tools: Covering Win
Runner, Silk Test, Load Runner, JMeter and Test Director With Case Studies”,
Reference tutorials:
1. Spoken Tutorials on Selenium Software Testing Framework at http://spokentutorial.
org/tutorial-search/?search_foss=Selenium&search_language=English
We must do partial testing because we only have enough resources (time and money) to run relatively
few test cases
Partial testing can never prove the absence of defects if the system passes all your test cases, there
could still be defects; you just need more or better test cases to find them.
Effective testing lies in intelligently choosing the relatively few test cases that will actually be executed
–Test all requirements and features defined in the requirements spec. and functional spec.
–Test cases should not be redundant (i.e., each one should follow a different path through the code)
–Analyze the program’s design and code to find potential weak areas
–Analyze all points at which data enters the system and look for ways to attack it.
Approaches for test case design are generally divided into two broad categories: Black Box Testing and
White Box Testing
•Black Box testing the tester has limited knowledge of the inner workings of the item being tested
Test cases are based on the specification of the item's external behavior
•White Box Testing The tester has knowledge of the inner workings of the item being tested
Test cases are based on the specification of the item's external behavior AND knowledge of its internal
implementation.
Testing is unlike other software development activities because the goal is to break the software rather
than to create it
•Effective testing requires the assumption that you will find defects
•If you think you won't find defects, or you don't want to, you will have set up a self-fulfilling prophecy
•Testing by both developers and an independent testing group are essential They have different
perspectives and motivations
They do different kinds of tests (developer does white box, test team does black box), which tend to
discover different types of defects.
One goal of testing is to identify the most problematic modules Redesign may be needed if there is an
inherent design flaw
How many defects should you expect to find? It depends on your development process
Automation of test cases is essential to make frequent re-running of test cases feasible
•A lot of the interesting testing work is found in inventing and creating ways to automate test cases
(i.e., create programs whose purpose is to test other programs)
•Automation requires a lot of software design and implementation (sometimes called “Test
Engineering”)
There could be various opinions & definitions. We follow the text & references for
Given specifications.
Bug / defect / Fault: it is a deviation from the expected functionality. However, it is not
Purpose of Testing
1. It is done to catch bugs.
Bugs arise due to imperfect communication among the members of the development
Team regarding specifications, design and low level functionality. Statistics show that
If insufficient effort (cost) is spent in QA (includes testing), the rejection ration will
Be high. Rework & recycling will be high and hence the net cost. (Usually the testing
& QA costs are 2% for a consumer product and are about 80% for critical software
The biggest part of s/w cost is the cost of bugs and the corresponding rework.
Testing, as part of QA, should focus on bug prevention. The art of test design one
Of the best bug preventers have known. Test design and tests should provide clear
first and then writing the code) can discover and eliminate bugs at every stage of the
SDLC. To the extent that the testing fails to achieve its primary goal (bur
The last pair of definitions is testing and quality assurance (sometimes shortened to QA).
These two terms are the ones most often used to describe either the group or the process that’s
Verifying and validating the software. In Chapter 20, “Software Quality Assurance,” you’ll
learn more about software quality assurance, but for now, consider these definitions:
• The goal of a software tester is to find bugs, find them as early as possible, and make
Sure they get fixed.
• A software quality assurance person’s main responsibility is to create and enforce standards
and methods to improve the development process and to prevent bugs from ever
Occurring.
Of course, there is overlap. Some testers will do a few QA tasks and some QA-errs will perform
A bit of testing. The two jobs and their tasks are intertwined. What’s important is that you know
what your primary job responsibilities are and communicate that information to the rest of the
Development team. Confusion among the team members about who’s testing and who’s not has
Caused lots of process pain in many projects. The purpose of SQA is to find and report defects AND
succeed in getting them fixed
What is a software defect?
Definition #1: There is a mismatch between the program and its requirements spec or functional spec
This definition is fine if a requirements specification exists and is complete and correct (not always true)
Definition #2: The program does not do what its end user reasonably expects it to do this definition
always applies, even when there's no specification.
The longer defects remain in the system, the more expensive they become The cost of a defect grows
dramatically the longer it remains in the system
What is the cost of a defect in the requirements specification if it’s found during requirements phase?
During implementation?
•SQA should be performed throughout the software development life cycle it’s not something you do
only at the end after everything's pretty much done.
Software Testing
Formal Verification
The concept of independent Testing did not prevail during the initial days of Software development. It
was believed that whatever the developers were doing was the best way of producing product & customer was
expected to use it as it is. If there were many problems reported by customer / user, they would be fixed by
developers, & the application would be given back to user. The primary responsibility of testing was with
customer/ user during production phase.
Glenford Myer introduced Software Testing as a separate phase in SDLC. According to him, Tester was
expected to test software with all possible combinations. The main intention was to create software which
would never fail in production. Tester was expected to have an attitude to break the software so that it would
be eventually corrected & would never fail during use. This approach separated debugging from testing &
independent testing community was created as a different phase of STLC.
During the initial phase, testing was considered as a part of development, developers were expected to
perform debugging on the application, which they were building. Tests were not document & were mainly
done in a heuristic way. Generally testing was completely positive testing to see whether the implementation
was working correctly or not.
This approach was basis of Glenford, Myers theory of s/w testing. As per this approach it was not
sufficient to only test the s/w positively but users must also be protected from any conceivable failure of
application. The tester’s responsibility changed from proving that s/w works under normal conditions to
proving that s/w does not fail at some abnormal instances.
Often the testers were too imaginative in breaking the s/w, defects for which there were no feasibility
of happening were reported as defects which was quite frusteting to s/w developers. Testers as considered as
demons and testing was considered as hurdle to be past before delivering application to costumer.
Approaches to testing
There are many approaches to s/w testing defined by experts in s/w quality and testing. The
approaches may differ significantly as per customer requirements, type of system being developed as well as
management thinking about s/w development life cycle followed by type of project and customer with
maturity of development team.
It can be considered simply as s/w testing life cycle (STLC). With following phases
1) Requirement testing
2) Design testing
3) Code testing
4) Test scenario and test case
5) Defect and report generation
S/w testing is disciplined approach it execute s/w and finds defects in it. The intention of s/w testing is to
find all possible failures, so that eventually these are eliminated and a good product is given to the
customer. It works on the principle that no s/w is defect free but less risk s/w is better and acceptable to the
users.
The tester’s job is to find out defect so that it will be eventually fixed by developers before handovering
product to the user. Completion of testing must yield number of defects which can be analyzed to find the
weaker areas in process. s/w testing is viewed as exercise of doing a SWOT analysis of s/w product were we
can build s/w on basis of strength, weakness , strong process present in these areas supporting
development of good product . We can always rely on this process and try to deploy then in other areas
Weaknesses: - The area of s/w was requirement compliance is on verge of failure may represent week
areas. It may not be failure of that movement but, it may be on boundary condition of compliance and if
Opportunities: - Some areas of s/w which satisfy requirement as defined by customer or implied
requirements but with enough space available for improving it further. This improvement can lead to
customer delight. These improvements represent ability of developing organization to help the customer
and give competitive advantages. It decides capability of developing organization to provide expert advice
and help to customer for doing something better.
Threats: - Threats are the problems or defects with the s/w which result into failures. They represent
possible loss.
3) If the test foes not detect a defect in present system, it is an unsuccessful test.
4) A test that detects a defect is valuable investment for valuable development and it helps to improve
a product.
At many places s/w testing is termed quality assurance activity. In reality testing is quality control
activity similar many misconceptions about s/w testing are listed below.
Testing needs to be performed according to process defined for it. It needs skilled and trained people
to break the application and demonstrate problems in s/w product, some key points in testing are (principles)
Test policy defines the action part of test policy. It defines the ways and means to achieve test policy.
There is single test policy at organization level for product organizations, while test strategy may differ
from product to product, customer to customer and time to time.
Ex. 1) Definitation of coverage like requirement coverage or functional coverage define for particular
product, project and customer
How much testing would be done manually and what can be automated, number of developers to
tester
TEST PLANNING:-
It is the first activity of test team. If one does not plan for testing then he/she is planning for failures.
Test planes are intended to plan for testing throughout testing development life cycle. Test plans are defined in
framework created by test strategy and established by test policy.
Test planes are made for execution which involves various stages of s/w testing associated with s/w
development process. Test planes should be realistic and talk about limitations, risk and assumptions about
testing
CHALLENGES IN TESTING:-
Methods:-
3) Identify tactical risks related to development (structural risk, technical risk, size risk).
Testing process made of many milestones. Testers need to achieve them one by one. Following are few
milestones.
10) performing retesting, regression testing when defects are resolved by development team
TEST METHODOLOGIES/APPROACHES:-
Testing need a disciplined approach. A tester is the person interested by an organization to work as
devils agent. He/ she are a person working for client finding defects in process and the product.
The main purpose of testing is to demonstrate the defect are present and point towards weaker areas
in s/w as well as process, so tester must have following skills.
A) General skills:-
3) Facilitation skill
B) Technical skill
2) Concepts of testing
3) Levels of testing
• Interpersonal Skills – remain sensitive and constructive in complex, political, stressful situations
• Creative, Intuitive, Critical Thinking Skills – understanding the possibilities of the product and how it can
fail
What is Defect?
1. Failure
Symptom of a fault
2. Fault
Symptom of an error
3. Error
Testing Development
Defect Open
Fixed
Ready for Deferred
Closed Retest
Re-Open
1.New
2.Open
3.Assign
4.Fixed
5. Ready for Re-Test
6.Closed
7. Re-open
8. Rejected
9. Deferred
1. New: When the bug is posted for the first time, the status of the Bug will be “NEW”.
2. Open: After the tester has posted a bug, the technical lead approves it, if the bug is genuine and
changes the state to “OPEN”.
3. Assign: The Technical lead assigns the bug to respective developer, and changes the status to
“ASSIGN”.
4. Fixed: Responsible Developer fixes the bug.
5. Ready for Re-Test: Once the developer fixes the bug, the modified build is sent to the testing team.
The corresponding Test Lead changes the status to “Ready for Re-Test”
6. Closed: During Re-Testing, if the Bug is fixed properly, then the Tester will change the status to
Closed.
7. Reopened: During Re-Testing, if the Bug is not fixed properly, then the Tester will change the status
to Re-Opened, and sends it back to the corresponding developer/development team
8. Rejected: If the Bug is not a genuine one, then the developer changes the Status of the Bug to
“Rejected”.
9. Deferred: When the bug does not stand importance to this particular release and it can be fixed in
the later releases, then the developers will change the status to “Deferred”.
Defect categories
Extra: A requirement incorporated into the product that was not specified.
People with different attitudes see things differently. Here is my favorite example: When 3 friends – a
carpenter, a painter, and a biologist visited a forest and shared their observations, they found out that:
– For the user, it’s an instrument to solve specific tasks or to satisfy a need (e.g., to transfer
money or to read the news)
– For the PM, it’s a way to bring to life the ideas imprinted in the specs.
Brain Positioning
The expression, “Software has bugs,” is not a joke or an exaggeration; it’s the Universal Cosmic Law.
So, let’s open our hearts to that Law! Let’s have an unquestionable belief that:
– If software seems to work, it’s not normal – it means that something is wrong
We’ve all heard these words of wisdom: “Ask and you shall receive.” As testers, we “ask” for bugs:
The less we trust software, the more bugs we find. The more bugs we find, the less we trust
software. Nice cycle!
The lack of trust towards software is connected with another tester’s virtue: destructive thinking! We
are not silent observers of this buggy substance called “software”; we actively find ways to demonstrate that
the software is breakable. And what is the best way to show that the software is breakable? By finding bugs!