INFO -6055 -2

Non-functional testing
Thusrday – 10 am to 12 pm
+ 2h async
 Contact information
• Hany ElYamany, PhD
• FOL email: [email protected]
• Office hours:
• Send me an email for an appointment.
 Agenda
• Introduction
• Tell me about yourself
• Course policy and guidelines
• Course evaluations strategy
• Introduction to non-functional testing
• Class discussion – Case Studies
 Laptops
• You will need your laptop for:
• Tests
• Working with testing tools
• Research projects
• You may also use laptops for taking notes or view the slides during
class.
• What about Webcam?
 Class format
• 4 hours per week (2h sync + 2h async)
• Lectures and discussions of course materials
• Help with projects
• Tests
• Students are expected to attend all online classes until a further note
 Non-Functional Testing - Intro
• What is it?
• non-functional requirement
• A requirement that describes how the component or system will do what it is intended
to do.
• non-functional testing
• Testing conducted to evaluate the compliance of a component or system with non-
functional requirements.
• Behavior versus operability
• Aspects of non-functional testing
• Cost of testing
• Cost of not testing
 Measuring Quality

• How do we measure quality?

• Does the software / hardware do what we want it to
do?
• Correct behavior is quality?
• Rich behavior is quality?
• So we can measure quality by the function and
feature of the application.
• Is there more than functional quality?
 Measuring Quality

• If all the functional aspects are as expected, is there

anything that can still impact quality?
• We have tested how the system behaves (functional)
and it passed.
• What about how it operates?
• Can non-behavioral aspects affect quality?
• What are some non-functional aspects of your cell
phone?
 Non-Functional Testing
• What do we test against during functional testing?

Requirements!!!!
• What do we test against during non-functional testing?

Requirements!!!!
We always test against requirements
 Non-functional testing
• Understand the Requirement
• Understand how to detect a pass
• Design a test
• Execute the test
• Record the results
• Sound familiar?
 What is non-functional testing?

• Non-functional testing validates non-functional

aspects (performance, usability, reliability, etc.) of an
application or device.
• It is explicitly designed to test the readiness of a
system as per non-functional requirements which are
never addressed by functional testing.
• It shows “How good the system works”.
 What is non-functional testing?

• Non-functional testing is equally important as

functional testing and affects quality.
• Non-functional requirements can impact perceived
and actual quality.
• Examples of perceived quality (i.e., customers’
perceptions)?
• Examples of actual quality?
 So what are the non-functional
aspects?
• How does the application perform under normal circumstances?
• How does the application behave when too many users log in
concurrently?
• Can the application handle stress?
• How secure is the application?
• Can the application recover from any disaster?
• Can the application behave in the same way in a different
environment or OS?
• How easy is to port the application in a different system?
• Are the documents/user manual provided with the application easy to
understand?
 Objectives of non-functional testing

• Should increase usability, efficiency, maintainability, and

portability of the product.
• Reduce production risk and cost associated with non-functional
aspects of the product.
• Optimize the way the product is installed, setup, executes,
managed and monitored.
• Collect and produce measurements, and metrics for internal
research and development.
• Improve and enhance knowledge of the product behavior and
technologies in use.
 Characteristics of Non-functional
testing
• Non-functional testing should be measurable, so there is no
place for subjective characterization like good, better, best,
etc.
• Exact numbers are unlikely to be known at the start of the
requirement process (baseline?)
• Important to prioritize the requirements
• Ensure that quality attributes are identified correctly
 Serious Non Functional
Pitfalls
• Infrastructure that cannot scale.
• Applications that cannot scale.
• Limited ability to diagnose problems in real time.
• Unexpected production behaviour patterns.
• Applications that exhaust available hardware capacity.
• Failure of the automated failover systems that result in outages.
• Sudden degradation of application performance.
• Low availability – reduced service.
• Complex operation – systems that can only be operated by a highly skilled
team of specialized resources.
 Non-Functional as an Afterthought

• Non functional requirements remain an afterthought on most IT

projects.
• The scope of non functional requirements tends to be limited to
performance and load testing, excluding such things as capacity
planning, operability, monitoring, system health checks, failover,
memory management, and sustainability.
• User and business communities are better at defining their
functional requirements but are not able to articulate the non
functional characteristics of the system.
• Its up to the technology community to build the tools and
methodologies needed to ensure sufficient non functional test
coverage.
 Importance of non-functional
• Non-functional testing is as important as functional testing.
• Non- functional testing ensures that the application works
good enough to meet the business expectations.
• What kind of non-functional expectations might the business
have?
 Comparing Functional and non-
functional
Functional Testing Non Functional Testing
Functional Testing verifies how well the system Non functional testing verifies how well the system
works. responds.
Functional testing verifies the functionality of Non functional testing verifies the Performance of the
the system or how the system behaves. system.

Functional testing is based on the customer Non functional testing is based on the customer
requirement. expectation.
Functional testing has following types: Non functional testing includes:
• Unit testing • Performance testing
• Integration testing • Load Testing
• System Testing • Stress testing
• Acceptance Testing • Volume testing
• Security testing
• Installation testing
• Recovery testing
 Comparing Functional and non-functional

Functional Testing Non Functional Testing

Example of Function testing: Providing Example of Non Function testing: Providing username and
username and password on login screen should password on login page should allow to log into site within
allow the user to log into application. 3 seconds.
Customer feedback helps in reducing risk Customer feedback is more valuable for non functional
factors of the product. testing as it helps to improve and let the tester know the
expectation of the customer.
Functional testing does not necessarily require Non functional testing can be more appropriate/efficient
tools. It can be done effectively manually. using tools.

Functional testing is done based on business Non- functional testing is done based on customer
requirement. expectation and Performance requirement.
 Black Box or White Box
• Non-functional testing comes under black box testing
technique.
• You do not need to have the code
• You do not always need to understand the design
• You do not always need to know the technologies
involved
• In some cases you do need to know some or all of the
above (e.g., the case of the Puzzlingly Poor Performance)
Types of Non-Functional Aspects of Systems
 Manual Testing or Tools?
• Some tests can be doe manually but many cannot.
• There are several tools available in the market for non-functional
testing.
Jmeter Forecast
Loadster Load Complete
Loadrunner Webserver Stress Tool
Loadstorm WebLoad Professional
Neoload Loadtracer
vPerformer
 The cost of Non functional testing

• If you had unlimited resources and unlimited time,

you could increase the scope of non functional
testing.
• However, the reality is, that is not usually the case.
• You will need to make judgement calls on the level of
risk and return on investment, by minimizing the risk
through non functional testing.
 Real World Function
• Business and application development teams place a
real emphasis on the richness of functionality, but
overlook the non functional aspects. However, rich
functionality is of little to no value if users are unable
to access the application.

• The case of the puzzling poor performance.

• The case of the disappearing database.
 The cost of not performing non-functional tests

• What are some non-functional aspects of your,

• mobile phone?
• Sports car
• Badminton racket
• FOL
• How do these aspects affect your satisfaction with
these products?
• Menti.com