BDD Scenarios in a Testing & Traceability Strategy (Webinar 19/02/2021)

19/02/2021
Gáspár Nagy
coach • trainer • bdd addict • creator of specflow
http://bddbooks.com • “Discovery” • “Formulation”
@gasparnagy • gaspar@specsolutions.eu

Copyright © Gaspar Nagy
Gáspár Nagy
coach, trainer and bdd addict
creator of SpecFlow
gaspar@specsolutions.eu
https://specsolutions.eu
@gasparnagy
http://bddbooks.com
bdd addict
given.when.then
CAUTION!
on air

Today…
• Super brief BDD & SpecSync intro
• The SpecSync quality strategy
• Multi-target BDD scenarios
• Living Documentation and Traceability
• Wrap-up, Q&A

This is a BDD scenario
Scenario: Discount is applied when item added to the cart
Given a user without submitted order
And they selected a book that costs €50
When the user adds the book to the cart
Then the cart sub-total should be €45

Behaviour Driven Development is about
understanding, documenting & validating
business requirements
through illustrative examples

BDD Practices
Image source: https://cucumber.io/docs/bdd/
Discovery
Shared understanding is established through
collaboration and structured conversations
Formulation
The examples of system behaviour are
documented as scenarios
Automation
Scenarios are automated to be able to verify
the behaviour of the system

SpecSync for Azure DevOps
synchronize • share • trace
<TRX>
<JUnit XML>
<Cucumber JSON>
<…>
synchronize
(push)
(pull)
run tests
BUG
1234
share
(publish test results)
trace

Started in 2015
Cross-platform (Win/Linux/macOS/Docker)
Supports all major BDD Frameworks (SpecFlow, Cucumber, Behave, …)
50000+ line of code (C#)
Survived 2 major updates and an ADO API change
Build with BDD/SpecFlow (180+ scenarios, 300+ BDD tests)
Uses SpecSync for traceability – eat your own dogfood

Rule: Can create new Test Case work items from scenarios
Scenario: A new scenario is linked to a new Test Case
Given there is an Azure DevOps project
And there is a scenario in the local repository
"""
Scenario: Sample scenario
Given there is something
When I do something
Then something will happen
"""
When the local repository is synchronized with push
Then a new Test Case work item "Scenario: Sample scenario" is created in the project
And the Test Case contains the following test steps
| step |
| Given there is something |
| When I do something |
| Then something will happen |
And a tag "@tc:[id-of-new-test-case]" is added to the scenario in the local repository

The
Pyramid
is
not
enough
Test Automation Pyramid: Mike Cohn – Succeeding with Agile
Pic: http://gasparnagy.com/2017/02/balancing-scenarios-and-unit-tests-case-study-with-specflow/

There
is
a
carnival
of
pyramids!

More Agile Testing: One-page test plan
Source: More Agile Testing, page 95, Bernice Niel Ruhland

The four-page quality strategy
• A quality strategy is like an overall test plan – gives a guidance for the whole
team about how we address quality
• Custom & specific to your project, but keep it short
• SpecSync’s strategy* has the following sections:
• Challenges (Risks)
• Quality Aspects
• Testing Methods
• Each with: purpose, automation strategy, structure, out-of-scope
• Feedback channels
*Download current snapshot from: http://speclink.me/webinar190221-strategy

Challenges (Risks)
• SpecSync is an integration product
• The quality cannot be fully ensured
• Quality changes externally (e.g., new version of ADO comes out)
• The majority of the logic cannot be tested without integration
• Supports 8 platforms, 4 target ADO versions, 15 BDD Frameworks and 8 SpecFlow
versions
• ADO versions: TFS 2017, TFS 2018, ADO 2019, ADO Services (cloud)
• Critical for clients:
• chance of blocking traceability (e.g., traceability cannot be enforced if SpecSync does not
work)
• chance of obscuring traceability (e.g., generating unnecessary changes)
• chance of data loss (e.g., SpecSync deletes important data due to a bug)

Why the Pyramid® is not enough as a strategy

Breaking down the undefinable quality:
quality aspects (sample)
Functional
• Works as
expected
• Expectations
are good
• Expectations
are
documented
Operational
• Secure
• Fast
• Convenient
• Pretty
• Consistent
• Predictable
Strategic
• Maintainable
• Architecture
• Code quality
• Easy to
integrate
• Flexible
Source: Kevlin Henney

SpecSync quality aspects
Functional
• Features documented and
work
• Supports different BDD
frameworks
• Supports different ADO
versions
• Handles variations in
configuration
• Handles variations in BDD
framework test results
• Handles ADO integration
edge cases (version, size
specialties)
Operational
• Works on different platforms
• Secure
• Understandable output, easy
to diagnose
user/configuration errors
• Provides guidance for
configuration
• Works with acceptable speed
(processes at least 100
scenario per minute)
Strategic
• Has a fast & offline
development environment
• Easy to make a fix, hotfix,
proposed fix or spike
• Easy to maintain long-term
• Flexible to support custom
development processes of the
clients
• Easily extensible for new BDD
frameworks and new ADO
versions

Choosing
test
methods
Automated BDD Scenarios
(SpecFlow, Integration)
SpecSync
ADO
ADO
2019
TFS
2017
TFS
2018
STUB
ADO
SpecSync Algorithmic Logic
Tests (MsTest, Unit)
ADO Integration Tests
(MsTest, Integration)
sync
engine
service classes
console API
Smoke Tests
(SpecFlow, E2E)
Exploratory
Features documented and work
Supports different ADO versions
Has fast & offline dev env
Works on different platforms
Handles variations in BDD frw test results Handles ADO integration edge cases
Understandable output

Testing
methods
&
Feedback
channels

Same test – different target
Rule: Establish work item links based on scenario tags
@tc:155
Scenario: A work item link tag added to the scenario
Given there is an Azure DevOps project with a Bug work item
And the synchronizer is configured as
| setting | value |
| synchronization/links[]/tagPrefix | bug |
And a @bug tag has been added to an existing synchronized scenario as:
"""
@tc:[id-of-test-case] @bug:[id-of-bug]
When I do something
"""
Then the test case is linked to the Bug work item

Before “Gateways and Drivers”
Automated BDD Scenarios (SpecFlow, Integration)
SpecSync
sync
eng.
ADO
ADO
2019
TFS
2017
TFS
2018
STUB
ADO
scenario steps
step definitions
service classes
Given 2
When 2
Given 1
Then
And there is a scenario that has been synchronized already
When a @bug:[bug-id] tag is added to the scenario
And SpecSync push is executed
file
system
When 1

Gateways and Drivers
Automated BDD Scenarios (SpecFlow, Integration)
SpecSync
sync
eng.
ADO gateway
ADO
ADO
2019
TFS
2017
TFS
2018
STUB
ADO
scenario steps
step definitions
ADO driver interface
app driver
stub driver ADO driver
service classes
local project
driver
file
system

The
Injector

Living Documentation
Scenarios describe the requirements in a business-readable form. As they are
automated, by running them as tests, they can tell us when the implementation
diverges from our expectations – so it becomes living documentation that the
whole team can rely on.
• We are going to check
• Structure
• Level of details
• Tags
• Traceability

The living documentation is a documentation!
http://speclink.me/specsyncdoc

Feature – Rule – Scenario
Feature: Linking Work Items and other artifacts using tags
Scenario: A work item link tag added to the feature
Rule: Link type can be specified
Scenario: "Tests" link is created by default
Scenario: The work item is linked as "Parent“
Scenario: The link type has changed
Rule: Existing links are preserved and links should remain when tag is removed
Scenario: The work item tag is removed from the scenario
Rule: Multiple prefixes can be linked
Scenario: A product backlog link and a bug link tag is added to the scenario
Rule: Pull Requests can be linked […]
Rule: Adding a work item link should trigger an update […]

Details: hide incidental, show real data
@tc:155
| setting | value |
"""
@tc:[id-of-test-case] @bug:[id-of-pbi]
When I do something
"""

Details: hide incidental, show real data
@tc:155
| setting | value |
"""
@tc:[id-of-test-case] @bug:[id-of-pbi]
When I do something
"""
--- table step argument ---
| setting | value |
Applying config settings:
{
"synchronization": {
"links": [
{
"tagPrefix": "bug"
}
]
}
}
-> done: ConfigurationSteps.GivenTheSynchronizerIsConfiguredAs(<table>) (0.3s)

Tags – break out from strict hierarchy
Three examples from SpecSync tags:
• @customization – cross-cutting feature
• Describes “Customization” features: used to support special development process
requirements of the SpecSync clients.
• @notsupported-tfs2017 – exclusion criteria
• The particular scenario is not supported on the target platform (here TFS2017). These
scenarios are excluded from the CI builds testing that target
• @formulated – SpecSync development process
• Allows specifying (formulating) scenarios and commit them before automation. These
scenarios are excluded from CI builds (they would fail anyway).

Show synchronized Test Case
Link to a ticket
Publish test results
Target success matrix
Demo: Traceability

• How to build and document a quality strategy
• BDD scenarios are phrased using the business language, and therefore the same
scenario can be executed against different targets
• Well-structured BDD scenarios can form a living documentation and enable
traceability
What have we seen?

Check out our books!
• BDD books for everyone (PO, BA, dev, tester)
• Practical guide
• Demonstrate good collaboration and
formulation techniques, illustrated by concrete
examples
Find them on Amazon* and on Leanpub
through http://bddbooks.com!
* Formulation is currently only on Leanpub
Use Leanpub coupon for 50% off: webinar021921

https://www.specsolutions.eu/specsync

Gáspár Nagy
coach • trainer • bdd addict • creator of specflow
http://bddbooks.com • “Discovery” • “Formulation”
@gasparnagy • gaspar@specsolutions.eu

BDD Scenarios in a Testing & Traceability Strategy (Webinar 19/02/2021)

More Related Content

What's hot (20)

Similar to BDD Scenarios in a Testing & Traceability Strategy (Webinar 19/02/2021) (20)

More from Gáspár Nagy (12)

Recently uploaded (20)

BDD Scenarios in a Testing & Traceability Strategy (Webinar 19/02/2021)