Agile ALM Tool Comparison

Vendor Landscape: Agile ALM
ALMs for the poor, and for the rich.

Info-Tech Research Group 1

Introduction

As development platforms, coding methodologies, and devices increase in
number, Agile Application Life Management (ALM) tools support integrations
with an ever-increasing range of systems.

This Research Is Designed For: This Research Will Help You:

Enterprises seeking to select a solution for Agile Understand what’s new in the Agile ALM market.
Application Lifecycle Management (ALM) technology
Evaluate Agile ALM vendors and products for your
enterprise needs.
Their Agile ALM use case may include:
Determine which products are most appropriate for
• Providing testing support for a quality-centric particular use cases and scenarios.
development environment.

• Supporting Agile development.
• Integrating lifecycle management with Integrated
Development Environments.


Executive Summary

Info-Tech evaluated eight competitors in the Agile ALM Info-Tech Insight
market, including the following notable performers:
1. ALM and BPM start to intermingle:
Champions:
Premium offerings give the ability to
• CollabNet TeamForge, a comprehensive, full-featured ALM tool customize process. Evaluate your need
at a reasonable price. Despite its impressive range of offerings, for, and the cost of, process customization
CollabNet supports the SMB. before you make your purchase.
• Micro Focus Borland ALM, a long-standing leader in the market
with a broad feature offering and strong product support. 2. The range of platforms is growing:
• Serena Suite, a robust offering backed by strong customer Premium products support development
support and a broad reseller network. across the growing range of platforms and
development methodologies, including
Value Award: mobile, web, and infrastructure.
• TechExcel DevSuite, a comprehensive offering with a price tag
far below its comparable competitors. 3. New entrants offer attractive options:
Niche offerings can serve your needs at a
fraction of the cost of a premium tool. For
example, consider specialized tools for
Agile and testing integration.


Market Overview

How it got here Where it’s going
• Long-standing vendors in the ALM space can trace their • Agile ALM has moved into the Cloud. Major vendors of
roots to the 1980s, or even earlier. cloud collaboration software have moved into the ALM
space and new SaaS-only ALM tools have appeared.
• Since the 1990s, ALM tools have played a key part in
the project manager’s arsenal, allowing him or her to • Process flexibility has become key. Instead of a strict
track project status and progress towards objectives. adherence to Agile or waterfall development, most
firms have pursued a middle path and customized
• Managers have paid top dollar for ALM suites that could their methodology to meet their own needs. Some
track project data and provide meaningful reporting. Agile ALM tools cater to the need for flexibility.

• As development environments have evolved, the tools • Quality control has become a core part of ALM.
and components being used to manage this process Several major ALM tools are built around testing tools
have grown to become integrated and convenient, and process maturity.
covering all phases of the development lifecycle
including architecture, testing, and deployment with a • Going forward, expect stronger integration between
single common interface. ALM tools and the ecosystem of products supporting
development, such as testing, PLM tools, and IDEs.

As the market evolves, capabilities that were once cutting edge become default, and new functionality
becomes differentiating. Traceability has become a table stakes capability and should no longer be used
to differentiate solutions. Instead, focus on workflow and reporting to get the best fit for your requirements.


Agile ALM Vendor Landscape selection / knock-out criteria:
Market share, mind share, and market consolidation
• The Agile ALM space continues to see new entrants, new products, and new features. Ongoing demand for development
on mobile and web platforms creates new opportunities for vendors focused on a particular market niche.
• For this Vendor Landscape, Info-Tech focused on those vendors that have a strong market presence and/or reputational
presence among small to mid-sized enterprises.

Included in the Vendor Landscape:

• CollabNet TeamForge. A relatively recent entrant with a strong offering and a reasonable price tag.

• Micro Focus Borland ALM. A large, ALM-focused vendor that provides strong support for its product.

• PTC Integrity. A strong ALM offering tightly integrated into a PLM platform.

• Parasoft Concerto. A product targeted towards development shops with a high level of process maturity.

• Serena Suite. The complete ALM package controlled by a complete business process engine.

• SmarteSoft SmartSuite. ALM tools built around a quality-control platform.

• TechExcel DevSuite. A flexible tool that supports development on a variety of platforms.

• ThoughtWorks Suite (Mingle, Twist, Go). A strong, Agile-focused product offering that provides some flexibility
within the Agile framework.


Agile ALM Criteria & Weighting Factors

Product Evaluation Features Usability
30% 30%
The solution provides basic
Features and advanced feature/functionality.

The solution’s dashboard and reporting tools Architecture 20% 20% Affordability
Usability are intuitive and easy to use.

The five year TCO of the solution is Product
Affordability economical.
50%
The delivery method of the solution aligns with
Architecture what is expected within the space.

Vendor Evaluation
50%
Vendor is profitable, knowledgeable, and will Vendor
Viability be around for the long term.

Vendor is committed to the space and has a Viability Strategy
Strategy future product and portfolio roadmap. 25% 30%

Vendor offers global coverage and is able to
Reach sell and provide post-sales support. 15%
Channel Reach
30%
Vendor channel strategy is appropriate and the
Channel channels themselves are strong.


The Info-Tech Agile ALM Vendor Landscape

Champions receive high scores for most
evaluation criteria and offer excellent value.
They have a strong market presence and
are usually the trend setters for the industry.

Innovators have demonstrated innovative
product strengths that act as their
competitive advantage in appealing to niche
segments of the market.

Market Pillars are established players with
very strong vendor credentials, but with
more average product scores.

Emerging players are newer vendors who
are starting to gain a foothold in the
marketplace. They balance product and
vendor attributes, though score lower
relative to market Champions.

For an explanation of how the Info-Tech Vendor Landscape is created please see the slide entitled “Vendor Evaluation Methodology” in the appendix.


Every vendor has its strengths & weaknesses;
Pick the one that works best for you

Product Vendor
Afford- Archi-
Overall Features Usability Overall Viability Strategy Reach Channel
ability tecture

CollabNet

PTC

Micro Focus

Parasoft

Serena

SmarteSoft

TechExcel

ThoughtWorks

Legend =Exemplary =Good =Adequate =Inadequate =Poor

For an explanation of how the Info-Tech Harvey Balls are calculated please see the slide entitled “Vendor Evaluation Methodology” in the appendix.


TechExcel captures the highest value score with a
comprehensive, affordable offering
What is a Value Score? On a relative basis, TechExcel
Champion
maintained the highest Info-Tech Value
ScoreTM of the vendor group. Vendors
The Value Score indexes each vendor’s product
were indexed against TechExcel’s
offering and business strength relative to their
performance to provide a complete,
price point. It does not indicate vendor ranking.
relative view of their product offerings.
Vendors that score high offer more bang for the
buck (e.g. features, usability, stability, etc.) than
the average vendor, while the inverse is true for
those that score lower.

Price-conscious enterprises may wish to give the
100
Value Score more consideration than those who
are more focused on specific vendor/product
attributes.
64 Average Score: 31.5
55

22
13
0 0 0
TechExcel CollabNet Serena SmarteSoft Micro *PTC *Parasoft *Thought-
Focus Works
*Vendor declined to provide pricing.

For an explanation of how the Info-Tech Value Index is calculated please see the slide entitled “Value Index Ranking Methodology” in the appendix.
For an explanation of how normalized pricing is determined please see the slide entitled “Product Pricing Scenario & Methodology” in the appendix.


Table Stakes represent the minimum standard; without these
a product doesn’t even get reviewed

The Table Stakes What Does This Mean?

Feature Description The products assessed in this Vendor
LandscapeTM meet, at the very least, the
Basic task The product allows the user to create
requirements outlined as Table Stakes.
management tasks, assign resources to those tasks, and
view project status reporting.
Many of the vendors go above and beyond the
Graphical displays The product displays tasks and other outlined Table Stakes, some even do so in
objects as editable, graphical objects. multiple categories. This section aims to highlight
the products capabilities in excess of the criteria
Traceability The product ties together objects created listed here.
at various stages of the lifecycle, showing
how tasks relate to requirements, etc.
Links to external The product allows the user to link to
documents external documents for requirements and
other content.
Central data store The product stores data in a centralized
repository, not on local disk.

If Table Stakes are all you need from your Agile ALM solution, the only true differentiator for the
organization is price. Otherwise, dig deeper to find the best price to value for your needs.


Advanced Features are the market differentiators that make or
break a product
Scoring Methodology Advanced Features

Info-Tech scored each vendor’s features offering
Feature What We Looked For
as a summation of their individual scores across Requirements Users can enter requirements in a
the listed advanced features. Vendors were given Management hierarchical structure.
one point for each feature the product inherently
provided. Some categories were scored on a Change Management The product packages and traces changes.
more granular scale with vendors receiving half
points. Workflow The product provides automated workflow.

Source Code The product manages source code or
Management integrates with SCM solutions.
Task Management The product manages timelines and tasks.

Testing The product provides test management.

Defect/Bug Tracking The product manages and traces defects.

Resource The product permissions users to edit
Management objects and evaluates resource capacity.
Reporting & Analytics The product analyzes project progress and
provides meaningful reporting.
Release Management The product helps with release planning.

Content Management The product organizes documents.


Each vendor offers a different feature set; concentrate on what
you need

Evaluated Features
Req. Change Work- Task Defect Res. Report- Rel. Content
SCM Testing
Mgmt Mgmt flow Mgmt Track Mgmt ing Mgmt Mgmt
Collab-
Net

PTC

Micro
Focus

Parasoft

Serena

Smarte-
Soft
Tech-
Excel
Thought-
Works
Legend =Feature fully present =Feature partially present/pending =Feature absent


CollabNet offers a strong product with a broad range of
features and a reasonable price tag
Champion Overview
• A recent entrant to the ALM space (but in SCM since 1999),
Product: TeamForge
CollabNet offers the range of functionality normally expected
Employees: Unavailable
by large organizations, but targets the SMB market with a
Headquarters: Brisbane, CA
relatively affordable price tag.
Website: Collab.net
Founded: 1999 Strengths
Presence: Private company
• TeamForge provides all of the functionality associated with full-
featured ALM, from requirements through to release.
• Despite its robust offering and some well-known clients,
CollabNet has a large number of SMB clients as well.
• CollabNet has strong support offerings, both for its clients and
its channel partners.

Challenges
• CollabNet’s limited revenue and emphasis on North America
sales restricts its global reach. However, the company has
experienced strong revenue growth.
• TeamForge’s native resource management solution lacks
$1 $1M+
functionality, although it offers basic capacity management.
3 Year TCO: Tier 8; between $250K and $500K

Info-Tech Recommends:
CollabNet offers a very strong product at a reasonable price, supported by a recognized vendor in the
source code management space.


Micro Focus provides full-featured ALM with strong customer
and reseller support
Champion Overview
• With 30 years of experience in application management, Micro
Product: Borland ALM
Focus continues to lead the market with a strong offering.
Employees: 1,200
Headquarters: Newbury, UK
Website: microfocus.com
Presence: LON: MCRO
FY11 Revenue: $436M • The Borland ALM suite benefits from Micro Focus’s strong
global reach and extensive experience in ALM.
• Micro Focus combines large size with a focus on ALM.
• Micro Focus offers strong support for its products, including
multi-language, 24x7 support.
• Micro Focus’s reseller networks stretch across four continents.

Challenges
• The Borland ALM suite excludes some key functionality such
as resource utilization management, assuming that the user
will integrate with a project management tool.
• Borland ALM does not have burn-down reports or agile task
$1 $1M+
management reporting.

Organizations seeking a premium product and willing to pay a substantial amount should consider
Micro Focus.


Serena provides all the ALM tools, packaged within a powerful
process orchestration engine
Champion Overview
• Serena continues to achieve competitive differentiation
Product: Serena Suite
through its Serena Business Manager, a full-fledged process
Employees: 700
management solution that orchestrates development.
Headquarters: Redwood City, CA
Website: serena.com
• Serena includes all of the functionality associated with full-
featured ALM, from requirements to release management.
• Serena has proven longevity in the ALM space, having entered
the market over 30 years ago.
• Serena’s support offices and reseller network stretch across
four continents. Support is available in ten languages.

Challenges
• Serena support centers provide only callback service on
weekends.

$1 $1M+

Serena will make sense for organizations needing a high level of process customizability and
workflow, and willing to make a substantial investment.


TechExcel offers flexibility and a range of ALM features;
support offering is weak
Innovator Overview
• Despite being a long-standing player in the ALM business,
Product: DevSuite
TechExcel has only a small share of the market.
Employees: 155
Headquarters: Lafayette, CA
Website: techexcel.com
• DevSuite includes most of the functionality associated with full-
featured ALM, from requirements to task management.
• DevSuite offers a high degree of flexibility, making it useful for
development on a range of platforms.
• TechExcel integrates with a variety of development tools and
supports development using diverse methodologies.

Challenges
• TechExcel offers relatively weak support for a major player,
providing telephone support only during business hours.
• TechExcel lacks key release management functionality and
some reporting and analytics capabilities as well.
$1 $1M+
• Despite its longstanding presence in the market, TechExcel
3 Year TCO: Tier 7; between $100K and $250K lacks a strong global revenue footprint.

Organizations seeking a full-featured ALM solution should consider TechExcel, particularly if they
have limited support needs.


Concerto provides a range of functionality and targets high-
maturity development shops
Market Pillar Overview
• A long-standing player in the ALM market, Parasoft has a wide
Product: Concerto
array of patent holdings that gives it a competitive edge in the
marketplace.
Headquarters: Monrovia, CA
Website: parasoft.com
• Concerto includes most of the functionality associated with full-
featured ALM, including requirements and task management.
• Concerto can integrate with .NET and Eclipse IDEs.
• Concerto allows the user to create custom workflows in UML.
• Parasoft has a strong market focus on the ALM space.

Challenges
• Concerto does not include release management functionality
and Parasoft has no intention of adding this.
• While Parasoft has clients in the SMB space, Concerto targets
a level of process maturity (CMMI 3 or above) that most Info-
$1 $1M+
Tech SMBs do not have.
Pricing was not made available • Parasoft has a limited footprint compared to larger players.

Concerto will provide strong support for organizations of any size that require a high level of
development process maturity.


PTC Integrity provides a range of ALM offerings within a
Project Lifecycle Management context
Emerging Player Overview
• PTC acquired MKS in 2011 to integrate MKS’s Integrity ALM
Product: Integrity
offerings into PTC’s suite of Product Lifecycle Management
Employees: 5,000
(PLM) tools.
Headquarters: Needham, MA
Website: ptc.com
Presence: NASDAQ: PMTC
FY10 Revenue: $1B • Integrity provides almost all of the functionality associated with
full-featured ALM, from requirements to task management.
• Integrity supports both Agile and waterfall methodologies, plus
the ability to graphically edit processes.
• Integrity benefits from strong integration with other lifecycle
management tools sold by PTC.

Challenges
• The target client for the product has at least 25 software
developers, making it a difficult proposition for many SMBs.
• PTC limits phone support to weekdays, unusual for a vendor
of this size.
$1 $1M+
• While PTC has a long history in the PLM space, its ability to
Pricing was not made available manage the ALM side of the business remains unproven.

Integrity has a strong offering for larger organizations, or any organization seeking to integrate ALM
with a broader PLM suite.


SmarteSoft provides full-featured ALM at a substantial price;
support offering is weak
• SmarteSoft entered the ALM field in 2005 with its SmarteSuite
Product: SmarteSuite
of tools.
Employees: 50
• The suite centers around SmarteQM, a quality-oriented ALM
Headquarters: Austin, TX
and testing tool.
Website: smartesoft.com
• Despite its positioning as a test automation tool, SmarteQM
has most of the functionality associated with ALM.
• Half of SmarteSoft’s clients are still mid-sized businesses, a
relatively strong focus on the SMB client.
• Customizable workflow in SmarteQM allows users to adapt to
various development platforms and methodologies.

Challenges
• SmarteSoft offers relatively weak support given the cost of
SmarteQM: initial contact is usually via email with a 24-hour
turnaround time.
• Given its recent entry, SmarteSoft lacks proven longevity in the
$1 $1M+
space, a concern given the high price tag.

SmarteSoft’s credentials in test management software will make this a suitable choice for users with
heavy testing and quality management needs.


ThoughtWorks offers a strong offering for Agile development
but limited support for waterfall or other methods
• A recent entrant into the Agile ALM market, ThoughtWorks
Product: Mingle, Twist, Go
provides a robust offering but limited support for non-Agile
development activity.
Headquarters: Chicago, IL
Website: thoughtworks-studios.com
• ThoughtWorks includes most of the functionality associated
with full-featured ALM, from requirements to task
management.
• ThoughtWorks has an organizational focus on Agile ALM and
long-standing experience in Agile development.
• ThoughtWorks testing is designed for a variety of platforms.

Challenges
• ThoughtWorks offers email-only support and a one-day
turnaround, a relatively weak support offering.
• ThoughtWorks has a strong focus on the Agile methodology,
limiting the usefulness of the tool for waterfall or other
$1 $1M+
methods.
Pricing was not made available • As well, integration with .NET is not offered.

ThoughtWorks can make sense for Agile-oriented organizations, or organizations seeking training and
help in moving towards Agile.


Identify leading candidates with the Agile ALM Vendor
Shortlist Tool
The Info-Tech Agile ALM Vendor Shortlist Tool is designed to generate a
customized shortlist of vendors based on your key priorities.

This tool offers the ability to modify:
• Overall Vendor vs. Product Weightings

• Top-level weighting of product vs. vendor
criteria

• Individual product criteria weightings:
Features
Usability
Affordability
Architecture

• Individual vendor criteria weightings:
Viability
Strategy
Reach
Channel


Quality-sensitive development shops require tight integration
with a top-notch testing tool

Look for a vendor that provides a full-featured testing solution as part of the
ALM offering.
Exemplary Performers

1 Quality-sensitive
development

Viable Performers
2 Agile focused

3 Development integrated Adequate Performers

4

Agile-focused environments should seek a tool with strong
support for Agile

Seek an ALM product with native support for Agile processes and artifacts.


1 Quality-sensitive
development

Viable Performers
2 Agile focused


4

Organizations requiring development integration should seek
flexible integration capabilities

Look for native integrations as well API-based integrations for niche IDEs.


1 Quality-sensitive
development

Viable Performers
2 Agile focused


4

Appendix

• Vendor Evaluation Methodology
• Value Index Ranking Methodology
• Product Pricing Scenario & Methodology


Vendor Evaluation Methodology

Info-Tech Research Group’s Vendor Landscape market evaluations are a part of a larger program of vendor evaluations that includes Solution
Sets that provide both Vendor Landscapes and broader Selection Advice.

From the domain experience of our analysts, as well as through consultation with our clients, a vendor/product shortlist is established. Product
briefings are requested from each of these vendors, asking for information on the company, products, technology, customers, partners, sales
models, and pricing.

Our analysts then score each vendor and product across a variety of categories, on a scale of 0-10 points. The raw scores for each vendor are
then normalized to the other vendors’ scores to provide a sufficient degree of separation for a meaningful comparison. These scores are then
weighted according to weighting factors that our analysts believe represent the weight that an average client should apply to each criteria. The
weighted scores are then averaged for each of two high level categories: vendor score and product score. A plot of these two resulting scores
is generated to place vendors in one of four categories: Champion, Innovator, Market Pillar, and Emerging Player.

For a more granular category by category comparison, analysts convert the individual scores (absolute, non-normalized) for each
vendor/product in each evaluated category to a scale of zero to four whereby exceptional performance receives a score of four and poor
performance receives a score of zero. These scores are represented with “Harvey Balls,” ranging from an open circle for a score of zero to a
filled in circle for a score of four. Harvey Ball scores are indicative of absolute performance by category, but are not an exact correlation to
overall performance.

Individual scorecards are then sent to the vendors for factual review, and to ensure no information is under embargo. We will make corrections
where factual errors exist (e.g. pricing, features, technical specifications). We will consider suggestions concerning benefits, functional quality,
value, etc; however, these suggestions must be validated by feedback from our customers. We do not accept changes that are not
corroborated by actual client experience or wording changes that are purely part of a vendor’s market messaging or positioning. Any resulting
changes to final scores are then made as needed, before publishing the results to Info-Tech clients.

Vendor Landscapes are refreshed every 12 to 24 months, depending upon the dynamics of each individual market.


Value Index Ranking Methodology

Info-Tech Research Group’s Value Index is part of a larger program of vendor evaluations that includes Solution Sets that provide both Vendor
Landscapes and broader Selection Advice.

The Value Index is an indexed ranking of value per dollar as determined by the raw scores given to each vendor by analysts. To perform the
calculation, Affordability is removed from the Product score and the entire Product category is reweighted to represent the same proportions.
The Product and Vendor scores are then summed, and multiplied by the Affordability raw score to come up with Value Score. Vendors are
then indexed to the highest performing vendor by dividing their score into that of the highest scorer, resulting in an indexed ranking with a top
score of 100 assigned to the leading vendor.

The Value Index calculation is then repeated on the raw score of each category against Affordability, creating a series of indexes for Features,
Usability, Viability, Strategy and Support, with each being indexed against the highest score in that category. The results for each vendor are
displayed in tandem with the average score in each category to provide an idea of over and under performance.

The Value Index, where applicable, is refreshed every 12 to 24 months, depending upon the dynamics of each individual market.


Product Pricing Scenario & Methodology

Info-Tech Research Group provided each vendor with a common pricing scenario to enable normalized scoring of Affordability, calculation of
Value Index rankings, and identification of the appropriate solution pricing tier as displayed on each vendor scorecard.
Vendors were asked to provide list costs for Agile ALM software licensing to address the needs of a reference organization described in the
pricing scenario.
Additional consulting, deployment, and training services were explicitly out of scope of the pricing request, as was the cost of enhanced
support options, though vendors were encouraged to highlight any such items included with the base product acquisition. Vendors were asked
to prepare a three-year total acquisition cost for their respective Agile ALM solutions. This three-year total acquisition cost is the basis of the
solution pricing tier indicated for each vendor.
Finally, the vendors’ three-year total acquisition costs were normalized to produce the Affordability raw scores and calculate Value Index
ratings for each solution.

Key elements of the common pricing scenario provided to Agile ALM vendors included:
• A six-site organization with 2,200 employees, located at two locations in each of the US, England, and France. The US locations create
software to be used internally, as well as for external clients, while the development resources in England and France are focused almost
entirely on externally facing applications, including mobile solutions.
• The development organization has grown rapidly through acquisitions. While the teams work well together, projects are often seen as
chaotic when crossing office boundaries.
• The corporate development group has determined that implementing ALM consistently across all locations and projects would greatly
improve the overall efficiency of the collective global development group.
• The corporate development group would like to create visibility for the corporate stakeholders into all projects being executed with real-time
(or near real-time) access to data. Dashboards and reports should be able to be filtered by project, by development group, and across the
entire corporation.
• Development projects are typically run with an Agile approach, following two-week sprints for most projects.


Agile ALM Tool Comparison

Recommended

More Related Content

What's hot (20)

Viewers also liked (7)

Similar to Agile ALM Tool Comparison (20)

More from Venkat Janardhanam, MS, MBA (11)

Agile ALM Tool Comparison