NOVEMBER 2013

A PRINCIPLED TECHNOLOGIES ANALYSIS

Commissioned by CA Technologies
THE VALUE OF ANALYTICS


SUMMARY
IT is under pressure to deliver fast and reliable application performance and satisfying
customer experiences on complex, dynamic, and mission-critical applications. Todays
applications are multi-platform, multi-tier, and span physical, virtual, and hybrid cloud
environments. Monitoring these new Web, cloud, and mobile apps for customers and
staff add to an already burdened IT workload. The manual processes and legacy
application-and transactional-monitoring tools of the past cant keep up with the
complexity of mission-critical applications. They dont give IT staff the support they need
to identify and solve problems quickly or deliver on performance goals.
To keep up with complex, dynamic, and interconnected applications, analysts
and administrators need application performance management (APM) tools with
advanced analytic capabilities. These modern tools are designed to better anticipate
potential problems, solve problems quickly, keep downtime to a minimum, and support
IT in delivering the quality of service that mission-critical applications demand.





A Principled Technologies analysis 2


The value of analytics
BIG STAKES: MANY APPLICATIONS, MANY PLATFORMS, MANY
PROBLEMS
Large organizations have dozens or perhaps hundreds of applications running on
a variety of platforms, ranging from mail applications, order-entry systems, HR systems,
to development stacks. These applications may be on physical servers, VMs, or a
combination of both. They may exist on new private, public, or hybrid cloud platforms
or on legacy, siloed environments. Modern organizations must support applications on a
range of devices including notebooks, tablets, and phones.
A comprehensive APM solution for such an organization must measure how well
each application performs on all those different platforms. It must measure constantly,
extracting multiple data points per application per minute from multiple points of view.
Ideally, it follows transactions from the users device through all the physical, virtual,
and cloud components, and then back to the user. A tool that captures all these data
points, in terms of end-user experience monitoring and performance monitoring on all
the servers, storage, and other components a transaction touches, is the ideal tool.
Collecting all that data and making it visible and available to IT staff is what an APM
solution does. Making sense of all of that data so that IT can use it is what the analytics
does.
Data quantities and data explosion
Most organizations already run traditional APM or other monitoring products to
collect data and pinpoint problems. In fact, theyre likely running multiple such
products. Sixty-eight percent of respondents in one survey reported using more than
three different application and transaction management tools to track performance.
1

These tools send out warnings and error messages by the hundreds or thousands, every
day. Youve asked them to, defining and setting dozens (or hundreds) of thresholds for
acceptable performance that the tools monitor faithfully. These messages tell you when
processor performance peaks over 80 percent, when any Web page takes more than a
second to load, when a disk drive hits capacity, and so on. The more you wish to
monitor, the more thresholds you set in your APM or transaction management tools,
and the more errors they report. This leads to mailboxes full of unsorted error reports.
Assigned staff members sort the critical errors from the merely annoying, from the its
over a threshold, but its not a problem errors, to the its not really a bug, its a
feature reports.
Theres no shortage of this data, which means your staff must spend even more
time taking deep dives into your growing data pool to try to solve the critical problems.

1
http://www.cioinsight.com/c/a/Latest-News/Application-Management-Hinders-IT-Operations-Report-
876209/#sthash.S3t8W25d.dpuf




A Principled Technologies analysis 3


The value of analytics
They have millions of data points to choose from because, even faster than its reporting
errors, your faithful APM program collects current data and retains historical data for
error diagnosis. This data is deep and wide and out of control. Theres enough data to
solve most problems, but also enough to drown in before an error can be fixed.
Its no wonder that many problems take weeks, even months to solve one
recent survey noted that nearly 31 percent of issues take more than a month to solve.
2

While some of those problems may not need to be resolved quickly, others definitely
do. And if they arent solved, there are consequencesoften big ones. A malfunctioning
application can hurt worker productivity, lose customer transactions, create dissatisfied
customers, get you in trouble with regulators, or take you out of compliance with SLAs.
If the only consequence of a problem is that its eating into IT staff time, youre lucky.
Your current arsenal of fire-fighting tools isnt enough to contain the blazes.
Threshold-based APMs are great tools, but not enough for todays multi-platform, multi-
site application environments.
How did we get here?
APM data can be too much of a good thing. Youve been running APM tools for
a long time, so you have vast amounts of monitoring data on hand. Over 80 percent of
the respondents in one survey reported that their metric data collection had grown 300
percent or more in the last four years alone. Monitoring tools have created not just
databases, but a big data problem.
3

The performance data you collect on your application infrastructures is growing
as you collect more data using more metrics on more devices, and you collect the data
more frequently. One survey seeks to quantify this data explosion:
4


More devices Device counts have multiplied in recent years. Twelve percent
of respondents in a survey said they now managed very large infrastructures of
50,000 or more elements; 13 percent managed large infrastructures of 10-50
thousand elements, 35 percent managed mid-sized infrastructures of 2-10
thousand managed elements. Only 40 percent ran smaller infrastructures with
fewer than 2,000 managed elements.
5



2
Forrester study commissioned by RiverBed: http://apmdigest.com/forrester-study-highlights-application-performance-
challenges
3
Survey sponsored of IT professionals in the US and UK by Netuitive: http://apmdigest.com/application-performance-
management-big-data-challenge
4
Ibid.
5
http://apmdigest.com/application-performance-management-big-data-challenge
Theres enough
data to solve most
problems, but also
enough to drown in
before an error can
be fixed.




A Principled Technologies analysis 4


The value of analytics
More metrics Slightly over half of the respondents (54 percent) collect
between 10 and 30 metrics per component, with 72 percent reporting growth in
this number over the past decade.
6


More frequent data Five minutes was the most common data collection
frequency, up from 15 minutes 10 years ago. Nearly a quarter of the
respondents collect data every minute or less.
7


More monitoring tools One survey pointed to another variable for the data
explosionmultiple monitoring tools. Sixty-eight percent of the IT professionals
surveyed have invested in more than three distinct application and transaction
management tools.
8
Another survey also found that multiple tools caused
problems, with multiple consoles with no data normalization or time alignment
between them, making it impossible to gain a holistic view of problems.

Too much data to analyze Even with multiple tools, many of the survey
respondents reported a lack of complete coverage and visibility in their
environment.
9
More metrics and more results-per-metric created big data that
Netuitive survey respondents found that they couldnt analyze effectively with
traditional APM tools. Eighty-eight percent of those survey respondents said
they were able to analyze less than half the metric data they collected, with 45
percent analyzing less than a quarter.
10

What is the solution?
The solution is better analytics. Traditional APM tools are providing data, but
lack capabilities to sift through the data to pinpoint and resolve problems. One recent
survey found that the greater complexity of data and lack of analysis tools had an
impact on business. Over 75 percent of respondents were dissatisfied with their ability
to correlate business, customer experience, and IT metrics. The study found that larger
companies, those with 5,000 or more managed elements, were more likely to be

6
Ibid.
7
Ibid.
8
http://www.cioinsight.com/c/a/Latest-News/Application-Management-Hinders-IT-Operations-Report-
876209/#sthash.S3t8W25d.dpuf
9
Summary: http://apmdigest.com/forrester-study-highlights-application-performance-challenges. Sign up here to
download full report: http://www.riverbed.com/about/document-repository/Forresters-Ideal-Tool-Set-For-Application-
Performance-Management-For-Better-Business-Performance.html?asset_name=Document%20Repository%20-
%20Forrester%E2%80%99s%20Ideal%20Tool%20Set%20For%20Application%20Performance%20Management%20For%20B
etter%20Business%20Performance&cookie_value=New&action_name=AssetClicked
10
http://apmdigest.com/application-performance-management-big-data-challenge
Another survey
also found that
multiple tools
caused problems,
with multiple
consoles with no
data normalization
or time alignment
between them,
making it
impossible to gain
a holistic view of
problems.




A Principled Technologies analysis 5


The value of analytics
dissatisfied with their analytics capabilities.
11
Respondents of another survey were also
dissatisfied: even with their APM monitoring tools, 71 percent of the businesses
surveyed still could not pinpoint the source of business-impacting slowdowns.
12
All
these surveys point to the same conclusion, one we agree with: IT needs better APM
data analytics to turn the APM data explosion from a problem into IT solutions.
The evolution of applications
Applications are more complex than ever, and change constantly. Buyers are
using more mobile devices, and use both Web and app front-ends. Applications are
multi-platform, multi-tier, and dynamic. They run on various combinations of physical
devices, cloud environments, and on VMs. They have more components in their delivery
chain than ever each may intertwine with an array of Web servers and services,
middleware, databases, third-party components, and software platforms. Applications
interconnect: for example, your email client may launch an Office app for an
attachment, synchronize to cloud storage, connect to collaboration applications, and
synchronize with other devices. Your Web browser app may hit multiple disparate APIs
simultaneously. It is hard sometimes to see the lines between these types of
applications, which makes it difficult for IT to solve problems.
Another issue is the transient nature of both application development efforts
and operations and resource management initiatives. Many organizations have evolved
the nature of their application development and deployment, using the DevOps model
of quick release cycles. Developer teams want to keep applications fresh, current, and
ever changing with frequent updates. Operations groups may move resources,
balancing the load among clouds instances and cloud providers, and move virtual
machines and storage.
Todays application is different than yesterdays, and tomorrows application
will continue to evolve. Older versions of APM and APM analytics solutions havent kept
up with these developments in the application and operations stack, and often require
many manual processes. Instead of solving problems proactively, it may take hours or
more to identify a problem thats already occurred, learn the relevant data, and
pinpoint and implement a solution. This delay leaves users vulnerable to performance
problems that affect end-user satisfaction and staff productivity.
So what is the answer? IT needs a solution that includes both advanced APM
and APM analytics tools. The answer is finding the appropriate application monitoring
toolsets that can analyze and learn your big data, not just collect it.

11
Ibid.
12
http://www.cioinsight.com/c/a/Latest-News/Application-Management-Hinders-IT-Operations-Report-
876209/#sthash.S3t8W25d.dpuf




A Principled Technologies analysis 6


The value of analytics
The shift from infrastructure to applications
Applications have become a central focus for organizations. Well-performing
applications bring in revenue, increase customer loyalty and retention, and improve
staff productivity. To keep them successful in these areas you need tools that provide a
complete view of user experience and application behavior so that you can make sure
the applications perform well. Organizations measure the success of organizational
units, including IT, by how well they deliver applications and how users experience
them.
Approximately two years ago, IT operations shifted from an infrastructure focus
toward a more user-and application-centric view. A 2011 report noted this at the time,
reporting that 20 percent of the Global 2000 were shifting the emphasis of their IT
operations to centrally monitor and manage applications, rather than the
infrastructure.
13
This shift showed not just the organization-wide application focus, but
also the undercurrent of applications shift to the cloud, which arent well managed by
infrastructure-based performance monitoring toolsets.
The transition is still underway. A 2013 survey reported that 65 percent of
respondents had or planned to adopt on-premises application monitoring. This was
fewer than the numbers for network server and database-performance management
tools.
14
Adoption may be lagging older products because early-generation APM tools
lack critical features. However, interest is high. Eighty percent of IT decision makers
surveyed believed that management tools were important in managing application
performance.
15
Both of these surveys suggest that adoption rate will continue to rise.
Complex multi-platform applications
The problem isnt just the explosion in data. Just as IT shifts its focus from an
infrastructure-centric point of view to a more application-centric one, innovations have
made applications more complex and more difficult to manage.
In the last few years, applications have become more complex, dynamic,
distributed, and multi-platform, which make them harder to monitor and makes their
data more complex and more difficult to analyze. Data centers have faced huge growth
in scale and complexity. Meanwhile, a massive shift to cloud and virtualization
technologies have helped make the application the focus of performance analysis,
rather than the infrastructure alone.

13
http://webperformanceguru.files.wordpress.com/2011/10/gartner-magic-quadrant-for-application-performance-
monitoring-sept-2011.pdf
14
: http://www.riverbednews.com/2013/07/forrester-application-performance/
15
http://www.riverbed.com/about/document-repository/Forrester-Think-Youve-Mastered-Application-Performance-Think-
Again.html
Forty-six percent
of companies say
they have double
or tripled the IT
components
(servers, storage,
network, app
servers, etc.) they
had a decade ago.




A Principled Technologies analysis 7


The value of analytics
How many applications, running on how many platforms and components make
up your business? You have applications on servers in your data center, on public,
private, and hybrid clouds, backups on a variety of platforms at other cloud providers,
and a disaster recovery site. They can include mainframe servers, application servers,
Web servers, appliance servers, and database servers. They may support desktop,
server, and storage virtualization, use software-as-a-service, or use multiple network
and storage solutions. Your data center is now multi-platform, multi-site, multi-vendor,
hybridized, and virtualized. And its growing, even into the public cloud spaceeach
time a developer adds another public cloud VM for testing, your virtual data center
expands.
A survey last year of IT professionals found that 46 percent of companies say
they have double or tripled the IT components (servers, storage, network, app servers,
etc.) they had a decade ago. Another 26 percent reported a four or five fold increase
over that same period.
16

The interdependencies among the components have grown as well. Any one
application may span many of those components and include application servers,
database servers, Web servers, and backup servers. Additionally, the application may
have components in one or more private or public clouds, count on third-party
products, or link with other applications.
Applications have a lot of moving parts. A 2012 survey of IT executives found
that 64 percent of responding organizations managed seven or more distinct
technologies within their application systems, with virtual environments and cloud-
based applications adding even more complexity.
17
Its simple: Interdependency adds
complexity.
Application complexity understandably contributes to performance issues. In a
recent survey, the majority of respondents identified application complexity as the
primary cause of performance issues (51 percent).
18

Intangible and tangible costs of problems
The costs of application performance problems extend beyond the obvious and
can be separated into three main areas: the IT costs of solving the problem, lost
productivity and business, and customer satisfaction.


16
http://apmdigest.com/application-performance-management-big-data-challenge
17
http://www.cioinsight.com/c/a/Latest-News/Application-Management-Hinders-IT-Operations-Report-
876209/#sthash.S3t8W25d.dpuf
18
Forrester study commissioned by RiverBed: http://apmdigest.com/forrester-study-highlights-application-performance-
challenges
The majority of
respondents
identified
application
complexity as the
primary cause of
performance
issues.




A Principled Technologies analysis 8


The value of analytics
The high cost of problem solving
Infrastructure and application performance problems devour huge amounts IT
staff time, which consumes IT budgets. Individual problems take a long time to repair
with manual process and traditional tools. Most respondents to one recent survey of IT
professionals reported taking an average of 60 minutes to repair each performance
issue.
19
The majority of respondents to another survey reported that it took more than
24 hours to resolve a quarter or more of their problems. Almost a third of those
respondents reported that some problems took a month or more to resolve.
20
These
costs reflect the complexity of the applications these groups are dealing with.
Add the staff costs of all the infrastructure and application problems and you
get a significant part of the IT budget. Most IT operations groups in one survey reported
spending from 10 to 30 percent of their resources fixing infrastructure and application
issues.
21
Another reported that a full 72 percent of respondents IT budgets was spent
operating and maintaining existing IT services.
22

In addition to staff costs, you may spend money for unnecessary upgrades, not
knowing where the true problem lies. You can remedy or hide many performance
problems such as high WAN traffic, or an overactive VM with costly infrastructure
upgrades. If instead you could dig down into the performance issues and find the root
cause of the problem, you could avoid those costs and instead choose a less expensive
and quicker solution, such as offloading work to underutilized resources or giving a
mission-critical application the priority it needs.
The IT dream is to identify and resolve problems before they affect end-users or
while the effect is still minimal. This entails identifying potential problems, finding a
solution, and reacting quickly. One study found that only 16 percent of organizations are
able to proactively prevent performance issues for 80 percent or more of performance
incidents.
23
The cost of failure is high.
Mission-critical impacts
IT leans on APM solutions to help with key applications, but can be burdened if
these solutions cant help deliver on application performance objectives. Application
problems that result in failures to meet SLA requirements or regulatory requirements
may trigger fines and penalties, reduce your bottom line, or even endanger the financial
health of end users.

19
TRAC Research study reported in APMdigest: http://apmdigest.com/it-performance-monitoring-in-2013
20
Forrester study commissioned by RiverBed: http://apmdigest.com/forrester-study-highlights-application-performance-
challenges
21
Summary of Forrester study: http://apmdigest.com/forrester-study-highlights-application-performance-challenges
22
http://spectrum.trac-research.com/2013APM/?section=market_context
23
http://spectrum.trac-research.com/2013APM/?section=market_context




A Principled Technologies analysis 9


The value of analytics
A recent survey found that enterprises deploy application management
solutions most often to improve performance and user experience for mission-critical
applications. It identified email, line of business, document management, database
applications, and customer relations management (CRM) applications as the top five of
these applications. The stakes are especially high for these applications, which need
deliver top performance around the clock in order for the business to run smoothly and
deliver quality service to customers.
24

Highly regulated industry verticals, such as healthcare, energy, and banking also
frequently deploy application management systems to protect high-stakes, mission-
critical applications specific to their industry. For healthcare applications such as
electronic patient record apps and prescription-management systems, the stakes
include the quality of patient care. Other regulated verticals are affected as well. An
energy company recently estimated its downtime costs at up to $1 million an hour in
operational costs, with a single minute of downtime costing more than $16,000.
25

Loss of productivity and revenue
If customers or end users cant access applications or if applications
underperform, your customers and end users lose efficiency, which leads to lost
productivity and sales, and ultimately lost revenue.
A recent survey found that 30 percent of respondents identify productivity loss
for business users as the major consequence of application performance issues.
26
Staff
rely on applications to do their work and need constant and secure access to their email,
office suite, the enterprise VPN, browsers, chat, social networks, and other applications.
They require predictable and fast responses from each of these applications not just at
work, but at home and on the road as well. They lose productivity if these applications
have problems or they cant access them. Lost staff productivity comes at a high dollar
value. One study last year put the cost to organizations of downtime for critical
applications at an average of $138,000 per hour.
27
Another estimated that organizations
lose $72,000 per minute of unplanned network downtime.
28

Businesses that rely on a direct outward facing customer interface could benefit
from advanced analytics APM. Data from a 2011 study suggests that an e-commerce site

24
Achieving top performance for business critical enterprise applications, Aberdeen Group, March 2013.
http://www.aberdeen.com/Aberdeen-Library/8415/RB-application-performance-management.aspx
25
http://www.forbes.com/sites/netapp/2013/10/07/whats-it-downtime-cost/
26
Forrester study commissioned by RiverBed: http://apmdigest.com/forrester-study-highlights-application-performance-
challenges
27
http://blogs.aberdeen.com/it-infrastructure/ahhahaha-stayin-alive-just-virtualize/
28
http://spectrum.trac-research.com/2013APM/?section=market_context




A Principled Technologies analysis 10


The value of analytics
with $100,000 in revenue per day could lose $2.5 million in sales every year with an
added 1-second delay per page.
29

Proper APM tools can also aid the productivity of your development teams. You
lose productivity and snappy release schedules when developers, testers, and
operations staff lack tools to assess app performance, debug problems, and move
application releases smoothly from development to deployment. One study found that
applications management tools improve programmer productivity more than 20
percent.
30

The costs of failures escalate when IT cant solve them quickly customers go to
your competitor, employees arent able to do their jobs, and IT costs rise, among other
consequences. One study of IT reported user productivity loss as the major consequence
of application performance issues.
31
Thirty percent of IT professionals responding to the
survey named that as their top concern.
Customer satisfaction suffers
Applications are the face of business for many customers. Customers learn
about the organization through applications, contact it via applications, and use
applications to buy from it. A key measure of business success, individual business unit
success, and IT success, is how well the business delivers applications to users. If all is
not well in the application world, the organization suffers. Most of the organizations
responding to one survey reported that over half of their performance issues had an
impact on end-users. This intangible damage is often significant to the long-term health
of companies.
First-generation application performance monitoring
Confusion in the APM market
There is no industry standard for what an APM tool is. Gartner identifies the
components as end-user experience monitoring, application runtime architecture, user-
defined transaction profiling, deep-dive monitoring, and analytics.
32

The APM offerings from different vendors arent directly comparable, which
may cause confusion for those looking to make a purchase. Trac Research recently
studied the vendors offering application performance monitoring and application
performance management and found that few offered full end-to-end APM, instead
specializing in subsets of APM tool capabilities such as code-level solutions, end user
experience monitoring, network performance monitoring, APM analytics, cloud

29
2011 data; http://apmdigest.com/how-loading-time-affects-your-bottom-line
30
http://www.asg.com/Solutions/Applications-Management/Application-Understanding-and-APM.aspx
31
http://www.riverbed.com/about/document-repository/Forrester-Think-Youve-Mastered-Application-Performance-Think-
Again.html
32
http://innetworktech.com/wp-content/uploads/2012/08/2012-Magic-Quadrant-for-Application-Performance-
Monitoring.pdf




A Principled Technologies analysis 11


The value of analytics
performance, and virtualization management. Other vendors focused on the needs of
specific industry verticals, such as healthcare.
33

The limits of first-generation APM tools
Even just a few years ago, you could get good value from an APM tool that
tracked application metrics across the server, storage, and network resources in your
data center. These tools compared collected data to thresholds you set, and reported
whenever metrics escaped those thresholds. It was fine that these tools didnt capture
information on virtualized or cloud infrastructures because you used the APM tools for
mission-critical applications that were not yet on VMs or in the cloud. Now that youve
virtualized these apps or placed the in the cloud, many tools cant keep up.
Traditional APM systems presented IT with the dual problem of creating too
much data with the potential for data gaps. These tools monitored and analyzed data on
thousands of performance metrics on applications, transactions, and infrastructure
components. It kept the data over time so you could analyze performance over time.
But now this data volume turns into a big data problem there is too much data for IT
staff to reasonably understand and analyze manually, and the first-generation APM
toolset may not have analytical capabilities.
Long mean time to repair
Mean time to repair (MTTR) is a measure of how long it takes on average to
solve a problem from the time the problem occurs. It includes time to detect the
problem, figure out a solution to the problem, test the solution in the test environment,
and implement the solution in production. For some problems, the MTTR never gets a
value because the problem defies solution or repeatedly drops to the bottom of the list
of things to solve. Weve already discussed the long repair time for problems in
applications not monitored by advanced APM solutions with analytics, and the resulting
issues: higher IT costs, lost revenue and productivity, and reduced customer satisfaction.
Tricky problem triage
Traditional wisdom says you determine the potential impact of problems and
allocate resources based on the severity. This front-end triage can use time that you
could spend fixing problems. If an end-user reports a problem, the triage team has to
measure the severity and compare its severity to that of all other problems on the triage
list. Triage determines the order to solve problems based on evaluation of how
expensive the problem is in terms of user productivity and satisfaction effects, revenue
potential, and SLA and regulatory impact, among other considerations. If the traditional
APM solution reports the problem, it may do some basic triage, but leaves a lot of the

33
http://www.new.trac-research.com/blog/2013/11/05/five-areas-shaping-the-application-performance-management-
market-in-2014/




A Principled Technologies analysis 12


The value of analytics
research work to IT. Solutions with analytics can successfully diagnose impacts and
remove this burden from IT staff.
Benefits of next-generation APM tools with analytics
Advanced APM tools with analytics
Todays IT department needs APM tools that offer full views of application and
transaction performance and of user experience. IT needs advanced APM analytics tools
with sophisticated capabilities to make sense of the data and help identify, prioritize,
and solve problems. IT needs tools that take the initiative of understanding the data
without having to have it defined for them. These tools should be quick to setup, quick
to provide value, and take over manual processes. IT needs tools to help solve problems
quickly, keep downtime minimal, and deliver the quality of service that mission-critical
applications demand. These tools should provide performance improvements and cut
down the cost of application performance management.
Even with good APM tools, companies need the ability to analyze the resulting
data. Without analytics in the solution, IT must spend huge amounts of time reactively
diagnosing performance problems without being able to solve problems before they
hurt user experience.
APM analytics tools can harness all that data to help identify emerging
problems, point to their root cause, and provide supplementary data to help triage the
problem. You want an APM analytics solution that helps with triage so that you can
deploy your resources where theyll do the most good, solving the significant problems
first.
Advanced APM analytics work to offload problem analysis from IT staff, lower
MTTR for problems, and avoid the end-user dissatisfaction and other costs of lingering
problems. It should provide solutions based on both transaction and end-user
perspectives:
Transaction: APM should continue look at applications from the
transactions point of view, but should track those transactions across
the entire physical, virtual, and cloud infrastructure so that IT has the
data it needs to find the root cause of problems.
End user: It should also include data on how well the application
performs for end-users. This would include not just traditional measures
such as Web page responsiveness, but also information of interest to
marketing and managers such as abandoned transactions, response to
ads and promotions, and customer retention rates.





A Principled Technologies analysis 13


The value of analytics
Detecting anomalies
A key technology for the new generation of APM is anomaly detection. With
first-generation APM, IT needed to set up the APM by defining behavior goals using
thresholds and writing rules for alerts. The APM solution would collect the data so that
it could respond to the thresholds and alerts, invoking a manual triaging process by IT
staff. These manual steps were time-consuming and had to be repeated frequently as
the applications changed. The advanced APM toolset is less IT-directed and more self-
directed. Rather than IT learning and defining expected behavior, the advanced APM
tool find relationships among the metrics to identify normal behavior and then to watch
ongoing performance for anomalies. It is constantly reanalyzing and mining the data to
refine its expectations of normal. It can process far more data than can IT staff and find
more sophisticated relationships among the data than can IT. It should trim the amount
of time it takes IT to set up and run the tool compared to previous generations. It should
also do a better job of finding the root cause of application performance problems and
solve them quickly and proactively.
Weakness in some approaches to APM analytics
Traditional APM tools havent been able to keep up with cloud and
virtualization, which means that many major performance problems go unsolved. In late
2012, Information Week published a gloomy analysis of APM utilization, declaring that
the dynamic nature of virtualized and cloud environments rendered most installed APM
systems useless.
34

Somewhat less harsh is the 2013 survey by TRAC Research in which IT
organizations reported that their performance visibility into user experience and
application availability dropped after they implemented public or private clouds or
desktop or server virtualization projects. The biggest drop they reported was a 61
percent average decline in performance visibility into user experience after public cloud
projects.
35

Virtualization and cloud environments have been blind spots for traditional
APM, making it hard for IT to use these tools to diagnose, analyze, and repair
performance problems for virtualized and cloud applications.
Traditional APM solutions measure performance of physical servers, storage,
and networks using performance indicators such as CPU and memory utilization. You
need different metrics for virtual and cloud environments. In virtual and cloud
environments, you need to track metrics such as hypervisor CPU usage in the VM and
guest OS layers.

34
http://reports.informationweek.com/abstract/1/9375/Application-Performance-Optimization/Research:-The-Next-Wave-
In-APM.html?cid=pub_analyt__iwk_20130904
35
TRAC Research study reported in APMdigest: http://apmdigest.com/it-performance-monitoring-in-2013




A Principled Technologies analysis 14


The value of analytics
IT teams need to troubleshoot performance problems across physical, virtual,
mainframe, mobile, and cloud infrastructure environments. They need an APM tool and
APM analytics that can see into all those environments, their networks, and
infrastructures, and can provide visibility into the relationships among components to
detect problems quickly and reduce MTTR.
The misfit between traditional APM and cloud solutions is only now emerging as
a big problem, because only recently have we seen many mission-critical applications
move to the cloud. One study points to the increased importance of APM analytics. It
found that organizations spend more time identifying the root cause of performance
issues than on repairing the problems.
36

That survey also pointed out that some APMs have blind spots. Surveyed
organizations reported that Web browsers and VMs were the top two blind spots for
monitoring application performance.
37
Mainframes, user devices, public Internet, and
application components followed.
38

POWERFUL APPLICATION PERFORMANCE MONITORING ANALYTICS
FEATURES
Gartner defines APM analytics as the marshaling of a variety of techniques
(including behavior learning engines, complex-event processing (CEP) platforms, log
analysis and multidimensional database analysis) to discover meaningful and actionable
patterns in the typically large datasets generated by [APM].
39

Prerequisite Advanced APM data availability
Advanced APM analytics are able to track performance problems and results
through the network, into the cloud, deep into VMs, over the Web, and down to the
end user platform but only if the collected data is available. Advanced APM analytics
relies on comprehensive data from advanced APM including application performance,
transaction performance, SLA compliance, and end-user experience across a multi-
platform delivery model. If an APM tool isnt able to collect data across multiple
platforms, your analytics solution wont have the data it needs to identify and triage
problems and point out their root causes.
Advanced APM data collection should be built into the entire application
lifecycle and infrastructure stack, from development to testing to production, so that

36
http://apmdigest.com/it-performance-monitoring-in-2013
37
http://apmdigest.com/it-performance-monitoring-in-2013
38
http://apmdigest.com/it-performance-monitoring-in-2013
39
http://innetworktech.com/wp-content/uploads/2012/08/2012-Magic-Quadrant-for-Application-Performance-
Monitoring.pdf




A Principled Technologies analysis 15


The value of analytics
issues such as code, configuration, and scale errors can be caught early and fixed before
they hit production and affect users.
Provide multi-dimensional analytics engines
Traditional APM tools kept to the data center, tracking performance on server,
storage, and network hardware. Advanced APM analytics goes outside the bounds of
the data center, and combines measurements from multiple dimensions. APM analytics
combines application, infrastructure, cloud, network, VM, and user-experience
monitoring into a single solution with an application focus.
Transactions are distinct. One transaction might follow one execution path and
set of resources, while another similar transaction may, moments later, follow another
code path or use different resources, touching different back-end servers, routers, VMs,
even cloud providers. An advanced APM analytics solution needs to measure
performance and track transactions throughout the entire application delivery chain
including servers, storage, middleware, backend systems, databases, third-party services
and platforms. Your APM needs to keep up, so if something goes wrong, your APM
analytics tool can pinpoint the exact transaction affected, follow its path, and pinpoint
its failure.
Quantify real-user experience
For customer or staff-facing applications, superior end-user experience is the
mission. With all these components, there is a lot that can go wrong (or right) that
affects end-user experience, so its important that your APM analytics can assess the
situation. Advanced APM analytics should be able to assess the experience of real users
with the application and the applications success in delivering customer satisfaction.
This means that advanced APM needs to assess application performance from the point
of view of the application and the transaction, but also look at the application from the
end users point of view.
End-user experience APM measures user satisfaction and its effect on business
results by analyzing transaction size, revenue, shopping cart or page abandonment, and
other indicators. This data can help triage of problems and in making business decisions
based on customer-experience facts. It can help staff move a revenue-related problem
to the top of the bug list to fix, or a slowdown on a little-used Web page down to the
bottom.
Harness the data in close to real time
Real-time monitoring generates big data it requires real-time analytics to sort,
analyze, and make inferences from that data .The ideal is to catch problems before
users notice them. For that, you need timely data and quick analysis, so that you can
react. You want to bypass a malfunctioning component before it becomes a bottleneck,
add additional VMs to handle unexpected traffic before users abandon slow-loading




A Principled Technologies analysis 16


The value of analytics
pages, or expand the size of a database file before it stops processing purchases. You
can bump a malfunctioning third-party ad off a Web page, nudge application priority to
give a revenue-producing application more throughput, and otherwise manage the
application delivery pipeline so that each customer has a great experience accessing
your mission-critical applications.
Real-time analytics also has to do triage, to sort through the real-time
monitoring data and point out whats important. You dont want to know about
frequent but unimportant events, but you may care about both frequent and rare
critical events. You might not care about a 30 percent burst in some metric, but you may
want to know that a driver update resulted in a small but steady uptick in that same
metric. Youd like the analytics tool to ignore the first (or mark it as of low interest) but
highlight the second.
Assures safe updates and changes
Another role of APM analytics is to provide safer change management. Code
changes can come quickly. With DevOps and Agile methodologies and their focus on
continuous delivery, code changes move into testing and production quickly. Your APM
tool needs to provide real-time peeks into a changing code base so you know which
code version transactions passed through, but also so you can measure the impact of
code changes.
APM analytics isnt just for spotting problems. Its for measuring the
effectiveness of solutions. It can help you verify that your data center and services have
the expected positive effect. You dont want to add servers, increase your VM count,
add cloud instances and find later that the changes diminished performance or user
experience. Analytics can point out if a change improves (or cuts) performance or SLA
compliance and by how much. You can use the data to justify the improvement,
pointing to the revenue increase that a million dollar server investment brought for
example. On the other hand, you can use the data to quickly fix any problems that arise
in consequence of the upgrade. Analytics help your change management, your reaction
time and perhaps your job security.
Solve problems quickly
APM analytics saves IT time and effort by quickly finding the root cause of
performance problems. It can identify problems proactively based on variations from
normal behavior that its anomaly detector identifies. The tool, not IT staff, does the
grunt work of digging through hundreds of thousands or millions of pieces of data to
analyze patterns and identify problems and their causes so that IT can quickly fix the
problem. APM analytics then needs to help IT verify the effectiveness of the fix in testing
and finally in production. One study found that organizations that deployed advanced
APM tools and analytics saw 10 times faster problem isolation and an average reduction
One study found
that organizations
that deployed
advanced APM
tools and analytics
saw 10 times faster
problem isolation
and an average
reduction of 27
percent in MTTR.




A Principled Technologies analysis 17


The value of analytics
of 27 percent in MTTR.
40
Research shows that IT spends more time identifying the root
cause of application performance issues or slowdowns than on repairing the problems.
41

POTENTIAL USE CASES FOR ADVANCED APM ANALYTICS
Many potential use cases exist for advanced APM analytics solutions. Below we
list just a few.
Web site monitoring: APM Analytics could prove to be invaluable in
providing end-user experience tracking and monitoring availability and
performance of an applications Web site. It could analyze end-user
transactions in real time, looking at how long it takes to process
purchases, display pages, and open links.
Cloud infrastructure monitoring: Traditional APM tools focused on the
physical compute devices data center. Advanced APM analytics tools
need to follow applications out of the data center including into the
public, private, and hybrid cloud infrastructures. An effective APM
analytics tool needs to be able to measure performance across a variety
of cloud providers, hypervisors, and management interfaces.
Enterprise IT performance: As we noted earlier, all enterprises use a
variety of applications that drive their business ranging from HR
applications, development applications, order-entry systems, and so on.
An ideal APM analytics solution can be an asset to a large enterprise.
SLA compliance: When you have certain SLAs to meet with your end-
users or customers, an APM analytics solution is critical to show you
where you are in danger of slipping out of the SLA boundaries.
WHAT TO LOOK FOR IN AN APPLICATION PERFORMANCE MONITORING
ANALYTICS SOLUTION
You want some of the same things in an analytics tool that you do in a new team
member you want analytic skills, organizational ability, and a quick time-to-value,
among others.
Self-learning and pattern matching
With earlier versions of APM analytics, you had to spend significant time during
installation teaching the APM tool about the applications. Youd identify metrics and
values and the tool would report if the application ever exceeded that set value. With
advanced APM analytics, instead of you learning the application, deciding which metrics

40
http://apmdigest.com/10-business-benefits-of-apm-application-performance-management
41
http://spectrum.trac-research.com/2013APM/?section=market_context




A Principled Technologies analysis 18


The value of analytics
to track, and then and teaching them to the APM tool, you install the APM then let it
sort through the data it has collected. It looks for the patterns of normal behavior, and
compares future behavior to those baseline patterns. When it finds problems, it alerts
you to developing issues, and highlights the metrics and events.
Views and perspectives
End-user experience
Monitoring with end-user experience in mind is an important facet of the APM
conversation. Typical monitoring includes standard measures on page load times, error
counts, and other basic measures of how well pages deliver to users, but these may not
provide the depth needed to pinpoint potential issues that affect your customers. APM
with analytics allows the support organization to monitor the environment more in
depth with the end users perspective in mind. Are there certain pages with loading
issues because of an unexpected shift in customer demand? Do more users follow one
path through opening pages versus another path? Should you focus on and optimize
your mobile application front-end or your PC device application front-end? These are all
end-user-focused questions that a successful APM and analytics solution could help to
answer.
Horizontal and vertical
The ideal APM analytics solution collects and analyzes data from across your
environment from side to side, or horizontally. For example, all hypervisors, cloud
environments, and network infrastructure types should be measured. The tool should
also have the capabilities to dig deep vertically and see deep into a certain segment of
the application topology.
Streamlined end-user experience
Out-of-box experience and dashboard capabilities
Most of the earlier generation APM tools were complex to install, set up, and
use. Only IT staff with deep knowledge of important indicators for the monitored
applications and set policies and rules for good behavior could complete setup. The
tools compared performance against these indicators. The tools then may have required
constant tuning to update the indicators as applications and infrastructures changed.
This onboarding process was a costly and painful one for organizations.
A recent study found that 73 percent of organizations reported that time-to-
value is the key selection criteria for APM solutions. That shows more tool intelligence
and more automation arent just nice options, but are necessary features.
42


42
http://apmdigest.com/it-performance-monitoring-in-2013




A Principled Technologies analysis 19


The value of analytics
An advanced APM analytic tool should be easy to use and immediately useful.
Once started up, it should look through already collected APM data and start monitoring
real time data to discover applications and their baseline patterns. It should then start
hunting for anomalies, suggest triage status, and get you started solving problems. It
should provide immediate value after a short install.
Alerting and reporting
Dashboards provide real time analysis for quick decision making. The tool
should come installed with pre-configured dashboards, but also allow for customization
of new dashboards for specific monitoring roles. Dashboards should show the state of
key status indicators such as response time, application errors, overview, and highlight
problems in performance and capacity.
While dashboards show real-time analysis data, reports show trends over the
last few days or weeks. They can show responses to events such as application updates
or infrastructure expansion. APM analytics tools should include built-in report templates
that you can customize.
Features such as per-constructed and customized reports and dashboards let
not just IT but also application managers and Web and application developers view
problems. It should provide tools so you can do your own analysis and dig deep into the
big data database to track root causes and solutions.
Intelligence
Earlier generation tools relied on rule and threshold based analytics, where IT
entered rules that the tools compared to performance. Modern APM analytics tools
advance from that static definition of performance, which couldnt necessarily keep up
with applications rate of change or the scale of available data. Instead, advanced APM
tools figure out normal application behavior and adjust that definition over time. The
tool keeps up with application change, while still giving IT the flexibility of adding their
own intelligence on top of the solutions.
Advanced APM tools offload analysis from IT staff to the monitoring software,
which unlike even the most capable IT staff, can process hundreds of thousands of
indicators a minute to look for trends, interconnections, patterns, and problems.
Self-learning and anomaly detection
The Advanced APM analytics tools should be self-learning, studying real-time
data, setting baselines, and keeping current behavior to those baselines. It should adjust
baselines over time as applications evolve. It should learn whats normal and then point
out deviations from the normal. It should be able to give each anomaly a triage score
based on its severity and impact so IT can prioritize its work fixing problems.




A Principled Technologies analysis 20


The value of analytics
Advanced statistical capabilities, pattern matching, heat maps
These tools use multivariate analysis and other statistical capabilities to discern
patterns in big data. It spots anomalies in current behavior patterns and uses
dashboards to display heat maps or other triage indicators that point IT to the problems
it needs to solve now.
Triage and remediation
You need analysis, but you also need remediation capabilities and assistance. If
your analysis tool is just spewing out error messages, its not helping. Even if it is sorting
them into red, green, and yellow priority groupings, its not helping enough. You need a
finer level of triage, one that points you to key problems as they occur and helps you
with root cause analysis and problem solving.
Thresholds, latency, and response time metrics, the bread and butter of
traditional APM, are a still a valid way to identify errors in modern applications, but can
lack the depth and understanding that an advanced APM analytics tool brings to the
table, with its more sophisticated algorithms and anomaly detection.
CONCLUSION
Without advanced APM analytics solutions, IT staff must monitor, manipulate,
and analyze data on thousands of performance metrics and tens of thousands or more
transactions for each application to keep things running. The best and brightest of your
IT staff are overburdened with this understandably challenging exercise. The evolution
in application complexity, the number of diverse platforms, and the onset of the huge
datasets all contribute to a big data problem. IT staff have enormous databases of
multiple performance and transaction monitoring tools, but have no way of analyzing
the data.
More advanced APM analytics solutions are the key solution to this problem.
These tools can bring new points of view, and measure not just transaction performance
and component performance, but also end-user performance. These tools take
advantage of modern servers ability to process huge amounts of data in close to real
time and use advanced statistical techniques to make sense of the data. Analytics
solutions present IT with actionable intelligence on application performance problems.
This gives IT a step up to solve problems proactively before they affect end-users and
enables a faster MTTR for critical problems on mission-critical applications than ever
before.
The ideal modern APM analytics solution reduces the frequency and severity of
downtime, effectively triages and allows automated remediation capabilities, ensures
SLA compliance, enables superior user experience, installs and is managed easily, all
with the ultimate goal of increased revenue for your organization.




A Principled Technologies analysis 21


The value of analytics
ABOUT PRINCIPLED TECHNOLOGIES


Principled Technologies, Inc.
1007 Slater Road, Suite 300
Durham, NC, 27703
www.principledtechnologies.com
We provide industry-leading technology assessment and fact-based marketing
services. We bring to every assignment extensive experience with and expertise
in all aspects of technology testing and analysis, from researching new
technologies, to developing new methodologies, to testing with existing and new
tools.

When the assessment is complete, we know how to present the results to a
broad range of target audiences. We provide our clients with the materials they
need, from market-focused data to use in their own collateral to custom sales
aids, such as test reports, performance assessments, and white papers. Every
document reflects the results of our trusted independent analysis.

We provide customized services that focus on our clients individual
requirements. Whether the technology involves hardware, software, Web sites,
or services, we offer the experience, expertise, and tools to help our clients
assess how it will fare against its competition, its performance, its market
readiness, and its quality and reliability.

Our founders, Mark L. Van Name and Bill Catchings, have worked together in
technology assessment for over 20 years. As journalists, they published over a
thousand articles on a wide array of technology subjects. They created and led
the Ziff-Davis Benchmark Operation, which developed such industry-standard
benchmarks as Ziff Davis Medias Winstone and WebBench. They founded and
led eTesting Labs, and after the acquisition of that company by Lionbridge
Technologies were the head and CTO of VeriTest.

Principled Technologies is a registered trademark of Principled Technologies, Inc.
All other product names are the trademarks of their respective owners.
Disclaimer of Warranties; Limitation of Liability:
PRINCIPLED TECHNOLOGIES, INC. HAS MADE REASONABLE EFFORTS TO ENSURE THE ACCURACY AND VALIDITY OF ITS TESTING, HOWEVER, PRINCIPLED
TECHNOLOGIES, INC. SPECIFICALLY DISCLAIMS ANY WARRANTY, EXPRESSED OR IMPLIED, RELATING TO THE TEST RESULTS AND ANALYSIS, THEIR ACCURACY,
COMPLETENESS OR QUALITY, INCLUDING ANY IMPLIED WARRANTY OF FITNESS FOR ANY PARTICULAR PURPOSE. ALL PERSONS OR ENTITIES RELYING ON THE
RESULTS OF ANY TESTING DO SO AT THEIR OWN RISK, AND AGREE THAT PRINCIPLED TECHNOLOGIES, INC., ITS EMPLOYEES AND ITS SUBCONTRACTORS SHALL
HAVE NO LIABILITY WHATSOEVER FROM ANY CLAIM OF LOSS OR DAMAGE ON ACCOUNT OF ANY ALLEGED ERROR OR DEFECT IN ANY TESTING PROCEDURE OR
RESULT.

IN NO EVENT SHALL PRINCIPLED TECHNOLOGIES, INC. BE LIABLE FOR INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES IN CONNECTION WITH ITS
TESTING, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. IN NO EVENT SHALL PRINCIPLED TECHNOLOGIES, INC.S LIABILITY, INCLUDING FOR DIRECT
DAMAGES, EXCEED THE AMOUNTS PAID IN CONNECTION WITH PRINCIPLED TECHNOLOGIES, INC.S TESTING. CUSTOMERS SOLE AND EXCLUSIVE REMEDIES ARE
AS SET FORTH HEREIN.