Aveta Six Sigma Online Black Belt Course

SAMPLE
Six Sigma Black Belt Project Report

1
Aveta Six Sigma Online Black Belt Course

Executive Summary

As a new subsidiary of an established company moving into a new business, establishing a reputation for
solid customer service at a low price is paramount for the Security Department’s new serivce. Along the
way, the business manager discovered that the idea of a solid customer service reputation was in jeopardy,
but felt the personal service could not be beat. Under that presumption, he solicited help to use Six Sigma
processes to 1) validate the perception, 2) propose a recommendation for solutions and 3) put processes in
place to ensure control in the long run.

The Problem
After research and survey, we found that site managers were pleased to be moving to the new video
surveillance technology, but they were disappointed in the number of outages they were seeing in the
equipment. After measuring the process, we found there were several reasons for outages:

1) Network service issues

2) Software update issues
3) User training issues
4) Equipment failure

Of these, the latter proved to be the most costly and unexplainable. According to information from the
manufacturer, we should not see the kinds of problems we were having. The estimated cost for these
repairs exceeds $70,000 annually and has an untold impact on Operations.

The Process
Using the Six Sigma process of Define-Measure-Analyze-Improve-Control, we determined what the
priority is for the Security technical group. We then chartered a project and proceeded to understand and
define the problem. Through customer surveys, site visits and data gathering began to focus on the real
issues at hand. To help clarify the situation, we mapped the installation and support processes for the
security equipment finding and correcting issues as we went.

Eventually, the measurement and analysis processes pointed to the fact that we had a real issue that was
out of the ordinary for the equipment in service. Fortunately, we had clear concise records and good
support from the manufacturer to help research and resolve the problem.

Findings
As the old adage goes from a well-known Houston media personality, we had “slime in the ice machine!”
More importantly we had significant amounts of dirt in the machines. A by-product of the process/project
was that we discovered the Information Technology team was having the same issue, just not quite to the
same extreme.

With some research, the team did find a low cost solution to protect the machines. While it is early in the
post-solution phase, we are seeing improvements.

Conclusion
By using a systematic approach, we identified several areas in the process that needed modification. The
net result is a longer machine life with an expected savings of more than $70,000 annually as well as
improved customer service. Based on the proactive response to their issues, the Operations Management
teams of both the internal and external customers are beyond satisfied and are serving as excellent
references for the Security Department.

2
Aveta Six Sigma Online Black Belt Course

Definition Phase

Project Opportunities
As a new subsidiary business within a well established, large company, the Security Department
has several competing projects for a small staff. They find themselves needing to balance on-
going operations, roll out of more services to the parent company and new sales opportunities for
external customers. For internal customers, the goal is to provide excellent security service -
burglar, fire and equipment monitoring - as well as video coverage (with historical recordings) of
the operations area. For external customers, their strategy is to utilize their existing staff to
provide low cost services for monitoring alarms. Their goal continues to be low price and
excellent customer service. Excellent customer service is defined as reducing false alarms and
properly maintaining equipment for legitimate alarms as well as providing video services for
those wanting to pay for it. In order to fully understand what was important to the customers, we
conducted a survey to understand what people are expecting from the Security Department.
From that survey, table 1 lists potential projects1 identified from on-going operations and for
future customer sales.

Weight Criteria and weights

4 2 3 10
Profitability Easy to Low Customer Total
Do Cost Satisfaction
Add new model DVR support 3 6 8 7 118
Resolve perceived DVR failure issue 8 2 3 10 145
Automatic video retrieval on alarm 6 3 3 6 99
Implement IP alarm monitoring function 6 4 5 8 127
Table 1: Security Project List & Impact

In discussions with senior management, internal customers, call center employees and potential
external customers, the Security Department used a weighting to help determine what to focus on
first. Based on input, they weighed customer satisfaction as critical since their predominant
advertising mechanism is going to be word of mouth for the first year of the new subsidiary. By
using the results from Table 1, the Security Department decided to investigate the perception of
DVR failures since it had a direct link to customer services. As a side note, this was not even on
their list of potential projects prior to the survey.

Voice of the Customer

With the relative surprise response of a high rate of Digital Video Recorder (DVR) failure
problems, I felt it was necessary for the department management (potential project sponsor) to
really understand the nature of what the customer had to say. So rather than just presenting
statistics and summaries from surveys, we discussed the comment section from each response.

Site Manager (parent company) - “Being able to review and monitor transaction
processing via video is critical. I never would have thought I would say that since I
avoided the technology. Now, I can’t live without it. When the camera systems are
down, it costs me money. If everyone knows it is down, I can’t image what they are
doing.”

1
Actual list is longer. The list was truncated to simplify this report
3
Aveta Six Sigma Online Black Belt Course

Safety Manager (parent company) - “By having video access (and more importantly
recording), I have ‘eyes in the back of my head’. This technology extends my ability
to cover a site and find issues before they have a negative impact on the business. An
outage, especially an extended one, raises a great deal of concern.”

Security Director (external customer, multiple sites) – “The potential of the service is
incredible, but the number of camera failures is frustrating. If we do not resolve this
problem, then I will have to cut services back to the basics.”

While these represent the extreme comments, several people made similar comments. The last
one (and others like it) is what made Management sit up and take notice of a problem. Based on
the scoring, conversations regarding customer service and potential impacts to their reputation of
service challenges, the Company decided to authorize a project to investigate the perception and
implement solutions if the allegations proved to be true. Remember this is a new subsidiary and
working under “word of mouth” advertising.

Project Charter

A summary (retyped version to simplify project submission) is shown in Table 2 below. The
original has full company and employee names along with telephone numbers and signatures.

PROJECT INFORMATION
Project Name: Chartered Date:
Investigate & Resolve DVR Failure 6/1/06

Sponsoring Organization: Project Start Date:

Security Department 6/2/06

Revision Number: Target Completion:

2 6/30/06

TEAM SPONSORS
Project Sponsor: Contact Number:
Doug – Security Monitoring Director

Project Black Belt Contact Number

Tom

Project Green Belt Contact Number

None available

ADDITIONAL TEAM MEMBERS

Name Role Contact Number
Matt Security Supervisor
Alex Technician
Sharron Customer Service Rep

4
Aveta Six Sigma Online Black Belt Course

ADDITIONAL TEAM MEMBERS (continued)

Barry Installation Specialist

Jeff IT Analyst
Nick DVR Vendor Representative

PRINCIPLE STAKEHOLDERS
Name Role
Zack VP – Security
Bill ABC Customer – Security Manager
Doug Security Monitoring Director & Sponsor
SLT Senior Leadership Team

PROJECT GOALS
Identify any deficiency in DVR performance. Assuming a deficiency, implement
corrective actions to resolve the problem and return performance to documented
performance standards. In the absence of a deficiency, document performance
expectations and discuss with the customers.

The following was added in revision 2 of the project charter after finding
installation issues:

Correct the installation problems and return DVR performance to published

performance standards as well as reinstate customer confidence in both the
equipment and customer service.

PROCESS PROBLEM
The following was added in revision 2 of the project charter after finding
installation issues:

Under certain installation conditions, the DVRs are failing prematurely. The
apparent cause of the issue is dirt. Certain operational environments are outside
recommended standards – the actual offices are temporary spaces that have a high
volume of traffic in a construction type environment. The result is the server fans
are pulling in too much dust and dirt for the servers to function properly.

SCOPE OF PROJECT
The following are the items that have been identified as the scope requirements:

1) Conduct further surveys to understand the nature of DVR failures

2) Investigate performance standards and expectations of the various DVR
brands
3) Compare installed DVRs and maintenance issues against the performance
standards.
4) If performance is found to be substandard, investigate what is causing the
problem and recommend solutions.
5) Pending findings and approval for #4, implement recommended solutions.

5
Aveta Six Sigma Online Black Belt Course

PROCESS IMPORTANCE

The video support function is a significant value added service provided by the
Security department. Some of the potential savings and findings are listed below:

Monitor cash point of sales to discover fraud. One site reported savings of
$500-$750 per day. The reports indicated that it had been going on for years.
While that kind of fraud is extreme and rare, it does occur.

Allows Internal Audit in performing site audits on viewing scale transactions

with point of sale entries. Each IA trip is estimated to cost $5000. Using
reliable video, IA could save at least 10 trips annually.

In addition to these findings, the external customers are expecting to save a great
deal on safety audits and inventory shrinkage discovery. While they have not
documented their savings to the company, it is clear they are expecting to save
over $100,000 annually. As the project team interviews the stakeholders to
confirm their interest in the project, they will document more regarding cost
savings.

AUTHORIZED RESOURCES

Project sponsor – Doug, Security Monitoring Director.

As project sponsor, Doug is responsible for the following:
1) ensuring staff members are available to work as a team member,
2) coordinating funding with the management team using the figures
provided by the project manager and black belt candidate,
3) coordinating participation from the customer representative. Doug’s
role also has him serving in a sales role, so he maintains the customer
relationship.

Key Stakeholder – Bill, ABC Customer Security Manager.

As customer, Bill is a key stakeholder in buying the service. He has a vested
interest in seeing the project work as planned due to the amount of hard dollar
savings. Additionally, there are some un-quantified savings that he is expecting
to achieve. His role on the project is to provide as much detail as possible about
the operating environment of his company as well as answer questions regarding
expectations and understanding service deliveries.

Key Stakeholder – Zack, VP – Security.

In addition to being a key stakeholder, Zack is also responsible for the
entire Secure Department budget (he has fiscal and functional
responsibility for the Security Department). As a stakeholder, his interest
is to ensure that customer service is maintained. He services as one of the
key people in regard to sales and is a major proponent of the group.

While the sponsor will facilitate getting a funding stream, Zack will also
help. He is also responsible for answering to the Senior Leadership Team

6
Aveta Six Sigma Online Black Belt Course

AUTHORIZED RESOURCES (continued):

in regard to external sales. As a result, his role will be to help drive an

understanding within the leadership team of 1) problem resolution
techniques and 2) capital funding for implementing the project resolution.

Key Stakeholder – Senior Leadership Team (6 people)

The senior leadership team represents the stockholders and outside interest
of the company. They have authorized and provided funding for the new
subsidiary. As such, they want to receive a return on investment (profit)
for that funding; however, they owe support to the operation to ensure that
the best quality of service can be delivered. In the event that it cannot
produce the profit as expected, they will own the decision to either
continue or cease operations.

Team Members –
Matt Security Supervisor
Alex Technician
Sharron Customer Service Representative
Barry Installation Specialist
Jeff IT Analyst
Nick DVR Vendor Representative

Duties are to be defined as the project proceeds.

Outside Needs
Some travel to locations may be required as part of understanding the installation
process. The point of having experienced team members (Jeff, Nick and Barry)
on the team is to minimize that need since they have worked in the various sites.

Technical expertise regarding the equipment in service will be required. As a

representative of the company providing the equipment, Nick has access to that
information. A test system with cameras may be needed for experimentation and
configuration changes if required.

Training needs are unknown at this time, but expected to be minimal based on
the experience of those selected to participate. They were chosen under the
theory that they have “seen everything”.
Table 2: Project Charter

7
Aveta Six Sigma Online Black Belt Course

Project Schedule
Figure 1 is a schedule drafted early in the discovery process the edited as data became available. The
control phase is documented in later sections.

ID Task Name Duration Week 1

1 Definition Phase 23.32 days

2 High level business need survey 5 days

3 Review findings 1 day

4 Meet with business leaders regarding findings 0.33 days

5 Detailed follow-up on customer issue regarding DVRs 3 days

6 Follow-up meeting with business leaders regarding problem 0.33 days

7 Identify customers, sponsors and process owners 0.33 days

8 Identify resource requirements - high level 0.34 days

9 Team selection 0.33 days

10 Draft project schedule 2 days

11 Draft project budget 1 day

12 Review with sponsor 0.33 days

13 Project Charter approval 0.33 days

14 Detail project plan 2 days

15 Process Mapping 4 days

16 Develop maps 1 day

17 Resolve immediate issues (clear gaps in process) 3 days

18 Develop success metrics (CTQs) 1 day

19 Evaluate existing data for applicability 1 day

20 Sponsor review 1 day

21 Measure Phase 5.32 days

22 Validate metrics 0.33 days

23 Review existing data 1 day

24 Develop checklist measurement & sample sizes 0.33 days

25 Prep base analysis charts 1 day

26 Re-evaluate source data. 2 days

27 Analyze Phase 3.32 days

28 Evaluate preliminary chart results 0.33 days

29 Benchmark to published standards 0.33 days

30 Compare results across location type 0.33 days

31 Compare results with vendor 0.33 days

32 Review results with business leaders 0.33 days

33 Improve Phase 28.33 days

34 Team brainstorming sessions 0.5 days

35 Options analysis (costs & time) 0.5 days

36 Review options with business leaders 0.33 days

37 Capital approval documentation 2 days

38 Pilot on 2 sites 5 days

39 Rollout to remaining sites 20 days

40 Control Phase 2.16 days

41 Determine control variables 0.33 days

Figure 1: Project Schedule

8
Aveta Six Sigma Online Black Belt Course

Process map

Since the results of the original survey were a surprise to the security team, we documented the setup
process (figure 2) as well as the trouble shooting and repair processes (figure 3) as well.

DVR Installation Process

Request review by
Cost review by site
Security Dept Implement Security orders
management & Yes
Request from Site (cost & equipment Video? equipment
Security
for Video estimate)

No

Equipment staging
(separate process
Counterproposal chart for this)
Yes Cost Issue?
for other options

No
Site prepares for
No implementation data and power
requirements

Local IT or Security arranges

Ship equipment to
Site testing Site configuration contractor installs for local
site
equipment installation

No

Remote testing Yes Test OK? Production Ready

Figure 2: Setup Process

Special notes:
As a by product of creating the process maps for system setup, we discovered three significant points.
First, some items were not being configured consistently or at all. Some of that depended on the
technician handling the process. The second item is that the sites were not necessarily prepared for the
implementation process. In many cases, the local installation technician needed to arrange for power and
data cabling once on site. That loop is shown in the setup process above. The third was the need to create
a knowledge base for troubleshooting (figure 3 – troubleshooting). In evaluating the process, one
technician was generally perceived as a bottleneck. In reality, he had a better process for handling issues
which resulted in a better resolution rate. That is not documented in the results for this particular project.

9
Aveta Six Sigma Online Black Belt Course

DVR Troubleshooting & Repair Process

Automatic
reporting
Technician calls
function
site to discuss
sends Return to
problem
trouble production
signal
No

Technician works
Problem Full system
Onsite personel Yes troubleshooting A
exists? outage?
notice issue and process*
call in Yes

No
Dispatch regional User resolvable Review camera Camera
No Yes
camera contractor with instruction? specific issue specific issue
Reboot DVR

No No
No Yes

Review DVR
Camera contract DVR Restart
processes with
contacts central Yes Functional normal?
Issue resolved user (training as
technician to Issue?
required)
diagnose &
resolve No

Yes
Review error
Return to messages
A No Issue resolved Yes Yes Yes
production

Resolve per
Microsoft or DVR
Software
Issue resolved vendor Yes
Error?
specifications &
standards

No
No

Hardware repaired
Customer ship unit Hardware
and returned per
to central support error?
set up process

* Common issues are camera outage, focus

issue, bad lighting (camera contrast), can not
search video (training issue).

Figure 3: Troubleshooting & Repair Process (Simplified)

Key Measures for Evaluation

From the troubleshooting and repair process, we found that several machines were being returned to the
central support group. The general perception is that anytime a machine is taken out to be shipped, the
overall outage is entirely too long and expensive. From that information and perception, we were in a
position to evaluate run times, environment and impact to the business. Some of the items that we felt are
critical to customer satisfaction or needed to be understood for comparison purposes are as follows (the
list grew after the first analysis iteration):

10
Aveta Six Sigma Online Black Belt Course

1) Types of outages for the DVRs.

a. Software issues
b. Hardware issues
c. Network outages (provider service)
d. Network issue (hardware other than DVR, non-provider)
e. User error
f. Miscellaneous
2) Vendor performance expectations
a. Warranty and service life for the unit
b. Operating conditions
3) Number of units returned to central support in various time frames
a. Weekly
b. Monthly
4) Total number of support calls resulting in on-site technical support
a. Weekly
b. Monthly
5) Total duration of outages over various time frames
a. Weekly
b. Monthly
6) Total number of units in production

In addition, we did discover that the central technical support staff was keeping detailed notes on the
various things they were finding with each returned machine. They were just not making a point of
discussing how many machines they were seeing with the management team. In the long run, the notes
saved a great deal of time in getting to the final solution. That is noted in the section discussing the
Analyze Phase to follow.

Lastly, we also wanted to make sure that we understood the impact to the customer for each outage. The
general opinion was certain outages were “less expensive” than others. We would use the Measurement
Phase to help drive where to look:

1) Hard dollar cost per repair

2) Soft dollar cost per hour outage
3) Total outage per site over various periods
a. Weekly
b. Monthly

Measure & Analyze Phases

Using the information from above regarding the key measures, we then determined that the
critical quality variable is system availability. That is generally stated as operating hours per
month at the DVR level or more importantly as the time in service before failure. In some cases,
the camera level data also becomes important, but not all cameras are equivalent. In order to
keep the measurement process at a definable level, we focused at the DVR level overall. Camera
level analysis at specific locations may become a target for review in the future.

Following the decision to treat issues at the DVR level, we worked to understand the relationship
among the different items as they relate to cost and business impact. The following are some of
the items that we discovered:

11
Aveta Six Sigma Online Black Belt Course

1) The hard dollar costs are a dependent item, so tracking them independently is not
required.
2) Network outages generally do not have a long term negative impact. The DVR
continues to record and the data can be accessed once the network is available. As a
result, we only used 4 weeks worth of data for analysis purposes. A later study will
work with the external network provider to improve that service.
3) Opinions on soft dollar outages vary significantly based on location type. For
example, sites with high volume of cash transactions (Operations) feel an outage is
significantly more expensive than an office environment. As such, those numbers are
not included in outage estimates in order to be more conservative on savings. Any
presentation to business leaders did discuss potential numbers around soft costs.

The end result is that the key variables for monitoring are the following:
1) Number of hours of outage – relates to total cost for the outage
2) Number of units returned to central support
3) Operating time to failure

The latter uses the definition of failure as some event that requires removing the machine for
maintenance (actual failure or diagnostics). Appendix A lists some of the raw data used to arrive
at the conclusions listed above and to create the Pareto chart shown in Figure 4 & 5. Due to the
correlation between labor effort and outage costs, Figure 5 pointed to the area needing attention –
specifically that the DVR hardware maintenance uses more labor. The next question needing
research was to find out how the DVRs were performing in relation to performance
specifications. Some of the data for figure 5 as well as subsequent analysis came directly from
the central support maintenance logs. A sample log sheet is shown in Appendix B.

Total Outage Counts by Type

Count of Cause of Outage

160

140

120

100

80

60

40

20

0
Network DVR Hardware User issue Software Network Hardware

Cause of Outage

Figure 4: Count of Outages by Type

Network outages for 4 weeks only. All other data for sample time frame

12
Aveta Six Sigma Online Black Belt Course

Total Technician Hours by Cause of Outage

Sum of Technician Hours

350

300

250

200

Total

150

100

50

0
DVR Hardware Network Hardware Software User issue Network

Cause of Outage

Figure 5: Total Technician Hours by Cause

Taking figure 5 a step further, we then analyzed the outages over the sample time. Figure 6
shows the findings. Using the outages per week against an upper limit (lower limit was set to
zero since it could not be negative), we found the actual outages to be “in control” meaning that
whatever perception site management may have, it was consistent and unless we changed
something, it would continue; however, the perception remains that there were too many outages
for their expectations.

Outages per week

5
Count of Outages

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Week

Outages Average UL

Figure 6: Outages per week

13
Aveta Six Sigma Online Black Belt Course

Taking this information and translating that into business impact, each outage has an average
cost of $597 in repairs (labor, parts, shipping), an average outage of 70.15 hours and 2.44
outages per week. (Appendices B and C show cost and outage information.) The distribution of
the outages is shown in Figure 6 previously. This does not attempt to quantify the “soft” dollar
costs since the various locations have such a diverse definition of what the cameras are saving
them. However, we do know that Internal Audit has a concern about the availability of the video
to the point that they have not cancelled any trip in lieu of video coverage.

In order for Internal Audit to “get comfortable” with video coverage and to ease site
managements concern about outages, the management team mandated cuting the outages from
an average of over 2 per week to 2 per month (1 every other week). That mandate equated to
more than $60,000 savings annually in hard dollars. Again, the soft dollar costs are not
documented. The questions needing to be addressed where 1) is this possible within the
environment and 2) what would it take? The first question addresses the issue around why are
the DVRs failing and is the failure within the manufacturing tolerances. The second question is
based on the assumption that there was something within the environment causing the problem
and not a manufacturing issue.

In order to understand how the failure rate compared to what should be expected, we went back
to the original installation information. While much of the information was available, there were
some assumptions needed:

1) DVRs would be in production 24 hours per day

2) DVRs would be in production every day from installation
3) Production is defined as turned on so that the hardware components were in use even
if not for the intended software package.
4) The exact installation dates are not available, so we used the first day of the month for
the most conservative estimates of time in production.

Using those assumptions, we then compared observed service life with expected performance.
While expected performance can be expressed in many terms, we elected to use a conservative
definition of warranty period.2 Based on that expectation, we found the following:

Number of Failures 39
Total Population 508
Average Time to Failure (days) 382.87
Warranty Period 1080

Maintenance records were sampled for a 16 week period from

1/29 to 5/19. Warranty is 3 years @ 360 days/year.

To further understand the outages and impact, we then analyzed which sites were experiencing
the problems. That also relates to the actual business impact of the outages. In theory, zero
revenue sites have less at stake (soft cost) than revenue sites. Table 3 shows the largest portion
of outages effect operational facilities. Table 4 gives the installation base as background.

14
Aveta Six Sigma Online Black Belt Course

Count of Cause of Outage

Location Type Cause of Outage Total
Not documented User issue 39
Not documented Total 39
Office DVR Hardware 2
Network Hardware 1
Software 2
Office Total 5
Operations DVR Hardware 34
Software 2
Operations Total 36
Transfer DVR Hardware 3
Network Hardware 1
Software 1
Transfer Total 5
Grand Total 85
Table 3: Outage counts by location type and cause

Count of DVR Number

Location Type Total % of Total
Office 26 5%
Operations 443 87%
Transfer 39 8%
Grand Total 508
Table 4: DVR counts by location type

We also took one more assumption – namely that all DVRs that have not failed will not fail
before reaching the warranty level (3 years). With that assumption, we then recalculated the
average time to failure as 1026.48 days with a standard deviation of 187.84 days. When plotting
that in a control chart, most all of the failed units are outside of the control limits. Figures 7 and
8 show that information (the tools are limited so that 508 points cannot be included in one chart).
These assumptions putting the failures outside of the control limits are conservative. Most
notably is that our sample mean is less than 3 years. The actual mean from the manufacturer is
more than three years – thus the warranty set at that point. A call to the manufacturer outlining
our problem got us this information. That call and subsequent discussions resulted in finding
that they are not seeing out of the ordinary challenges like this. After that call, they have
cooperated in our investigation by sending a systems engineer to work with us through the later
phases of the project.

15
Aveta Six Sigma Online Black Belt Course

DVR Production Days

1800

1600

1400

Days in Production
1200

1000

800

600

400

200

0
DVR (1-255)

Figure 7: Production Days by DVR (1-255)

DVR Production Days

1800

1600

1400
Days in Production

1200

1000

800

600

400

200

0
DVR (256-508)

Figure 8: Production Days by DVR (256-508)

DVR Production Days Against Manufacturer Information

2000

1800

1600

1400
Days in Production

1200

1000

800

600

400

200

0
Failed DVRs Time In Production

Figure 9: DVR Failures Against Manufacturer's Information

Based on this new data (or more accurately, based on this relevant comparison), we can conclude
that we are not seeing the expected life we should see. Something is causing the DVRs to fail

16
Aveta Six Sigma Online Black Belt Course

prematurely. At that point, the team used a brain storming session to develop the cause and
effect diagram shown in figure 7. Many of the items shown on the diagram were directly from
the technicians’ maintenance notes (sample form shown in Appendix B). While the
manufacturing area is a process that could have problems, we tended to focus on the items that
were handled internally. Everyone is aware that some machines will fail prematurely due to
manufacturing challenges. If we get to the point where that is the only cause, then we will
initiate another project to resolve that problem.

Manufacturing Installation

Damage in
shipping Improper
wiring

DVR
Failure
Power surges
No
Out of date Preventive
drivers Maintenance
Dirt
Missing virus
software Untrained
users

Configuration Production
& Staging Environment

Figure 10: DVR Failure Cause & Effect

Improvement Phase

Using the information from Figure 10 and the maintenance logs, we took each of the causes and
evaluated their true impact on the problem:

1) Out of date drivers and missing virus software

While these are a problem that this discovery process did find (and have since
corrected), this would not cause the kind of outages we were seeing. A follow-up
virus scan of the remaining DVRs in production turned up no infestations.
2) Damage in shipping
This could be an issue, but we use the shipping containers from the manufacturer to
send the machines to the location. The systems engineer confirmed that the
manufacture is not seeing out of the ordinary challenges.
3) Improper wiring & power surges
This does explain some of the outages we have seen. The maintenance notes on one
of the power supply outages clearly documented a surge. It was not a standard policy
to have surge protectors. That has changed.
4) No preventive maintenance
After analyzing this, the team concluded that this may have saved the faulted units by
finding the other conditions before they caused an outage.

17
Aveta Six Sigma Online Black Belt Course

5) Untrained users
This is an issue for the technicians handling telephone calls. It is not a cause of
hardware failure. The user security and software prevent the users from damaging the
data. This topic will be analyzed for a future project.
6) Dirt
After reviewing the maintenance notes, this jumped out as the most likely cause of
issues in the operating locations. All of the maintenance logs for operations locations
referenced an abnormal amount of dirt in the machines. Based on that, the
installation specialist, technician and systems engineer from the vendor went to a
field location – specifically to a site of a failed DVR. The following outlines their
statements:

Technician – “No wonder the machine failed, this place is a mess”

Systems engineer - “We test machines in extreme environments, but this is
beyond what we would expect to see normally.”
Installation specialist after hearing their comments – “This is normal for
operations”

The operating locations tend to be temporary buildings much like you would find on a
construction site. One of the external customers is actually a construction firm.
Generally, there is a lot of dirt movement with large equipment causing a dusty
environment all around. The other observation was that the machines were not being
carefully placed. Basically, the site personnel tend to direct where the machines will
be placed and thinking of the machine’s health is not high on their list.

Based on the maintenance notes and site observations, we needed to find a way to protect the
equipment. Once again, the team headed into a brain storming session. Unfortunately, this time
did not produce a lot of results; however, Google™ saved the day. There are several inexpensive
covers and keyboard skins on the market. The resulting action plan followed:

1) All DVRs (and servers after a discussion with the IT group) in operations will have
dust covers.
2) All keyboards will have keyboard skins
3) IT staff will perform routine cleaning of the DVRs and servers when on site.
Compressed air cans are now standard issue.

Further checks will be needed to see if the above recommendations are necessary at transfer
stations and office locations.

To pilot the process, we pulled two machines at the same site (some sites have multiple DVRs)
from service for a quick preventive maintenance routine and cleaned them thoroughly. As they
went back into service, one was covered with a dust cover and key board skin and the other was
not. After a week, the team went back to the site to review the state of the machines. While it is
not conclusive, there was a noticeable difference in the machines. Only a longer period of time
and close review of the control charts of outages will clarify the understanding.

18
Aveta Six Sigma Online Black Belt Course

Control Phase

As noted throughout this document, there were several places where we discovered minor deviations from
best practices. As part of the ongoing control, there are several steps that are in place (or soon will be) to
control and monitor the production environment:

1) A configuration check list has been standardized

2) Each technician completes the check list and signs it.
3) Someone other than the technician reviews the checklist for completeness and prepares the
machines for shipping. The same person did step 2 and 3 previously.
4) Field IT will now handle preventive maintenance when at the sites working normal support.
They will note any issues and report to the Security technicians.
5) Control charts like the one in Figure 6 are now posted for the top 4 items creating DVR outages.
Any time complete failures happen once per week, the staff reviews the machine to look for
issues.
6) Routine comparisons of equipment trouble tickets will be compared to what the IT group is
experiencing to watch for similarities. As a side note, the IT group has implemented several of
the things the Security department has found and while they have not published expected savings,
it is easily going to be more the $100,000 annually when combined with the Security Department.

As you can tell from the brevity of the control section, we have not had time to confirm the results. This
has literally just been completed. Early indications from Security and IT are things are definitely on the
right track. Continued observation will tell the tale.

Conclusion

The net result of the project was more than anticipated in hard dollar savings. While we have not proven
that we are down to one outage every other week, the early indications are good. In addition, the IT
group is also seeing a reduced hardware outage issue, but not quite to the same degree. They had a better
preventive maintenance program in place, but protecting the machines and discussing the issue with site
management has proven productive. At this point, the estimated savings from this project is in excess of
$70,000 hard dollars. As stated many times above, there is no estimate on soft dollars. In addition to
that, we have also implemented new policies and practices. Some of those lessons learned include the
following:

1) Virus protection & device drivers updates and maintenance

2) Proper site preparations (wiring)
3) Surge protection
4) Servers were having similar issues

By using a systematic approach and truly analyzing the problem, we improved customer service, proved
to be responsive and saved money.

19
Aveta Six Sigma Online Black Belt Course

APPENDIX A – Cost Structures & Support Hour Samples

The following are the cost and implementation elements used for estimating outage impacts:

Function Cost
Contract labor – on site camera work $75/hour
DVR Shipping $50 each way
Central Technical support $20/hour
Network connection (Frame) $1000/month/site
DVRs in production 508
Sites with DVRs 432

The following are the labor estimates for configuring, installing and troubleshooting (many of
these are subject to future study for improvement via Six Sigma):

Function Responsible Party Labor Estimate

DVR setup (from un-boxing to Central Technical support 2 hours
repackaging
Telephone Support Central Technical support 15 minutes/call
Software download & configuration Central support 30 minutes each
On-site setup & configuration of DVR Contract Labor 2 hours

20
Aveta Six Sigma Online Black Belt Course

APPENDIX B – DVR Maintenance Log Sheet

The following is a blank maintenance log sheet as generated by the maintenance ticket system.
The process has the technician manually complete the form. Eventually, the data is entered into
the tracking system.

MAINTENANCE LOG SHEET

Date Received: Repair Ticket Number: auto-generated
DVR Location: Configuration:
Manufacturer: ____________________
Model: _____________________
Disk Drive type/size: ________________
Operating System: __________________
Address used for return shipment (Y/N):
Equipment Status on Receipt:

Repair Comments:

Software Load: Date Returned:

Shipment Tracking Number:
Technician:

21
Aveta Six Sigma Online Black Belt Course

APPENDIX C – Data Summaries

Summary of outages by weeks of the sample:

Sum of Outage Duration in hours

Week of Outage Total
1 186
2 281
3 106
4 200
5 217
6 145
7 169
8 265
9 290
10 170
11 73
12 204
13 97
14 241
15 97
16 145
Grand Total 2883

Summary of Outage Hours by outage type

Sum of Outage Duration in hours
Cause of Outage Total
DVR Hardware 2736
Network 77
Network Hardware 36
Software 25
User issue 10
Grand Total 2883

Summary of Technician Hours by outage type

Sum of Technician Hours
Cause of Outage Total
DVR Hardware 312
Network Hardware 36
Software 25
User issue 9.75
Network 0
Grand Total 382.75

22
Aveta Six Sigma Online Black Belt Course

Count of Outages by Type

Count of Cause of Outage
Cause of Outage Total
Network 142
DVR Hardware 39
User issue 39
Software 5
Network Hardware 2
Grand Total 227

DVR Failure counts by week and location type

Count of Cause of Outage
Week of Outage Location Type Cause of Outage Total
1 Operations DVR Hardware 2
2 Operations DVR Hardware 3
Transfer DVR Hardware 1
3 Operations DVR Hardware 1
4 Operations DVR Hardware 2
5 Operations DVR Hardware 2
Transfer DVR Hardware 1
6 Operations DVR Hardware 2
7 Operations DVR Hardware 3
8 Office DVR Hardware 1
Operations DVR Hardware 2
9 Operations DVR Hardware 4
10 Operations DVR Hardware 2
11 Operations DVR Hardware 1
12 Operations DVR Hardware 2
Transfer DVR Hardware 1
13 Operations DVR Hardware 2
14 Office DVR Hardware 1
Operations DVR Hardware 3
15 Operations DVR Hardware 1
16 Operations DVR Hardware 2

23