Case Study - Pharmaceutical automation

Automation is a risky endeavor—improve your odds with

planning, requirements, testing, and documentation
FAST FORWARD

 In pharmaceutical automation projects, define and fix your requirements.

 Have a robust plan, obtain management support, and maintain the discipline to
execute the plan.
 Too little testing during the development process will result in missed mistakes
during application coding.

By Dave Adler

Did you ever wonder why it is so difficult to have a successful

pharmaceutical automation project? My definition of success is measured by
achieving the schedule milestones, meeting the cost estimate, satisfying the
system automation requirements, having the automation system work from
day one, and satisfying the facility’s business leaders. To satisfy all of these
measures is almost impossible. Pharmaceutical automation is tough, but
when successful, it is very rewarding.

Numerous studies of software projects have found success rates of less than
20%, where success was defined as achieving schedule, meeting cost
estimates, and satisfying requirements. Automation professionals who find project success
challenging have lots of company. There are many ways to do automation projects poorly,
but just a few ways to do them correctly.

I recently conducted research on pharmaceutical automation technology, costs, and benefits

for 24 facilities of 16 member companies of the Pharmaceutical Automation Roundtable
(PAR). This study analyzed the relative automation cost per input and output device (I/O).
The costs per I/O varied greater than a factor of three from the least expensive to the most
expensive automation system. The wide variance in automation cost per I/O in the study
indicates the opportunity exists to optimize the business processes to manage automation
projects.

Pharmaceutical companies have developed a rigorous methodology for automation systems.

The industry uses a life-cycle model known as computer system validation to ensure the
automation system does what it is supposed to do and can be expected to continue doing so in
the future. Before I lose my non-pharmaceutical industry readers, this is just a fancy way of
saying automation professionals need to:

1. Do upfront planning.
2. Define requirements for the automation system.
3. Test the automation system.
 4. Document the technical content.

I hope every automation professional does each of these activities on every project, but of
course not to the level of the pharmaceutical industry.

The pharmaceutical industry is regulated by the U.S. Food and Drug Administration (FDA)
and the international Ministry’s of Health. These regulations apply to the manufacturing of
drugs and medical devices including the use of computers to manufacture these products. In
1983, the FDA published its first guide to computer system validation. Since then, the
industry’s automation professionals have developed the business processes to support a
cradle to grave life-cycle approach to automation. The industry has had a lot of opportunity
over the years to use business processes to support the delivery of automation. An industry
trade group, International Society of Pharmaceutical Engineers has produced a reference
guide to Good Automation Manufacturing Practice that highlights one approach widely
adopted.

I have been involved with more than 20 major pharmaceutical automation projects in my
career, so I have had the opportunity to be on many critical and even a few troubled
automation projects. I have learned many painful lessons and now have many stories to tell.
These lessons are applicable to an automation professional in any industry. Your chances of
having a successful automation project can be greatly increased by using appropriate
planning, requirements, coding, testing, and documentation practices.

Planning can be guide to success

Step one in improving your odds of a successful automation project is developing a plan and
getting all the key stakeholders to buy into your approach. I hope by now I have convinced
you how difficult it is to have a successful automation project. A structured approach, starting
with a plan, can increase your odds of success.

An automation plan at a high level defines: the project drivers, the scope of work, the
automation system’s desired functionality, the operational strategy, the safety expectations,
the maintenance strategy, the schedule, and the cost estimate. In the pharmaceutical industry,
there is a regulatory requirement to have a validation master plan. An automation validation
master plan defines at a high level the expectations for quality, requirements, testing,
documentation, review, and approval. It would also cover expectations for security, change
control, contingency planning, and periodic reviews.

There are a number of guides available to help organize a plan. Business processes are
available for project managers such as those documented by the Project Management Institute
that have been used by automation professionals for our discipline. Guides are available for
scoping and estimating automation projects from the author.

During the initial planning of a proposed automation upgrade project, controlling cost was
identified as the number one issue with getting approval. The initial proposal by the facility
planning group was rejected by the management team based on the cost of other recent
upgrade projects. The automation team was asked to significantly reduce the estimated cost
for the proposed project. The planning effort required the automation team to think outside
the box. None of the existing automation business processes were immune from review. The
planning process took several months. A plan was developed that reduced the proposed
automation estimate by 25%. This cost reduction was due to changes in the business process
and not the overall scope of the work (e.g., fewer control loops). Highlights of the planning
process were to:

1. Choose experienced handpicked automation professionals.

2. Dedicate automation staff with no other responsibilities.
3. Co-locate all automation staff in one room.
4. Ensure tech service, process engineers, and operations personnel availability when
needed.
5. Define roles and responsibilities.
6. Develop a prototype for the entire software development process.
7. Replicate from previous projects.
8. Have well-defined scope and fixed requirements.
9. Create a “just say no” list of cost enhancers.

The facilities business leaders agreed and bought into this plan because they wanted the lower
cost. The automation upgrade project was approved. I will skip forward several months as
this project was recently completed. The automation project actually pleased the facility’s
business leaders. The software and hardware worked flawlessly during start-up, the project
was completed on schedule, and the final automation project cost was 20% less than the
revised project plan estimate. If you have a robust plan, obtain management support, and
maintain the discipline to execute the plan, you can achieve your targets.

Define, fix your requirements

You must accurately define your automation project requirements with the help of the users
of the automation system. The requirements should be measurable and testable. They need to
identify the business, equipment, and process needs. Define the requirements before you start
the design, and get the key stakeholders to agree to them. Communicate broadly these
requirements. You need to keep the requirements fixed during the course of the project
design, implementation, and through start-up. If you can minimize scope creep, you have
removed a major hurdle to automation project success. Scope creep can result in changes and
additions to requirements that can greatly lengthen and increase the cost of the project.

One of my early failures was a major pharmaceutical plant upgrade in the early 1990s that
was a fast track project. It looked like we would not make enough material to launch this
projected blockbuster product. The project manager and manufacturing executives wanted
automation to get started and get off the critical path of the overall project. The automation
team felt a lot of pressure to get started. We started ordering instruments without piping and
instrument drawings. We started writing automation application code without agreed to valid
requirements. In fact, we were still arguing about requirements during start-up. Not
surprisingly, this project went poorly, and start-up did not go well. Cost estimates were
missed, and schedule milestones were not met. Product was not made on time. This story
does not have a happy ending. Even though it was not a good experience for me, I learned
some valuable lessons.

Later in my career, in the mid 2000s, I was again assigned to a fast-track pharmaceutical
upgrade project. The business drivers mandated a compressed schedule. The project manager
and manufacturing executives wanted automation to get started and get off the critical path of
the overall project. The automation team felt a lot of pressure to get started. However, we
refused to write code and order instruments until we had requirements and piping and
instrument drawings. Of course, I was getting worried looks and phone calls from everyone
in management. We took two months to define requirements and locked in the piping and
instrument drawings before we started writing code and ordering instruments. I will make a
long and grueling story short and jump to the end of the story: Automation was done on
schedule and on budget, and it worked well. We had a successful start-up, and within six
months, automation facilitated some dramatic improvements in the operations of the facility.
It does pay to plan your project and get the requirements right before you start your work.

Testing reduces start-up issues

If a developer does too little testing during the development process, there will be mistakes in
the process control application code. The software will then have to be debugged later in the
development process or during start-up of the manufacturing equipment in the facility. It is
significantly more costly to debug software during start-up than during the development
process. The sooner a developer catches a mistake, the cheaper it is to fix. Appropriate testing
can reduce overall project cost and minimize rework during start-up. Of course, inappropriate
testing will increase cost and lengthen the development time.

Testing determines if the automation system meets the previously defined requirements. The
success of testing depends, in part, on good requirements. If a process automation
professional has solid and fixed requirements, it is much easier testing.

The development process consists of testing the software at each stage of the process. The
testing starts with the individual software module, and then the units should be individually
tested. This module and unit testing uses “test scripts” that test small portions of the whole
system software. It is important to define test scripts so the observed results are clear and
concise. This testing is frequently conducted in an off-line or development system.

Once all the individual units have been tested, the overall system is tested as a whole. A
portion of this may be achieved as a Factory Acceptance Test, or FAT. Final testing in the
pharmaceutical industry is often defined as on-site acceptance testing and includes
installation qualification, operation qualification, and performance qualification.

During a recent retrofit project, it was critical to minimize the time the facility was down. The
facility was producing a life-saving medicine. The needed production output would not allow
for an extended downtime. This retrofit required software and hardware changes to replace an
existing obsolete automation system, including its I/O with a new automation system. This
led to significant off-line testing of the automation software. In addition, a plan was
developed to optimize hardware changeover. Off-line testing of the hardware was desired. An
I/O room was built of plywood and plastic sheets to simulate the hardware changeover
process. This room was a full-scale mock-up of the actual I/O room including the elevated
floors. It allowed training of the electricians and trial runs of wiring efforts. The electricians
practiced disassembly of the old I/O system and reassembly of the new I/O system. It allowed
optimization of the number and activities of the electricians, pre-labeling terminations,
fabrication of panels, and even allowed some pre-cut wire to be made up to exact lengths.
This reduced the wiring effort during construction by over 75%. The actual hardware change-
out took days rather than the normal weeks to change out a large I/O system. The most
surprising outcome was the project actually was less expensive than planned because start-up
went so well with almost no hardware and software issues. Even with the extra testing
expense, this project was less expensive overall than other retrofit projects being done at the
same time.

Long-term viability

Documentation is critical to the success of an automation project. If the development team

does too little, the long-term support team will have a difficult time and exert additional effort
to maintain the automation system. If the development team does too much, the project could
be delayed and experience cost overruns.

When I first started in process automation in the early 1980s, I learned a painful lesson. On
one of my first projects, I did the minimum level of documentation. It resulted in frequent
phone calls from me to explain to my replacement what the design or software was doing.
Since the new engineer could not figure out what I did, I clearly had not documented the
automation system well. In future projects, I spent a lot more time and effort to document as I
designed and coded the automation applications.

By the late 1980s, the pharmaceutical industry was implementing computer system
validation. One of the unique features of the pharmaceutical industry is the actual medicine
you take sometimes cannot be completely tested. A sampling of medicine from each batch is
tested, but the testing itself often destroys the product. Obviously the pill that the patient
takes cannot undergo a destructive test. This has led to a business process to ensure quality
called validation. Quality has to be built into the process of manufacturing the product and
the software to control the equipment. Quality cannot be tested into a product, but it must be
built into the process for making the product. Industries’ response to build quality into the
manufacturing process resulted in progressively more rigorous and larger computer system
validation documentation packages on pharmaceutical projects. By the early 2000s,
automation projects were being implemented with only 10% of the effort on design and
coding, but 90% of the effort on testing and producing documentation.

By the mid 2000s, the pharmaceutical industry was taking a look at using a more appropriate
level of documentation. Numerous FDA investigators made comments that it was not the
thickness of the documentation that was important, but rather the business value of the
documentation. The FDA encouraged industry to take a critical look at its business processes
to ensure cost-effective drugs for its patients. Concepts such as using a “risk-based” approach
to the overall validation process and the use of in-line quality measurements through Process
Analytic Technology were initiated. This openness by the regulators to change resulted in the
automation discipline taking a critical look at all its business processes including
documentation.

Before the start of a major project in 2006, the level of documentation was identified as
important to a cost-effective and on-schedule project. The area’s management team set up an
improvement team to study the computer system validation process. It defined the problem,
measured the process, analyzed the data, improved the process, and set up a control system to
manage the new process. The process completely prototyped the workflows and insured they
functioned as intended. The process revised policies and procedures appropriately. Specific
examples of improvements were using checklists instead of detailed narratives to aid in
document assembly and preparation, smaller test scripts, minimizing the process to fix script
errors, reducing the number of reviewers and approvers, shorter targeted QC reviews, moving
to an electronic document management system from a paper system, and reducing the level of
documents to support unit level testing with more focus on system level testing
documentation. These efforts reduced the total level of effort on documentation by 50%, but
more importantly, it produced more useful content for the long-term system support and the
future optimization of the manufacturing process.

The details do matter

Contrary to a prevalent management myth that claims you do not need to worry about the
details, in pharmaceutical automation projects, it is the details that really matter. If you do not
worry about the details, your project will not be successful. When you work on a project you
need to sweat the details including: doing upfront planning, having well defined and fixed
requirements, conducting testing, and documenting the appropriate technical content.