CAPA Effectiveness Checks

101: How To Verify That Your

Actions Plans Are Successful
By Danielle DeLucy, ASA Training & Quality Consulting

In the highly regulated life sciences industry, serious quality issues can and do occur. Once an
investigation team convenes and performs a thorough root cause analysis (RCA), the focus then turns
to remedying the issue. Perhaps the most commonly used method to fix quality issues — and ensure
they don’t rear their ugly heads again — is to create thorough corrective and preventative action
(CAPA) plans. These plans take each root cause that was investigated and spell out specific steps to
address them.
However, before you can consider the issue corrected, the CAPA needs to be verified as effective and
complete, per international regulations and standards. For example, step 3.2.2 of ICH Q10 states:
“CAPA should be used and the effectiveness of the actions should be evaluated.” Thus, quality units
must perform effectiveness checks on their CAPA plans — measuring and determining that the
corrective action has (or has not) eliminated the problem.

There are main three reasons for taking CAPA plans through effectiveness checks:

1. To build confidence that the solution will work

2. To validate that the solution did work
3. To minimize any risks or potential problems that might cause reoccurrence

Verifying the effectiveness of a CAPA closes the loop between identifying a problem and completing
the actions to solve a problem.

Choosing The Appropriate Measurement

Determining the best verification approach for the wide range of problems that could occur can often
be challenging. Both qualitative and quantitative methods can be used to measure success of a CAPA.
Qualitative methods explain and gain insight and understanding of an issue through intensive
collection of narrative data, generate a hypothesis to be tested. Quantitative methods explain,
predict, and/or control phenomena through focused collection of numerical data and hypothesis
testing.

Some firms struggle to use quantitative methods in performing effectiveness checks, because certain
aspects of CAPAs can be harder to measure than others. However, you can measure just about
anything. For example, you can measure entry errors for operators who are deviating from good
documentation practices by identifying areas where quality records are reviewed and documentation
errors are measured. You can measure by operator, by type of documentation (e.g., lab record, batch
record, or cleaning log), by month, etc.

Another example is the measurement of transcription errors made when copying information from
sample ID labels to laboratory notebooks. Say an investigative team determines that the labels made
on the current label printer are frequently unreadable and suggests a CAPA plan that includes
replacing the current label printer with one that produces legible labels. A proper effectiveness check
would include returning to the area within a week to determine if the new printer is operating
correctly and the labels can be read. In addition, the team could also monitor the number of errors
made during the week and compare the rate with the number of errors made prior to solution being
implemented.

To help determine the most appropriate measurement for your effectiveness check, answer some/all
of these questions:

1. Who will measure it? The task should be assigned to the quality unit or other group in the
organization that ensures all quality system issues are resolved according to regulations.
2. What will be measured? Measurements can take on many different appearances. You can
measure the number of errors made once the solution is in place versus the number made
prior to its implementation. You can measure the types/amount of deviations that occur after
the solution is implemented, or you can measure the effectiveness of a new process on a
specific product. You can also measure the amount of over-the-limit environmental
monitoring results after incubation.
3. Where will it be measured? In most cases, the measurement can be documented using the
CAPA form or electronic tracking system.
4. When will it be measured? The timing of the measurement depends on the severity of the
defect. CAPAs with low severity defects, which are not harmful to the patient, can be put in
the “parking lot.” While more significant defects, such as those that may cause the patient
life-threatening harm, should be investigated and checked immediately.
5. How will it be measured? Measurement of effectiveness checks can be done qualitatively or
quantitatively, as discussed in the previous examples. Metric-based checks can account for
the quantitative aspect, whereas the qualitative approach can include revisions to a
standard operating procedure (SOP) or training of impacted employees. For example, once
the SOP has been revised, you can monitor employee performance and compliance to the
SOP revision by tracking the number of deviations to the step.
6. How will measurements be analyzed? Analysis of the measurements is conducted by the
quality unit, who will determine if the solution was effective and the CAPA plan can be
 closed. For instance, quality staff can conduct measurements by heading out onto the shop
floor to determine if corrective actions are being followed and if they are indeed beneficial to
the process.
7. Who will data analysis be communicated to? As in most organizations, this data analysis
should be discussed and presented to the quality unit executive management team in the
form of monthly metrics. Of course, the analysis should also be communicated to the owners
of the CAPA plans.

Once these questions have been answered, you can move on to developing ways to verify your CAPA
effectiveness.

Methods For Verifying Effectiveness

There are many highly effective tools and principles you can use to make your effectiveness checks
successful. Here are a few common approaches:

 Trend analysis — In cases of human error, training, environmental monitoring excursions,

cleaning deviations, testing errors, etc., trend analysis can help you determine if the
corrective action has remediated the issue. Review data over a predetermined timeframe
and determine if the problem or deviation has occurred again (or as frequently) after the
corrective action was implemented. In most cases, the data will tell you if the fix was
successful, or if another corrective action should be established.
 Periodic checks — This method involves scheduling a time for the quality unit to review the
process that was remediated. For example, if there was a CAPA implemented to improve
gowning practices, the auditor can observe the newly improved gowning practices prior to
the operator entering the clean room.
 Surprise audits — Audits are intended to ensure that the operators, process, or equipment is
following the prescribed corrective action documented in the CAPA. One other benefit of
surprise audits is that the reviewer gets to observe the daily occurrences in the area and the
how the solution is working in the area affected.
 Sampling — Sampling is a successful method to verify corrective actions pertaining to
environmental monitoring or lab testing. For example, if there was a deviation for an area
exceeding environmental action limits, and more robust cleaning practices were installed,
additional samples could be taken to verify that the new cleaning protocol was effective.

When To Schedule Effectiveness Checks

Another vital piece to ensuring a robust and properly managed CAPA effectiveness verification is
establishing the appropriate timeframes to perform the checks. When settling on a due date, it is
important to allow enough time for the corrective action to take effect, while also maintaining a sense
of urgency around ensuring that the issue is fixed and will not recur.

Some deviations or problems only require a simple procedure update, retraining of staff, or
recalibration of a piece of equipment. However, in many cases, CAPAs are opened to address more
difficult issues, such as changing the facility layout, altering gowning practices, increasing sample
sizes, or implementing a new piece of equipment. In these cases, it is best practice to set up several
effectiveness checks over a period of time. Perhaps the first check can be done rather quickly after
the CAPA plan has been approved, but others will be spread out over months.

Conclusion

CAPA effectiveness checks are a surefire way to track, trend, and remediate deviations that occur
during manufacturing and laboratory testing. By developing solid corrective action plans and
appropriately timed effectiveness checks, firms can begin the process of remediating issues that
could cause deviations. Having a strong quality unit in place to perform these checks is the key to
ensuring that these effectiveness checks are conducted in a timely fashion and are improving the
process. When done right, these checks demonstrate your firm’s commitment to quality and
compliance.

A SMART Approach To CAPA

Effectiveness Checks
By Mark Durivage, Quality Systems Compliance LLC

Corrective and preventive action (CAPA) issues continue to be one of the top Form 483 observational
findings by the FDA. Many times, CAPAs fail due to the structure and flow of the process and not
necessarily the efforts of those managing the CAPAs. Frequently, organizations do not fully document
the CAPA phases and confuse verification of implementation and verification of effectiveness
activities. (For an introduction to the CAPA phases, read my previous article “The 10 Phases Of An
Effective CAPA.”) This article will look at using the SMART
(specific, measurable, achievable, relevant, and time-bound) methodology for use in developing a
CAPA verification of effectiveness plan.

CAPA Requirements

Several international regulations and standards provide guidance on conducting CAPAs:

21 CFR 820 Quality System Regulation

Sec. 820.100 Corrective and preventive action:

(a) Each manufacturer shall establish and maintain procedures for implementing corrective
and preventive action. The procedures shall include requirements for:

(1) Analyzing processes, work operations, concessions, quality audit reports, quality
records, service records, complaints, returned product, and other sources of quality
data to identify existing and potential causes of nonconforming product, or other quality
problems. Appropriate statistical methodology shall be employed where necessary to
detect recurring quality problems;

(2) Investigating the cause of nonconformities relating to product, processes, and the
quality system;

(3) Identifying the action(s) needed to correct and prevent recurrence of nonconforming
product and other quality problems;

(4) Verifying or validating the corrective and preventive action to ensure that such
action is effective and does not adversely affect the finished device;

(5) Implementing and recording changes in methods and procedures needed to correct
and prevent identified quality problems;

(6) Ensuring that information related to quality problems or nonconforming product is

disseminated to those directly responsible for assuring the quality of such product or
the prevention of such problems; and

(7) Submitting relevant information on identified quality problems, as well as corrective

and preventive actions, for management review.

(b) All activities required under this section, and their results, shall be documented.

ISO 13485:2016 Medical devices — Quality management systems — Requirements for

regulatory purposes

8.5.2 Corrective action

The organization shall take action to eliminate the cause of nonconformities in order to prevent
recurrence. Any necessary corrective actions shall be taken without undue delay. Corrective actions
shall be proportionate to the effects of the nonconformities encountered.

The organization shall document a procedure to define requirements for:

a) reviewing nonconformities (including complaints);

b) determining the causes of nonconformities;

 c) evaluating the need for action to ensure that nonconformities do not recur;

d) planning and documenting action needed and implementing such action, including, as
appropriate, updating documentation;

e) verifying that the corrective action does not adversely affect the ability to meet applicable
regulatory requirements or the safety and performance of the medical device;

f) reviewing the effectiveness of corrective action taken.

Records of the results of any investigation and of action taken shall be maintained.

8.5.3 Preventive action

The organization shall determine action to eliminate the causes of potential nonconformities in order
to prevent their occurrence. Preventive actions shall be proportionate to the effects of the potential
problems.

The organization shall document a procedure to describe requirements for:

a) determining potential nonconformities and their causes;

b) evaluating the need for action to prevent occurrence of nonconformities;

c) planning and documenting action needed and implementing such action, including, as
appropriate, updating documentation;

d) verifying that the action does not adversely affect the ability to meet applicable regulatory
requirements or the safety and performance of the medical device;

e) reviewing the effectiveness of the preventive action taken, as appropriate. Records of the
results of any investigations and of action taken shall be maintained.

The Joint IPEC - PQG Good Manufacturing Practices Guide for Pharmaceutical Excipients

8.5.2 Corrective Action

The excipient manufacturer should establish, document and maintain procedures for:

 determining the root causes of nonconformities,

 ensuring that corrective actions are implemented and effective,
 implementing and recording changes in procedures resulting from corrective action.

8.5.3 Preventive Action

The excipient manufacturer should establish, document and maintain procedures for:

 initiating preventive actions to deal with problems at a level corresponding to the risks,
 implementing and recording changes in procedures resulting from preventive action.

ICH Harmonised Tripartite Guideline Pharmaceutical Quality System Q10

3.2.2 Corrective Action and Preventive Action (CAPA) System

The pharmaceutical company should have a system for implementing corrective actions and
preventive actions resulting from the investigation of complaints, product rejections, non-
conformances, recalls, deviations, audits, regulatory inspections and findings, and trends from
process performance and product quality monitoring. A structured approach to the investigation
process should be used with the objective of determining the root cause. The level of effort, formality,
and documentation of the investigation should be commensurate with the level of risk, in line with
ICH Q9. CAPA methodology should result in product and process improvements and enhanced
product and process understanding.
Applying The SMART Methodology To CAPAs

An effective complaint CAPA process consists of 10 distinct phases, as shown in the figure below.
Phase 8, verification of effectiveness plan (VOEP), is rarely used, yet it can cause a CAPA to fail
during the verification of effectiveness check. The VOEP is used to establish and document
predetermined criteria to verify that corrective and/or preventive actions were indeed effective.

George T. Doran is credited with first using the term SMART in the November 1981 issue
of Management Review. Peter Drucker's management by objectives concept utilized SMART
goals. Today, the SMART methodology is generally associated with personal goals and
performance objectives related to raises and annual bonuses, but the methodology can be readily
be applied to CAPA VOEP activities.
 When developing and evaluating a SMART VOEP, the following questions should be considered:

 Specific — Is the VOEP unambiguous, clear, and focused?

 Measurable — Is quantifiable data being used to assess the VOEP?
 Achievable — Is the VOEP feasible or practical?
 Relevant — Is the VOEP appropriate for the level of risk?
 Time-Bound — Does the VOEP have a realistic deadline?

Additionally, reasonable and realistic are acceptable substitutes for the R or relevant question.

Examples of a VOEP’s statements include:

 The problem has not recurred in the last three production lots.
 The problem has not recurred in the last three months.
 The first pass yield will increase from 90 to >99 percent during Q2 2020.
 The scrap rate will decrease by 5 percent (say, from 15 percent to 10 percent) over three
months.

The verification of effectiveness phase is the process of reviewing the criteria and documenting the
evidence that was predetermined in the VOEP prior to closure. Just as in process validation activities,
if the predetermined acceptance criteria are not met, the CAPA fails. When a verification of
effectiveness fails, close the CAPA as ineffective and open a new CAPA. Ensure the new CAPA
references the old CAPA.

VOEP Example

A short time ago, a mature stable package sealing process running on Heat Sealer 20 began
occasionally producing parts that were not meeting the approved pull test specification that was
necessary to maintain package integrity. A typical production run is approximately 1,000 packages,
which is easily completed during a normal workday for the facility.
The production manager opened CAPA 2020-005 as a result of the nonconforming package seals
being produced. A CAPA team was formed, an impact assessment was conducted, and containment
actions were performed due to the risk involved with the defective package seals. Production
records, nonconformances, calibrations, maintenance records, training records, and validation
documentation for the past year were reviewed, and the heat sealer machine operators were
interviewed. A root cause investigation was conducted, a root cause determined, and the corrective
actions were implemented.

During the review of the production records, nonconformances, calibrations, maintenance records,
training records, validation documentation, and operator interviews, the CAPA team’s investigation
concluded machine settings on Heat Sealer 20 were sometimes set outside the validated parameters
by new, inexperienced heat sealer machine operators. The CAPA team concluded this was the most
likely root cause of the nonconforming package seals.

The CAPA team decided to address the root cause by changing the programmable logic controller
user access permissions, which effectively restricted the operators’ ability to select parameters that
were outside the validated settings on the heat sealer. The heat seal machine operators will be
retrained to reinforce the importance of using the validated machine parameters and that these
validated settings are crucial to prevent package seal failures. Additionally, the CAPA team initiated a
change order to require the actual heat sealer machine settings to be recorded in the production
record at the beginning and end of the production record to verify the settings were within the
validated ranges.

The production manager, in conjunction with the CAPA team, used the SMART method to develop the
VOEP. The CAPA team decided on the following VOEP: The packages sealed on Heat Sealer 20 will
be monitored for conformance to specifications (seal integrity failures) over three consecutive days of
production (three lots) using a tightened level of inspection per the sampling procedure. The CAPA
will be deemed effective if all of the packages produced for the next three consecutive days of
production (three lots), meet the pull test specification, the operators are unable to select parameters
outside of the validated range, and the proper validated settings are recorded in the batch record.

The CAPA team performed a verification of implementation assessment to verify the heat sealer
operators were retrained, the programmable logic controller user access permissions restrict the
operators’ ability to select parameters that are outside the validated settings on the heat sealer, and
the production record documentation was updated to document the settings are within the validated
ranges. The CAPA team concluded the verification of implementation tasks was completed as
planned.

For the next step the CAPA team evaluated the VOEP to determine whether it’s SMART.

Specific — Is the VOEP unambiguous, clear, and focused? ☑ Yes ☐ No

 The packages produced for the next three consecutive days of production (three lots) on
Heat Sealer 20 must meet the pull test specification.
 The operators are unable to select parameters outside of the validated range.
 The proper validated settings are recorded in the batch record.

Measurable — Is quantifiable data being used to assess the VOEP? ☑ Yes ☐ No

 The packages produced for the next three consecutive days of production (three lots) on
Heat Sealer 20 must meet the pull test specification.
 The operators are unable to select parameters outside of the validated range.
 The proper validated settings are recorded in the batch record.

Achievable — Is the VOE feasible or practical? ☑ Yes ☐ No

 The packages produced for the next three consecutive days of production (three lots) on
Heat Sealer 20 must meet the pull test specification.

Relevant — Is the VOE appropriate for the level of risk? ☑ Yes ☐ No

 The packages sealed on Heat Sealer 20 will be monitored for conformance to specifications
(seal integrity failures) over three consecutive days of production (three lots) using a
tightened level of inspection per the sampling procedure.

Time-Bound — Does the VOE have a deadline? ☑Yes ☐ No

  The packages produced for the next three consecutive days of production (three lots)

The CAPA team performed the VOE following results:

☑ Yes ☐ No Heat Sealer 20 ran for three consecutive days of

production (three lots) using the validated settings.

☑ Yes ☐ No All packages meet the pull test specification.

☑ Yes ☐ No The validated heat sealer settings for the beginning and
end of each production run were properly recorded in the batch record.

The CAPA team decided the CAPA was complete and effective. The team also shared its success with
other departments so similar actions could be implemented.

The example demonstrates the appropriate use of the SMART method to plan and verify CAPA
effectiveness, ultimately resulting in CAPA closure. Adapting the SMART method can help improve
the effectiveness of your CAPA program and demonstrate compliance with regulations, standards,
and expectations.

Conclusion

Not only are CAPAs required by regulations, standards, and guidances, CAPAs, if performed
correctly, can help an organization improve its competitive position by reducing waste and improving
processes. Using the SMART methodology is an effective way to develop sound verification of
effectiveness plan, leading to a compliant CAPA system.