Quality Metrics Publications & Tools

ISPE Quality Metrics Pilot Program

The first public industry-regulator dialog on Quality Metrics was held at the ISPE-FDA cGMP Conference in June 2013. Feedback from these discussions identified that:

  • Those companies that collect metrics do so using different business processes and different definitions and with different objectives after review of data
  • Metric data often exist in different databases even within the same company and are difficult to collect, summarize and submit
  • Definitions of metrics chosen need to be extremely well established so that data are consistently generated

Given this complexity of gathering, analyzing and reviewing data, ISPE proposed to FDA in the whitepaper ISPE Proposals for FDA Quality Metrics Program  that a pilot program should be conducted within industry to further understand the implementation opportunities, challenges and benefits of a standardized quality metrics program.

ISPE, in cooperation with McKinsey and Company, undertook the industry most extensive quality metrics pilot program from 2014 – 2016. Findings from the pilot have enabled ISPE and industry to provide data-driven input to the FDA on their proposed metrics program.


ISPE Quality Metrics Pilot Project—Wave 1

Designed and developed by the ISPE Quality Metrics Core Team, the project drew on the knowledge and experience of cross-functional industry representatives, ex-regulators and academicians, with further insight gained in detailed discussions with a variety of industry associations at many industry meetings.

Wave 1 of the Pilot ran from June through November 2014 and included data collected at 44 sites from 18 participating companies. Nearly all metrics collected were reported at site level; three were collected at product level within each site.

The objective of Wave 1 was to provide real world experience with metrics definitions, data collection and reporting burden for the benefit of both industry and regulators.

Metrics tested in Wave 1:

  • Quantitative Metrics
    • Lot acceptance rate B 
    • Complaints rate (total B and critical)
    • Confirmed OOS rate B
    • US recall events (total and by class) B
    • Stability Failure rate B
    • Invalidated (unconfirmed)  OOS rate B
    • Right first time (Rework/Reprocessing) rate B
    • APQR reviews completed on time
    • Recurring deviations rate
    • CAPA effectiveness rate
    • Media fill (for sterile aseptic sites) failures B
    • Environmental monitoring (for sterile aseptic sites) B
  • Technology-specific Metrics
    • Media fill (for sterile aseptic sites) failures B
    • Environmental monitoring (for sterile aseptic sites) B
  • Survey-based metrics
    • Process capability
    • Quality culture
  • Two more quantitative metrics calculated from data collected for Wave 1:
    • Deviations rate
    • Incoming material OOS

ISPE Quality Metrics Pilot Project—Wave 2

Wave 2 of the pilot ran from August through November 2015 with expanded industry segments, geographies, and time to provide deeper understanding of the relationships revealed in Wave 1.

During the planning of the Wave 2 pilot, the USFDA issued its Request for Quality Metrics draft Guidance. The Wave 2 pilot was adjusted to include:

  • Testing the proposed FDA metrics as defined as well as testing alternative definitions applied to the FDA Draft Guidance metrics
  • Evaluating the logistics and effort of gathering data at a product-application level

Metrics Tested in Wave 2:

  • Qualitative metrics
    • Product Quality (Total) Complaint Rate
    • Critical Complaints Rate
    • Recall Events
    • Lot Acceptance Rate
    • Invalidated OOS Rate
    • Right-First-Time Rate
    • Deviations Rate1
    • Recurring Deviations Rate
  • Culture indicators
    • CAPAs with Preventive Actions
    • Planned Maintenance
    • Employee Turnover
    • Human Error Deviations
    • Deviations with no Assigned Root Cause
  • Surveys
    • Quality Culture2
    • Process Capability
    • Questions to assess whether standardized metrics can assist prediction of drug shortages

Insights from the Pilot

  • Definitions and terminology must be exact; in particular, denominators can have a big impact on a company’s ability to report on the metric with either ease or difficulty
  • Even common terms like “lot”, “deviations”, “complaints” and “reviews” must be specified in great detail to minimize multiple interpretations
  • Product and process differences will generate differences and variations in metric ranges
  • Commentary on data points is essential to data interpretation and analysis
  • Frequency of data collection (e.g. quarterly, annually) has a major impact; analytic and reporting burden is proportional to the number of data points collected
  • Site-level metrics are more easily systematized for collection
  • Product-level metrics are significantly more complicated to collect; typically require a company to pull data from multiple systems and can rapidly result in large data management requirements

Quality Metrics Pilot Program Reports

Learnings from the ISPE Quality Metrics Pilot Program are fully documented in two reports, including:

  • Analysis of metrics data and relationships to internal and external quality outcomes
  • The actual time and effort required to collect FDA Draft Guidance metrics as compared to the FDA estimate.
  • Aggregated data from the participating companies, useful for benchmarking
  • Culture findings and relationships
  • Process Capability landscape
  • Success factors for a quality metrics program

Download the Complete Reports:

ISPE Quality Metrics Pilot Program: Wave 1
Download Report
ISPE Quality Metrics Pilot Program: Wave 2
Download Report