Quality Metrics Publications & Tools

ISPE Quality Metrics Pilot Program

The first public industry-regulator dialog on Quality Metrics was held at the ISPE-FDA cGMP Conference in June 2013. Feedback from these discussions identified that:

  • Those companies that collect metrics do so using different business processes and different definitions and with different objectives after review of data
  • Metric data often exist in different databases even within the same company and are difficult to collect, summarize and submit
  • Definitions of metrics chosen need to be extremely well established so that data are consistently generated

Given this complexity of gathering, analyzing and reviewing data, ISPE proposed to FDA in the whitepaper ISPE Proposals for FDA Quality Metrics Program  that a pilot program should be conducted within industry to further understand the implementation opportunities, challenges and benefits of a standardized quality metrics program.

ISPE, in cooperation with McKinsey and Company, undertook the industry most extensive quality metrics pilot program from 2014 – 2016. Findings from the pilot have enabled ISPE and industry to provide data-driven input to the FDA on their proposed metrics program.

ISPE Quality Metrics Pilot Project—Wave 1

Designed and developed by the ISPE Quality Metrics Core Team, the project drew on the knowledge and experience of cross-functional industry representatives, ex-regulators and academicians, with further insight gained in detailed discussions with a variety of industry associations at many industry meetings.

Wave 1 of the Pilot ran from June through November 2014 and included data collected at 44 sites from 18 participating companies. Nearly all metrics collected were reported at site level; three were collected at product level within each site.

The objective of Wave 1 was to provide real world experience with metrics definitions, data collection and reporting burden for the benefit of both industry and regulators.

Metrics tested in Wave 1:

  • Quantitative Metrics
    • Lot acceptance rate B 
    • Complaints rate (total B and critical)
    • Confirmed OOS rate B
    • US recall events (total and by class) B
    • Stability Failure rate B
    • Invalidated (unconfirmed)  OOS rate B
    • Right first time (Rework/Reprocessing) rate B
    • APQR reviews completed on time
    • Recurring deviations rate
    • CAPA effectiveness rate
    • Media fill (for sterile aseptic sites) failures B
    • Environmental monitoring (for sterile aseptic sites) B
  • Technology-specific Metrics
    • Media fill (for sterile aseptic sites) failures B
    • Environmental monitoring (for sterile aseptic sites) B
  • Survey-based metrics
    • Process capability
    • Quality culture
  • Two more quantitative metrics calculated from data collected for Wave 1:
    • Deviations rate
    • Incoming material OOS

ISPE Quality Metrics Pilot Project—Wave 2

Wave 2 of the pilot ran from August through November 2015 with expanded industry segments, geographies, and time to provide deeper understanding of the relationships revealed in Wave 1.

During the planning of the Wave 2 pilot, the USFDA issued its Request for Quality Metrics draft Guidance. The Wave 2 pilot was adjusted to include:

  • Testing the proposed FDA metrics as defined as well as testing alternative definitions applied to the FDA Draft Guidance metrics
  • Evaluating the logistics and effort of gathering data at a product-application level

Metrics Tested in Wave 2:

  • Qualitative metrics
    • Product Quality (Total) Complaint Rate
    • Critical Complaints Rate
    • Recall Events
    • Lot Acceptance Rate
    • Invalidated OOS Rate
    • Right-First-Time Rate
    • Deviations Rate1
    • Recurring Deviations Rate
  • Culture indicators
    • CAPAs with Preventive Actions
    • Planned Maintenance
    • Employee Turnover
    • Human Error Deviations
    • Deviations with no Assigned Root Cause
  • Surveys
    • Quality Culture2
    • Process Capability
    • Questions to assess whether standardized metrics can assist prediction of drug shortages

Insights from the Pilot

  • Definitions and terminology must be exact; in particular, denominators can have a big impact on a company’s ability to report on the metric with either ease or difficulty
  • Even common terms like “lot”, “deviations”, “complaints” and “reviews” must be specified in great detail to minimize multiple interpretations
  • Product and process differences will generate differences and variations in metric ranges
  • Commentary on data points is essential to data interpretation and analysis
  • Frequency of data collection (e.g. quarterly, annually) has a major impact; analytic and reporting burden is proportional to the number of data points collected
  • Site-level metrics are more easily systematized for collection
  • Product-level metrics are significantly more complicated to collect; typically require a company to pull data from multiple systems and can rapidly result in large data management requirements

Quality Metrics Pilot Program Reports

Learnings from the ISPE Quality Metrics Pilot Program are fully documented in two reports, including:

  • Analysis of metrics data and relationships to internal and external quality outcomes
  • The actual time and effort required to collect FDA Draft Guidance metrics as compared to the FDA estimate.
  • Aggregated data from the participating companies, useful for benchmarking
  • Culture findings and relationships
  • Process Capability landscape
  • Success factors for a quality metrics program

Download the Complete Reports:

ISPE Quality Metrics Pilot Program: Wave 1
Download Report
ISPE Quality Metrics Pilot Program: Wave 2
Download Report

Input to Regulators

ISPE continues to support a quality metrics program that has value to regulators, industry and patients. ISPE’s input is experiential and data driven, drawn from:

  • A quality metrics initiative spanning more than 4 years and led by a core team and senior leaders representing 10 companies
  • An extensive pilot program conducted in cooperation with McKinsey and Company with participation from 28 companies and 83 sites.  These companies and sites represented a wide range of technologies and included contract manufacturing organizations (CMOs) and laboratories, and drug substance manufacturing sites.
  • Response team of knowledgeable practitioners, 21 members, 12 companies including CMOs and API manufacturers
  • Ongoing robust dialog with pilot companies and the industry at large at ISPE conferences, workshops, and forums

Insights gained over the past four years remain consistent:

  • Definitions and terminology must be exact in order to avoid multiple interpretations by participants, and to enable consistency in reporting and ability to conduct analysis
  • Many companies currently collect metric data at a site level and often utilize definitions different from the FDA or agreed ISPE harmonized definitions. Moreover, there are often different definitions or variations in interpretations of a definition between sites in the same company. Consequently, there is a significant burden for companies to collect data against harmonized definitions.
  • Frequency of data collection (e.g. quarterly, annually) has a major impact; analytic and reporting burden is proportional to the number of data points collected
  • Producing metric data on a product basis is difficult for products manufactured with complex supply chains and for companies with a large number of products. Historically, these data were not commonly produced or included in periodic product reviews (PPRs)

As the public discussion of quality metrics continues, the importance of value relative to the burden is being increasingly recognized. ISPE is uniquely positioned to facilitate the ongoing dialogue between regulators and industry.

Date Dialogue
Mar 2017 ISPE response to FDA revised draft Guidance, Submission of Quality Metrics Data
Sep 2016 ISPE response to FDA Quality Metrics Technical Conformance Guide
Nov 2015 ISPE response to FDA draft Guidance, Request for Quality Metrics
Aug 2015 ISPE presentation at FDA Quality Metrics Public Meeting
Dec 2013 ISPE whitepaper: ISPE Proposals for FDA Quality Metrics Program

Cultural Excellence

The influence of an organization’s quality culture has emerged as a key enabler from the recent public discourse on measuring performance and quality metrics for delivering outcomes that matter to the patient.

Cultural excellence requires that all employees are passionate about eliminating mistakes by making quality their driving principle.  It goes beyond following the content of the quality policies and procedures by creating an environment that is focused on continuous improvement and learning.

Examining the powerful force that culture exerts on day-to-day operations within organizations, ISPE’s Quality Culture Team has established that although for many the concept of quality culture remains abstract, the behavioral impacts are very real indeed.

ISPE’s Cultural Excellence Team has developed a collection of practical, powerful tools and a comprehensive behavior-based approach for improving quality culture as a means of delivering enhanced quality outcomes.

Published 2017, 100 pages
Download Report