Special Reports
September / October 2018

Metrics Reporting and Analysis

Jason Schneider
Metrics Reporting and Analysis - ISPE Pharmaceutical Engineering

An evaluation of the FDA quality metrics portal.

In mid-2017, ISPE established a cross-functional subteam (working title: “Metrics Reporting and Analysis”) under the Advancing Pharmaceutical Quality (APQ) core team to evaluate, summarize, and provide feedback on the FDA quality metrics portal. The results of that exercise are expected to be shared with ISPE members and companies. While this is still the primary charge of the subteam, delays in opening the portal gave the team an opportunity to take on a slightly broader and more thoughtful approach to the evaluation process.

In addition to simply “testing” the portal for the submission of the FDA-requested data, the team also shared approaches and best practices from across the industry regarding local collection, management, and use of data for metrics. These experiences, learnings, challenges, and opportunities were summarized and shared with industry and agency thought leaders during ISPE’s Quality Week, 4–8 June 2018, and are now being considered for inclusion in the broader APQ program. 

TEAM CHARTER

Choosing a well-structured approach to test the portal, the cross-functional subteam was chartered to ensure clarity among team members and alignment with the APQ program. Under this charter, the group went on to establish themselves as a data-reporting, solution-focused subteam tasked to: 

  • Discuss various approaches adopted by industry in preparation for the opening of the portal 
  • Agree upon best practices for consideration and inclusion in ISPE portal testing plans 
  • Evaluate the FDA quality metrics portal, once it is open for testing, through execution of representative case studies and scenarios 
  • Summarize and share testing results, experiences, and learnings with ISPE members

The results of testing and associated lessons learned would then serve as potential input to the broader APQ program. With the team chartered ahead of portal opening, they capitalized on the opportunity to validate expectations by sharing approaches taken by organizations further along the planning and preparation spectrum. 

INFORMATION SHARING

Since several organizations represented within the subteam had already taken significant steps toward preparation to participate in the voluntary phase of the FDA metrics reporting program, the group was able to conduct a number of demonstrations and presentations. These not only showed the overall direction taken by the presenting organizations, they also highlighted key focus areas, including where significant spend was incurred, time was required, and resource commitment and burden was at its highest.

Although these early results reinforced several initial concerns around burden, they also confirmed a number of the benefits provided by a robust internal metrics program for organizations that chose the path of early preparation. With the pending FDA program as the driver, several organizations capitalized on the opportunity to improve their internal metrics, reporting, and analytics initiatives though: 

  • Harmonization: Implementing process improvements in other functions and departments by establishing standard definitions, formulae, and processes 
  • Organization: Normalizing and “mapping” data elements to account for all applicable values across multiple standalone and disparate systems 
  • Consolidation: Create new data repositories, reports/exports, and scorecards for shared data use within the organization 
  • Innovation: Leverage validated data sources as inputs to the data submission to produce a leaner verification/validation process

GUIDING PRINCIPLES

Having gained insights regarding preparation for testing, the team began to establish basic guiding principles by which all testing would be executed. This aligned ISPE team members participating in the early testing phase of the metrics submission program. Because new or updated guidance could be released ahead of the portal opening, it was important to clarify the boundaries for testing governance: 

  • Testing will be bound by the parameters set forth in the FDA “Quality Metrics Technical Conformance Guide”1 and “Submission of Quality Metrics Data”2 guidance as currently written, until/unless feedback from the 2017 review cycle leads to additional changes and requires a process modification. 
  • Sufficient instructions and/or training on the use of the FDA portal will be provided prior to test initiation. 
  • To protect any company-specific data used in testing, no data used in testing would be utilized for any official analysis. 

RECOMMENDATIONS

With basic ground rules and operating parameters established, the team identified several items for FDA consideration with respect to how testing would be conducted. Beyond simply posting or uploading a file of three predefined metrics and associated data elements, the team wanted to explore the possibility of including additional and/or alternative value-added metrics and more extensive testing by being permitted to: 

  • Use “test” or otherwise “blinded” (yet known and traceable by the submitter) data where possible. 
  • Execute both “positive” and “negative” (challenge) testing, including the submission of partial data sets and stress testing the portal
  • Incorporate alternate testing approaches and additional context (e.g., additional metrics, data, charts, visualizations) in the development of testing plans, scenarios, and execution tasks
  • Leverage various IT solutions in data submissions (automation where possible), presuming the output aligns with the current TCG and/or portal training received
  • View the data once submitted (on the FDA side), in order to ensure successful transmission and aligned understanding of the data and what it represents (i.e., trends, patterns, relationships)
  • View summary results, lessons learned, and/or FDA takeaways that result from execution of the broader program (not only results of ISPE participation)
  • Receive details on the confirmation/rejection of data submission, the portal support model, and system release notes and/or change control details to ensure ongoing alignment with site systems and supporting processes

These recommendations not only support a more extensive testing opportunity, but allow for the inclusion of metrics and data elements that are more representative and indicative of what is currently in place across the industry. It was further agreed that a broader testing program scope (with a reduced burden by being allowed to leverage existing data “as is”) would be seen as a very positive step and increase the likelihood of success through increased participation by industry.

LESSONS LEARNED

With the portal not yet open as of the publishing of this article, lessons learned have been focused on and limited to content provided by those organizations that were early adopters and took significant steps toward preparing for the opening of the portal. Key and consistent themes seen throughout the knowledge sharing exercise included: 

  • The concept of “quality beyond compliance” is at the very foundation of a strong metrics program, whether industry or agency
    • While the voluntary FDA program may have been the driver for many, there are numerous internal benefits to be realized by implementing a robust and evolving analytics, reporting, and metrics program
  • The quality data landscape is critical and not to be underestimated 
    • Understand your data sources (including data availability, reliability, and accuracy)
    • Establish standard terminology, definitions, scope, and expectations within your organization to help minimize inconsistencies and discrepancies
    • Harmonize and consolidate data where possible (i.e., data repository, lake, warehouse, etc.) 
    • Ensure reporting continuity and accuracy through lean and robust business processes
  • Start small, with a well-thought-out proof of concept and pilot 
    • Establish and maintain a strong relationship with your data providers, stewards and process owners 
    • Confirm requirements, functionality and usability across a small number of products and locations 
    • Ensure agility, flexibility and scalability to keep pace with both internal and external (agency) process and requirement changes
  • Governance is critical to the success of any metrics program
    • Develop and sustain a clear (enterprise) understanding of the quality data and business process landscape to ensure the ongoing reliability, stability, and sustainability of the program
  • Clarity and final requirements can only improve the program 
    • Removing uncertainty and ambiguity around the benefits of the program will help increase industry participation 
    • Alignment and (testing) standardization will ensure a more successful pilot execution

The Team shared best practices from across the industry regarding local collection management, and use of data for metrics

Expanding on these lessons learned could help industry prepare for a site visit under the new FDA proposals or provide a road map for building a robust metrics reporting and analytics program within individual functions and organizations.