Conceptualizing a Streamlined Performance Assessment – The St.Gallen APQ Benchmarking Tool

The University of St. Gallen recently launched the Global Quality Benchmarking 2020 study (Pharmastudy) in collaboration with Dun & Bradstreet which is funded by the US FDA. It strives to analyze quality management maturity and operational data from approximately 2000 manufacturing establishments in 52 countries. The concise, user-friendly benchmarking module in this study was derived from the original St. Gallen Operational Excellence Benchmarking. All selected Performance Indicators and Enablers have been proven as meaningful for surrogating overall plant stability and performance best. In total, there are 13 Key Performance Indicators (KPIs) in four dimensions, 18 Maturity Questions in three categories and five Contextual Factors go into the study. Developing this set of measures required extensive research, primarily using St. Gallen’s databases containing operational performance data from more than 380 manufacturing sites and around 100 quality control labs. These databases were built over the last 15 years and contain the outcomes of the full St. Gallen OPEX benchmarking programs¹. See figure 1 for a summary of the research problem.

In a first steps a pre-selection of potential candidates has been derived based on the legacy research in the field of quality metrics published in three comprehensive reports². Furthermore, experience from supporting numerous pharmaceutical companies in developing and enhancing their internal metrics systems over the last years have been taken into account. Figure 2 shows the pre-selection of performance indicators derived.

The available data sets gathered in the full St.Gallen benchmarking programs were further used to analyze the contribution of all pre-selected metrics to overall performance. Therefor statistical exploration—such as correlation and regression analyses or t-tests—led to a subset of 13 metrics surrogating overall performance best. All of the 13 seleted metrics contribute to assessing performance in a balanced way.

Additional validation comprising the direct comparison of an abbreviated overall performance score calculated based on the chosen measures only, and the full performance score used in the legacy benchmarking provided confidence from a system perspective. Figure 4 shows the aggregated outcome of the analytics: The newly derived performance scroe aggregating the 13 identified metrics to be used in the assessment positively correlates with the full OPEX benchmark score calculated using legacy benchmarking data. A degree of determination around 39 % provides confidence that the selected set of metrics is able to meaningfully surrogate overall performance in pharmaceutical production.

The questionnaire used in the Global Quality Benchmarking study is completed with 18 enabler questions linked to quality management maturity. Those have been derived along the three categories Organization, Social/Employee and Technical/Tools together with the U.S. Food and Drug Administration. Thus, in addition to measuring the current outcome performance the result report provides insights about the implementation level of practices associated with striving for continouly improving performance. Furthermore, a set of contextual factors allows to consider production site’s specific characteristics when interpreting benchmarking results.

ISPE’s APQ coreteam and the University of St.Gallen agreed to enrich the above described study module with an ICH Q10 maturity assessment providing a high-level evaluation of the current level of maturity for each of the ICH Q10 Pharmaceutical Quality System elements. This comprises a newly derived benchmarking tool tailored to ISPE’s needs. It will serve as pre- and postassessment of the APQ Assess, Aspire, Act and Advance Framework and can be conducted in the form of an external Benchmarking against a tailored industry peer group from the Global Quality study or as internal self-evaluation. Figure 5 summarizes the content of the St.Gallen APQ Benchmarking tool.

The ability to offer a scientifically conceptualized baselining and to measure the impact of the deep-dive modules on quality and delivery performance adds significant value to the APQ process. We are delighted to support industry on its journey to advanced pharmaceutical quality and look forward to your engagement. Please contact Tami Frederick, Chair ISPE APQ team, or Marten Ritz, University of St.Gallen, to conduct the pre-assessment as part of your APQ journey. We are happy to support in case of any questions.