Artificial Intelligence (AI) Controls: Not a One-Size-Fits-All, Rather a Choice Among Options

Introduction

With the wider adoption of AI in GxP regulated areas, industry, and regulators seek to streamline their approaches to AI to further secure considerable business benefits while being acutely focussed on patient safety, product quality and data integrity. Often, the experience from implementing the first use cases within a company shows that implementation of AI solutions is more challenging than anticipated. Recurring challenges span organizational aspects such as workforce readiness and AI literacy to technical topics such as data deficiencies and a model’s suitability for a specific context of use.

To translate those first experiences into established, streamlined processes and practice, various questions may arise:

• How can we embed and implement AI-related life cycle activities within existing quality and validation frameworks?
• How can we develop a decision discipline that allows for comprehensive scaling of life cycle activities and defensible risk-based decisions?
• How can we reach consensus among stakeholders, such as regulators, regulated companies and suppliers, on the acceptable levels of control for a specific use case?

Reflecting on these questions, our main thesis is: Controls are context-dependent; methods are options, not answers. A holistic approach leveraging end-to-end life cycle thinking is necessary to address these questions.

This blog post aims to connect the dots of this discussion, from background information to the relation between “what” and “how” and an outlook on future activities.

Evolving Industry Guidance and Developing Regulation on AI

In recent years, various publications have discussed approaches and expectations on implementation of AI in regulated areas. Key publications include:

• US Food and Drug Administration (US FDA) draft guidance – Considerations for the Use of Artificial Intelligence To Support Regulatory Decision-Making for Drug and Biological Products1: In January 2025, US FDA provided draft guidance with an approach to demonstrate AI model credibility at its core, alongside operational maintenance. While conceptual guidance providing a seven-step approach, the guidance also illustrated regulators’ thinking through two use case examples.
• European Medicines Agency (EMA) Reflection paper on the use of Artificial Intelligence (AI) in the medicinal product lifecycle2: The reflection paper provides an overview of typical use cases throughout the medicinal product lifecycle and provides key considerations on technical aspects such as data management as well as governance-related and ethical topics.
• EU GMP Annex 22: Artificial Intelligence draft3: The Annex 22 describes key principles and requirements for the use of AI with a focus on critical use with direct impact to patient safety, product quality and data integrity in GMP environments. The draft also discourages suitability of certain technologies and system designs, such as dynamic systems, probabilistic systems and the use of large language models for critical GMP applications.
• Guiding Principles of Good AI Practice in Drug Development4: The publication provides a collection of guiding principles serving as high-level expression of mutual understanding and harmonization of major authorities regarding good AI practice.
• ISPE GAMP® 5 (Second Edition)5: The GAMP Guide includes an overview of typical life cycle activities in an AI context, and the related use of data, in its Appendix D11. Life cycle activities are organized through typical computerized system phases concept, project and operation.
• ISPE GAMP Guide: Artificial Intelligence6: The Guide provides a holistic framework to support safe and effective use of AI in GxP regulated areas, in alignment with general guidance to achieve computerized systems that are fit for intended use as per ISPE GAMP 5: (Second Edition)5.

Further publications are expected—not only formalizing some of the drafts outlined above but also covering further areas of interest such as the use of generative AI, or more detailed expectations and guidance on the collaboration between regulated users and suppliers.

Sometimes: A Striking Presence of Technical Details

Comparing these publications, the authors have noticed a striking presence of technical details and specific methods coupled with more high-level guidance. For instance,

• Technical methods such as SHAP and LIME are included in the EMA reflection paper, the ISPE GAMP Guide: Artificial Intelligence, and also in the EU GMP Annex 22 draft. They are meant as illustrative examples for methods to support interpretability and explainability. However, other methods like chain of thought to provide insights into the internal reasoning of a Large Language Model are not discussed on a similar level of detail.
• Data management forms a considerable part of AI-related activities, and according guidance. Here, we notice coupling of general requirements such as independence of data used for testing and for development purposes, with specific techniques such as the split into training, validation and test data sets. However, data considerations more relevant with the use of Generative AI such as contextual or supporting data are less commonly discussed.
• Guardrails is an umbrella term covering organizational measures such as validation frameworks to more technical aspects such as data input and model output controls. They are a key point of discussion for the update of Annex 22, as also raised through the Interested Parties exchange between industry and EMA.

Contrasting these observations with other publications such as EMA Annex 11 draft or US FDA 21 CFR Part 11, the “what”, i.e. principles and requirements, and the “how”, i.e. practices, methods and techniques, seem to be less clearly distinguished. These examples illustrate the underlying tension between principle and prescription, and the potential risk of stifling innovation and future patient benefit, while industry and regulators are on a learning journey in a quickly evolving field. Lessons from the past—like the infamous “typewriter rule” in the context of Part 11 which delayed electronic data being considered the original data—show unintended, and in hindsight undesirable consequences when practices become misaligned with the actual regulatory intent.

Industry Needs a Holistic Approach and Interconnected Thinking

The above examples serve to highlight the problem when such technical details are included; current, even draft documents seem to be outpaced by evolving technology; let alone how they may be perceived in final documents with revision cycles spanning years or even decades, rather than months and weeks. Remember that just 2022 or 2023, three to four years ago, industry only became aware of the potential of generative AI and its application on a wider basis, while the discussion had quickly shifted towards agentic AI by 2025.

This leads to key questions: What actually motivates the inclusion of such technical details?

Our view: Many stakeholders share a perception of being overwhelmed by the speed of change related to AI. With technology options and control concepts evolving fast, it may be tempting to try hunting for a catch-all solution: A simple answer to tough design questions, aiming for a checklist approach rather than critical thinking.

However, from our point of view, the need for the assessment of various options and application of good judgment to make decisions amongst the evident ambiguity are inherent to the use of AI in critical or complex regulated environments. The weaknesses of checklist-oriented approaches are exacerbated by the sheer number of options, and the speed of innovation.

Furthermore, even though some of the concepts listed above may often prove useful for risk control purposes, they are ambiguous in themselves. While they may work for some use cases, in others they may be irrelevant or even misleading and may only provide limited to no control or assurance, as also stated in the ISPE GAMP Guide: Artificial Intelligence6. One example may be SHAP values that are not tied to interpretable features, but rather technical characteristics. We may anticipate and evaluate their suitability ex ante, still their effectiveness needs to be demonstrated for the context of use.

Grounding in Product and Process Understanding, AI Literacy and Data Understanding

Product and process understanding is one of the five key concepts as per ISPE GAMP 5: (Second Edition)5 as a basis to determine computerized system validation approaches. This was emphasized in the ISPE GAMP Guide: Artificial Intelligence6, alongside AI literacy and data understanding:

• Product and process understanding is the basis for making science- and risk-based decisions to ensure that the system is fit for its intended use.5
• AI literacy includes foundational understanding of how AI methods function, the considerations required to activate such innovation effectively, and their implications within the context of use.6
• Data understanding includes knowledge about data, its origin, nuances and potential deficiencies (based on6).

Together, they support quality risk management approaches, with their relation to knowledge management and understanding highlighted as per ICH Q9 (R1)8 generally to computerized systems. Such understanding and literacy helps to derive a comprehensive and effective control strategy linked to identified risks. This control strategy may take advantage not only of a single method, but a fabric of control options that together are adequate for the context of use. Such fabric also accounts for human abilities for oversight purposes, system and AI sub-system complexity, and data deficiencies. An example for such thinking in the context of generative AI is provided in the article, “Seven Control Layers for Large Language Models (LLMs) in GMP Decision-Making”7. For instance, controls can also include raising and maintaining awareness for human biases; for instance, occasionally operators may be confronted with test instances that exhibit flaws, while related data would be separated from the actual operational data. Such thinking does not only support the choice of appropriate methods, but also other facets of critical thinking in an AI context, regarding where and how to apply and scale quality and compliance activities.

A holistic strategy also covers a line of sight across life cycle phases. In an AI context, the suitability and effectiveness of such methods not only need to be demonstrated during initial verification efforts, but also become part of the ongoing monitoring strategy, considering both technical performance and also the perception of humans involved for oversight purposes. Change management and change controls also apply to auxiliary methods such as for explainability purpose or in the case of guardrails.

This emphasizes the fact that there is no single method to stay on the safe side of the use of AI—rather, control is an ongoing life cycle concern. Such life cycle management should also consider advances in the state-of-the-art, as guardrail approaches, just like the underlying AI models themselves, may undergo rapid innovation and be subject to evolving trends in the future.

Still Work to Do: An Outlook

Recognizing the need for a more holistic view on the interplay of AI capabilities, its deficiencies, control options, and life cycle management is only the first step towards enablement of critical thinking and effective scaling of life cycle activities. A certain level of AI literacy, conceptual blueprints, and collected experience of use case implementations—possibly with a variety of control strategies—are required to build consensus across the stakeholder ecosystem. In this regard, we welcome EMA’s approach seeking deeper dialogue with industry with respect to the Annex 22 draft through collaboration with Interested Parties, going beyond regulatory commenting.

Here, the authors encourage regulators to make use of various instruments to accommodate needs of industry: For instance, principles and requirements fit well the logic of Annex 22, while more detailed technical discussions may be reserved for more flexible Q&A publications. For practitioners, this means shifting focus from selecting the “right” method to designing the right combination of controls for the specific context of use.

ISPE is committed to contributing in order to establish such consensus, through:

• Contributions to the recent EMA Interested Parties exchange in conjunction to the Annex 22 draft revisions
• Ongoing dialogue with regulators as further guidance is developed
• Continuation of conceptual work to support consensus across organizational boundaries, such as between regulated users
• Fostering discussions at conferences such as the AI in life sciences summit

ISPE Communities of Practice (CoPs), like the AI CoP with its various subcommittees and the GAMP CoP, including its Global Special Interest Group Software Automation and Artificial Intelligence, are open for further contributions.

ISPE members: View ISPE Communities of Practice.

Not an ISPE member? Join today.

Disclaimer

iSpeak blog posts provide an opportunity for the dissemination of ideas and opinions on topics impacting the pharmaceutical industry. Ideas and opinions expressed in iSpeak blog posts are those of the author(s) and publication thereof does not imply endorsement by ISPE.