The Role of Humans in the AI Age: GxP Areas

Introduction

Industry has seen AI advancement’s growing effects over the past several years. As more AI use cases have been explored, and more systems have been adopted, people have gained considerable knowledge about how to use this technology in a GxP-compliant way. Stakeholders began discussions to share insights from case studies, which sparked new concepts and ideas, which filtered up to inspire comprehensive industry guidance.

Major milestones on this journey include (see Figure 1):

• ISPE GAMP^® RDI Good Practice Guide: Data Integrity by Design,1 which provided a view of the relevance of data in and its life cycle for use in machine learning (ML)
• ISPE GAMP^® 5 (Second Edition),2 which introduced the idea of an AI/ML subsystem that is managed through the life-cycle phases of concept, project, and operation
• ISPE GAMP^® Guide: Artificial Intelligence,3 which established a holistic, comprehensive best-practice framework for implementing AI in GxP-regulated areas

Along this path, the industry shifted noticeably toward using advanced models. The models offered greater flexibility for handling input and more capabilities for providing output. When integrated into AI-enabled computerized systems, the models offered a more powerful means for automation.

AI use shifted from creating ML models from “scratch” using specific data, to fine-tuning approaches that took advantage of more general model capabilities, to general-purpose models that are instructed to serve a use case (“prompt engineering”), provided relevant context data (“context engineering”). This tendency was reflected in some case studies Global Special Interest Group members have published, including a use case on detection of caries,4 an overview of capabilities of large language models (LLMs),5 and a study on using synthetic avatars for informed consent in clinical trials.6

Although classical ML models still play an important role, the emergence of general-purpose models, typically accessed via natural language interfaces, has lowered the technical barriers for using AI. For example, the technical work of selecting, training, or fine-tuning a convolutional neural network on image data is typically reserved for highly specialized data scientists. However, user-facing interfaces to generative AI and integrating generative AI to accomplish specific tasks offer a chance to explore the technology’s possibilities and limitations with wider audience.

This leads to a heterogeneous environment with a range of AI approaches, from classical task-specific ML models, general-purpose generative AI components, and more advanced architectures such as retrieval-augmented generation and agentic workflows. Retrieval-augmented generation (RAG) is a technique for retrieving data from a knowledge base that might be suitable for answering a given question. This retrieved data is used to instruct a generative AI model to tap this knowledge to yield and answer. Agentic AI relies uses one or more agents to “collaborate” on solving a task. An agent typically has a dedicated role, admissible activities, and tools (such as web search functions). Many platforms offering generative AI capabilities also include web search capabilities, so that they can be seen as a form of agentic AI.

To understand this emerging technology’s possibilities and limitations, we need critical thinking; missing this step is dangerous. As reported during generative AI’s early adoption, for example, a lawyer who relied on AI to prepare for a case got hallucinatory output, which led him to cite fully fabricated cases in court.7 Today, we expect people to handle generative AI output with more caution, though the situation may not have been so clear for the lawyer. He may, for decades, have trusted that the cases he retrieved from registries were in fact registered. Then, he trusted new technology to deliver registered cases with equal reliability. In this light, his error in judgment becomes understandable. However, we have collectively built more understanding on these technologies and can critically reflect on their limitations. We have a new perspective for using AI responsibly.

This insight highlights the need to reconsider humans’ role in the current AI age for GxP areas—exploring what skills are required, how roles may shift, and how to strengthen collaboration to spur success.

Figure 1: Major milestones in establishing guidance on the use of AI in GxP areas.

Image

AI Capability Insights: Replacing or Augmenting Humans?

Human roles differ across AI use cases. Although humans may sometimes control every model output, they may apply less control elsewhere and let AI operate more autonomously. To structure this spectrum, the ISPE GAMP^® Guide: Artificial Intelligence3 has provided a conceptual framework, describing AI maturity and AI subsystem autonomy, refining the AI maturity concept originally presented in.8. According to ISPE GAMP^® Guide: Artificial Intelligence, autonomy can be categorized in five stages (see Figure 2):

• Autonomy stage 1: The AI subsystem runs in parallel to the process and is not used to make decisions. Input and output data of the AI subsystem and its model can be collected.
• Autonomy stage 2: The AI subsystem supports decision-making with an active human decision step for all model outputs.
• Autonomy stage 3: The AI subsystem executes a step or part of a process to make decisions with human user control and correction.
• Autonomy stage 4: The AI subsystem executes a step or part of a process to make decisions that are self-controlled, signaling users to intervene when input data or model output is outside the specified range.
• Autonomy stage 5: The AI subsystem executes a step or part of a process to make decisions and controls and corrects itself.

With this maturity model in mind, the following examples illustrate systems designed with differing degrees of human involvement.

Radiology Image Analysis in Medical Practice (Medical Devices)

ML models can help radiologists assess medical images, letting them work more quickly, find characteristics to design treatments, and monitor images accurately. Such use cases typically pertain to AI subsystem autonomy stage 2, because a radiologist would perform the final assessment of image data and draw conclusions for treatment purposes or health status monitoring.

Case Intake Pharmacovigilance (Good Pharmacovigilance Practices)

Pharmacovigilance processes operate on highly diverse input data, which may include articles from literature, reports from healthcare professionals or semistructured, complex information as obtained from clinical trials. LLMs offer the potential to efficiently and accurately extract information from diverse source data types. Many applications rely on a human verification step—hence, AI subsystem autonomy stage 2. But more advanced automation may be considered for recurring cases once relia-ble and consistent extraction of information has been demonstrated; this may yield an AI subsystem autonomy stage 3 or even stage 4. The primary result is an increase in efficiency, making it easier to manage growing workloads and case intake peaks.

AI-Enabled, Automated Visual Inspection (Good Manufacturing Practices)

Quality control of finished products, particularly for injectables, needs to consider potential product defects such as cracks or scratches in the container, particles in the solution, or out-of-specification filling levels. Though some processes still rely on manual inspection methods, automated options have been implemented for various manufacturing lines. ML may reduce false rejects by better distinguishing true defects from optical phenomena (e.g., stemming from the viscosity of the product). In such highly automated processes, the AI subsystem would be considered autonomy stage 4.

LLM-Supported Validation Activities (Supporting Activities)

Sophisticated, appropriately instructed LLMs can support validation activities such as stablishing requirements and specifications, creating test scripts, and executing tests. (This example is included for illustrative purposes of a prominent use cases, supporting regulated activities, whereas most of the validation artifacts are not directly covered under predicate rules.) Primary goals include raising efficiency of these activities while also aiming for consistent and complete capture of requirements or operationalizing risk-based approaches. With an LLM, draft validation package may be created within minutes—instead of weeks—so humans can simply verify the accuracy of requirements and refine them. AI subsystems in this area would typically be considered autonomy stage 3.

Figure 2: AI subsystem autonomy stage overview.

Image

Changes in human involvement can be substantial when introducing AI. Although raising the efficiency and use of human time is a common element of all use cases, the involvement in the target process and the impact differ, as does the knowledge and expertise required to support the process.

Wider Human Involvement in the AI-Enabled Computerized System Life Cycle

Human involvement is important for operational AI oversight but also factors critically in developing and controlling AI-enabled computerized systems. Controls applied throughout the life cycle help ensure that AI use aligns with ethical standards, doesn’t introduce bias, expands on model results, mitigates risks, and maintains effective and safe use when applying changes.

Ethical considerations aim for subsystems operating in alignment with human values such as autonomy, fairness, or sustainability; see also ISPE GAMP^® Guide: Artificial Intelligence.3 Careful evaluation of decisions is required, particularly when facing competing moral objectives. Contextual and cultural understanding is necessary for effective human oversight with respect to AI ethics.

Human oversight also aids in mitigating biases in algorithms, promoting fair, equitable outcomes. A critical element to avoid-ance bias is data understanding—assessing data’s fitness in the context of use and evaluating implications of potential deficiencies. These implications may also include the choice of human involvement and required capabilities of humans in operation of the system.

Humans can anticipate and mitigate potential risks throughout the system’s life cycle, thus ensuring the system’s robustness and safety during operation. Critical thinking is a key skill for effectively managing risks; fostering effective collaboration is fundamental for organizations to holistically understand risks.

In complex decision-making scenarios, AI methods may exhibit limitations. So, human judgment and contextual understanding with regard to the product, data, and process are essential to supplement technology. Human oversight is also required to fulfill accountability obligations for decisions taken throughout the life cycle, aiming for transparency toward stake-holders and recourse when errors or unintended consequences arise.

Human oversight also applies to change management, aiming for continued relevance, reliability, and system effectiveness. People must carefully consider the implications of changes while ensuring model capabilities and system design decisions remain consistent.

Human oversight, and appropriate controls, can strengthen end user trust and encourage broader adoption of AI technologies within a company. Human oversight also ensures that companies comply with regulatory frameworks and meet internal quality standards. From a life-cycle perspective, the following human involvement activities are relevant and broadly applicable across use cases.

Concept Phase

Humans identify and organize data sources, ensuring they are relevant, reliable, diverse, and fit their domains of use. Human expertise is critical for deriving consistently and accurately labeled data (e.g., for supervised learning methods, being mindful about complexities, accuracy of human judgment, and potential subjectiveness).

Project Phase

Feature engineering is not purely technical; it also relies on human understanding to extract or assess meaningful patterns and relationships from raw data. Expertise from subject matter experts and data scientists is essential for model selection and evaluation; these people choose the appropriate algorithms and hyperparameters to fit the specific problem and desired outcomes. Interpretability and bias mitigation require humans to interpret model outputs, detect biases, and address ethical concerns to prevent discriminatory results.

Operation Phase

Continued monitoring and evaluation are necessary in the life cycle’s operations phase. Humans ensure that AI subsystems perform as expected. They also identify biases, errors, or drifts and refine systems by managing changes.

The Role of Humans in AI: Insights from Use Cases

With this understanding of human oversight in mind, a richer picture of use cases can be developed, going beyond the human control within the process.

Radiology Image Analysis in Medical Practice

In the radiology image analysis case, many stakeholders are involved, all contributing understanding and dedicated expertise to ensure AI is used safely and effectively. Humans contribute particularly in technical areas, such as identifying and assessing data for suitability and gauging annotation quality. Humans also understand data that nuances (e.g., different resolutions, or artifacts stemming from the radiology imaging process) are of high importance. Although statistical means may help identify patterns, it is up to humans to determine how those patterns affect design workflows and spot signs that will help clinicians augment the model’s results.

Human oversight during operation also expands the quality control of practitioners; postmarket surveillance gains more importance to ensure that the model suits its purpose. You can find an example of when such data-supported oversight is required in the European AI Act9 in conjunction with the Medical Device Regulation.10 Although manufacturers must manage human oversight, stakeholders are also involved. Under said regime, notified bodies would verify that processes such as quality management, risk management, and validation are adequate. In these cases, human oversight should go beyond checking boxes; it should question assumptions and ensure that data used to produce performance indicators is of similar importance.

Humans ensure that AI subsystems perform as expected. They also identify biases, errors, or drifts and refine systems by managing changes.

Case Intake Pharmacovigilance

For AI to support case intake in pharmacovigilance, human oversight must go beyond verifying data extracted by the model. Because important signals or severe adverse events may be missed, the effectiveness of human verification and hence the performance of the human–AI team should be monitored. (The term “human–AI team” has been introduced as the guiding seventh principle of Good Machine Learning Practice for medical device development.11)

Cognitive biases need to be considered: Humans are prone to automation or confirmation bias/fatigue, particularly when under stress.12 Thus, monitoring should consider the fraction of changes or added information in operation—although a lower fraction of human actions may mean that AI biases are not caught and in fact are executed, it may also mean a better fit of the model for its intended task. It is also important to reflect on and understand the true meaning of such signals and to initiate action accordingly. This may ultimately lead to advanced training or awareness programs that highlight the limitations of the use of AI or a decision for a higher autonomy stage.

AI-Enabled, Automated Visual Inspection

For AI-enabled automated visual inspection, and its higher autonomy stage, implications on human oversight are two-fold. Human oversight needs to look forward during concept and project phases, to mitigate potential process risks, and take an aggregated view during operation. Understanding the environment and conditions of operation, and thoroughly understanding defect types and how they can manifest in data, requires careful consideration grounded in product and process understanding.

Foresighted planning also includes monitoring strategies during operation (e.g., adopting strategies to detect data drift that could hurt model performance). Companies must ensure that stakeholders understand how critically human oversight affects product quality and patient safety and contributes to decisions. Only then can introducing AI help achieve high-quality results.

LLM-Supported Validation Activities

Our last case of LLM-supported validation activities does not involve direct effects on patient safety, product quality, or data integrity. However, critical oversight and knowledge of limits still matters. Just as orchestra conductors must understand music and direct musicians, AI validation experts must understand all AI requirements, whether from regulations, statutes, or regulatory guidance or from organization-specific user needs.

Although an LLM-based approach may help companies achieve consistency and comprehensive requirements, companies will still need people to consider and communicate about competing objectives. Despite all performance gains, critical reflection and assurance take time—which requires support and understanding from management. Such “backing” is necessary for mitigating automation bias that may prompt more work or introduce regulatory risks.

Key Competencies of the Future Workforce

Although focus areas vary across use cases, human involvement’s effectiveness is grounded in three key competencies: critical thinking, AI literacy, and data understanding (see Figure 3). ISPE GAMP^® 5 (Second Edition)2 states that “critical thinking promotes informed decision-making and good judgment on where and how to apply and scale quality and compliance activities for computerized systems.” It is rooted in the understanding that a “check-the-box” approach (i.e., treating all aspects of a computerized system alike during critical validation activities) can lead to unnecessary work and compromise critical results.

Effective critical thinking requires thorough process and product understanding along with open-mindedness to risk-based approaches. In AI contexts, knowledge and understanding gain new facets, particularly AI literacy and data understanding. As per the ISPE GAMP® Guide: Artificial Intelligence,3 AI literacy includes foundational understanding of how AI methods function, how to activate them effectively, and how they fit within the context of use. Data understanding includes knowledge about data, its origin(s), other nuances, and potential deficiencies.

The “law of the instrument” describes the trap of a narrow view, particularly when new technology is introduced. If you have only a hammer, everything looks like a nail. In our case, the three foundational elements of critical thinking, AI literacy, and data understanding help meaningfully apply AI (i.e., the “hammer”) in a specific context of use and avoid adopting AI just to adopt it.

Establishing critical thinking, AI literacy, and data understanding as foundational competencies will require organization-wide time and effort, leadership support, and strategy. However, benefits can go beyond enabling AI. For instance, data understanding can also help staffs better understand how and where data is captured—which may inspire process improvements, whether AI is used or not.

Figure 3: Three key competencies for successfully implementing AI.

Image

Conclusion

AI is used in industry heterogeneously—from more traditional ML methods to more recent approaches based on generative AI. These technology options and computerized system design choices illustrate the need for a context-specific approach to human involvement throughout the life cycle. In some use cases, ethical considerations may be more pronounced, such as when using AI in medical devices or in a clinical setting, or assessing AI risks and suitability of AI for autonomous designs.

Although the need for humans will remain, their role will evolve over time, shifting from more tactical and/or laborious work to critical oversight and more strategic forward thinking. Although this shift brings opportunity for creativity, people will still need critical skill sets to apply the technology successfully. Critical thinking, AI literacy, and data understanding are necessary across the organization, both to support people in their daily routines and enable communication and collaboration.

A theoretical foundation and a structured approach, as presented in the ISPE GAMP^® Guide: Artificial Intelligence,3 are a solid starting point, but practical experience is equally important. Thus, organizations need to create learning opportunities for their employees, cooperating in cross-functional teams to learn from different perspectives. A strategic program includes raising critical thinking capabilities, AI literacy, and data understanding as those three key enablers.

The ISPE AI Community of Practice and the ISPE GAMP^® Global Software Automation and Artificial Intelligence Special Interest Group (SIG) provide a collaborative environment across organizational boundaries offering opportunities to further advance industry guidance. If you are interested in learning more or contributing to shaping the future of AI in life sciences, both groups are open to new members.

Acknowledgments

The authors would like to thank the ISPE GAMP^® Global Software Automation and Artificial Intelligence SIG for its support in this article’s creation. We would like to also thank Joanne Donald and Stuart Jones for their contributions to this topic during the development of the ISPE GAMP^® Guide: Artificial Intelligence.