Human‑in‑the‑Loop as an Illusion of Control?

Keywords: Human-in-the-Loop, Automation Bias, Review Fatigue, Computerized System Validation, GxP, Annex 11, EU AI Act, AI Governance

1. Trust as a Regulatory Key Resource – and Why Governance Creates It

In addition to traditional systems, AI is increasingly used in GxP environments to generate, assess, and pre-structure critical artifacts, including validation documentation, test evaluations, traceability analyses, and deviation assessments. These systems can significantly improve efficiency and consistency. At the same time, they introduce a fundamental challenge: outputs are often probabilistic, not fully transparent, and difficult to verify; yet humans are expected to review and verify them. This is referred to as HITL, a risk-based control concept whereby qualified personnel are explicitly integrated into automated or AI-supported processes to review, approve, or override outcomes, ensuring compliance, data integrity, and accountability in GxP systems.

For the purpose of this article, the terms AI-enabled and AI-supported are used with distinct meanings. AI-enabled refers to technological systems, computer systems, software, applications, or tools that incorporate AI functionality. AI-supported refers to GxP processes, computerized system contexts, including the associated process, workflows, reviews, tasks, activities, outputs, or artifact preparation that are supported or influenced by AI-enabled systems. Accordingly, the term AI-supported GxP processes is used to describe GxP processes in which AI-enabled systems generate, assess, recommend, monitor, or otherwise influence information, artifacts, decisions, workflows, or process outcomes.

In regulated environments, this creates a central requirement: decisions must remain explainable, traceable, and demonstrably controlled. In essence, the systems must create trustworthy output. Trust, therefore, is not an abstract concept but an operational necessity. It determines whether system outputs are accepted, and whether decisions can be based on them, and ultimately whether compliance and product quality can be ensured.

Regulatory frameworks explicitly address this challenge.

Regulation (EU) 2024/1689 establishes human oversight as a core requirement for high-risk AI systems, emphasizing the need for effective human intervention capabilities. In parallel, EU GMP Annex 11 requires clearly defined responsibilities, risk-based lifecycle management, validation, and data integrity controls for computerized systems. The emerging EU GMP framework is also moving towards explicit expectations for human oversight of AI-supported GxP processes. While still in draft status, the proposed EU GMP Annex 22 (“Artificial Intelligence”) identifies human oversight as a key control mechanism and establishes expectations for intended use, model governance, performance monitoring, change management, and ongoing human supervision of AI systems used in GMP-regulated environments. In parallel, the ongoing revision of Annex 11 (“Computerised Systems”) reinforces lifecycle management, risk-based controls, data integrity, and governance requirements for computerized systems. Together, these draft updates provide a clear indication of the regulatory direction of travel and highlight the increasing importance of effective HITL arrangements as a component of GxP compliance for AI-enabled systems.

Recent regulatory enforcement further illustrates the practical implications. In a 2026 warning letter, the US Food and Drug Administration (US FDA) identified critical deficiencies where AI agents were used to generate GMP-relevant records without sufficient verification and control mechanisms (US FDA, 2026; Warning Letter). The findings highlight a key regulatory expectation: the use of AI does not reduce accountability; it increases the need for demonstrable control over outputs.

Within this context, HITL is frequently positioned as the primary safeguard. Human review, approval, or override is expected to ensure that system-generated, AI-generated outputs are accurate, complete, and compliant. This model aligns well with established GxP principles of responsibility and oversight and is therefore widely adopted in practice.

However, this assumption requires careful examination.

HITL is often treated as an inherently reliable control mechanism, but its effectiveness depends on how humans actually behave under operational conditions. Especially in AI-supported processes, the human role shifts from active creation and analysis to monitoring and verification of pre-structured outputs. This shift fundamentally changes the cognitive conditions under which decisions are made.

Under these conditions, human oversight may not function as an effective detection control. Instead, it may become a formalized step that creates accountability without reliably ensuring error detection. The key question is therefore not whether HITL is present, but whether it is capable of providing effective and demonstrable control in the specific context in which it is applied.

This article addresses this question from a GxP and CSV perspective. It examines the psychological and operational mechanisms that can limit the effectiveness of human oversight in AI supported GxP processes and analyzes their implications for typical CSV use cases. Based on this analysis, it derives a multi-layered control approach that integrates human oversight with technical and organizational safeguards to ensure robust and auditable control in system- or AI-supported GxP environments.

2. Why Human Oversight in AI-Supported GxP Processes Is Often Overestimated

In practice, AI enabled systems are frequently implemented to support regulated processes by preparing artifacts, proposing decisions, or pre-structuring reviews, while humans perform a final review and approval. This pattern is attractive because it visibly assigns responsibility and aligns well with audit narratives. At the same time, it creates working conditions that are especially vulnerable from a human-factors perspective: humans shift from active problem solvers to passive monitors.

Research on the “use, misuse (overreliance), disuse (underutilization), and abuse (inappropriate application)” of automation describes precisely this role shift and its risks. It has been known for decades that, as automation appears increasingly reliable, vigilance and monitoring quality decline, and automation errors are less likely to be detected (Parasuraman and Riley, 1997).

For GxP and CSV, this defines the core tension: humans are expected to safeguard probabilistic, potentially flawed AI outputs, yet they are simultaneously placed in a cognitively unfavorable role. Even though one might think that the probabilistic nature of AI outputs would make humans more vigilant and engaged in the review process, there is currently no strong empirical evidence that probabilistic AI outputs inherently lead to more effective human oversight than deterministic outputs. Human review performance appears to be driven primarily by trust calibration, perceived reliability, workload, uncertainty communication, and workflow design. While explicit uncertainty information may promote greater reviewer engagement, the probabilistic nature of the underlying model alone should not be assumed to increase vigilance (Parasuraman, R. and Manzey, D.H. (2010); Beck et al., (2025); Romeo and Conti 2025).

HITL is therefore not automatically an effective detection control. Depending on its design, it may become an illusion of control: formally present but functionally fragile. The ability to detect errors and issues must be designed into the overall process into the HITL activities and tasks to ensure they are consistently reliable and value-adding.

3. Psychological Mechanisms Limiting the Effectiveness of HITL

3.1 Automation Bias and Over-Reliance

Automation bias describes the tendency to prefer recommendations from automated systems over one’s own judgment, even when contradictory information is available. A recent interdisciplinary review frames automation bias as the interaction of trust calibration, cognitive relief, expertise, and task design, with mechanisms such as reduced verification and declining user engagement (Romeo and Conti, 2025). The effect is well established in safety-critical domains and is explicitly highlighted in recent analyses as a fundamental limitation of HITL approaches (Kahn, Probasco, and Kinoshita, 2024).

A related but distinct phenomenon is large language model (LLM) sycophancy, which refers to the tendency of a large language model to affirm, reinforce, or align with a user’s stated beliefs, assumptions, or preferences, even when those assumptions may be incomplete, unsupported, or incorrect. Unlike automation bias and over-reliance, which are human factors, sycophancy is a characteristic of the AI system’s behavior. Automation bias arises when users defer to automated recommendations instead of applying independent judgment, while over-reliance describes the resulting behavioral dependence on the system. In contrast, sycophancy originates from the model’s tendency to provide agreeable or confirmatory responses. LLM sycophancy is a behavior specific to large language models and other conversational generative AI systems, where the model tends to affirm or reinforce a user’s beliefs, assumptions, or preferences. In contrast, automation bias and over-reliance are human factors that can occur when interacting with any computerized or automated system, including rule-based software, decision-support systems, expert systems, machine learning models, and LLMs.

Although conceptually different, LLM sycophancy can act as a contributing factor that increases the likelihood of automation bias and over-reliance. By presenting information in a highly affirmative, confident, or user-aligned manner, the model may reduce the user’s motivation to challenge, verify, or critically assess the output. This can reinforce trust beyond what is warranted, promote reduced vigilance, and increase the probability that users accept AI-generated recommendations without adequate scrutiny.

3.2 Cognitive Ease and Plausibility Traps

AI outputs, e.g., from an LLM, are often linguistically fluent, formally consistent, and visually convincing. These characteristics generate cognitive ease: information that is easy to process feels more plausible. In review situations, this can lead reviewers to stop actively evaluating content and instead unconsciously seek confirmation. Experimental evidence from human–AI review settings shows that reviewers correct AI outputs less frequently when correction requires additional effort, leading to increased acceptance of incorrect suggestions (Beck et al., 2025).

The risk is further amplified by the volume of content that generative AI systems can produce. Large outputs increase the effort required for comprehensive review and verification, encouraging reviewers to rely on surface plausibility, selective checking, or acceptance of conclusions without fully evaluating the underlying reasoning. As review workload increases, vigilance may decline, creating conditions that further promote automation bias and over-reliance.

3.3 Attitude and Expectation Effects

For CSV, it is particularly relevant that individual attitudes towards AI significantly influence review performance. In the cited large-scale study, pre-existing attitudes to automation were a strong predictor of error detection and correction behavior (Beck et al., 2025). Formal qualification alone does not ensure consistent control effectiveness if psychological dispositions and contextual factors are not addressed.

3.4 Review Fatigue and Automation Complacency

Review fatigue typically arises when humans are asked to assess large volumes of homogeneous, predominantly correct outputs. In medicine, the related phenomenon of automation complacency is discussed as a risk of clinical decision-support systems: over time, vigilance and critical distance erode, increasing the likelihood of uncritical acceptance (Saadeh et al., 2025). GxP workflows in which AI pre-reviews large volumes of data or artifacts create structurally comparable conditions.

3.5 Trust Calibration: The Risks of Under-Trust and Algorithm Aversion

While this article primarily focuses on the risks of automation bias, over-reliance, and complacency, the opposite failure mode should also be considered. Human–AI collaboration can be impaired by under-trust, algorithm aversion, and unnecessary overrides, where users reject or modify valid AI-generated outputs despite evidence that the system performs at least as well as, or better than, human judgment. Excessive skepticism may lead reviewers to spend disproportionate effort re-performing analyses, disregard valuable system-generated insights, or introduce new errors through unnecessary intervention. Consequently, effective HITL design is not simply a matter of reducing trust in AI, but of achieving appropriate trust calibration. The goal is neither unquestioning acceptance nor reflexive rejection of AI outputs, but a risk-based level of confidence that reflects the demonstrated capabilities, limitations, and uncertainty of the AI-enabled system, and the intended use of the process it supports. Organizations should therefore design controls, training, and review procedures that address both over-reliance and under-reliance, recognizing that either extreme can reduce process effectiveness and impair decision quality.

3.6 The Human–AI Loop: Feedback and Drift Risks

Oversight and review workflows also shape the data basis and, consequently, system behavior. Not only the model, but also documentation and decision practices may adapt to the expected logic of the AI. This effect is described in practice-oriented literature as the “human–AI loop.” Although often lacking peer-reviewed backing, it represents a plausible organizational risk for drift and data distortion that should not be ignored (Xenakis, 2025).

3.7 HITL and the gradual delegation of Responsibility to Systems

Beyond individual cognitive effects, HITL concepts are embedded in organizational routines to attribute responsibility and accountability. But often, organizations are implicitly shifting responsibility to systems. These non-intentional delegations subtly redefine what is considered “normal” or sufficient oversight, even though formal accountability structures may remain unchanged.

4. Evidence and Lessons from Aviation, Healthcare, and the Military

Aviation is widely regarded as a reference domain for human-factors research because highly automated systems have been used there for decades. Parasuraman and Riley (1997) show that misuse and monitoring failures become more likely as automation reliability increases. In military and civilian security contexts, a recent case-study analysis (including aviation and air-defense systems) demonstrates that HITL does not suffice on its own to prevent accidents or erroneous decisions and must therefore be combined with technical, organizational, and training measures (Kahn, Probasco, and Kinoshita, 2024).

Similar effects are observed in healthcare. While decision-support systems can enhance clinical decisions, they also risk undermining human judgment through over-trust and fatigue of the clinician. Saadeh et al. (2025) explicitly frame these risks as automation complacency and derive requirements for “mindful integration” that preserves vigilance rather than eroding it. In this context, “mindful integration” refers to the deliberate incorporation of AI decision-support systems into processes and practices so that they save cognitive effort without replacing human judgment, vigilance, accountability, or the decision-making role.

5. CSV Use Cases: Where Psychological Failure Modes Become Operational

5.1 AI-supported Test Review: From Active Test Analysis to Passive Approval

In an AI-supported test review, AI systems could analyze templates, test results, or deviations for completeness, consistency, or anomalies. Humans are expected to assess relevance and determine whether evidence is fit for purpose. Psychologically, this setting is risky because AI typically delivers structured, plausibly reasoned outputs. The validation expert’s role shifts from active reconstruction of test outcomes to confirmation of machine suggestions, a pattern identified in human-factors literature as prone to misuse and monitoring failures (Parasuraman and Riley, 1997). If correcting or overriding the AI requires more effort than accepting its output, correction rates demonstrably decline (Beck et al., 2025).

Short example: An AI marks an operational qualification test result as “OK” based on formal compliance with acceptance criteria. Cognitive ease arises: everything appears formally correct, and the reviewer approves. An implicit assumption, such as the representativeness of the dataset for the production configuration, remains unchallenged. The potential error is not technically complex, but cognitively silent.

An additional emerging risk is the growing practice of using one LLM to generate test cases and another LLM to review or verify them. While this may create the appearance of independent verification, both systems may share similar patterns of reasoning and assumptions, increasing the likelihood that weaknesses remain undetected. In practice, LLM-generated tests often degenerate into “smoke tests” that demonstrate successful execution or expected process flow but contain few substantive assertions capable of detecting incorrect behavior. As a result, the review process may confirm that a test ran successfully without establishing that the underlying requirement, business rule, or control objective was actually verified. Human reviewers may be particularly susceptible to approving such tests because they appear comprehensive, structured, and technically plausible.

5.2 Traceability Agents: Formal Completeness vs. Substantive Correctness

Traceability agents could analyze user requirement specifications (URS), functional requirement specifications, system design specifications, test artifacts, and trace matrices to identify gaps, inconsistencies, and missing links. They are AI systems that perform automated semantic and structural analysis of lifecycle documents to infer, evaluate, and visualize relationships between requirements and downstream artifacts. Unlike traditional traceability matrices that rely on manually maintained links, traceability agents attempt to assess completeness and consistency across large collections of documentation by combining pattern matching, semantic analysis, and contextual reasoning. These systems may generate outputs such as heatmaps, traffic-light indicators, or scores, such as a traceability completeness score (95 percent of requirements linked to downstream artifacts), that have strong plausibility effects. Automation bias is particularly likely here: a visually “clean” artifact is misinterpreted as evidence of substantive correctness. However, CSV relevance lies not merely in the existence of links, but in their necessity and justification. Medical decision support literature warns that seemingly unambiguous recommendations can obscure contextual information, leading to decision substitution rather than decision support (Saadeh et al., 2025).

Short example: A traceability agent links an audit trail URS requirement to a test case and marks traceability as complete because a formal reference exists and the test mentions the presence of an audit trail function. However, the test itself only verifies existence, not whether the required audit trail attributes (e.g., completeness, immutability, attribution) are fulfilled. This situation may arise either because the URS is not specified at a sufficiently granular level or because the agent is not designed to assess the semantic adequacy of test coverage but instead relies on keyword or structural matching (e.g., detecting “audit trail”). As a result, traceability appears complete from a formal perspective, while substantive verification remains unaddressed. Without a targeted review strategy that explicitly focuses on the adequacy and depth of testing, a human reviewer may overlook this mismatch, particularly if system indicators suggest a “green” status.

5.3 A Shared Pattern: Throughput Increases, Visible Error Rates Decline – Vigilance Declines

As AI-enabled system maturity increases, visible error rates may decline, while review volumes often rise. Importantly, this reduction in observed errors does not necessarily reflect a corresponding reduction in underlying risk, but rather a reduction in detectability. Reviews become increasingly monotonous, vigilance declines, and review fatigue becomes likely.

This dynamic constitutes a central argument against treating HITL as a standalone control in GxP decision-making use cases. However, the effect depends on the relative rates of improvement of the AI-enabled system, review process adaptation, reviewer training, workload, and control design.

At the same time, this relationship is not linear and should not be interpreted as evidence that declining visible error rates are inherently problematic. Improvements in model quality, domain specialization, retrieval mechanisms, and validation controls may genuinely reduce errors and improve the detection of defects that human reviewers would otherwise miss. In some cases, declining error rates may therefore reflect improved system performance rather than reduced detectability. Moreover, advanced AI systems may identify inconsistencies, omissions, and traceability gaps at a scale and speed that would be impractical for manual review.

6. Why “More Review” Is Not a Robust Solution

A common reaction to these risks is to increase the number of review stages. Empirical evidence suggests that this approach is unreliable. In controlled experiments, correction activity declined as soon as corrections became more effortful (Beck et al., 2025). Automation-bias research further highlights that explainability and transparency mechanisms may increase acceptance but do not necessarily improve decision/review quality. Effective interventions must actively promote engagement and verification (Romeo and Conti, 2025).

Increasing review layers without protecting cognitive autonomy risks turning oversight into ritual compliance — formally compliant, but substantively hollow.

7. Solution Approaches: HITL as Part of a Multi-Layered Control System (People–Process–Technology)

HITL is meant to detect issues and create responsibility and accountability. But detectability must be intentionally designed through architecture, controls, and governance mechanisms that remain effective even when human vigilance degrades.

Robust solutions combine human oversight with technical and organizational controls that operate independently of human vigilance. In GxP AI governance, guardrails, documented policies and procedures, and operational monitoring are recommended core elements to ensure that machine learning models continue to behave as intended after deployment (Mintanciyan et al., 2024, ISPE GAMP AI Guide, 2025). Guardrails are technical and procedural constraints embedded within the AI execution framework that govern permissible inputs, outputs, actions, escalation paths, and review triggers.

Practical implementation approaches for AI-enabled systems in regulated environments should use risk-based control bundles (severity, probability, detectability) linked to monitoring, audit trails, and change/release mechanisms (de Malliard et al., 2025).

Technical supplementary controls may include rule-based guardrails for critical GxP attributes, independent consistency checks, statistical anomaly detection, separate second-level reviews (“AI checks AI”), multi-run and multi-model disagreement checking (e.g., repeated execution with varying model parameters such as temperature, comparison of outputs across different models, or ensemble-based consensus approaches), as well as drift and data quality monitoring. To be effective and auditable, such controls should be implemented within the agent’s operational harness—including the system prompts, orchestration logic, tool-calling rules, and execution constraints that govern how the model is invoked and how outputs are processed—rather than as external add-on controls. Their configuration, execution, outcomes, and exceptions should be observable, logged, and subject to review to demonstrate and verify their effectiveness throughout the system lifecycle. Industry discussions on AI validation emphasize that AI-supported quality-control mechanisms are most effective when embedded in a risk-based validation and governance framework and complemented by subject matter expert oversight (Blanke, 2025), a concept that is also expressed throughout the ISPE GAMP Guides.

8. HITL Evaluation Matrix

The following table can be used as input for structured risk assessments to determine necessary additional controls to make the HITL a robust control. This operationalizes the psychological risk drivers described in the text (automation bias, cognitive ease, review fatigue) as evaluation criteria and links them to required supplementary controls.

Table 1: AI output character and typical psychological risk density.

It should be noted that Psychological Risk Density and output certainty are independent dimensions. Psychological Risk Density is the degree to which the characteristics of an (AI-) generated output (e.g., fluency, structure, completeness, confidence, visual presentation, certainty labels, or apparent authority) increase the likelihood of cognitive biases such as automation bias, over-reliance, complacency, confirmation bias, or reduced verification effort during human review. In this context, “certainty” refers to the degree to which an output can be objectively derived from, reproduced, and independently verified against source evidence.

In addition, the application of certainty-based triage, in which AI-generated outputs are classified according to the degree to which they can be objectively derived from and independently verified against underlying evidence. The primary objective is not to determine whether an output is correct, but to guide the allocation of human review effort toward activities where uncertainty, interpretation, and expert judgment play the greatest role.

At one end of the spectrum are high-certainty outputs. These are typically deterministic results that are directly derived from source data and can be independently verified by the AI-enabled system itself. Examples include calculations, completeness checks, reconciliation activities, traceability statistics, identification of missing references, or consistency checks against predefined rules. Because the outcome can be reproduced and objectively verified, the AI-enabled system can report such outputs with a comparatively high degree of certainty. These outputs may justify a lighter HITL review approach, allowing reviewers to focus their attention on areas of greater uncertainty.

At the opposite end of the spectrum are low-certainty outputs, which involve interpretation, contextual reasoning, semantic adequacy assessments, risk evaluations, or conclusions that cannot be derived directly from underlying data. Examples include assessing whether a test case provides adequate evidence for a requirement, determining the significance of a deviation, evaluating procedural compliance, or judging whether a requirement has been sufficiently implemented. Such outputs depend on assumptions, context, and expert interpretation, and therefore require greater human scrutiny. In these cases, HITL review intensity should increase because the AI is producing conclusions rather than verifiable facts.

Importantly, certainty-based triage and psychological risk are separate concepts. The psychological risk density associated with an output is not necessarily correlated with its objective certainty. In some cases, outputs that are deterministic, reproducible, and independently verifiable may create particularly strong automation bias because they are explicitly classified as high certainty. Human reviewers may unconsciously interpret the certainty label as evidence of correctness and reduce their verification effort accordingly. As a result, the likelihood of over-trust and insufficient scrutiny may increase even when the technical uncertainty is low.

Consequently, certainty classifications should be used to prioritize review activities rather than eliminate critical review. High-certainty outputs may justify reduced review scope from a process perspective, but they should still be subject to periodic verification, challenge testing, and oversight to ensure that the assumptions, calculations, and controls underlying the certainty assessment remain valid. Conversely, low-certainty outputs should trigger intensified HITL activities because they rely on inference, judgment, and contextual interpretation.

A further consideration is that the degree of structure in an AI-enabled system output affects not only certainty classification but also the efficiency and effectiveness of human review. Highly structured outputs—such as tabular summaries, traceability matrices, exception lists, discrepancy reports, or categorized findings—are generally easier for reviewers to interpret and compare against source evidence than large volumes of unstructured narrative text. By condensing information and presenting it in a consistent format, structured outputs can reduce cognitive load and improve the visibility of anomalies, missing elements, inconsistencies, and outliers. As a result, structured presentation may enhance a reviewer’s ability to verify outputs and detect unexpected patterns. However, this benefit should not be confused with evidence of correctness. A well-structured and visually coherent representation can increase plausibility and user trust, potentially reinforcing automation bias if reviewers equate presentation quality with substantive validity. Consequently, structured outputs should be viewed as a means of improving review efficiency and anomaly detection, while maintaining the expectation that reviewers critically assess the underlying evidence and assumptions on which the output is based.

Depending on the output character, the criticality of the output in the GxP-process context, and the psychological risk density, the following risk drivers should be evaluated for HITL-tasks or activities:

Table 2: HITL risk drivers, typical failure modes, and required additional controls.

Depending on the nature and criticality of the process/system, additional controls may include systematic “AI-checking-AI” mechanisms, where independent or adversarial AI models review outputs to detect inconsistencies, bias, or unexpected behavior. The effectiveness of “AI-checks-AI” depends on independence, diversity of models, adversarial design, and disagreement mechanisms. Controlled challenge datasets or seeded test cases in non-production environments can be used deliberately to measure detection rates and assess the robustness of both the primary AI model and the oversight mechanisms. For critical areas, it may be required to establish appropriate controls to measure the continuous effectiveness of the HITL activities and tasks, in addition to the performance of the computerized system, which supports the process.

An example:

In pharmacovigilance, individual case safety reports (ICSRs) capture adverse events reported for a medicinal product. Traditionally, a specialist reviews source documents, extracts the relevant information, enters it into a database, and assesses whether the case meets regulatory reporting criteria. In AI-supported processes, the extraction and structuring step is automated: the system reads source material and produces a fully completed, submission-ready case. The human reviewer then performs the final check and approval.

At first glance, this seems robust because “a human signs off.” However, this situation poses a high risk for HITL control. The AI output is already structured, plausible, and, in most cases, correct, thereby subtly shifting the human role from an active assessor to a passive monitor. Over time, reviewers tend to trust the system more and more (automation bias), accept plausible outputs without deeper analysis (cognitive ease), and, especially under high workload, approve cases more mechanically (review fatigue). Controls to address automation bias may require active verification of critical fields, such as seriousness or outcome, rather than allowing passive acceptance. To prevent cognitive ease, the process may enforce justification to ensure that reviewers not only confirm a classification but also explain it, ideally while being exposed to the original narrative and uncertainty indicators. Review fatigue could be addressed by moving away from full, uniform review towards risk-based triggers, controlling workload, or introducing targeted audits.

9. Practical Recommendations for Implementation

A pragmatic implementation begins with functional differentiation of HITL roles: reviewing, deciding, or correcting. The more passive the role, the more susceptible it is to automation bias and cognitive ease. Where HITL serves as a control, it should therefore be linked to explicit decision questions (e.g., documented justification for acceptance or rejection).

Second, review volumes per person should be limited, as fatigue is a systemic effect. Risk-based triggers (boundary cases, anomalies, and behavioral changes in the system) help focus human attention on situations where it genuinely enhances detectability. To ensure that such situations are consistently identified, these risk-based triggers should be encoded directly into the AI-enabled system’s guardrails and operational logic as explicit instructions, escalation criteria, and review conditions. Rather than relying on ad hoc human recognition, the system should be designed to actively detect predefined risk patterns and route them for enhanced human review.

Organizations should also consider the opposite failure mode of automation bias. Under-trust, algorithm aversion, and unnecessary overrides can reduce process effectiveness when reviewers repeatedly reject or rework correct AI-supported outputs. HITL design should therefore aim for trust calibration rather than trust reduction, ensuring that review effort is proportionate to the uncertainty, criticality, and demonstrated performance of the AI-enabled system.

Third, in decision-proximate GxP use cases, where AI outputs directly influence or substantially shape GxP-relevant decisions, HITL should never stand alone: it should be supplemented by independent technical controls that remain effective even when vigilance declines.

Fourth, user interface design is part of compliance: traffic-light indicators and scores can reinforce automation bias, whereas uncertainty, data gaps and alternative hypotheses should be made visible.

Finally, psychological risk drivers should be considered and documented in CSV risk assessments of all systems, rather than assuming detectability by default.

In regulated environments, the central question is not whether humans should remain in the loop. It is how system- or AI-supported GxP environments must be designed so that human judgment remains meaningful in practice.

Oversight must be auditable and actively supported by the system. Mere formal or symbolic human involvement is insufficient.

References

Beck, J., S. Eckman, C. Kern, and F. Kreuter. 2025. “Bias in the Loop: How Humans Evaluate AI-Generated Suggestions.” arXiv preprint arXiv:2509.08514.

Blanke, M. 2025. “AI Validation in Pharma: Maintaining Compliance and Trust.” EY Switzerland, October 21, 2025.

de Malliard, P., B. Xavier, C. Smith, I. Sutcliffe, and K. Ambrosini. 2025. “A Guide to Building AI Agents in GxP Environments.” AWS Machine Learning Blog, November 11, 2025.

European Commission. 2011. EudraLex Volume 4 GMP: Annex 11, Computerised Systems (Revision 1). Brussels: European Commission.

European Commission. 2025. Draft EudraLex Volume 4 GMP: Annex 22, Artificial Intelligence. Public Consultation Version. Brussels: European Commission.

European Commission. 2025. Draft Revision of EudraLex Volume 4 GMP: Annex 11, Computerised Systems. Public Consultation Version. Brussels: European Commission.

International Society for Pharmaceutical Engineering (ISPE). 2025. ISPE GAMP® Guide: Artificial Intelligence. Tampa, FL: ISPE.

Kahn, L., E. Probasco, and R. Kinoshita. 2024. AI Safety and Automation Bias: The Downside of Human-in-the-Loop. Washington, DC: Center for Security and Emerging Technology, Georgetown University.

Mintanciyan, A., R. Budihandojo, J. English, O. Lopez, J. E. Matos, and R. McDowall. 2024. “Artificial Intelligence Governance in GxP Environments.” Pharmaceutical Engineering, July/August 2024.

Parasuraman, R., and V. Riley. 1997. “Humans and Automation: Use, Misuse, Disuse, Abuse.” Human Factors 39, no. 2: 230–53.

Parasuraman, R., and D. H. Manzey. 2010. “Complacency and Bias in Human Use of Automation: An Attentional Integration.” Human Factors 52, no. 3: 381–410.

Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act). 2024. Official Journal of the European Union, L 2024/1689.

Romeo, G., and D. Conti. 2025. “Exploring Automation Bias in Human-AI Collaboration: A Review and Implications for Explainable AI.” AI & Society 41 (2026): 259–278.

Saadeh, M. I., J. Janhonen, E. Beer, C. Castelyn, and D. N. Hoffman. 2025. “Automation Complacency: Risks of Abdicating Medical Decision Making.” AI and Ethics 5: 5783–5793.

US Food and Drug Administration. 2026. “Warning Letter: Purolea Cosmetics Lab, Reference No. 722591.” April 2, 2026.

Xenakis, Alan. 2025. “Signature Article Series #2: The Human-AI Loop, Why Cognitive Bias Still Rules the Algorithm.” LinkedIn, November 26, 2025.