Seven Control Layers for LLMs in GMP Decision-Making

This article proposes a GAMP-aligned framework with layered risk control strategies to realize these benefits while balancing the associated risks such as overreliance, hallucinations, and limited explainability.

Background

Since the emergence of LLMs like ChatGPT, the pharmaceutical industry has explored their use for decision support such as brainstorming, writing assistance, or summarization. 1 These early decision support applications rely on human competence, responsibility, and artificial intelligence (AI) literacy to ensure safe use within regulated environments despite risks of overreliance..2^, 3 Recent advances in foundation models, especially multimodal LLMs, now demonstrate the ability to process data in various forms. These include text, images, and audio inputs simultaneously. Coupled with longer task horizons and improved reasoning abilities, this expanded capability shows promise for automating GMP workflows with diverse data types and assessments previously not possible with traditional deterministic systems (identical input produces identical output).4^, 5

These advances place the pharmaceutical industry at a transition point from decision support toward automated decision-making in GMP environments. To ensure compliant and robust systems, such LLM-enabled computerized systems require risk controls tailored to the nondeterministic nature of LLMs where identical inputs may yield different outputs and factual hallucinations are possible. Building on the ISPE GAMP^® Guide: Artificial Intelligence, this article proposes a layered defense-in-depth framework for mitigating those risks.6 It defines seven complementary control layers spanning input and output guardrails, domain knowledge, LLM selection, monitoring and explainability, and transparency. Considerations on each based on intended use and risks can provide a path to compliant automation while maintaining or improving product quality, patient safety, and data integrity within GMP or as inspiration for other GxP areas..7^, 8

Risks of Using Llm for Automated Decision-Making

Regulatory authorities are actively defining expectations to address AI in GMP environments, with particular focus on trustworthiness.9^, 10 Some regulatory authorities advocate risk-based approaches for AI use-case implementation, whereas others are considering prohibiting LLMs in GMP decision-making due to their nondeterministic nature..7^, 9 The US Food and Drug Administration has specifically identified hallucinations as a challenge.11 This necessitates full human oversight for LLM subsystems without controls (see Figure 1).

AI-enabled computerized systems are already used for automated GMP decision-making use cases but depend primarily on traditional deterministic machine learning or deep learning neural networks. This is seen within computer vision applications with known outcome classifications and structured data sources like automatic visual inspection (AVI). 12.

A Defense-in-Depth Approach

The ISPE GAMP^® Guide on AI helps organizations identify and assess AI subsystem risk by assessing maturity through the lens of adaptiveness (static/dynamic models) and autonomy (decision support and making).6 This framework focuses on AI maturity level 3–5 risks related to static systems with high autonomy. To this end, the five risk control strategies from the ISPE GAMP^® AI Guide are considered throughout the layers of the framework. These include elimination, substitution, technical controls, procedural controls, and behavioral controls.6 Additionally, this framework is supported by the concept of defense in depth from information security and LLM development.13 Defense in depth employs multiple protective layers throughout the system’s data flow. By implementing sequential controls at different points, the redundant protective layers compensate for each other’s limitations (see Figure 2).

Workflow for Automated Decision-Making

This proposed layered framework consists of seven control layers. These can work together as a complementary risk mitigation strategy depending on the use case and identified risks. The amount and type of controls in each layer should be risk-based, and not all seven layers are required for every use case. The framework layers follow the workflow from input to output using domain knowledge, the selected LLM, and potential fine-tuning, and subsequent monitoring and explainability.

Operational Benefits

Even though it’s becoming a regulatory expectation, the final implemented LLM-enabled system should equal or reduce false negatives (missed quality issues) compared to previous processes.7 The additional potential benefit is reducing false positives (or unnecessary investigations) that consume quality resources and attention depending on the use case. The implemented system should also aim to equal or reduce assessment variability compared to both LLM systems without controls and traditional human assessments. Other benefits include scalable implementation with centralized control.

Organizations can deploy standardized LLM subsystems across either multiple processes or manufacturing sites and maintain centralized governance over control layers and quality thresholds. However, each new process or site application must assess if it’s within the intended use, including considerations on model language capabilities. This centralized approach promotes scalability and enforces consistent quality standards across geographically distributed operations and may allow for site-specific customizations where appropriate within intended use.

The framework focuses on a single workflow, but larger tasks could be broken into smaller subtasks with more nuanced controls for each subtask workflow, often increasing the performance.14 This could also potentially decrease risk by having more suitable controls for each subtask. The structured monitoring (in layer six) also generates quantifiable performance data that enables systematic improvement of both the LLM-enabled subsystems and the underlying business processes. Finally, these controls transform potentially opaque “black box” systems into more transparent and trustworthy decision-making tools by implementing control techniques commensurate with system risk, regardless of whether the system is used for decision support or decision-making.

Use Cases for Automated Decision-Making in GMP

With the increase in capabilities and a layered risk mitigation, automated decision-making use cases within GMP can include:15

• Documentation review, including batch records and certificates of analysis
• Quality management system activities like subtasks related to deviations, change control, and complaint handling
• Equipment maintenance processes

As an example, complaint handling tasks could range between smaller subtasks like complaint categorization coding to larger and more complex tasks like complaint investigations. These listed areas feature workflows with defined decision pathways and established risk understanding that make automation feasible.

The Seven Layers in Detail

For each layer, this article proposes suggestions for practical implementations using existing and tried techniques inside and outside of GxP. These are kept at a high-level to allow for future developments, especially as explainability remains an evolving field and is without considerations to costs, compute usage, and duration per workflow. This is because they are constantly evolving and use-case dependent. A workflow illustration with all the layers applied is shown in Figure 3.

Layer 1: Input Guardrails

Before the input reaches the model, preprocessing controls could verify input appropriateness. This layer resembles controls of incoming goods ensuring manufacturing materials meet specifications prior to advancing to the next workflow steps. Guardrails is a generic term for detective controls but is used here as a layer to describe enforceable constraints on LLM behavior by analysis of either the input or output.

Input guardrails could cover:

• Input data validation (quality assurance) against predefined acceptance criteria like ensuring valid batch numbers8
• Structured input via templates14
• Implementation of technical control techniques to confine queries to the intended context of use14
• Input analysis using statistical similarity comparing inputs to previous inputs or test data used during validation16
• Anomaly detection for unusual or potentially problematic requests, building upon a robust cybersecurity foundation and becoming particularly important when processing inputs from external sources such as customer complaints

Some use cases may require a more deterministic approach where either the LLM is configured specifically for deterministic responses or with techniques that identify whether an input has been previously processed and, if so, returns the same previous output from an external data source.17^, 18 This ensures that identical inputs consistently produce identical outputs but adds complexity during development and deployment of new system versions as previous outputs or configurations could be based on different model training, context, or other variations.

Layer 2: Domain Knowledge Through Retrieval-Augmented Generation

Domain knowledge can be introduced to the LLM-enabled system by different techniques going from light interventions like prompt engineering and providing examples to retrieval-augmented generation (RAG). The current primary tool for larger corpora of domain knowledge is using RAG. RAG re-trieves relevant information from external knowledge bases based on the input query, then provides this retrieved context to the LLM during output generation.6 This provides control by anchoring responses in verified information. RAG can reduce hallucination risks through context control, though it does not eliminate them entirely.

The RAG implementation should integrate with fit-for-use data sources and could, for example, contain data sources like:

• Data sources in the form of specifications and manufacturing execution system or other manufacturing data (this could also include knowledge graphs)
• Content-related data sources from quality management system data like deviations, changes, and complaints along with internal procedures and regulatory guidance

RAG effectiveness is supported by input guardrails to direct to the correct context of use and ensuring appropriate knowledge bases are used. The database itself should naturally adhere to common data integrity principles including periodic review, version control, and formal change management procedures.8

Layer 3: LLM Selection and Capabilities

Numerous LLM suppliers offer models with varying transparency, performance, and guardrail capabilities. These can include:

• Alignment processes to ensure outputs conform to established ethical guidelines19
• Emerging hallucination reduction methods to improve factual accuracy20
• Adversarial input detection to block potential security exploits (e.g., prompt injection, data poisoning)21
• Transparency of models including architecture, weights, data management practices, system prompts, and change management procedures
• Ways of supporting explainability, like chain of thought (see layer 7)

Organizations should thoroughly document these capabilities and limitations when assessing and justifying an LLM supplier and model for fitness of use. Regulatory authorities increasingly expect documentation regarding cloud providers. This potentially also translates to LLM providers, despite the challenges in acquiring detailed information from proprietary model suppliers.8 This transparency gap represents an evolving compliance consideration for manufacturers.

The evaluation process should follow a structured approach like the general supplier good practices. This is further described in the ISPE GAMP® Guide: Artificial Intelligence.6 Organizations can consider using the guide’s risk control strategy of substitution and elimination when choosing a suitable model. This can respectively support explainability with lesser capable models or avoiding models that have potential security exploits.

Layer 4: LLM Fine-Tuning

Model fine-tuning represents a control layer after LLM supplier and model selection that can increase the probability of outputs consistently meeting predetermined specifications or forms.14^, 22 This enables customization of the model for the specific use case and is available for some open source and proprietary models.

Fine-tuning enables predetermined specifications that are case-specific and to define output structures via schemas/coding like complaint categorization coding or certain language use like deviation conclusions. For some intended uses fine-tuning may be necessary and can include:

• Training with fit-for-use “gold standard” examples representing correct responses
• Fine-tuning language for emphasis on case-specific terminology or regulatory compliance

The development of use-case-specific “gold standard” datasets require careful considerations and subject matter expertise from different disciplines.7^, 23 Different intended uses present varying risk profiles that directly impact model decision-making through decision boundaries and acceptable error rates. Although fine-tuning increases consistency, the data used for fine-tuning should be controlled. This may increase complexity of model life cycle processes.6

Fine-tuning approaches vary by deployment model. This is because open source models can be fine-tuned on-premises with greater control of data. However, some cloud-hosted proprietary models offer fine-tuning capabilities via API where training data is processed by the supplier. The approach selected should be documented as part of the system’s risk assessment, with appropriate controls justified.

Layer 5: Output Guardrails

Multiple output control methods can support output credibility akin to the multiple analytical methods used in quality control. Output guardrails include:

• Verification ensuring consistency of critical information such as batch numbers or drug names16
• Confidence thresholds that prevent low-confidence outputs from reaching operators16
• Confirmation that the output lives up to the predetermined fine-tuned output structure (layer 4)24
• Starting several isolated identical LLMs and technically assessing the differences between the outputs to increase consistency25
• A separate “LLM-as-a-judge” built and fine-tuned for quality control of the primary LLM (for example as a citation checker between output and RAG data sources or a consistency checker that output follows an internal procedure or guideline)14
• Output assessment for performance degradation with technical means (like LLM-as-a-judge) or input similarity scoring16

The LLM-as-a-judge approach is becoming common outside the pharmaceutical industry.26 However, two LLMs can have overlapping vulnerabilities and should only be considered a supplement in GMP processes.

Layer 6: System Monitoring

Operational LLM subsystems should be monitored using appropriate techniques and metrics, as is expected with traditional machine learning (ML) subsystems..7^, 14 At implementation, the system operates with a configuration and performance fit for intended use. As production continues, inputs reflect real-world data and could evolve beyond the initial validation dataset. This is similar to how manufacturing processes expect ongoing monitoring to maintain their validated state. This change can impact model performance through data drift (changes in input distribution).

The system monitoring layer can provide technical, procedural, and behavioral controls to avoid drift by ensuring performance metrics are tracking against validated baseline performance7^, 14 and verifying that considerations on human assessment or intervention depend on the use case. This verification can be accomplished by:

• Automated direction of cases to human operators when model or guardrail confidence falls below acceptable levels16
• Giving both the system and a human assessor the same input/task on a sampling basis to collect information about the human–AI agreement rate (similar to in-process controls to identify improvement opportunities and potential system weaknesses)14
• Human intervention with justification if a system recommendation is to be changed after decision-making due to unforeseen circumstances (e.g., deviations) or if detected by the system monitoring after output guardrails

This monitoring approach complements the static controls implemented during system development.

Layer 7: Explainability and Transparency

System and model explainability (XAI) contribute to addressing the challenge of interpreting model output as AI-enabled system complexity increases.6 Explainability focuses on providing meaningful explanations with explanation accuracy and knowledge limit for specific LLM outputs. Transparency refers to visibility into the system’s overall architecture and operations like model input, output, and decisions. Both qualities are preferable when decisions affect product quality or patient safety. The audit trail enhances LLM subsystem transparency by logging results from each workflow step and control layer, including data and function requests, to other systems. For some models and use cases, explainability can be supported with emerging methods like:

• Chain-of-thought analysis for reasoning models with technical methods like LLM-as-a-judge.27 This analysis could, for example, be relevant for understanding how a complaint investigation was performed.
• Mechanistic interpretability like attribution graphs that trace model pathways to an output28 or probes as potential means to observe overexpression of a concerning concept, like recalls, in complaint handling. 29
• Feature attribution methods like SHapley Additive exPlanations (SHAP) values for technical analysis of input importance and explainability.

Quality performance remains the primary objective, with explainability and traceability approaches providing supporting evidence on how conclusions were reached. These methods can be necessary for some use cases as complexity increases. Explainability features and audit trails enable later reviews of existing controls to verify adequacy and support continual improvement. The approach aligns with GMP requirements and data integrity principles.

Conclusion

The pharmaceutical industry stands at a transition point where LLM can extend beyond decision support toward automated decision-making, leveraging improved capabilities in new use cases. Their nondeterministic nature introduces risks such as overreliance, hallucinations, and limited explainability. However, these can be assessed and mitigated through a layered, GAMP-aligned risk control approach like guardrails, LLM selection, fine-tuning, system domain knowledge, monitoring and explainability. This opens opportunities for pharmaceutical manufacturers to develop LLM-enabled computerized systems for automated decision-making, where appropriate risk controls ensure such systems maintain or improve upon current processes to advance product quality, patient safety, and data integrity.