US Food and Drug Administration (US FDA) on Artificial Intelligence (AI), Critical Thinking, and the Enduring Principles of Quality

Toms began his talk by acknowledging ISPE’s long-standing role in advancing technology and innovation, then framed AI not as a completely unprecedented disruption, but as the latest in a series of changes that have prompted familiar questions about trust, risk, oversight, and responsibility. Toms has seen technology evolve during his more than 20 years in the industry. He noted that each transition has created excitement, but also uncertainty about whether systems perform as intended, whether risks are understood, and whether organizations can maintain confidence in the decisions being made.

The central message is that while technology changes, the responsibilities of regulated organizations do not. AI may be powerful, but it does not reduce the obligation of the organization to understand processes, manage risk, exercise sound scientific judgment, and ensure product quality.

Organizations need to keep understanding the capabilities and limitations of the systems they deploy, evaluate performance against how those systems are actually intended to be used, and maintain oversight for as long as the system is in service, not just at launch.

Products must still be safe, effective, and meet established quality requirements. Processes must still be controlled, risks must still be identified and managed, and decisions must still be scientifically justified. These expectations existed long before AI and will remain after today’s AI tools evolve into something else. Toms emphasized that organizations remain accountable for the systems they deploy and for the outcomes those systems influence.

Where AI Actually Shines

Toms’s presentation acknowledged AI’s significant potential across drug discovery, clinical development, manufacturing, quality systems, supply chain management, pharmacovigilance, and regulatory operations. AI can analyze large volumes of information, identify patterns, summarize data, generate recommendations, and support decision-making. In many cases, the industry’s challenge is not a lack of data, but an excess of it, and the difficulty of converting that data into knowledge and understanding. Helping organizations learn from the enormous amounts of data they already collect is one of AI’s most promising uses, said Toms. The real bottleneck has been turning that data into knowledge.

Used appropriately, AI can strengthen process knowledge, help detect trends, improve monitoring, support knowledge management, and focus attention on higher-risk activities.

Success is not determined by the capabilities of the technology; it is determined by how effective people apply it.

Seneca Toms

National Expert, Drugs

U.S. Food and Drug Administration (FDA)

However, Toms stressed that the key question is not whether AI can create opportunities, but how those opportunities can be realized responsibly. Risk-based thinking and critical thinking remain essential because not every AI use case carries the same level of risk. A tool that summarizes meeting notes does not require the same assurance as a system that supports decisions affecting product quality or patient safety. The appropriate level of oversight, assurance, and scrutiny depends on the role the AI system plays in the decision-making process and the potential consequences if its output is wrong. As the significance of a decision increases, so should the level of scrutiny applied to the system supporting it.

The Risk Isn't the Technology. It's Us.

Toms also cautioned that many AI-related risks are not purely technical; they are human. People have long tended to over-trust systems that appear sophisticated, familiar, or authoritative. This phenomenon, observed in aviation, manufacturing, healthcare, and other fields, can lead to diminished skepticism and insufficient evaluation of system outputs. Toms recalls inspection experiences where people trusted systems simply because they had been in place for years. The issue was not necessarily system failure, but the loss of questioning and critical review. The same risk exists with AI if users accept recommendations because they are convenient or appear credible.

AI Should Sharpen Thinking, Not Replace It

A recurring theme is that AI should augment, not replace, critical thinking. Toms used everyday examples such as GPS directions and autocorrect errors to illustrate that decision-support tools are useful but still require human judgment. In regulated environments, AI may identify a manufacturing trend that would otherwise go unnoticed, but identifying a trend is not the same as understanding its cause, determining its significance, or deciding what action to take. Those steps still require process knowledge, scientific understanding, professional judgment, and accountability. Toms summarizes this principle with a phrase used in a prior ISPE presentation on outsourcing: “You can outsource a lot of things, but you cannot outsource your common sense.”

Oversight Means Understanding—Not Just Approving

In his presentation, Toms also distinguished meaningful oversight from mere review. Approving an output without understanding it, or accepting a recommendation without questioning it, is not oversight. True oversight requires context, evaluation of evidence, consideration of uncertainty, and application of professional judgment. These are enduring features of an effective quality system and remain necessary in an AI-enabled environment. Organizations must continue to evaluate AI performance, understand limitations, and maintain lifecycle oversight rather than assuming confidence in a system is established once and never revisited.

Toms expressed optimism about AI’s future in pharma and life sciences, particularly its potential to improve understanding, strengthen decisions, advance quality, and benefit patients. At the same time, he reminded the audience that the “current” in cGMP reflects an expectation that industry keep pace with advancements in science and technology. Being current does not mean adopting every new tool simply because it exists, but it also does not mean standing still while waiting for perfection. New technologies should be evaluated against the current state, not an impossible standard. Paper records, electronic systems, legacy tools, and humans all have limitations; the relevant question is whether AI can provide meaningful improvement while risks are understood and managed.

The Rules Haven't Changed, Only the Tools Have

The core message that Toms wanted regulated organizations to take away was that AI doesn't lower the bar. It doesn't reduce the obligation to understand processes, manage risk, or exercise sound scientific judgment. Products still need to be safe and effective. Processes still need to be controlled. Risks still need to be identified and managed. Decisions still need to be scientifically justified.

Ultimately, Toms said, AI does not change the fundamental questions the industry must ask; it offers new ways to answer them. Success will not be determined by the sophistication of the technology alone, but by how effectively people apply it. AI may change how information is generated, analyzed, and presented, but it does not change responsibility for ensuring appropriate decisions and safe, effective, high-quality products. Patients will not know what algorithms were used, nor should they need to. They expect products that meet the highest standards. Therefore, as organizations adopt AI, they must preserve sound judgment, risk management, lifecycle oversight, and accountability. The technology may change, but the responsibility to patients does not. AI will continue to evolve into whatever comes next. What's new is the tool, but what remains constant is responsibility. “So, our responsibility continues despite what the technology change is, despite what the system change is; our responsibility continues forever,” he concluded.