How to Use AI Agents in a GMP Lab Without Putting the LLM in the Critical Path

You can use AI agents in a GMP-regulated lab today, but not by letting the model touch the product. Draft EU GMP Annex 22 covers only static, frozen, deterministic models for critical applications, and states that generative AI and large language models "should not be used in critical GMP applications" - the objection being models that produce different outputs from identical inputs. The way through is architectural, not regulatory: put the LLM where it proposes and never disposes, and make the layer that actually moves the instrument deterministic, qualifiable, and boring.

This post is about software architecture, not regulatory advice. We are instrument-software engineers, not a regulatory consultancy - we cite what the drafts say and we do not interpret them for your submission. What we can tell you is how to build a system that does not put you in front of that question in the first place.

Key Takeaways

• The regulator's objection is non-determinism, not AI. Annex 22 admits static models "where all parameters have been finally set" and that do "not adapt their performance during use." An LLM fails that test by construction.
• The draft does not cover generative AI at all - it states that "the document does not apply to Generative AI and Large Language Models (LLM)" - and says they should not be used in critical GMP applications. For non-critical use, "qualified and trained personnel should be responsible for ensuring outputs are suitable for the intended use."
• This is already being enforced. In April 2026, FDA issued its first AI-related warning letter, citing 21 CFR 211.22(c) after AI agents produced CGMP specifications, procedures and records without Quality Unit review.
• The architecture answer: the agent's output is a document, not an action. A deterministic validator and a deterministic executor sit between the proposal and the instrument. Those two are what you qualify.
• This is not a workaround, it is better engineering. The same split gives you reproducible execution, a real audit trail, and dry-run capability you want whether or not you are regulated.
• The clock is real. Consultation closed 7 October 2025, no final text exists as of August 2026, and EMA is targeting Q4 2026 to deliver final text to the European Commission.

What Draft Annex 22 Actually Says

Annex 22 is a supplementary annex to the EU GMP Guide, published for consultation alongside a PIC/S counterpart. Consultation closed on 7 October 2025. As of August 2026 no final text has been adopted, and EMA is targeting Q4 2026 to deliver one to the European Commission.

Its scope is narrow and deliberate. It applies to AI in critical GMP applications - those with direct impact on patient safety, product quality or data integrity - and within that, to models that are static and deterministic: all parameters finally set, no adaptation during use.

What it excludes matters more than what it includes:

• Dynamic models that "continuously and automatically learn and adapt performance during use" should not be used in critical GMP applications.
• Probabilistic-output models - those that may return different outputs for identical inputs.
• Generative AI and LLMs, which the draft states it does not apply to, and which should not be used in critical applications.

For non-critical applications, generative AI and LLMs are not banned. The expectation is that "qualified and trained personnel should be responsible for ensuring outputs are suitable for the intended use" - a documented point at which a human reviews and can intervene.

The rest of the draft is recognisably GMP: models under change control before deployment, explainability via feature attribution (SHAP, LIME) reviewed as part of test-result approval, confidence scores logged where applicable with low-confidence outcomes considered "undecided" rather than guessed, defined metrics for monitoring drift in input data against the original training envelope, and a four-eyes separation so that whoever curates training data is not the person certifying the test results.

There is movement, too. An EMA workshop in June and July 2026 signalled stakeholder support for enabling generative AI under a risk-based approach built on ICH Q9(R1) principles. The direction of travel is toward permitting more, under conditions. It has not arrived.

Verdict: the regulator's objection is not to artificial intelligence. It is to a component whose behaviour you cannot pin down, sitting somewhere that affects the product.

The First FDA AI Warning Letter Made It Concrete

In April 2026 the FDA issued what is widely reported as its first warning letter citing AI misuse as a GMP violation, to Purolea Cosmetic Lab. AI agents had been used to produce drug specifications, procedures and master production and control records, and those documents went into use without human review.

FDA did not write a new AI rule. It applied 21 CFR 211.22(c) - the existing requirement that the Quality Unit review and approve such records - and stated that if AI is used to support CGMP activities, all results or recommendations from an AI agent must be reviewed and approved by an authorised representative of the Quality Unit. (Pharmaceutical Technology analysis)

Two things are worth noticing. The enforcement action was not about an autonomous laboratory or a robot arm - it was about AI output entering a controlled record without a human between. And the remedy is a checkpoint, not a ban.

That is the whole design brief, handed over by an enforcement action: there must be a defined point where the model's output is a proposal, and a defined act by which it becomes authorised.

Verdict: regulators are not asking whether you used a model. They are asking what stood between the model and the record.

The Architecture: The Agent Proposes, A Deterministic Layer Disposes

Here is the shape that follows. It is not a compliance trick - it is the architecture you would want anyway, and we have written about each piece separately without ever naming the regulatory reason it matters.

The diagram shows four layers and one boundary. Above the line, the agent interprets a request and emits a proposed plan. Nothing has moved; the output is a document. Below the line, everything is deterministic and qualifiable: a validator checks the proposal against the instrument's declared limits, a human authorises it where the application is critical, and an executor expands the approved plan into instrument commands by fixed rules. Every step writes to an audit trail.

Layer 1 - Intent, probabilistic, outside the critical path. The LLM reads what the scientist asked for and proposes a structured plan. It has no ability to actuate anything. Its output is data.

Layer 2 - Validation, deterministic. A rule-based validator checks the proposed plan against the instrument's declared capabilities and physical limits - volumes, capacities, tip state, reachability, labware compatibility. Same input, same verdict, every time. This is the component you qualify, and it is ordinary software: no model, no sampling, no temperature.

Layer 3 - Authorisation. For critical applications, the validated plan is presented for approval before it executes. This is the checkpoint the Purolea letter is about, and it is where the "qualified and trained personnel responsible for the output" expectation is discharged.

Layer 4 - Execution, deterministic. Intent-level tools expand the approved plan into the instrument's own commands. The expansion is code. The same approved plan produces the same command sequence every time, which is precisely the property the annex asks for.

Underneath all four sits the record: model version, prompt, proposed plan, validator verdict, approver identity, instrument response. That is an ALCOA+ shaped audit trail, and you get it because the plan was an artifact before it was an action.

Verdict: the LLM is not made compliant. It is made irrelevant to the part that has to be.

Why This Is Better Engineering Regardless

The uncomfortable question is whether this is regulatory theatre that slows everything down. It is not, and we would build it this way in an unregulated lab.

You can dry-run. Because the plan exists as data before it executes, you can run it against a digital twin first and see what it would do. The twin's state is the ground truth, it resets deterministically, and a plan that fails validation there costs nothing. That is a qualification tool and a productivity tool at the same time.

You can produce evidence instead of opinions. A scenario corpus run repeatedly against the deterministic layers gives you reliability numbers rather than assurances. The metric that matters is not "did it work once" but did it work on every repeat - and in published agent benchmarks that distinction is brutal: τ-bench reports state-of-the-art function-calling agents succeeding on under 50% of tasks, with pass^8 below 25% in the retail domain.

You constrain the failure that actually happens. The dominant agent failure in current research is silent: an agent confidently reporting completion on a task it did not complete, which accounts for 45-48% of failures on τ²-bench single-control domains and 75.8% of self-assessing coding-agent trajectories on AppWorld. A deterministic validator between proposal and actuation does not care how confidently the model phrased its summary.

And it is where the industry is stuck anyway. Gartner's 2026 CIO survey puts deployed-agent adoption at 17%, and roughly 60% of organisational leaders name legacy system integration as their primary AI challenge, with 35% calling it the single biggest barrier to scaling - the stated reason being that existing systems were designed for human operators rather than autonomous agents. The layer described here is exactly that missing piece, and building it is not optional whether or not a regulator is asking.

Verdict: the split that satisfies the annex is the same split that makes agentic lab automation work at all.

Where an LLM Is Still Genuinely Useful

None of this argues for less AI. It argues for putting it where its strengths are not liabilities:

• Interpreting the request. Turning "set up a dilution series across the plate at these concentrations" into a structured, checkable plan is exactly what a language model is good at, and getting it wrong costs nothing because the validator is next.
• Reading the documents. SOPs, method files, condition tables. This is language work, and it is non-critical until something acts on it.
• Explaining a failure. When a run fails, summarising what happened from the audit trail is high-value and touches nothing.
• Drafting, for a human to approve. Which is permitted, and which the Purolea letter tells you exactly how to handle: a named person reviews and approves before it enters a controlled record.

What an LLM should not be is the thing that decides a volume and sends it to a pipette with nothing in between.

What To Do If You Are Buying Or Building Right Now

1. Draw your critical path. Write down every component between a user request and a physical action or a controlled record. Anything on that line has to be deterministic and qualifiable. Most teams have never drawn this and are surprised by what is on it.
2. Check whether your vendor's agent can propose without executing. If the model calls the instrument directly, there is no boundary, and you cannot place a checkpoint where the FDA has just said one belongs.
3. Ask what the validation layer actually validates. "The model is trained on lab data" is not an answer. Limits enforced before actuation is an answer.
4. Ask for reliability evidence, not a demo. Pass rate over repeated runs on your workflows, not a recorded happy path.
5. Start non-critical. Document interpretation, run summarisation, method drafting with human approval. Real value, no critical-path exposure, and it builds the audit trail you will want later.
6. Do not wait for the final text to fix your architecture. The deterministic-execution split is required by good engineering regardless, and retrofitting it after you have built on direct model-to-instrument calls is expensive.

Frequently Asked Questions

Can you use AI agents in a GMP-regulated lab?

Yes, provided the model is not in the critical path. Draft Annex 22 covers only static deterministic models for critical GMP applications and states that generative AI and LLMs should not be used there, while permitting them in non-critical applications with qualified personnel responsible for the outputs. An architecture where the agent proposes a plan and a deterministic layer validates and executes it keeps the model on the permitted side of that line.

What does draft Annex 22 say about LLMs?

That the document does not apply to generative AI and large language models, and that they should not be used in critical GMP applications. The underlying objection is non-determinism - models that can return different outputs for identical inputs, alongside models that continuously learn and adapt during use. For non-critical applications the expectation is qualified, trained personnel responsible for ensuring outputs are suitable for their intended use.

Is Annex 22 in force?

Not as of August 2026. The consultation closed on 7 October 2025 and no final text has been adopted; EMA is targeting Q4 2026 to deliver final text to the European Commission, and an EMA workshop in mid-2026 signalled possible movement toward a risk-based opening for generative AI. Treat it as direction of travel rather than settled law, and note that the FDA has already enforced the same accountability principle under existing CGMP rules.

What did the FDA's first AI warning letter actually require?

It required that where AI supports CGMP activities, all results or recommendations from an AI agent be reviewed and approved by an authorised representative of the Quality Unit before use. FDA applied the existing 21 CFR 211.22(c) rather than any AI-specific regulation, after AI-generated specifications, procedures and production records were used without human review.

Does keeping the LLM out of the critical path make the system less capable?

No, it changes where the intelligence sits. The model still interprets the request, reads the documents and explains failures - the work language models are genuinely good at. What it stops doing is deciding a physical parameter and actuating on it unchecked, which is also the failure mode the research literature identifies as the most common and most dangerous.

---

We build the deterministic layer. Our SiLA 2 and MCP tooling is open source, and the validate-then-execute split described here is how our own instrument software is built - including the digital twin you dry-run against and the evaluation harness that produces the reliability evidence. Point us at one instrument and one workflow, and we will draw your critical path with you and tell you what is on it. Get in touch.

This article describes software architecture. It is not regulatory advice, and QPillars is not a regulatory consultancy - verify any compliance interpretation with your own quality function.

---

Written by Iacob Marian, Technical Lead and Co-founder at QPillars, where he builds the infrastructure that lets AI agents operate laboratory instruments safely and reliably. Published 2026-08-10.