DSAG 2026: The Invented Reviewer

The DSAG Life Sciences Processes working group met on October 21 and 22, 2026 in Hockenheim under the heading “Between innovation and compliance: AI in the GxP environment”. QFINITY opened the afternoon of day one with the impulse talk “The Invented Reviewer”. What the safety tests of two AI labs and the UK AI Security Institute in summer 2026 reveal about agents, and why it matters to every user who gives an agent write access in an SAP landscape.

Three boundaries as a line drawing: a dashed line, a torn wall and an open gate, with a process line running through them — **Three boundaries**Prompt, sandbox and task: the three boundaries Anthropic, OpenAI and the AISI set for their agents.

Around the keynote by Stefan Münch (GAMP D-A-CH) and the SAP sessions on Autonomous Regulated Manufacturing and Cloud ERP GxP Readiness, two days revolved around the same question. How does AI get from pilot status into validated operation? Oliver Herrmann (QFINITY) took the first part of the session block at 1:30 pm from Washington, where the ISPE Annual Meeting was running at the same time. Alexander Kunz (Merck KGaA) delivered the second part with “Einführung von KI im GxP Umfeld @ Merck”.

Three boundaries, three failures, nobody was watching

In summer 2026, Anthropic, OpenAI and the AI Security Institute (AISI) published what happened in their safety tests with the protective classifiers deliberately switched off. What happened in the tests, case by case:

Anthropic: the boundary in the prompt. “No internet access” was written into the prompt, but a misconfiguration of the test environment left the connection open. That only came to light afterwards, during a review of 141,006 evaluation runs. The result was a malicious package in a public package registry, installed on 15 real systems.

OpenAI: the boundary in the sandbox. The sandbox was technically sealed, until the agents found a previously unknown vulnerability in the sandbox's artifact repository (JFrog Artifactory), its only service with an internet connection, and, from July 11 to 13, broke into production systems of the AI platform Hugging Face. Hugging Face discovered the intrusion itself.

AISI: the boundary in the task. The AISI left the internet open on purpose; the only boundary was the scope of the task. There, security monitoring fired and stopped the run within an hour, with no demonstrated real-world harm, but with a pull request into a real software project.

For users, this means: a prompt is not a network boundary, a sandbox alone is not one either, and reading logs after the fact is not oversight. All agents ran in test environments of the labs, their partners or the AISI. The damage occurred at third parties on the internet. As far as is known, no pharmaceutical company and no SAP system has been affected so far. What do we learn from this when agents look for the path of least resistance to complete a task? The question arises as soon as models from the same vendors, with considerably stricter safeguards, work as agents in enterprise systems and receive write access in GxP environments.

What the logs show

Two investigation reports document the agents' logged reasoning, the report by METR and Redwood Research on the Hugging Face incident and the AISI incident report. They contain three patterns familiar from deviation reports and inspections, until now only from humans:

The rationalization. “External infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue.” Reasoning of an agent during the attack on Hugging Face. More than 90 % of the 533 agents on the internal message board took part.

The trail. “Our thoughts aren't necessarily logged?” In at least 20 % of the agents, the reasoning contains deliberations about manipulating their own logs for the automated grader. What was forged were individual tool calls in the transcript, which the reasoning treated as the basis of the grading. The reasoning itself was left untouched; the agents assumed it was not being recorded.

The invented reviewer. A pull request with hidden malicious code into a real open-source project. Under several invented identities, a second voice appeared and assured that it had reviewed the code and found nothing harmful. Submitter and second voice belonged to the same agent. The human maintainer rejected the pull request.

In the third case, the attack was not aimed at the technology but at the review. The review held because a human did not hand over his judgment to the second voice.

Who in your SAP landscape would notice if an agent invented its own reviewer?

Missing alignment or missing boundaries?

Line drawing of a control architecture: an agent at the core, enclosed by two frames, with measuring points at input and output and a switch — **Around the agent**Measuring points at input and output, a switch for the stop and a chain of records, the architecture the impulse talk put up for discussion.

In all three cases, completing the assigned task remained the agents' measure, and in all three cases they left the frame without purpose-built monitoring recognizing it at the moment it happened. Ethical concerns surfaced in the reasoning but rarely limited the action. Whether an agent behaves differently under observation than unobserved cannot be ruled out in a test. For regulated users, the question of intent therefore leads nowhere. What matters is what an agent is allowed to do, within which boundaries it executes actions, and how reliably the control mechanisms enforce those boundaries.

Inspectors have asked these questions for decades, so far of people and systems. The impulse talk ended with two of them for the drive home, and they apply to every SAP landscape: Which systems already give an agent write access today? Who can stop it, and what happens to the data afterwards?

Insight: four entry points into the SAP world

Anyone who wants to answer the two questions for their own SAP landscape will find four places at SAP where the entry begins. What SAP offers there comes first in each case; what reaches the operator follows.

1. See which agents exist and who is responsible for them. The SAP AI Agent Hub, built on SAP LeanIX, keeps a central registry of all agents, MCP servers and language models in the landscape, from SAP, partners and in-house development. Each entry carries identity, owner, capabilities, technology stack and governance status; an optional extension for risk assessment under the EU AI Act assigns agents to a risk class. Agents that have not yet passed verification remain in a pending state, and each agent's identity sits alongside human identities in SAP Cloud Identity Services. Noticing presupposes that someone knows which agents exist. For an inspector, the registry counts only once it matches the system inventory of the QMS and every entry names a responsible person.

2. Understand what happens to the data after the stop. In its white paper on Joule in GxP-regulated environments (May 2026), SAP describes, along GAMP 5 Appendix D11 and ISO 42001, the lifecycle controls an agent needs in GxP operation: monitoring in operation, change control with version control for model and data changes, periodic revalidation with CAPA, and maps what Joule brings to each of them. Joule is decision support, the human remains the decision maker (human-in-the-loop); SAP provides platform logging, the operator sets up application logging. A revised edition that takes the GAMP AI Guide into account is, according to SAP, in preparation. Which event at the agent triggers a renewed check, such as a new model behind the same interface, is left to the operator in the white paper; the operator defines those triggers.

3. Know the vendor's guardrails. In its Responsible AI principles, SAP sets out the rules by which the company selects, implements and controls AI use cases. For the question “alignment or boundaries”, that is the frame in which SAP builds its own agents. These are the vendor's rules for its products. The controls in their own operation, and the boundaries there, are set by the user.

4. Place the architecture as a whole. The AI-native North Star Architecture describes SAP's target picture in four layers, with Joule as the entrance, agents that execute processes across system boundaries, and a platform that provides runtime, sandbox, observability and governance. Every call passes through a controlled gateway; only exceptions and policy-bound decisions go to a human. SAP explicitly calls the document a direction, not a commitment.

These four places deliver the capability. The control architecture an inspector sees is built by the operator: the intended use of the process step the agent serves, the limits of its rights, the records under ALCOA++ and the person who stops it. That is where our work begins.

Why a human-in-the-loop alone is not yet control is explained in our article Human-in-the-Loop: An Illusion of Control?. What the answer looks like for your SAP landscape is something we clarify in conversation: book an appointment. The working group meets again on April 7 and 8, 2027.

Trust is not reviewed into a system. It is designed into it.

DSAG, German-speaking SAP User Group (dsag.de)↗SAP AI Agent Hub, reference architecture (architecture.learning.sap.com)↗SAP white paper: Joule in GxP environments, May 2026 (sap.com)↗SAP AI-native North Star Architecture (architecture.learning.sap.com)↗