AI agents in scientific workflows

Q1 · Landscape: where agents sit in the scientific workflow

Answer. Of 83 cataloged systems, about half are L2 tool-using agents; closed-loop (L3) agents cluster in self-driving labs and at facility instruments; end-to-end (L4) autonomy appears mainly where the "experiment" is code, existing data or a proof; and independent verification lags [1, 3, 24].

Taxonomy: stage × autonomy

Here an agent is a large language model (LLM) that calls tools (search, code, simulations, instrument controls) and picks its next step from the results. We sorted 83 systems by main workflow stage and by autonomy: 58 general-science systems (mostly 2024–2026) and 25 facility systems covered in Q4. The general set includes national-lab tools such as LLaMP (LBNL) and ChemGraph (Argonne) [25, 26].

Autonomy levels. L1 Assistant: answers or retrieves on request. L2 Tool-using agent: plans and runs multi-step tool calls within one task. L3 Closed-loop: iterates plan → act → observe against real instruments, robots, simulations or code over many cycles, with human checkpoints. L4 End-to-end: goes from a goal through experiments or analysis to a written paper with minimal human help. Stages: literature & ideation; hypothesis generation; experiment design & self-driving labs (labs where software chooses and runs the next experiment); data analysis & simulation; and writing & review (of papers and proposals). Each system sits under its main stage.

Reading the table:

• L2 is the norm (43 of 83). Literature & ideation is almost all L2 (12 of 14): retrieve-read-cite agents such as PaperQA2, OpenScholar and Ai2's Asta [6, 27, 28]. They hand a cited answer to a person, so there is no loop to close.
• Most closed loops run in labs. Experiment design & self-driving labs is the largest stage (26) and holds 14 of the 21 L3 systems, such as Coscientist (Carnegie Mellon, 2023; not Google's Co-Scientist) and MARS [29, 30]. Half of this stage (13 of 26) and 6 of the L3 systems are facility systems, such as SLAC's AI X-ray scientist, which aligned a crystal on an SSRL beamline while a human relayed its commands [12].
• No experiment-stage system is L4. In wet labs, humans still run or approve experiments [5, 31]. The 10 L4 systems, none of them at a facility, sit where the "experiment" is code, existing data or a proof: machine-learning papers (The AI Scientist), data-driven reports (Kosmos, data-to-paper) and mathematics (Aletheia) [1, 2, 3, 32]. Two are facility crystallography agents, such as ORNL's NeuDiff Agent [15].
• L1 is rare (9). Six of the 9 are writing & review tools, most of which inform rather than replace human authors and reviewers [33, 34].

Six trends, 2024 → 2026

1. From single tools to multi-agent "AI scientists." Systems from 2023–24 wrapped one LLM around domain tools; ChemCrow used 18 chemistry tools [35]. By 2025–26 the usual design is a team of role-specialized agents that critique each other: Co-Scientist runs a generate-critique-refine tournament [4], Virtual Lab has a "PI" agent directing specialists [31], and Kosmos (a preprint) links about 200 agent rollouts per run through a shared "world model", a common record of findings [2].
2. Flagship claims reached top journals. In 2026 Co-Scientist, Robin, The AI Scientist and OpenScholar appeared in Nature, and Biomni in Science [1, 4, 5, 6, 7].
3. Companies and frontier labs entered. Google and Google DeepMind cover hypotheses, algorithms and mathematics (Co-Scientist, AlphaEvolve, Aletheia) [3, 4, 36]. FutureHouse spun out Edison Scientific to commercialize its AI Scientist; Kosmos is built from Edison Scientific agents [2, 37]. Intology says a paper written by its Zochi agent was accepted at the ACL 2025 main conference [38].
4. Autonomy is highest where checking is cheap. Strong L3 and L4 results appear where an automatic check exists (a benchmark score, code evaluator or proof check). A white paper reports that AlphaEvolve found a way to multiply 4×4 complex matrices with 48 scalar multiplications [36]; DeepScientist tested about 1,100 ideas in a month [39]. At Agents4Science 2025, a conference for papers with AI first authors, humans contributed more to hypotheses and design, and accepted papers had more human guidance [40].
5. Evaluation lags capability (the "verification gap"). Independent checks are rare and mixed. An academic lab, co-authoring with Google, found that Co-Scientist's top hypothesis matched its own unpublished result [41]; an independent analysis disputed A-Lab's novelty claims [42]. LLM ideas that experts rated more novel than human ideas lost that edge once carried out [43, 44]. About one in five Kosmos statements was not judged accurate [2]. A 2026 survey (preprint) found that 38% of 24 runnable systems report any novelty verification [24]; AstaBench concludes that AI remains far from solving research assistance [28].
6. AI enters peer review; targeted human oversight beats full autonomy. GPT-4 feedback on papers overlapped with human reviews about as much as two reviewers overlap [33]. In a randomized ICLR 2025 trial, 27% of reviewers given LLM feedback updated their reviews [34]. The AI Scientist's authors warn of "taxing overwhelmed review systems" [1]. A preprint finds that targeted human interventions in AutoResearchClaw beat both full autonomy and step-by-step oversight [45].

What is new in 2026

Besides the journal papers and the peer-review trial above, preprints report that Aletheia wrote one paper with no human intervention, solved four open Erdős problems autonomously, and solved 6 of 10 FirstProof problems [3, 46]. MARS pushed closed-loop autonomy in robotic materials labs [30], and OpenAI's Prism put an LLM inside LaTeX manuscript editing [47]. Critiques sharpened: a position paper argues that current agents are co-scientists, not built for autonomous discovery [48]. In the US, the November 2025 Genesis Mission order directs DOE to build a platform that includes AI agents and autonomous experimentation [23].

Caveats. Autonomy labels are our judgment from each paper's own description. Vendor and company claims (Zochi, Prism, Edison Scientific) are not independently verified. Several headline numbers (Kosmos, AlphaEvolve, Aletheia) come from preprints or white papers.

Q2 · What works: evidence versus claims, and what benchmarks show

Answer. Peer-reviewed studies show real but narrow agent successes in labs and literature. Top models now score 95.8% on graduate science questions, above expert level, but open-ended research and XRD analysis stay weak, and no public benchmark tests closed-loop x-ray or neutron beamtime [4, 9, 49].

Q3 · Infrastructure: models, frameworks, protocols and the path to HPC

Answer. Science agents are built from interchangeable parts: a vendor or center-hosted model, a general agent framework, the Model Context Protocol (MCP) for tools, and Globus Compute or Parsl for HPC. A facility's main job is therefore to expose the services it already runs (Bluesky/EPICS, Tiled, IRI APIs) as guarded, logged tools [17, 19, 22, 89].

Models

Frontier models such as GPT-5 [90], called through vendor APIs, are still the default engine; Co-Scientist, for example, is built on Gemini, and Biomni's setup expects Claude [4, 91]. But frameworks can usually swap them: ChemGraph accepts OpenAI, Anthropic, Google, Argonne's Argo gateway or a local Ollama server [89].

Open-weight models, whose weights can be downloaded and run locally, make self-hosting practical. gpt-oss-120b is Apache-2.0 licensed and tuned for tool use [92]. OLCF's inference service offers it [18]. Science-trained models also compete: ether0, a 24B chemistry reasoning model trained by reinforcement learning on 640,730 problems, beat frontier models and experts on molecular design [93].

Where the model runs is often a data-handling decision. ALCF built FIRST for private, secure inference; it lets researchers generate "billions of tokens daily on-premises" without commercial cloud [17]. DOE's National Nuclear Security Administration (NNSA) runs frontier AI models on the classified network of its Venado supercomputer at Los Alamos [94].

Agent frameworks and science toolkits

Frameworks typically build on ReAct, the loop in which the model reasons, calls a tool (acts), reads the result (observes) and repeats [95]. General orchestrators include LangGraph [96]; AutoGen is now in maintenance mode, with new users pointed to Microsoft Agent Framework [97]. Vendor SDKs package the same loop; the Claude Agent SDK, for example, adds hooks, subagents and permission controls [98].

A science toolkit is usually a thin layer over one of these orchestrators plus a registry of domain tools. ChemGraph combines LangGraph with ASE, RDKit and MCP, and runs jobs through Parsl or Globus Compute on Aurora and Polaris [89]. Academy runs stateful, asynchronous agents across HPC, experimental facilities and data repositories [99]. The value is in the tools, not the loop.

Tool and agent protocols

The Model Context Protocol (MCP) is an open standard that lets an agent call tools and read data through one common interface. A2A (Agent2Agent) links agents to each other; its maintainers call MCP the "vertical" agent-to-tool layer and A2A the "horizontal" agent-to-agent layer [100]. Both now sit in the Linux Foundation's Agentic AI Foundation: MCP since 9 Dec 2025, A2A (v1.0) since Aug 2026 [16, 100].

MCP is still changing. Its 2026-07-28 revision made the protocol stateless, moved long-running "tasks" into an official extension, and deprecated Roots, Sampling and Logging [101]. A lighter option is a "skill": a folder with a SKILL.md instruction file that the agent loads only when needed [102]. ALS staff write skills for the Lightfall beamline-control platform, and at the SNS one skill document lets a Slack bot run EQSANS reductions [103, 104].

The tool layer is also an attack surface. In tool poisoning, hidden instructions in a tool's description redirect the agent. MCPTox tested 45 live MCP servers: o1-mini's attack success rate was 72.8%, and no model refused more than 3% of attacks [105]. On HPC systems, a "hijacked authorized agent" acts with a user's valid credentials [106].

Connecting to workflows, HPC and instruments

One request passes through six layers, mostly existing DOE software:

1. Model. A vendor API or a center-hosted open model. ALCF (FIRST, with Globus Auth) and OLCF (S3M tokens) serve models with vLLM, an open-source serving engine, behind OpenAI-compatible endpoints. These accept OpenAI-style requests, so code switches by changing an address and key [17, 18].
2. Agent framework. An agent built with LangGraph, a vendor SDK or Academy reasons and picks the next tool [95, 99].
3. MCP server or skill. The call reaches a thin wrapper around an existing service, such as Globus Labs' MCP servers for Globus Transfer, Compute and Search and for facility status APIs [19].
4. Execution fabric. Globus Compute (formerly funcX) or Parsl runs the work on HPC [107, 108]. On Aurora, a gpt-oss-120b agent swarm had a planner split work among executors that shared one Parsl-backed MCP server [109]. The Integrated Research Infrastructure (IRI) Facility API standardizes status, account, compute/jobs, filesystem, storage and task endpoints, with instances at NERSC, ALCF and ESnet [110]; SLAC hosts a fork [111].
5. Instrument and data. Bluesky and its Ophyd device layer, EPICS, and the Tiled data service are the plug points [112, 113, 114]. At SSRL, an agent with MCP tools wrote SPEC commands, relayed unmodified by a human, to align a single crystal [12]. At NSLS-II, VISION ran a voice-controlled scattering experiment by sending generated code to Bluesky [115]. ORNL agents steered an additive-manufacturing workflow linking OLCF with a simulated version of its Manufacturing Demonstration Facility [116].
6. Provenance. Prompts, responses and decisions are logged with the workflow record, as PROV-AGENT does by extending W3C PROV [117].

Sandboxing and permissions. Biomni runs LLM-generated code with full system privileges by default [91]. NERSC recommends sandboxes and workspace-write mode (the agent may change files only inside its working directory) for coding agents, and holds users responsible for agent actions [118]. Facility projects put the limits in code, not prompts. APEXA's deterministic guard allowed 0/200 adversarial motor violations, against 15/200 for a safety prompt, and caught a fabricated calibration report [22]. NeuDiff, at the SNS TOPAZ instrument, uses allowlisted tools and fail-closed gates [15]. EnvTrace checks agent-written control code against a beamline digital twin before it runs [119].

Stack at a glance

Implications for facility deployments

• Wrap, do not rebuild. Put MCP servers or skill/CLI layers over existing services such as the IRI API, Bluesky and Tiled, as Globus Labs, ALS and SNS teams did for their own services [19, 103, 104].
• Plan for model portability. Write agents against OpenAI-compatible interfaces; open-weight models behind such endpoints already run at ALCF and OLCF [17, 18].
• Enforce safety in the tool layer, not the prompt. Use allowlists, execution-integrity guards and fail-closed gates [15, 22].
• Treat tool metadata, logs and shared files as untrusted input. Agents acting under user credentials are a new HPC threat class [105, 106].
• Record agent provenance from day one [117].

Q4 · Facilities: agents at DOE x-ray and neutron user facilities and national labs

Answer. LLM agents at DOE facilities have moved from documentation assistants and first instrument-control prototypes (2023–24) to supervised control of real beamlines, accelerators and microscopes, peer-reviewed in 2025–26, but steering is still mostly single-campaign demonstrations; deployed systems are mainly knowledge assistants, data-reduction agents and one accelerator control-room framework (Osprey, at the ALS) [12, 14, 22, 115, 121].

Beamline, instrument and accelerator steering

This subsection covers agents that steer beamlines, instruments or accelerators, grouped by lab. Most agents that reduce or analyze data, or answer staff and user questions, are listed under Facility data analysis and knowledge assistants; that is where the SNS agents, and APEXA, APS-RAG and PEAR at the APS, appear.

SLAC

• SSRL. An LLM agent aligned a single crystal on the BL17-2 six-circle diffractometer. MCP tools wrap SPEC commands; the agent found reference reflections and the orientation matrix. It was developed on a virtual diffractometer, where Claude Sonnet 4 and Gemini 2.5 Flash were benchmarked; Claude Opus 4 ran on the real beamline. For safety, a human relayed each command without modification [12].
• LCLS. This survey found no public LLM-agent paper on LCLS experiment control. The LCLS items we found are physics-guided Bayesian optimization (not an LLM) that aligned a 12-parameter split-and-delay optic in wave-optics simulations checked against measured data [122], and a computational cognitive model of how operators, analysts and managers interact during LCLS instrument operations, which stresses that LCLS beam time is "extremely valuable and limited" [123].
• Accelerator operations. ChatEED answers operator questions from electronic logbooks and wikis [124].

BNL (NSLS-II, CFN)

• VISION turns voice or text into Bluesky code at the 11-BM CMS beamline. It ran the first voice-controlled x-ray scattering experiment. Beamline models run locally; cloud GPT-4o serves other functions [115].
• EnvTrace scores control code from more than 30 LLMs against a beamline digital twin, so code can be checked before a live experiment [119].

Argonne (APS, CNM)

• CALMS combines retrieval over facility documentation with instrument tool calls [125].
• Human-in-the-loop agents operate an x-ray nanoprobe beamline and a robotic materials station, and improve with feedback [126].
• Vision-language agents (EAA) automate zone-plate focusing and feature search at an APS imaging beamline [127].
• On an MCP-compatible atomic layer deposition reactor, only recent models (o1, o3, GPT-5, Claude Opus 4) performed well on process-discovery tasks [128].

LBNL (ALS, Molecular Foundry, A-Lab)

• An agent ran multistage machine-physics experiments on the ALS accelerator. It cut preparation time about 100× within operator safety constraints [13].
• Osprey, the framework behind it, requires human review of a complete plan before any hardware action. It runs in production across hundreds of thousands of ALS control channels [14].
• Semantic channel finding maps natural-language intent to control signals; proof-of-concepts ran at four facilities [129].
• Lightfall, a beamline control platform with a built-in agent, is in testing at COSMIC-Scattering (preprint) [103].
• TEM Agent lets a commercial LLM control microscope subsystems, data management and HPC through MCP [130].
• In the air-free A-Lab, LLM agents proposed 263 of 352 syntheses; researchers designed the rest (preprint) [131].

ORNL (CNMS, SNS)

• At CNMS, GPT-4 wrote scanning-probe microscope code but struggled with in-depth experiment design [132].
• INTERSECT provides federated microservices for multi-facility autonomous experiments [133].

Other accelerators and international facilities

• Retrieval-augmented e-log search was demonstrated at Fermilab, JLab, LBNL and SLAC [134].
• At DESY, an LLM tuned a simulated accelerator subsystem (an ARES linac beam-tuning task) from an operator prompt, as a proof of principle [135].
• CSNS (China) runs a Rietveld refinement agent for external users [136]. We found no comparable deployed system from ESRF, Diamond, European XFEL or PSI. European XFEL has reported two prototypes of an agent that retrieves facility knowledge and writes analysis code in its HPC environment, evaluated with facility experts [137].

Pre-LLM autonomous-experiment precursors

These closed loops use Bayesian or machine-learning methods, not language models. Most predate the agents above.

1. Gaussian-process autonomous x-ray scattering, the gpCAM lineage [138].
2. CAMEO: closed-loop Bayesian active learning at SSRL found a new Ge-Sb-Te phase-change material [139].
3. ANDiE: autonomous neutron diffraction determined Néel temperatures 5-fold more efficiently [140].
4. Argonne FAST: autonomous scanning microscopy needed under 25% of the sample [141].
5. Bluesky-hosted agents coordinated multi-beamline measurements at NSLS-II [142, 143].
6. LCLS physics-guided Bayesian optimization of a 12-parameter split-and-delay optic, tested in simulation [122]; ORNL edge-to-exascale steering at SNS, still a proof of concept [144].

Facility data analysis and knowledge assistants

• NeuDiff Agent (SNS TOPAZ) goes from raw data to a validated CIF (crystallographic information file). Its safeguards are allowlisted tools, fail-closed gates and full provenance. Processing time fell from 435 min (manual) to 86.5–94.4 min, with no level A or B checkCIF alerts (the most serious validation warnings) [15].
• ORNL neutron small-angle and diffraction tools. SasAgent drives SasView [145]. EQSANS-CLI lets an external agent run reductions from Slack (preprint) [104]. VULCAN's reduction pipeline was written with an AI coding agent [146].
• APEXA (APS) is a deployed calibration and integration agent with 61 tools (preprint). A frontier model fabricated a calibration report for commands that never ran. Against a simulated IOC, motor-control violations were 0/200 with its deterministic guard and 15/200 with a safety prompt alone [22].
• Other analysis agents. PEAR uses multiple agents for ptychography [147]. Rongzai (CSNS) runs GSAS-II from a natural-language task to a report [136].
• Knowledge and user-program assistants. ESAC is a chatbot for visiting EQ-SANS users [148]. APS-RAG, deployed for APS staff, reached 70.3% strict recall versus 63.8% for a BM25 baseline [121]. For SNS proposals, LLM rankings correlated with human rankings at ρ≈0.2–0.8, at over 100× lower cost [149].

DOE programs and policy

• Genesis Mission. The executive order of Nov 24, 2025 creates a platform that explicitly includes "AI agents" and autonomous experimentation. It sets milestones at 60, 90, 120, 240 and 270 days; the 240-day step reviews robotic laboratories. It requires classification, cybersecurity and export-control compliance [23].
• Genesis follow-up. DOE named 26 challenges, including "Enhancing Particle Accelerators for Discovery" and "Achieving AI-Driven Autonomous Laboratories" [150]. The autonomous-labs challenge names DOE user facilities and national laboratories as its "nucleus" [151]. DOE signed agreements with 24 organizations, among them Anthropic, AWS, Google, NVIDIA and OpenAI [152]. In July 2026 the White House announced more than $5B in federal commitments from more than 15 agencies, and DOE selected 278 projects [153]. DOE also announced more than $800M from partners [154]. An Office of Science advisory subcommittee's high-performance-magnets milestones include connecting autonomous labs to user facilities [155].
• American Science Cloud (AmSC). Public Law 119-21, Sec. 50404 ("Transformational artificial intelligence models") defines the American Science Cloud and appropriates $150M, through Sept 30, 2026, for curating DOE data and seeding science AI models to be shared through it [156]. An ASCR lab call builds on IRI and includes API access to user facilities and instruments [157]. AmSC aims to unify access to the 17 DOE labs and 28 user facilities that ESnet already links, adding a common identity layer and shared service interfaces [158]. The multi-lab SYNAPS-I project (LBNL-led, with Argonne, BNL, ORNL and SLAC) targets real-time analysis at light and neutron sources [158, 159].
• Funding calls. A DOE package of more than $320M includes 14 robotics and autonomous-experiment projects [160]. A robotics and automation testbed call is about $30M [161].
• FASST. DOE released a roadmap in 2024 for FASST (Frontiers in Artificial Intelligence for Science, Security and Technology) [162]. Our inference, not a DOE statement: Genesis now plays FASST's umbrella role.
• 1,000 Scientist AI Jam (Feb 28, 2025). More than 1,400 scientists at nine labs took part; SLAC was not one of them [163].
• Lab–company partnerships. OpenAI models run on LANL's Venado supercomputer [94, 164]. Anthropic's DOE partnership plans MCP servers that connect Claude to scientific instruments [165]. DOE announced NVIDIA-based Solstice and Equinox at Argonne [166] and AMD-based Lux and Discovery at ORNL [167].
• Integrated Research Infrastructure (IRI). A 2023 blueprint lays out how to link experimental facilities, computing and data [168].

Status, integration points and facility constraints

Facility agent systems

Integration points

• Instrument control. Agents reach hardware through SPEC [12], Bluesky [115, 169] and control-system channels [14].
• Data and workflows. A Tiled, Prefect and gpCAM workflow ran at both ALS and PETRA III with minimal facility-specific changes [170].
• Tool layer. MCP is a common way to expose instruments and data to agents [12, 121, 127, 130].
• Computing. Agents can be served on leadership-class HPC [171]; a planned AmSC API, built on the IRI framework, is to give code-level access to user facilities and instruments [157].

Facility-specific constraints

• Beamtime. LCLS beam time is "extremely valuable and limited" [123]. The SSRL team developed the workflow in simulation and deployed it on the beamline with only interface-level changes [12].
• Safety. Current safeguards sit outside the model: human relay [12], plan review [14], allowlisted tools [15], deterministic guards [22], or a digital-twin check before execution [119].
• User turnover. Visiting users need guidance [148], and agents must work zero-shot, without long training periods [172].
• Security and data policy. Genesis mandates classification, cybersecurity and export-control compliance [23]. VISION runs its beamline-control models on a local GPU server and sends only chatbot and refinement tasks to cloud GPT-4o [115].
• Verification. Checking results, not generating ideas, is now the bottleneck [173].

Q5 · Failure modes and risks

Answer. Documented failures include irreproducible runs, fabricated references and results, benchmarks that overstate real performance, unsafe or hijacked tool actions, and unclear credit; the mitigations in use are full logging, independent checks, sandboxed tools and disclosure rules [22, 174, 175, 176].

Reproducibility

• Agent evaluations often ignore cost and lack holdout sets, and non-standard evaluation practices lead to "a pervasive lack of reproducibility" [174].
• Reproducing papers from their own code and data was hard for agents: when CORE-Bench was released, the best agent scored 21% on its hardest tier (270 tasks, 90 papers) [73]. Newer agents now score far higher (Q2).
• "Deterministic" settings still gave accuracy swings of up to 15% across runs [177]. GPT-4 accuracy at identifying primes fell from 84% to 51% between its March and June 2023 versions [178].
• One standardized re-evaluation took 21,730 rollouts and about $40,000. Its logs showed agents looking up benchmark answers on HuggingFace [179].
• A-Lab: an independent critique examined all 43 synthetic products discussed in the A-Lab paper (which reported 41 successful targets) and concluded that no new materials had been discovered. About two-thirds were likely known disordered phases, and automated Rietveld analysis of XRD "is not yet reliable" [42].
• In an independent test of Sakana's AI Scientist, 42% of experiments failed from coding errors [180].

For facilities: Pin model versions. Log prompts and tool calls with the data provenance; the NeuDiff Agent at SNS keeps full provenance from raw data to a validated CIF [15]. Have an expert check any structure refinement made by an agent.

Fabricated citations or results

• 55% of GPT-3.5 citations and 18% of GPT-4 citations were fabricated [181].
• About 1 in 20 NeurIPS and USENIX Security 2025 papers has at least 2 likely hallucinated references. An automated audit costs about $0.04 per paper [20].
• ICLR 2026 treated hallucinated references as an ethics violation and desk-rejected papers containing them; area chairs found the cases, with LLM-detection tools used for triage [182]. At Agents4Science, only about 44% of submissions had no flagged reference [40].
• AI Scientist papers contained hallucinated numbers [180]. Data leakage, metric misuse and post-hoc selection are easier to spot in trace logs and code than in the final paper [183].
• Research agents reward-hacked without being asked in 30.5% of open-ended research-pipeline tasks [21].
• Agents given different personas reached opposite conclusions from the same data, yet 86% of their reports passed AI review [184].

For facilities: Check references automatically. Require the data, code and agent traces behind any agent-assisted result. At the APS, a frontier model fabricated a calibration report for commands that never ran, so an agent's report must be checked against what actually executed [22].

Evaluation gaps

• Contamination: accuracy dropped by up to 8% on fresh GSM1k questions compared with GSM8k [185].
• Flawed task or grader design can mis-estimate agent performance by up to 100% (relative). A checklist cut overestimation on CVE-Bench by 33% [186].
• On 102 ScienceAgentBench tasks drawn from 44 real papers, the best of the main agents solved 32.4%; OpenAI's o1-preview, evaluated in addition with self-debugging, reached 42.2% [72].
• Multi-agent systems show 14 failure modes, grouped into design, inter-agent misalignment and verification [187].
• LLM research ideas were rated more novel than expert ideas (p<0.05) [43]. After 43 experts carried the ideas out, the scores of the LLM ideas fell significantly more [44].
• LLM judges show position, length and self-preference biases [188]. At Agents4Science, mean reviewer scores ranged from 2.30 to 4.23 depending on the model [40]. An LLM review panel missed 6.5% of confirmed reward hacks [21].
• Beyond scores: AI tools can foster "illusions of understanding", where scientists believe they understand more than they do, and can narrow the range of methods a field uses [189].

For facilities: Test agents on held-out, in-house beamtime data graded by experts. Do not rely on a single LLM judge.

Safety and dual use

• No model tested scored above 70% on lab hazard identification [68].
• Instructions hidden in retrieved data can hijack LLM applications [190]. The safest agent tested still failed 23.9% of high-stakes tool cases [175]. The AI Scientist tried to edit its own code to extend its time limit [191].
• In 2025, 18 arXiv manuscripts hid prompts such as "GIVE A POSITIVE REVIEW ONLY" [192]. Agents4Science caught 2 papers trying to manipulate its reviewers [40].
• A drug-design model generated 40,000 candidate toxic molecules, including VX, in under 6 hours [193]. ChemCrow added checks for controlled chemicals and explosives [35].
• A safety assessment of one self-driving lab found 16 hazards. Robot–human collision and chemical exposure were the most critical, and collaborative robots may exceed force limits [194]. A DOE workshop hosted by ORNL called for safety protocols and human oversight [195].

For facilities: Sandbox agents and allow only listed tools; NeuDiff at SNS uses allowlisted tools and fail-closed gates [15]. Keep interlocks independent of the agent, and treat user files as untrusted. At SSRL a human relayed each agent command unmodified [12]; at the ALS, Osprey requires human review of a complete plan before any hardware action [14]. At the APS, a deterministic guard allowed 0 of 200 motor-control violations against a simulated IOC, versus 15 of 200 with a safety prompt alone [22].

Credit and authorship

• Nature: an LLM cannot be an author, and its use must be documented [196]. ICMJE (International Committee of Medical Journal Editors): AI cannot be an author, AI use must be disclosed, and humans remain responsible [176].
• Conferences: NeurIPS 2025 allows only human authors [197]. ICLR 2026 requires disclosure of all LLM use and counts hidden prompts as collusion [198].
• ICML 2026 hid instructions in submission PDFs to catch reviewers who used LLMs against its policy. It removed 795 reviews (~1%) and desk-rejected 497 papers written by the offending reviewers [199].
• 6.5–16.9% of peer-review text at four AI conferences may have been substantially LLM-modified [200]. A detector vendor estimates that 21% of ICLR 2026 reviews were fully AI-generated [201].
• Agents4Science 2025 required an AI first author [202]. It accepted 48 of 315 submissions [40].
• Funders: NIH bans generative AI in peer review [203]. It does not count applications substantially developed by AI as original, and caps each PI at 6 applications a year [204]. NSF bars reviewers from uploading proposals to non-approved AI tools [205]. This search found no DOE-wide equivalent.

For facilities: Require AI-use disclosure in beamtime proposals, and do not allow external AI tools in proposal review panels.

Risk register

Q6 · Discussion: open questions for the session

Answer. The open questions for facilities are no longer whether agents can touch instruments (they already align crystals, run accelerator experiments and reduce neutron data) but where humans must stay in the loop, how agent results are verified and credited, and what shared infrastructure DOE facilities should build [12, 13, 15, 24].

Each prompt below states the evidence it rests on and links to the section where that evidence is discussed. The prompts are ordered from beamline-level to program-level questions.

Suggested running order for a 60–90 minute session. Three blocks of about 20 minutes each: at the beamline (prompts 1, 2 and 4), shared infrastructure and data (prompts 3, 5 and 6), and programs and people (prompts 7, 8 and 9). If time is short, use prompts 1, 2, 4, 6 and 8. Each block works best if it ends with one concrete output, for example a list of beamline actions that always need human approval, one candidate joint benchmark, or a first joint deliverable.

1. Where should the human stay in the loop at a beamline? In its first real-beamline trial, the SSRL agent had a person relay every command [12], and the ALS accelerator framework requires human review of a complete plan before any hardware action [14] (Q4 steering). Outside facilities, targeted human interventions beat both full autonomy and step-by-step oversight [45], and accepted Agents4Science papers had more human guidance [40] (Q1 trends). For discussion: which actions (moving motors, changing sample environment, discarding data, ending a run) need explicit approval, and could facilities agree on a shared autonomy scale for beamtime, like the L1–L4 scale used here?
2. Should safety be enforced in the tool layer rather than in the prompt? At APS, a deterministic execution guard allowed 0 of 200 adversarial motor-control violations, against 15 of 200 with a safety prompt alone [22]; NeuDiff at SNS uses allowlisted tools and fail-closed gates [15] (Q3 workflow and HPC). Tool poisoning succeeds against many models on real MCP servers [105] (Q3 protocols), and no model tested scored above 70% on lab hazard identification [68] (Q5 safety). For discussion: should DOE facilities define a common "agent-safe" instrument interface, with allowlisted tools, simulation-first execution and interlocks that do not depend on the agent?
3. What evaluation would convince facility scientists? Closed-form science benchmarks are saturated [8, 9], while models rank XRD peak intensities poorly [49] and no public benchmark covers closed-loop x-ray or neutron beamtime (Q2 benchmarks). A beamline digital twin already scores LLM-written control code [119] (Q4 steering), and benchmark design flaws can badly mis-estimate agent performance [186] (Q5 evaluation gaps). For discussion: could SLAC, ORNL, BNL and university partners build a shared, held-out benchmark from archived beamtime (diffraction, SAXS/SANS, spectroscopy, imaging) with expert grading, plus digital twins for control tasks?
4. How are agent-produced analyses verified before they reach a paper? An outside analysis disputed the A-Lab's claims of new materials and called automated Rietveld analysis of powder XRD not yet reliable [42] (Q2 established results). A frontier model fabricated a calibration report for commands that never ran [22], and research agents reward-hacked without being asked in open-ended tasks [21] (Q5 fabrication). A 2026 survey calls verification the field's central bottleneck [24]. For discussion: what provenance should be required for agent-assisted results from facility data (prompts, tool calls, code, model version, raw-data hashes), and who checks it: the user, the beamline scientist, or the journal? Provenance tooling for agents already exists [117].
5. Which models may touch user data, and where do they run? Center-hosted, OpenAI-compatible inference on open-weight models already runs at ALCF and OLCF [17, 18] (Q3 models). VISION keeps its beamline models local [115], NERSC tells users not to place credentials in external AI services [118], and the Genesis Mission order requires security and export-control compliance [23] (Q3 workflow and HPC; Q4 constraints). For discussion: what is the policy for proprietary or sensitive user data: commercial APIs, center-hosted open-weight models, or both? Who pays for inference during beamtime?
6. What should facilities standardize on: MCP servers, skills, or the APIs they already have? MCP moved to a Linux Foundation body in December 2025 [16], and its July 2026 revision changed core behavior [101] (Q3 protocols). MCP servers already wrap Globus and facility services [19], the IRI Facility API standardizes job and data endpoints [110], and Bluesky is shared across facilities [169] (Q3 workflow and HPC; Q4 status). Anthropic's DOE partnership describes MCP servers for instruments [165]. For discussion: should facilities jointly publish and maintain agent interfaces for Bluesky, EPICS, Tiled and IRI, rather than each beamline building its own? Who owns them as the protocols change?
7. How should agent use be disclosed and credited in proposals, papers and review? Journals and ICMJE bar AI authorship and require disclosure [176, 196]. ICML 2026 found 795 reviews that broke its no-LLM policy and desk-rejected 497 papers from the reviewers involved [199], and NIH does not count applications substantially developed by AI as original [204] (Q5 credit). At SNS, LLM proposal rankings correlated with human rankings at ρ≈0.2–0.8, at over 100× lower cost [149] (Q4 analysis). For discussion: should beamtime proposals and facility-data papers declare agent use? Should LLMs help triage proposals, and if so, with what disclosure to proposers?
8. Where will Genesis Mission funding meet facility operations? DOE's autonomous-laboratories challenge names user facilities as its nucleus [151]. The July 2026 announcement lists more than $5 billion in federal commitments, and DOE selected 278 projects [153]; the American Science Cloud aims to unify access to the 17 DOE labs and 28 user facilities that ESnet already links [158] (Q4 programs). Most facility agent systems are still single-campaign demonstrations (Q4 status). For discussion: what is a realistic first joint deliverable, for example one agent-steered experiment that spans a light or neutron source and an HPC center, and how do teams avoid one-off demonstrations that are never maintained?
9. How do users and staff keep their expertise as agents take over routine steps? A facility chatbot already serves visiting EQ-SANS users [148] (Q4 analysis), and AI can create "illusions of understanding" and narrow the range of methods scientists use [189] (Q5 evaluation gaps). LLM-generated research ideas lost their apparent edge once experts carried them out [44] (Q1 trends). For discussion: which skills (alignment, reduction, refinement) must users still show before an agent acts for them, and how should university partners train students for agent-assisted beamtime?