Google’s Paper Assistant Reviewed 10,000 Papers at ICML 2026 — and Caught 398 Cheating Reviewers

TL;DR — ICML 2026 opened July 6 in Seoul with a record 23,918 submissions and became the largest live deployment of an AI peer-review system ever attempted. Google’s Paper Assistant Tool (PAT) reviewed more than 10,000 manuscripts in ~30 minutes each, catching bugs that had evaded authors for months. In parallel, the conference ran a sting operation using watermarked PDFs to detect LLM-generated reviews — catching 398 reviewers and desk-rejecting 497 of their papers. The two events, unfolding at the same conference, mark a turning point: agentic AI is no longer just the subject of ML research; it is now the infrastructure running the review process itself.

Introduction: A Conference at War with Its Own Scale

When ICML 2026 opened its doors at the COEX Convention & Exhibition Center in Seoul on July 6, the 11,000-plus researchers arriving faced a conference that looked very different from the ICML of even 18 months ago. The raw numbers tell part of the story: 23,918 submissions — more than double the 12,107 received in 2025, which had itself been a record (Source: Tech Times — ICML 2026 Opens Monday in Seoul). Program chairs Alekh Agarwal, Miroslav Dudik, Sharon Li, and Martin Jaggi accepted 6,352 papers at a 26.6% rate, with just 168 (0.7%) receiving Oral slots.

But the numbers only hint at the deeper structural shift. Two experiments unfolded at ICML 2026 that, taken together, represent the most consequential stress test of AI in scientific publishing yet conducted:

Google’s Paper Assistant Tool (PAT) — an agentic AI framework that ingested, analyzed, and critiqued over 10,000 full-length PDFs across STOC and ICML 2026, delivering structured, section-by-section feedback in roughly 30 minutes per paper.
ICML’s LLM-detection sting — a prompt-injection scheme that embedded machine-readable instructions in every submitted PDF. It caught 398 reviewers who used LLMs to write reviews despite explicitly agreeing not to — resulting in the desk rejection of 497 of their own submissions.

These are not separate stories. They are two sides of the same phenomenon: the automation of the academic pipeline is happening at both ends simultaneously, and the conference system has no playbook for this.

PAT: How Google’s Agentic Reviewer Actually Works

The Paper Assistant Tool is not a chatbot asked to “review this paper.” It is an agentic pipeline built on Gemini reasoning models with inference scaling — a term that describes allocating more compute at test time to improve reasoning quality.

The architecture, described in the June 26 preprint “Towards Automating Scientific Review with Google’s Paper Assistant Tool” (arXiv:2606.28277), works as follows (Source: arXiv — Towards Automating Scientific Review with Google’s Paper Assistant Tool):

PDF ingestion — PAT ingests the full manuscript, not just the abstract or introduction. It segments the document into logical sections (abstract, introduction, related work, methodology, results, conclusion).
Per-section critique — For each section, the agent writes a targeted evaluation. It checks theoretical results, validates experiments, suggests improvements, and identifies potential flaws. This is not a single forward pass through a model — it is an agentic loop that can spawn sub-processes for symbolic verification.
Summary surfacing — The per-section critiques are aggregated into a structured summary that surfaces the most significant issues.
Stateless, no training — Critically, PAT does not fine-tune on submitted manuscripts. It is inference-only. All data is deleted within seven days post-program. Each author receives one voucher per submission cycle, preventing bulk submissions for competitive intelligence.

The technical leap over standard zero-shot prompting is significant. On the SPOT benchmark for mathematical errors, PAT achieves a 34% recall improvement — meaning it catches over a third more errors than simply asking a frontier model to review the same paper. This improvement stems from the agentic architecture: zero-shot models lack the runtime budget to execute deep checks, whereas PAT can spin up sub-processes to run symbolic verification (Source: ai|expert — Google’s Paper Assistant Reviews 10,000 Papers in 30 Minutes).

From STOC to ICML to NeurIPS: The Deployment Timeline

PAT did not appear fully formed at ICML. Its deployment followed a deliberate ramp:

Conference	Deployment	Papers	Turnaround	Key Finding
STOC 2026	Pilot (opt-in)	~2,000	~2 days	80% opted in; 97% found feedback helpful; one author said PAT caught “a critical bug that made our proof entirely incorrect… an embarrassingly simple bug that evaded us for months”
ICML 2026	Full deployment	10,000+	~30 min	92.1% would use again; 35.4% identified significant theory gaps; 31% ran new experiments before human reviewers touched the paper
NeurIPS 2026	Adopted (December)	TBD	TBD	Same data model; one voucher per submission cycle

The acceleration from STOC’s two-day turnaround to ICML’s 30 minutes represents both engineering optimization and the shift to a larger compute allocation. Large PDFs at ICML still caused latency spikes beyond the 30-minute median, but the system held.

Among 869 ICML survey respondents, only 1.6% rated the tool “not useful.” Among authors with theoretical results, 35.4% said PAT identified significant theory gaps requiring more than an hour to fix — before any human reviewer had seen the paper. Among authors with experimental components, 31% ran new experiments in direct response to PAT feedback (Source: ai|expert — Google’s Paper Assistant).

This is the number that matters most for the future of scientific publishing: a third of experimental authors changed their papers based on automated feedback before human review began. The agent is not just a linting tool — it is substantively reshaping the content of accepted research.

The Sting: How ICML Caught 398 Reviewers Using LLMs They Had Agreed Not to Use

While Google’s PAT was reviewing papers, ICML was running a parallel operation in the opposite direction: catching human reviewers who were using LLMs to write their reviews.

The enforcement mechanism was technically precise. ICML implemented a two-track LLM policy:

Policy A: No LLM use in review writing.
Policy B: Limited LLM assistance allowed (understanding papers, polishing prose) — but not judging quality or drafting the review.

Reviewers chose their track. Those assigned to Policy A had either explicitly selected it or selected “either policy.”

To detect violations, the program committee used a technique based on research by Rao, Kumar, Lakkaraju, and Shah (published in PLOS ONE, September 2025). The method:

Created a dictionary of 170,000 unique phrases.
Sampled two random phrases for each submitted paper. The probability of any pair being coincidentally reproduced was smaller than one in ten billion.
Embedded these phrases as machine-readable instructions in the PDF — invisible to human readers, but visible to any LLM processing the document. The instructions directed the LLM to include both phrases in its review.
Any review containing both injected phrases was manually inspected by an organizing committee member.

The result: 795 reviews (~1% of all reviews submitted) were flagged as LLM-generated by Policy A reviewers. Of these, 506 unique reviewers were caught. The 398 “reciprocal reviewers” — those whose own submission to ICML depended on their serving as a reviewer — had 497 of their papers desk-rejected, roughly 2% of all submissions (Source: Tech Times — ICML 2026).

The program chairs were transparent about the method’s limits: “This is not a difficult measure to circumvent,” they wrote on the ICML blog, “particularly if it is known publicly — which was the case for almost the entire review period.” The technique only catches reviewers who feed the entire PDF to an LLM and copy-paste the output directly. Sophisticated users who paraphrase, use LLMs for partial drafts, or strip the injected prompt before processing would not be detected.

The implication: if 1% of reviews were caught by a method that, by the organizers’ own admission, catches only the most careless users, the actual rate of LLM use in Policy A reviews is substantially higher than 1%.

Nihar Shah, scientific integrity chair at CMU, told The Transmitter: “I have been working on conference peer review for several years, and I have hardly seen such strong support for anything. People were really tired of reviewers copy-pasting AI-generated reviews without putting any effort” (Source: The Transmitter — Scientists decry conferences’ use of hidden prompts).

Not everyone approved. Researcher Sören Auer called hidden prompts a problematic enforcement mechanism, arguing that “it’s not good to prohibit the use of AI — we should rather have a discussion on how to use it.” Sara Atito of the University of Surrey described the technique as a “poor mechanism” that filters some violations without addressing structural problems in peer review (Source: The Transmitter).

The Closed Loop: When AI Writes Papers Designed to Satisfy AI Reviewers

The ICML sting and PAT deployment are not independent events. They are converging into a structural problem that the academic community is only beginning to name.

If conference organizers deploy automated agents to filter and review submissions — as ICML, STOC, and now NeurIPS are doing — authors will, with mathematical inevitability, use matching agents to write papers optimized for those reviewers. The result is a closed loop: AI models write papers designed to score well on AI review pipelines, evaluated for internal consistency rather than external truth.

An agentic reviewer can catch a mismatched formula or a missing citation. It cannot judge whether a theory corresponds to physical reality. It evaluates logical coherence, not factual accuracy. This distinction matters because the volume economics of scientific publishing now favor automated pipelines at both ends.

Consider the numbers: ICML received 23,918 submissions. At a 30-minute review time per paper, PAT can process the entire submission pool in roughly 12,000 compute-hours — less than two weeks on a modest cluster. Human reviewers, by contrast, are already overwhelmed. If submission counts continue doubling every year, the only viable path for large-scale conferences will be to mandate agentic pre-filtering before any human reviewer sees a page.

This pre-filtering will become the gatekeeper of scientific credibility. Papers that fail automated checks — mathematical consistency, structural completeness, citation coverage — will be rejected instantly. Only a fraction will reach human evaluation. The risk is not that AI will lower review quality; it is that AI will homogenize it, favoring familiar methodologies and penalizing radical approaches that do not align with training data.

As Mark Jordan wrote in a sharp analysis for Singularity Moments: “By the end of 2027, the academic publishing model will split into two distinct tracks. One track of fast-path journals where papers are written, reviewed, and published entirely by agentic systems in days. The other track will be slow-path human-only verification, requiring physical replication or live presentations” (Source: Singularity Moments — AI peer-reviewers are about to break scientific publishing).

The Economics Are Already Unequal

One dimension of PAT’s deployment that received less attention at ICML is the cost structure. Running an agentic loop that performs deep symbolic verification for 30 minutes requires significant compute. While a zero-shot prompt costs fractions of a cent, a multi-step verification agent can cost several dollars per run.

This creates a resource gap: wealthy industrial labs (Google, DeepMind, OpenAI, Anthropic) can afford to run PAT-style verification on every preprint they produce or review. Independent academic researchers at smaller institutions cannot. If pre-submission verification becomes a de facto requirement for passing automated filtering, it becomes a tax that only well-funded labs can pay.

The PAT team addressed this partially with the NeurIPS deployment model — one voucher per submission cycle, preventing bulk use for competitive intelligence — but the underlying asymmetry remains: the same companies building the review infrastructure are also the companies submitting papers.

Agentic AI’s Structural Arrival in ML Research

Beyond PAT and the sting, the clearest signal about where ML research is heading came from the workshop proposals. Workshop chairs Gergely Neu and Courtney Paquette noted that some variation of “agentic AI” appeared in 60 of 247 workshop proposals — nearly a quarter of all proposals. The final program accepted 44 workshops plus 4 affinity workshops. Accepted events include “Agents in the Wild: Safety, Security, and Accountability,” “Statistical Frameworks for Uncertainty in Agentic Systems,” and “Failure Modes in Agentic AI (FAGEN)” (Source: ICML 2026 Workshops).

This concentration signals that autonomous, tool-using AI systems are moving from a niche subfield to the core of machine learning research — and the reliability and safety questions that accompany them are being taken seriously at the highest levels of the field.

FAQ

Q: Is PAT replacing human reviewers?

No. PAT is explicitly not a gate. Google describes four progressive levels of AI-human collaboration in scientific evaluation; current deployments sit at the lower end — pre-submission augmentation, not automated accept/reject. PAT surfaces errors, but adjudication stays entirely with human reviewers. Reviewers, area chairs, and program chairs see none of the PAT output.

Q: Can reviewers game the LLM-detection system?

Yes, and the ICML chairs acknowledged this openly. The prompt-injection technique only catches reviewers who feed the entire PDF to an LLM and copy-paste the output. Anyone who paraphrases, removes invisible text, or uses LLMs for partial drafts evades detection. The 1% caught rate is almost certainly a lower bound.

Q: What happens when authors optimize papers for AI reviewers?

This is the central risk. If automated pre-filtering becomes mandatory, authors will use AI to write papers that pass AI checks. The result is syntactically perfect, mathematically consistent, but potentially empty research. The academic community has not yet designed a defense against this.

Q: How much does PAT cost per paper?

The PAT team has not published per-paper cost figures. Anecdotal evidence from other agentic verification systems suggests $1–5 per deep verification run, compared to fractions of a cent for zero-shot prompting. This cost asymmetry favors well-funded labs.

Q: What conferences are adopting PAT next?

NeurIPS 2026 (December, Sydney) has adopted PAT under the same stateless, inference-only model. No other conferences have been publicly confirmed, but the trajectory from STOC → ICML → NeurIPS suggests major ML venues are the target.

Google's Paper Assistant Reviewed 10,000 Papers at ICML 2026 — and Caught 398 Cheating Reviewers