DigitalOcean Sells the Agent Feedback Loop: Traces, Signals, Simulations in One Cloud

TL;DR

DigitalOcean shipped three AI pieces in a week: Agent Data Services (private preview, Oct 12), Insights (observability) and Signals + Simulations (public preview, Oct 12), all under an agent-learning umbrella.
Signals triages production sessions across three axes — interaction, execution, environmental — and reports an 82% informativeness rate with 1.52× efficiency over random sampling on τ-bench.
Simulations replay a candidate change through multi-turn scenario episodes in an isolated sandbox, on the same harness and tools as production, then grade every trajectory with an LLM judge.
The strategy is the story: whoever holds the traces also holds the eval set, the regression suite and the fine-tuning data. DigitalOcean is buying the loop, not a feature.

Three products, one loop

The pieces were announced days apart and look unrelated until you put them end to end.

Agent Data Services is a managed context substrate: Data Flows ingest multimodal content and agent execution traces into a single governed catalog with lineage, queryable through SQL, vector search or an API. It captures runtime traces and behavioural telemetry, which is why it sits under data and learning rather than storage. Private preview, no SLA. (Source : DigitalOcean — Introducing Agent Data Services)

Signals watches live traffic on Managed Agents and Inference Engine and flags the sessions worth reading. Simulations turns a flagged session — or a scenario you describe in plain language — into a test a candidate change has to survive before it ships. Both public preview. (Source : DigitalOcean — Introducing Signals and Simulations)

Read as a pipeline: production → traces → triage → scenarios → simulated replay → graded diff → ship. That is a feedback loop, and it is the first time a mid-tier cloud has packaged it as a product line rather than four blog-post templates.

Why triage, not judging, is the hard part

The engineering problem DigitalOcean names is volume. Raw trace volume can exceed 100 GB/s, which makes grading every session with an LLM judge economically absurd and manual review arithmetically impossible. Signals does a cheap first pass, stitches turns into sessions, and surfaces only the ones likely to be informative.

The method is published: Signals: Trajectory Sampling and Triage for Agentic Interactions reports 82% informativeness and a 1.52× efficiency gain over random sampling in a controlled study on τ-bench. (Source : arXiv — Signals: Trajectory Sampling and Triage for Agentic Interactions)

The taxonomy is the quietly useful part. Interaction signals capture user sentiment — corrections, escalations, abandonment. Execution signals capture tool reliability — invalid arguments, retry loops. Environmental signals capture infrastructure — timeouts, rate limits, context overflow.

That split maps onto a real diagnostic failure. Most teams run a single “the agent is broken” alert, when the three causes need three different fixes: a prompt, a schema, or a quota. Separating them at triage is what makes one-click conversion into an eval scenario possible — a cluster of wrong-region database queries becomes a test case instead of a ticket.

Two constraints worth repeating: Signals is opt-in, redacts PII before processing, and keeps traces inside your team boundary rather than using them for cross-customer training. And it works best when your agent runs on Managed Agents with models on DigitalOcean Inference Engine — a sentence that is technically about compatibility and commercially about lock-in.

Simulations: the part that changes habits

Evaluations grade single responses. Agents fail across turns: a prompt tweak fixes step two and quietly breaks step seven. Simulations attacks that by running a user simulator against your candidate turn by turn, inside an isolated sandbox on Harness Runtime, with the same configuration and Action Gateway tools as production. Tools that write data or cost money can be mocked, so the test never touches live systems.

Each scenario plays multiple times; each play is an episode producing one trajectory, and the simulator takes different paths — which is how one scenario surfaces several ways a conversation can go. An LLM judge then scores every trajectory on goal completion, knowledge retention and tool-use accuracy, plus custom metrics. Because every tool call is recorded, the report separates a reasoning failure from a tool-use failure: the distinction engineers actually need.

Supported harnesses in preview: Claude Code, Codex, OpenCode. Not supported: LangChain, Hermes, CrewAI. If you only want to compare two models on fixed prompts, single-turn Evaluations remains the right tool.

There is a commercially clever detail: Simulations runs on Agent Droplets plans, and Pro gets a 15% discount, Team 20%, applied to scenario generation, user simulation and LLM judging. The eval loop is metered like compute, because it is compute.

The strategic read

Three things are happening at once, and only one is about DigitalOcean.

Observability is merging with evaluation. Tracing vendors tell you what happened; eval vendors tell you whether it should have. The loop needs both plus a data store, and whoever owns the traces owns the cheapest path to the test set. That is why this arrives as a bundle.

The moat moves from models to data. Inference is commoditising fast — the same week, DeepSeek shipped an MIT-licensed checkpoint that beats frontier models on half the agentic benchmarks published. When weights are free, the differentiator is your own traces, your own scenario library, your own regression suite. A cloud that stores and grades them has a business that survives commoditised intelligence.

The mid-tier cloud has a real wedge. Hyperscalers sell the same loop as five products with five IAM models and a six-figure commit. DigitalOcean sells the whole thing to a team of three, minus the compliance certifications. The preview terms are explicit: no SLA, no support commitment, not covered by standard compliance certifications, pricing and features may change. That is a bet on developer gravity, not on enterprise procurement.

The open question is where the loop closes. Scenarios are versioned by locking a set once used, with in-place versioning listed as next — without versioned scenarios a regression suite slowly drifts into folklore. And without exporting graded trajectories into training, the loop ends at evaluation rather than improvement. Signals data is downloadable for fine-tuning and as RL reward data, which is the door to the second half. (Source : DigitalOcean — Introducing Agent Data Services)

What builders should take away

If you run agents in production and cannot answer “what changed in behaviour last week, and did it get better or worse,” the loop is the gap. You do not need DigitalOcean to build it: traces into a warehouse, triage by heuristics, scenarios from real sessions, an LLM judge, a locked regression set. That is a two-week project for one engineer, and it is the highest-leverage two weeks available to an agent team right now.

What you should not do is confuse observability with evaluation. A dashboard showing 99.9% uptime and 400ms latency tells you nothing about whether the agent chose the right tool. DigitalOcean’s own pitch opens with that failure — a tool call that returned 200 with the wrong region — because it is the failure everyone has shipped at least once.

FAQ

Is this generally available? No. Agent Data Services is private preview; Signals and Simulations are public preview. Neither carries an SLA.

Does Signals work with any agent framework? It is designed around Managed Agents and Inference Engine events, so it works best there. Simulations in preview supports Claude Code, Codex and OpenCode; LangChain, Hermes and CrewAI are explicitly unsupported for now.

What does it cost? No public pricing. Simulations consume Agent Droplets allowances, with 15% (Pro) and 20% (Team) discounts on scenario generation, user simulation and judging.

Do my traces train someone else’s model? DigitalOcean says traces and trace-derived evaluations stay inside your team boundary and are not used for cross-customer training. Signals is opt-in and redacts PII before processing.

Can I take the data out? Yes — Signals data is downloadable for fine-tuning or as RL reward and preference data. That export is what makes the loop portable rather than rented.