AWS Infinity Fits 75,000 Idle Agents in 8 GB — Until You Count the Turns

TL;DR

Infinity, AWS and UC Berkeley’s open-source Rust runtime, saves an agent’s conversation and suspends it at tool calls rather than blocking a worker. (Source : AWS Open Source — Introducing Infinity: A Runtime for Extreme Agent Concurrency)
AWS’s 75,000-idle-agents-per-8-GB estimate is reproducible—but applies to short histories. Each turn adds state, reducing capacity. (Source : AWS Open Source — Introducing Infinity: A Runtime for Extreme Agent Concurrency; AgentOps Guide — AWS Infinity Agent Runtime)
Economically, waiting costs retained state, not a live process multiplied by tool latency. Infinity is early research, not a production service. (Source : AWS Open Source — Introducing Infinity: A Runtime for Extreme Agent Concurrency)

Introduction

A six-hour webhook wait normally means keeping a worker alive or checkpointing. Infinity runs a bounded slice, saves the conversation, and stops until an event resumes it. The headline: 75,000 idle agents in 8 GB. The question is how costs change as histories grow. (Source : AWS Open Source — Introducing Infinity: A Runtime for Extreme Agent Concurrency)

A tool call becomes a suspension point

Infinity is an Apache-2.0 Rust runtime released October 23 within Hydro, an AWS–UC Berkeley collaboration. Its crates are version 0.2.0; AWS calls it an early research release for feedback, not a managed or supported production product. (Source : AWS Open Source — Introducing Infinity: A Runtime for Extreme Agent Concurrency; Infinity repository)

Each bounded slice loads a conversation from a pluggable store, runs one model completion, persists its output, and dispatches at most one tool call. It never blocks on a tool, timer, or external operation. When a result arrives, another slice resumes from saved state: the tool call behaves like an await, while the continuation is stored conversation data rather than a suspended native stack. (Source : AWS Open Source — Introducing Infinity: A Runtime for Extreme Agent Concurrency)

Each conversation has its own queue and thread_id, preserving order while other threads run in parallel. spawn_thread and report_to_parent handle agent-to-agent work; Infinity also includes synthetic tool calls, RAP, and the coding agent Infinity Code. (Source : AWS Open Source — Introducing Infinity: A Runtime for Extreme Agent Concurrency)

What 75,000 proves—and doesn’t

AWS ran 80,000 agents for 20 turns—3.2 million completions—on one AMD EPYC 9R14 thread with in-memory stores. At about 100 KB per idle agent, the estimate scales to 75,000 per 8 GB. Each turn made a ~1.2 KB tool call and a second completion after the queued result. Raspberry Pi 5 is an extrapolation, not the test machine. (Source : AWS Open Source — Introducing Infinity: A Runtime for Extreme Agent Concurrency)

The rerun: 10,000 agents × 20 turns, 400,000 completions in 32.3 seconds, 1,056 MB or 105.6 KB per agent—about 75,700 per 8 GB. Each turn adds ~5 KB; at 160 turns, estimated capacity is 10,100 and throughput drops from ~18,200 to ~2,500 completions per second. (Source : AgentOps Guide — AWS Infinity Agent Runtime)

The rerun used a scripted model, in-memory stores, one thread, compaction disabled, and 340-character tool results—not real inference or large outputs. So 75,000 applies to compact idle state, not every long-running workload. (Source : AgentOps Guide — AWS Infinity Agent Runtime)

The cost unit changes

A blocking agent occupies a process throughout an external wait: process footprint multiplied by wait duration. With suspension, no agent code runs; its history and resume metadata remain. The cost model shifts from “process time × latency” to “state size × retention.” Infinity does not make state free; it separates it from an executing worker. (Source : AWS Open Source — Introducing Infinity: A Runtime for Extreme Agent Concurrency; AgentOps Guide — AWS Infinity Agent Runtime)

At 105.6 KB per agent, six hours of retention is ~634 KB-hours; three days is ~7.6 MB-hours, twelve times more. This is not a cloud-bill estimate: the rerun used memory stores, while a deployment may pay for durable storage and I/O. A three-day approval need not hold a worker for three days, but its history still occupies capacity somewhere. (Source : AgentOps Guide — AWS Infinity Agent Runtime)

Serverless fits the slice model

A slice can dispatch work and let its Lambda terminate; a queue event later starts the next slice. Function time then tracks work, not the tool wait. AWS’s reference stack uses SQS FIFO, Lambda, Aurora DSQL, and DynamoDB. Model, queue, and storage calls still cost money, but no worker idles for hours or days. (Source : AWS Open Source — Introducing Infinity: A Runtime for Extreme Agent Concurrency)

At 500 turns, especially with large tool outputs, transcript-as-state gets costly. Infinity shifts pressure to storage and compaction; the rerun disabled compaction and did not test summarization. Model inference usually costs more than memory, so Infinity attacks waiting rather than thinking. At the reported snapshot the repo had 24 stars and 54 open issues: research, not a production verdict. (Source : AgentOps Guide — AWS Infinity Agent Runtime; Infinity repository)

FAQ

Did AWS run 75,000 agents on a Raspberry Pi 5?

No. Both benchmarks ran on a server; 8 GB is extrapolated. (Source : AWS Open Source — Introducing Infinity: A Runtime for Extreme Agent Concurrency; AgentOps Guide — AWS Infinity Agent Runtime)

Does a suspended agent use no resources?

No. Its conversation remains in a store, incurring memory or storage costs. (Source : AWS Open Source — Introducing Infinity: A Runtime for Extreme Agent Concurrency)

Is Infinity production-ready?

No. AWS calls it early research, not a managed or supported product. The benchmark excludes real inference, compaction, and large outputs. (Source : AWS Open Source — Introducing Infinity: A Runtime for Extreme Agent Concurrency; AgentOps Guide — AWS Infinity Agent Runtime)

What should a follow-up benchmark test?

Test longer histories, realistic outputs, compaction, durable stores, and real model calls. (Source : AgentOps Guide — AWS Infinity Agent Runtime)