6.7 Resource Governance & Concurrency

What you'll learn

• Four control points: max_iterations, per-turn timeout, tool timeout, token budget
• Session pool patterns with TTL + LRU eviction
• Graceful shutdown that doesn't drop in-flight turns

Agents are unpredictable workloads — a simple Q&A may take 200 ms; a complex task may run 10 minutes and call dozens of tools. Without governance you either DoS yourself or light the cloud bill on fire.

Four resources to cap separately

Resource	Unbounded →	Primary control
LLM tokens	Runaway bill	max_context_tokens + max_iterations + budgets
Time	Request pile-up / timeout storm	Per-chat() + per-tool timeouts
Memory	OOM crash	Session pool limit + TTL eviction
FDs / subprocesses	Resource leaks	close() + MCP subprocess cleanup

Control 1 · Context window

agent = Agentao(
    max_context_tokens=128_000,   # default 200_000
)

Exceeding triggers context compression — the LLM summarizes old messages. Compression itself costs time and tokens.

Rules of thumb:

Model	Suggested max_context_tokens
gpt-5.4 / gpt-4.1	100_000–128_000
claude-sonnet-4	150_000–200_000
1M-context class	500_000–800_000
Cheap small models	4_000–16_000

Don't set to the model's actual max — leave 20% as compression headroom.

Control 2 · Iteration cap

reply = agent.chat(msg, max_iterations=50)   # default 100

Pair with SdkTransport(on_max_iterations=...) (4.6) for bailout.

Agentao also tracks repeated tool-call failures inside a turn. The third identical tool call (same name, same raw arguments) trips doom-loop protection. A potential write that is new this turn (a call whose tool is not read-only, not denied, with arguments not seen before this turn; a foreground sub-agent call also counts outside read-only mode) clears the other counts, so read_file or the same pytest after an edit is not a loop. A repeated identical write does not clear them again. Repeated unparseable argument payloads for the same tool are answered with a role:tool halt message so the next LLM request remains protocol-valid instead of spinning forever.

Tune by scenario:

Task	max_iterations
Simple Q&A	20–30
Code editing / analysis	80–150
Research / deep task	200–500

Control 3 · Per-turn timeout

Agentao has no built-in chat() timeout — it runs forever. Enforce at the host:

import asyncio

async def chat_with_timeout(agent, msg, timeout_s: float = 120):
    try:
        # Cancelling the awaiting task cancels the turn's token (2.6.5)
        return await asyncio.wait_for(agent.arun(msg), timeout=timeout_s)
    except asyncio.TimeoutError:
        return "[Timeout: agent took too long]"

Tiers:

• Interactive: 60–120s (past that, users leave)
• Background: 5–30 min
• Batch: per-task

Control 4 · Tool timeout

Custom tools must time out:

class ApiCallTool(Tool):
    def execute(self, **kwargs) -> str:
        try:
            r = httpx.get(kwargs["url"], timeout=10.0)
            return r.text
        except httpx.TimeoutException:
            return "Request timed out"

An MCP tool call is bounded by timeout.request. A plain number in timeout bounds only connecting and the handshake, so it does not limit calls:

{"mcpServers": {"x": {"command": "...", "timeout": {"startup": 60, "request": 30}}}}

A call past request fails with an error. The server keeps running, and so may the work the call started.

Concurrency: session pools

Pattern A · One instance per request (naive)

@app.post("/chat")
async def chat(req):
    workdir = Path(f"/tmp/ephemeral-{uuid.uuid4()}")
    try:
        async with Agentao(working_directory=workdir) as agent:
            return await agent.arun(req.message)
    finally:
        shutil.rmtree(workdir, ignore_errors=True)

Fits: stateless (each question independent, no cross-turn context), low QPS. Doesn't fit: stateful conversations, MCP-heavy stacks (MCP init is expensive).

Pattern B · Session pool + TTL eviction

import asyncio
from time import monotonic
from asyncio import Lock

class AgentPool:
    def __init__(self, max_sessions: int = 500, ttl_s: float = 1800):
        self._pool: dict = {}
        self._global_lock = Lock()
        self.max_sessions = max_sessions
        self.ttl_s = ttl_s

    async def get(self, session_id: str, workdir: Path) -> tuple[Agentao, Lock]:
        async with self._global_lock:
            now = monotonic()
            victims = self._pop_expired(now)
            while session_id not in self._pool and len(self._pool) >= self.max_sessions:
                victims.append(self._pop_lru())
            # Close inside the global lock and before building anything: a
            # get() for an evicted session (this one included) must not build
            # a second agent on its workdir while the old one is still
            # closing. return_exceptions, since close() logs its own teardown
            # errors and one victim's failure must not fail this caller's
            # get(). A cancel waits for the closes before it leaves the lock.
            closing = asyncio.gather(
                *(a.aclose() for a in victims), return_exceptions=True
            )
            try:
                await asyncio.shield(closing)
            except asyncio.CancelledError:
                await closing  # keep the lock until the old agents are closed
                raise
            if session_id not in self._pool:
                agent = Agentao(working_directory=workdir)
                self._pool[session_id] = (agent, Lock(), now)
            entry = list(self._pool[session_id])
            entry[2] = now
            self._pool[session_id] = tuple(entry)
        return entry[0], entry[1]

    def _pop_expired(self, now):
        expired = [sid for sid, (_, _, last) in self._pool.items()
                   if now - last > self.ttl_s]
        return [self._pool.pop(sid)[0] for sid in expired]

    def _pop_lru(self):
        sid = min(self._pool, key=lambda k: self._pool[k][2])
        return self._pool.pop(sid)[0]

Key points:

• Lock() guarantees serial turns within a session (agents aren't thread-safe — see 2.3)
• TTL eviction + LRU cap together to prevent unbounded growth
• Remember await agent.aclose() during eviction to reap MCP

Pattern C · ACP subprocess pool

ACP gives you one subprocess per session — natural process isolation. Good for:

• Multi-tenant SaaS
• Strict compliance
• Tenants needing different Python deps

Cost: cold start ~1–2 s, tens of MB per process.

Token budgets

Cap per-user/tenant total token burn:

class TokenBudget:
    def __init__(self, daily_limit: int):
        self.daily_limit = daily_limit
        self.used_today: dict = {}

    def try_reserve(self, user_id: str, tokens: int) -> bool:
        used = self.used_today.get(user_id, 0)
        if used + tokens > self.daily_limit:
            return False
        self.used_today[user_id] = used + tokens
        return True

budget = TokenBudget(daily_limit=1_000_000)

def on_event(ev):
    if ev.type == EventType.LLM_TEXT:
        budget.used_today[current_user] = \
            budget.used_today.get(current_user, 0) + len(ev.data["chunk"]) // 4

Exact counting: tiktoken via pip install 'agentao[tokenizer]'.

Memory footprint

Per-agent memory:

Component	~ size
Agent core + tool registry	3–5 MB
Conversation history (full context)	0.5–2 MB
Memory DB (open SQLite)	< 1 MB
MCP subprocess (stdio)	50–200 MB per child

Formula: total ≈ sessions × 10 MB + MCP × 100 MB

500 sessions + 3 resident MCPs ≈ 5 GB.

Graceful shutdown

import signal, sys

pool = AgentPool(...)

def shutdown(*_):
    print("Shutting down...")
    for sid, (agent, _, _) in list(pool._pool.items()):
        try:
            agent.close()
        except Exception as e:
            print(f"Error closing {sid}: {e}")
    sys.exit(0)

signal.signal(signal.SIGTERM, shutdown)
signal.signal(signal.SIGINT, shutdown)

Without this, SIGTERM leaves MCP subprocesses orphaned.

Pre-load-test checklist

• [ ] Max concurrent sessions per user/tenant
• [ ] Daily token budget per session/user
• [ ] Per-chat() timeout
• [ ] Sensible max_iterations defaults
• [ ] Session pool TTL and max_sessions
• [ ] Graceful shutdown
• [ ] Monitoring: active sessions, MCP subprocesses, memory per session

TL;DR

• Four caps to set explicitly: max_iterations (loop), per-turn timeout via wait_for (wall clock), per-tool timeout (HTTP / shell), token budget per session/tenant.
• Session pool: TTL + LRU eviction, keyed by (tenant_id, session_id). Always close() evicted entries to free MCP subprocesses.
• Graceful shutdown: stop accepting new turns → wait for in-flight ≤ shutdown timeout → force-cancel via tokens → close pools. SIGKILL only as last resort.
• Track active sessions, MCP subprocess count, memory-per-session. Alert when any exceeds your capacity plan.

→ 6.8 Deployment, Canary & Rollback