Agent Backend Architecture at a Glance: How APIs, Queues, Workers, State, and SSE Fit Together

The previous article, Agent Pattern Design, covered control-flow patterns inside a run: tool use, parallelism, planning, HITL, retry, and reflection.

This article has a different job. It is not how to implement a production agent backend, but how to split responsibilities and boundaries so later implementation articles have a shared map.

1. When agent runs get long, the problem changes

At first, running an agent inside a FastAPI handler is fine.

When a run takes three minutes instead of thirty seconds, the story changes. The user may close the browser, a Pod may restart, or the same request may arrive twice. The backend stops being a simple LLM app and becomes a small distributed system.

Production questions follow quickly:

• Should an HTTP handler hold the run until it finishes?
• When the user closes a tab, should execution stop?
• Where do progress and checkpoints live, and what do we show in the browser?
• If an API Pod restarts, how do we rediscover in-flight runs?

The mental model below is the starting point.

API        = accept work
Queue      = preserve execution intent
Worker     = execute the run
PostgreSQL = durable state / source of truth
Redis      = runtime state / coordination / short-lived events
SSE        = deliver progress to the user

The main questions this article answers:

Why do we need a queue?
Why does duplicate execution happen?
Why is the gap between DB and queue dangerous?
Why must SSE reconnect be designed for?

2. The architecture map

Solid arrows are how the user commands execution. Dashed arrows are how the browser observes progress.

Browser → API → Queue → Worker
              ↓
         PostgreSQL · Redis → SSE → Browser

Command:     Browser ──POST──> API ──publish──> Queue ──claim──> Worker
Observation: Browser <──SSE── API <──read──── Redis / PostgreSQL <──write── Worker

The worker runs LLM calls, tools, and checkpoints. The API mostly accepts runs, publishes to the queue, serves reads, and relays SSE. That split lets API and worker replica counts track different resources.

Command path vs observation path

3. The API accepts; it does not execute

Demos often start with:

asyncio.create_task(run_agent(request))
return {"status": "started"}

From a pattern view, a loop is running. From a backend view, run_id, failure lookup, and survival across deploy are still undefined.

Starting a long run usually follows:

1. Validate and authorize
2. Insert a run row in PostgreSQL
3. Publish an execution message to the queue
4. Return 202 Accepted and run_id

@app.post("/runs", status_code=202)
async def create_run(request: RunRequest, user: User = Depends(current_user)):
    authorize(user, request)
    run_id = new_run_id()
    await run_store.create(run_id=run_id, user_id=user.id, status="submitting", input=request)
    await job_queue.publish({"run_id": run_id, "input": request.model_dump()})
    await run_store.mark_queued(run_id)
    return {"run_id": run_id, "status": "queued"}

This snippet is a minimal illustration of responsibility split. In production you must handle the dual-write gap between DB writes and queue publish.

202 means the request was accepted and processing continues asynchronously—not that the browser connection guarantees survival. Queues, workers, and durable state provide that.

FastAPI Background Tasks suit short post-response work; agent runs usually belong in a queue.

4. The queue separates execution lifecycle

The queue’s job is not to run work faster, but to separate the API request lifecycle from the agent execution lifecycle.

• The API submits a job.
• The worker consumes it.
• If the worker dies, the job may be redelivered.
• Duplicate execution is therefore a normal failure mode.

Steps with side effects must ask “is this the first processing of this message?” via run rows, attempts, or idempotency keys. Can an Agent Tool Execute Exactly Once? continues that thread.

5. The worker owns the agent run

The worker role claims queue jobs and owns the current attempt lifecycle: LLM calls, tool loop, checkpoints, and terminal status commits.

Calls to external LLMs or business databases are not the same as internal state transitions—they need timeouts, retries, and idempotent side effects.

Closing a tab or dropping SSE removes the observation path only. An accepted run typically keeps going. Cancellation is product policy; the default is run lifetime ≠ HTTP connection lifetime.

6. Where does state live?

Do not put every kind of state in one store.

PostgreSQL holds recoverable truth. Redis holds fast-changing progress and coordination.

Retrieval storage (embeddings, RAG indexes) is a separate concern; this article covers execution architecture only.

BEGIN;
  INSERT INTO agent_runs (...);
  INSERT INTO outbox (topic, payload) VALUES ('run.execute', ...);
COMMIT;
──> publisher worker reads outbox, publishes to queue, sets processed_at

Mid-article recap: The API accepts work, the queue delivers it to a worker, durable state records outcomes, and SSE shows that state to the user.

7. Separate execution from observation — SSE

SSE is independent of agent execution.

• If the browser disconnects, the worker should keep running.
• On return, the user needs the current snapshot or recent events again.

Worker → State / Event Store → SSE endpoint → Browser

The observation path sends id, event, and data per the WHATWG SSE format. When a generator sees disconnect, it stops HTTP chunks only. A broken stream is not a run rollback.

8. What if two workers run the same run?

Queues and leases alone do not prove an old worker is gone. Design for multiple workers touching the same run.

Worker A → fencing token 41
Worker B → fencing token 42

Storage accepts only token >= current token

A write rejected · B write accepted

9. Pod, process, worker, thread

When Kubernetes, Python, and agent vocabulary land in one sentence, designs wobble. A worker is not a Pod.

When a Pod restarts, its processes and asyncio tasks go with it. The worker role is designed so queue messages and PostgreSQL runs let another Pod resume. A “thread pool worker” is not the same as a queue consumer.

10. Async, thread, process — inside the worker

Python GIL and executor details continue in Why Agent Systems Need Async and Async, Threads, and Processes.

11. Why split API and worker deployments

Locally, API and worker in one process or Pod is enough to learn. In production, split Deployments.

Putting everything in one Pod makes independent scaling hard.

12. Graceful shutdown

Pod termination is not just “kill the process”—it is an in-flight run ownership problem.

1. Stop accepting new jobs
2. Finish or hand off running jobs
3. Save checkpoints
4. Decide ack vs requeue
5. Exit

After SIGTERM, align lease release and ack order with Pod Lifecycle behavior. Do not ack a queue message before committing an important result to PostgreSQL.

13. Architecture recap

API        → accept (202 + run_id)
Queue      → separate lifecycle, redelivery possible
Worker     → own attempt, run tool loop
PostgreSQL → durable truth
Redis      → runtime / coordination / short observation
SSE        → observation only, not execution owner

Failures on the same axes:

Tool loops, HITL, and retry from pattern design run inside the worker. This map fixes which infrastructure owns lifecycle around that loop.

14. Next: implementation

This article defines architecture boundaries only. Follow-ups implement, in code:

• FastAPI submit endpoint
• Queue publisher (or outbox relay)
• Worker process
• PostgreSQL run state
• Redis progress state
• SSE streaming endpoint
• Retry and idempotency

Next, Why Agent Systems Need Async covers overlapping I/O waits inside the worker. The full path is on the Agent Engineering series page.