Long-running production side project · 2015–present
Telegram agent architecture: from commands to asynchronous workers
This bot has lived in the same group chats since 2015. It started as one command handler and is now a TypeScript monorepo with a routing-only webhook, three queue-backed workers, a fail-closed authorization gate and an agent loop with tools and scoped memory. Most of its boundaries exist to keep the webhook fast or to survive a redelivery; few of them touch the model.
- 2015
- first commit
- 14 min
- idempotency lease
- 3
- queues and workers
- 10s
- webhook budget
Most of the work happens before the model
In an active group chat the hard questions come before any LLM call: should the bot answer at all, what fits inside Telegram's webhook timeout, and what happens when the queue delivers the same message twice. The model call is one step near the end.
Webhook, queues, workers
The Telegram-facing Lambda checks Telegram's secret header, makes one cached authorization read, and routes. Every message is enqueued for the activity worker, registered commands go to the reply worker, and anything that could reach the agent goes to the agent worker. Then it returns. It never waits for a model, a render or a database write, which is what keeps the webhook inside its budget: a slow provider can't push it past the 10-second limit.
Three separate queues mean a stuck agent turn cannot delay statistics, and a broken command cannot block agent replies. Each lane has its own dead-letter queue; more than three visible messages in any of them sends an email.
What an admitted message walks through
Group chats default to ignoring, and most of this pipeline exists to drop work as early and as cheaply as possible. Three of the six stages can end the turn; only a message that passes all three reaches history, tools and the main model.
- 01 enabled check chat-configuration · 5s cache · consistent read on a miss disabled → stop
- 02 idempotency lease redis · 14 minutes · outlives the 13-minute worker timeout duplicate → stop
- 03 reply gate one classification with chat memory · engage or ignore · default ignore ignore → stop
- 04 context 24h history, tools and media, loaded only now
- 05 model and tools GPT-6 Astra, Gemini 3.8 Flash fallback, typed tool registry
- 06 delivery reply sent, lease swapped for a three-hour marker
Registered agent commands skip the reply gate, since the command is the request, but first pass a cyber-abuse check that fails closed.
Reply gating and context assembly
The gate is one small model call returning engage or ignore, and it runs on every eligible message with the chat's memory as context. Mention and reply-to flags go in too, marked as unreliable rather than used as a shortcut; otherwise anyone could summon the bot by typing its name while talking about it. History, tools and media load only after admission, so an ignored message costs a cached memory read and one cheap classification. The typed tool registry covers web and image search, image, video, music and voice generation, weather, code execution, LaTeX and SVG rendering, history lookup, memory updates and per-chat tools the bot can define itself. Execution order, timeouts and call limits belong to the runtime, not to the model.
Design decisions
order per chat, parallel across chats
Every queue is FIFO with the Telegram chat id as MessageGroupId, so one chat stays ordered while unrelated chats run concurrently. Workers use batchSize 1 and partial batch responses, so a poison message is retried on its own and moves to its dead-letter queue after five attempts.
fail closed on authorization
Ingress reads one DynamoDB item before enqueuing agent work: an owner-level allow flag and an admin-level toggle, cached five seconds per warm instance with a strongly consistent read on a miss. If DynamoDB errors the message is skipped rather than let through, and the failure raises an alarm.
assume every message arrives twice
SQS delivery is at-least-once and a sent Telegram message cannot be recalled. Reply and agent jobs take a 14-minute Redis lease before doing anything; it outlives the longest worker timeout (13 minutes, for the agent worker), so it needs no heartbeat. Success swaps it for a three-hour completed marker, failure releases it for a clean retry.
let the data be its own idempotency key
The activity worker needs no lease at all. Its chat event is written in one transaction with the message counter, conditional on the event key being free, and that key is derived from the message id. Replaying a message cancels the whole transaction, so counters cannot drift.
treat mentions and replies as hints
A mention or a reply only suggests that someone is talking to the bot. Both go into the gate prompt marked as unreliable, so naming the bot in a sentence about it does not trigger a reply.
attribute failures to a stage
Model and tool calls record status, latency, provider and fallback source. When something breaks, the metrics show which stage failed and whether a fallback ran.
What it runs on
- runtime
- TypeScript monorepo · Bun workspaces · Serverless Framework
- ingress
- grammY webhook on AWS Lambda, routing only
- transport
- three FIFO SQS queues, one dead-letter queue each
- durable
- DynamoDB — chat events, per-user counters, authorization
- fast state
- Upstash Redis — chat memory, 24h history, 30-day metrics, leases
- models
- GPT-6 Astra for the agent loop, GPT-6 Luna for the gate and safety check, Gemini fallbacks
- frontend
- Next.js on Vercel, private live statistics
How it got here
01 Single-process command bot
Currency, weather and search handled inline in one Telegram process. Fine while everything was fast and state was local.
02 Command registry
Handlers moved behind a registry with shared validation and integration helpers, which decoupled Telegram routing from feature code.
03 Thin ingress, async workers
The webhook became a routing Lambda. Model calls, statistics and media moved out, so Telegram acknowledgement stopped depending on downstream latency.
04 Durable queues between them
Direct invocations became FIFO SQS queues with dead-letter queues and idempotency markers. A worker crash now costs a retry instead of a dropped message.
05 Gated agent execution
The agent path added reply gating, provider routing, tools and scoped memory behind a fail-closed authorization check in DynamoDB.
Earlier architecture diagrams
What's missing
Repeatable evaluation. Metrics say which stage failed, not whether an answer got better after a prompt or model change. A replay corpus built from redacted production conversations would let me measure a prompt or model change before deploying it. That is the next piece of work.