Long-running production side project · 2015–present
Telegram agent architecture: from commands to asynchronous workers
This bot has lived in the same group chats since 2015. It started as one command handler and is now a TypeScript monorepo with a routing-only webhook, three queue-backed workers, a fail-closed authorization gate and an agent loop with tools and scoped memory. Almost every boundary in it exists to protect the webhook or to survive a redelivery — not to make the model smarter.
- 2015
- first commit
- 9.6M
- chat events stored
- 3
- queues and workers
- 10s
- webhook budget
The model call is the easy part
In an active group chat the hard questions sit upstream of any LLM. Should the bot answer at all? Telegram wants an acknowledgement in seconds, so what fits in that budget? The queue will hand you the same message twice — what happens the second time? Those questions shaped the architecture. The model call is one step near the end of it.
Webhook, queues, workers
The Telegram-facing Lambda does two things: one cached authorization read, and routing. Every update is enqueued for the activity worker, registered commands go to the reply worker, and anything that could reach the agent goes to the agent worker. Then it returns. It never waits for a model, a render or a database write, which is what keeps the webhook inside its budget no matter how slow a provider is that day.
Three separate queues mean a stuck agent turn cannot delay statistics, and a broken command cannot block agent replies. Each lane has its own dead-letter queue; more than three visible messages in any of them sends an email.
What an admitted message walks through
Group chats default to ignoring, and most of this pipeline exists to drop work as early and as cheaply as possible. Three of the six stages can end the turn; only a message that survives all three is allowed to cost anything.
- 01 enabled check chat-configuration · 5s cache · consistent read on a miss disabled → stop
- 02 idempotency lease redis · six minutes · outlives the lambda timeout duplicate → stop
- 03 reply gate one classification · engage or ignore · default ignore ignore → stop
- 04 context 24h history and chat-scoped memory, loaded only now
- 05 model and tools GPT-5.6 Luna, Gemini fallback, typed tool registry
- 06 delivery reply sent, lease swapped for a three-hour marker
Registered agent commands skip the gate — an explicit command is already an explicit ask.
Reply gating and context assembly
The gate is one small model call returning engage or ignore, and it runs on every eligible message. Mention and reply-to flags are handed to it as context explicitly marked unreliable, rather than used as a shortcut — otherwise anyone could summon the bot by typing its name while talking about it. History, memory and tool definitions load only after admission, so an ignored message costs one cheap classification and nothing else. The typed tool registry covers web and image search, media generation, weather, code execution, history lookup and memory updates; execution order, timeouts and rate limits belong to the runtime, not to the model.
Decisions that keep it debuggable
order per chat, parallel across chats
Every queue is FIFO with the Telegram chat id as MessageGroupId, so one chat stays ordered while unrelated chats run concurrently. Workers use batchSize 1 and partial batch responses, so a poisoned message cannot take its neighbours down with it.
fail closed on authorization
Ingress reads one DynamoDB item before enqueuing agent work: an owner-level allow flag and an admin-level toggle, cached five seconds per warm instance with a strongly consistent read on a miss. If DynamoDB errors the message is skipped rather than let through, and the failure raises an alarm.
assume every message arrives twice
SQS delivery is at-least-once and a sent Telegram message cannot be recalled. Reply and agent jobs take a six-minute Redis lease before doing anything; it outlives the five-minute Lambda timeout, so it needs no heartbeat. Success swaps it for a three-hour completed marker, failure releases it for a clean retry.
let the data be its own idempotency key
The activity worker needs no lease at all. Its chat event is written in one transaction with the message counter, conditional on the event key being free, and that key is derived from the message id. Replaying a message cancels the whole transaction, so counters cannot drift.
treat routing signals as hints, not proof
A mention or a reply-to is evidence that someone might be talking to the bot, not that they are. Both are passed into the gate prompt and explicitly marked unreliable, so typing the bot name in a sentence about the bot does not earn an answer.
attribute failures to a stage
Model and tool calls record status, latency, provider and fallback source. When something breaks at 2am the question is which stage failed, not whether the bot feels broken.
What it runs on
- runtime
- TypeScript monorepo · Bun workspaces · Serverless Framework
- ingress
- grammY webhook on AWS Lambda, routing only
- transport
- three FIFO SQS queues, one dead-letter queue each
- durable
- DynamoDB — chat events, per-user counters, authorization
- ephemeral
- Upstash Redis — memory, 24h history, metrics, leases
- models
- GPT-5.6 Luna primary, Gemini declared fallback
- frontend
- Next.js on Vercel, private live statistics
How it got here
01 Single-process command bot
Currency, weather and search handled inline in one Telegram process. Fine while everything was fast and state was local.
02 Command registry
Handlers moved behind a registry with shared validation and integration helpers, which decoupled Telegram routing from feature code.
03 Thin ingress, async workers
The webhook became a routing Lambda. Model calls, statistics and media moved out, so Telegram acknowledgement stopped depending on downstream latency.
04 Durable queues between them
Direct invocations became FIFO SQS queues with dead-letter queues and idempotency markers. A worker crash now costs a retry instead of a dropped message.
05 Gated agent execution
The agent path added reply gating, provider routing, tools and scoped memory behind a fail-closed authorization check in DynamoDB.
Earlier architecture diagrams
What's missing
Repeatable evaluation. Metrics say which stage failed, not whether an answer got better after a prompt or model change. A replay corpus built from redacted production conversations would turn "feels smarter" into something measurable before deploying. That is the next piece of work.