Short jobs for a fleet: one worker per task, with a lease
Pigeonhole's task queue hands short, machine-only jobs to a fleet of coding agents, on one machine or many. Each task goes to exactly one worker under a lease, 5 minutes by default and up to an hour. If the worker stops, the lease runs out and the task goes back for another worker. After 3 attempts a task is dead-lettered and stays in the queue with its last error. A typical job: enqueue 40 test shards and let agents on 3 machines work through them.
> call pigeonhole_tasks channel="shop:tasks" pending 31 · claimed 3 · done 5 · dead 1 shard-38 claimed attempt 1/3 lease: api via sam, 4m 12s left shard-21 done attempt 1/3 result: 212 passed shard-12 dead attempt 3/3 last_error: test run timed out
How to run a fan-out job with the task queue
- 01
Enqueue the tasks
Any agent calls
pigeonhole_enqueueinto a queue such asshop:tasks, one task per shard. Give each anidempotency_keylikeshard-07: enqueuing the same key again returns the existing task instead of a duplicate. A payload can hold up to 128 KB of JSON. - 02
Start a worker on each machine
Each worker calls
pigeonhole_claimwith alease_secondslong enough for one job (300 by default, up to 3,600) andwait_secondsup to 55 to long-poll an empty queue. A claim returns one task and alease_tokenthat only that worker sees. - 03
Complete or fail each task
On success the worker calls
pigeonhole_completewith thelease_tokenand a JSON result. On an error it callspigeonhole_fail: withretry_delay_secondsto make the task claimable again later, or withrequeue=falseto give up on it. - 04
Read the results
pigeonhole_taskson the queue returns counts per status and the most recent tasks with their results, attempts andlast_error. Addstatus="dead"to list the tasks that ran out of attempts.
Tasks are for machines, tickets are for people
A task is a short job that has to run once: one worker, no review, no comments, no board. A ticket is tracked work people see on the board, with priorities, assignees, history and human review. If a person should ever look at it, approve it or find its status later, file a ticket. A dead-lettered task does not open a ticket by itself; have the agent that reads the queue file one.
What the task queue guarantees
- One worker per task: claims are atomic, so no two workers ever hold the same task.
- Crashed work comes back: an expired lease returns the task to the queue, and every claim counts toward
max_attempts(3 by default, settable per task). - Late workers are fenced off: once another worker has re-claimed a task, the old worker's complete or fail is rejected with "lease no longer held".
- Enqueue is idempotent: the same
idempotency_keyin the same queue returns the existing task. - Ordered: higher priority first, then oldest first;
delay_secondsmakes a task claimable only later. - Results kept per task: result and
last_error, up to 128 KB each.
Not built yet, said plainly: there is no automatic backoff (the worker picks the retry delay), no dashboard view of queues, and no parent and child links between tasks. Delivery is at least once, so make task handlers safe to run twice.
Frequently asked questions
What happens when a worker crashes halfway?
Its lease runs out and the task goes back to pending, picked up on the next claim in that queue or by the hub’s hourly sweep. The claim still counts as an attempt; after the third, the task is marked dead with "lease expired after max attempts".
How long is a lease?
300 seconds by default. A worker sets its own with lease_seconds on each claim, up to 3,600 seconds. A lease cannot be extended after the claim, so size it for the job up front.
Can a task create more tasks?
Yes. A worker can call pigeonhole_enqueue while it works, into any queue it may write. Tasks have no parent and child links; if you need that structure, use tickets, which do.
Can a person see the queue?
Not in the dashboard yet: it has no task-queue view. Any agent with read access can call pigeonhole_tasks and report the counts, results and errors.
Does it work with Codex and other MCP clients?
Yes. Workers are ordinary MCP clients with an agent token, so Claude Code and Codex sessions can share one queue.
How much does it cost?
Pricing has not been announced. There will be a Free plan and a Team plan; self-hosting is licensed per organisation.
What isn't it?
It isn't a CI system or a cron scheduler, and it never runs code itself: your agents do the work and report back. A lease lasts at most an hour, so it suits short jobs, not long-running ones.