0028: Jobs that resume after a restart¶
Status: proposed for M4. Completes the jobs work from proposal 0013.
Problem¶
A background job runs in the process that started it. If that process stops, the job's lease runs out and jobs.get reports interrupted, and the caller has to start the work again by hand. The job was a Python closure, so no other replica could run it.
Proposal¶
Named jobs. A job registered by name, with its arguments stored as JSON, can be run by any replica:
@server.job("report.build", max_attempts=3)
async def build_report(args: StartInput) -> JobResult: # or (args, attempt)
...
job = await ctx.start_job("report.build", args.model_dump(mode="json"))
- Resuming. The next
jobs.getafter the lease runs out, on any replica sharing the store, starts the job again from the beginning and reportsrunningwithattempt: 2. - One restart per attempt. The store has no way to list keys, so jobs resume when polled rather than at startup. A claim per attempt (
put_if_absentonjob-claim/<id>#<attempt>) means two replicas polling at the same moment start it once. - Attempt limit. After
max_attemptsruns (default 3),jobs.getreportsinterruptedwith how many attempts were used. - Repeatable work only. A named job must be safe to run more than once: reports, exports, reads. Writes need their own idempotency.
- Checks at start.
- Arguments must be JSON (they are stored); anything else is refused when the job starts.
- Unknown job names are refused.
- A job's first parameter, if annotated with a Pydantic model, receives a validated model.
- Closure jobs (
ctx.start_job(work)) behave as before and are reportedinterrupted. - The report pilot now uses a named job, and its test checks that a report lost in a restart is built again on another replica.
Not included: MCP tasks¶
MCP's tasks extension (SEP-2663, io.modelcontextprotocol/tasks) lets a client run any tool call as a task and poll it. FastMCP supports it through the separate fastmcp-tasks package, which OunceMCP does not depend on. Mapping jobs onto MCP tasks would make jobs.get unnecessary for clients that support tasks. It needs a decision on adding that dependency, so it is left for the owner.
Alternatives considered¶
- Resuming at startup. It needs a store that can list keys, which
KeyValueStorebackends do not promise. Resuming on poll needs no new store capability, and the caller is waiting for exactly that job anyway. - Checkpointing inside jobs. It would let long jobs continue rather than restart, but it needs an API for jobs to save progress. It can come later if real jobs are long enough to need it.
Impact on tokens and task success¶
jobs.get gains an optional attempt field (a few output-schema tokens). Callers no longer have to start interrupted reports again themselves.
Migration notes¶
None. Existing closure jobs work unchanged. Move a job to @server.job to make it resumable.