Celery and FastAPI: answering before the work is done
A server that processes an image during the request eventually fails. Separating the acknowledgement from the processing changes the nature of the failure, and raises three questions the synchronous version concealed.
The need was ordinary: receive photographs of receipts, extract the lines, return a usable result. The first prototype did everything inside the request. It worked while the images were small and traffic was light, that is, until it went live. An extraction takes several seconds, an HTTP request lasting several seconds ties up a worker, and a hundred simultaneous uploads are enough for the server to stop responding, including to lightweight requests.
FastAPI offers `BackgroundTasks`, and it is the most tempting trap. The task does start after the response, but in the same process. It shares the processor with live requests, and if the process stops (deployment, restart, out of memory), the work is lost without a trace. That is acceptable for sending an email, not for processing promised to a user.
The server accepts the request, puts a message on the queue and responds immediately. A separate worker consumes the queue. One rule is almost always forgotten the first time: **the image does not go through the queue**. A message broker carries instructions, not megabytes. Write the image to object storage and pass only its key.
from celery import Celery
from fastapi import FastAPI, UploadFile
from fastapi.responses import JSONResponse
celery = Celery("receipts", broker="redis://redis:6379/0", backend="redis://redis:6379/1")
api = FastAPI()
@api.post("/receipts", status_code=202)
async def submit(upload: UploadFile) -> JSONResponse:
# The image goes to object storage; the queue only ever gets a key.
key = await store(upload)
task = extract.delay(key)
# 202 and not 200: the request is accepted, it is not fulfilled.
return JSONResponse({"id": task.id}, status_code=202)
@api.get("/receipts/{task_id}")
def status(task_id: str) -> dict:
r = celery.AsyncResult(task_id)
return {"state": r.state, "result": r.result if r.successful() else None}Celery’s defaults are designed for short, numerous tasks. Image processing is long and expensive. Three of those settings then cause problems, and they are better changed before an incident than during one.
celery.conf.update(
# Defaults to 4: a worker reserves four messages ahead and holds them while
# processing one. On long tasks, three images wait behind the current one
# while another worker sits idle.
worker_prefetch_multiplier=1,
# Defaults to False: the message is acknowledged on receipt, so it is lost
# if the worker dies midway. At True it is only acknowledged at the end —
# at the cost of a possibly repeated execution.
task_acks_late=True,
task_reject_on_worker_lost=True,
# With no limit, one pathological image holds a worker indefinitely. The
# soft limit raises a catchable exception; the hard one kills.
task_soft_time_limit=110,
task_time_limit=120,
)The third setting has a consequence to plan for from the start. If a message can be replayed, the task must be replayable too. Writing the result under a key derived from the input rather than appending it, and checking for it before processing, costs three lines and prevents a duplicate in a client report.
I do not cover the choice of broker. Redis is suitable as long as its delivery guarantee, weaker than a transactional broker’s, is acceptable; above a certain level of stakes the question comes up again, with arguments I did not have to settle here. I also leave aside progress reporting, the percentage that climbs while the user waits. It is feasible but almost always unnecessary: the person who uploaded an image wants to know when to come back for the result.