Polars and Delta from a Durable Function
A transactional table format and a function you do not control fit together badly: the first assumes a stable writing process, the second can be replayed at any moment. What that requires, and where the limit lies.
The need did not justify a distributed engine: a few hundred thousand rows per run, a simple transformation, a write to a data lake table. Starting a cluster for that is an expensive reflex. It takes three minutes to start, costs money while starting, and is justified by a volume that is not there. A library working in memory in a single process is enough, and the serverless function hosting it costs only what it uses.
A Durable Function contains two kinds of code that cannot be mixed. The orchestrator describes the sequence and is **replayed from the beginning** at every resumption. Everything in it must therefore be deterministic: no clock, no randomness, and above all no I/O. Activities do the work and run only once per successful call. Writing to the lake from the orchestrator produces as many writes as there are replays, and replaying is part of normal operation.
import azure.durable_functions as df
def orchestrator(context: df.DurableOrchestrationContext):
# Sequencing and nothing else: this block is replayed on every resumption.
days = yield context.call_activity("list_days", context.get_input())
# The activities fan out; the wait itself stays deterministic.
tasks = [context.call_activity("write_day", d) for d in days]
written = yield context.task_all(tasks)
return {"partitions": len(written)}
main = df.Orchestrator.create(orchestrator)The activity reads, transforms and writes. The write mode determines what happens when the same activity is called twice. In append mode, a replay doubles the rows. In partition-replace mode, it rewrites them identically, and that is the only one of the two modes compatible with replay.
import os
import polars as pl
STORAGE = {
"account_name": os.environ["ADLS_ACCOUNT"],
# Managed identity rather than a key: nothing to rotate, nothing to leak.
"use_azure_cli": "false",
"azure_storage_use_emulator": "false",
}
def write_day(day: str) -> str:
frame = (
pl.scan_parquet(f"abfss://raw@{os.environ['ADLS_ACCOUNT']}.dfs.core.windows.net/{day}/*.parquet")
.filter(pl.col("amount") > 0)
.group_by("store", "item")
.agg(pl.col("amount").sum().alias("revenue"))
.with_columns(pl.lit(day).alias("day"))
.collect()
)
frame.write_delta(
"abfss://refined@account.dfs.core.windows.net/sales",
mode="overwrite",
storage_options=STORAGE,
# Replace only the day's partition, not the table: that is what makes
# the activity replayable without doubling the rows.
delta_write_options={"partition_by": ["day"], "predicate": f"day = '{day}'"},
)
return dayThis setup leaves one question open: two activities writing the same table at the same moment. The format guarantees that only one of the two transactions succeeds and the other fails on a conflict, which is correct provided the caller retries. That is why the activities are split by partition rather than by batch. Two separate partitions do not conflict, and a conflict then becomes an exception.
This approach has a clear limit: everything must fit in the memory of a single function. Beyond that, you either split more finely, which has its own limits, or go back to a distributed engine, which then becomes the cheaper option again. Nor do I cover table maintenance. Compacting small files and cleaning up old versions are essential and are not done from the function that writes. They need a separate periodic job, which this setup does not provide.