SageMaker and EMR from Airflow: an orchestrator that computes nothing
A DAG on a managed service has neither the memory nor the permissions for a training run. It triggers, it waits, it alerts somebody, and that last part is always written last, often after the incident.
On a managed Airflow, the temptation is to run the computation inside the DAG: a `PythonOperator` that loads a dataset and trains a model. It works on a sample and fails afterwards. A managed service’s workers are sized to schedule, not to compute. They share memory with the scheduler, and a training run that uses too much blocks the scheduling of every other pipeline.
Since then I apply a simple rule: the orchestrator computes nothing. It describes a job, hands it to a service designed to run it, and monitors its progress. The rest (machine sizes, permissions, the cluster’s lifetime) becomes configuration instead of code.
Two cases recur. Training goes to the model service, with its own image and its own instance type. Heavy preparation goes to an ephemeral cluster that is created, given steps to run, then shut down. The last operator, the shutdown, is the one most often forgotten, and that is the most expensive omission.
from airflow.providers.amazon.aws.operators.emr import (
EmrAddStepsOperator, EmrCreateJobFlowOperator, EmrTerminateJobFlowOperator,
)
from airflow.providers.amazon.aws.operators.sagemaker import SageMakerTrainingOperator
from airflow.providers.amazon.aws.sensors.emr import EmrStepSensor
cluster = EmrCreateJobFlowOperator(task_id="cluster", job_flow_overrides=EMR_CONFIG)
prepare = EmrAddStepsOperator(
task_id="prepare",
job_flow_id=cluster.output,
steps=SPARK_STEPS,
)
# The sensor is what makes the dependency true: without it the next task starts
# as soon as the step is *submitted*, not when it has finished.
wait = EmrStepSensor(
task_id="wait",
job_flow_id=cluster.output,
step_id=prepare.output[0],
mode="reschedule", # gives the slot back instead of holding it
poke_interval=60,
)
training = SageMakerTrainingOperator(
task_id="training",
config=TRAINING_CONFIG,
wait_for_completion=True,
)
shutdown = EmrTerminateJobFlowOperator(
task_id="shutdown",
job_flow_id=cluster.output,
trigger_rule="all_done", # shut down even if the preparation failed
)
cluster >> prepare >> wait >> training
wait >> shutdownTwo details deserve a paragraph. A sensor in `reschedule` mode frees its slot between checks instead of holding it for two hours. On a managed service where slots are limited and billed, the difference shows on the invoice as well as in throughput. And the `all_done` trigger rule on the shutdown step is the only one that guarantees a cluster costing several euros an hour does not stay up after the step it served has failed.
On a managed Airflow you choose neither the version of the dependencies nor when the environment restarts. Packages are declared in a file placed on object storage, and any change triggers a restart of the environment, which means tens of minutes without scheduling. In practice you therefore do not add a dependency lightly, and you prefer a provider operator to a library that would need installing.
A failure email is not enough as an alert: nobody reads it at three in the morning. The pipeline feeding a model in production justifies an on-call rota; the others do not. Choosing which pipelines to watch matters more than the tool: a team woken up for a pipeline with no consequences ends up ignoring the alerts that matter.
from airflow.providers.pagerduty.hooks.pagerduty_events import PagerdutyEventsHook
def page_on_call(context) -> None:
instance = context["task_instance"]
PagerdutyEventsHook(pagerduty_events_conn_id="pagerduty").create_event(
summary=f"{instance.dag_id}.{instance.task_id} failed",
severity="critical",
source="airflow",
# The dedup key avoids opening one incident per attempt: the three tries
# of a single run wake exactly one person.
dedup_key=f"{instance.dag_id}:{instance.task_id}:{context['run_id']}",
custom_details={"log": instance.log_url},
)
# Set on the task that matters, not on the whole DAG: the sorting is what keeps
# an on-call rota credible.
training.on_failure_callback = page_on_callI do not compare orchestrators. The subject is interesting, but a single engagement is not enough to settle it. What mattered here was the constraint of the managed service, which applies whatever the tool. Nor do I cover resuming an interrupted training run: we restarted from the beginning, which was acceptable at that scale and would not have been beyond it.