← writing
17 June 2025

dbt-fabric called from a Fabric notebook

An integrated platform does not always provide somewhere to run a dbt project. dbt’s Python entry point provides one, provided you accept a few peculiarities.

On an integrated data platform, the warehouse exposes a SQL endpoint and the matching dbt adapter exists. One simple question remains blocking: where to run the command. There is no machine on which to put a project and run a command at five in the morning, only notebooks and a scheduler able to trigger them. The solution therefore lies in a library rather than in infrastructure.

01dbt as a library

It is often forgotten that dbt can be called from Python. `dbtRunner` runs the same commands as the command line and returns an object instead of an exit code: the list of models, their status, their duration. In a notebook, this avoids parsing text output to find out what failed.

python
from dbt.cli.main import dbtRunner, dbtRunnerResult

# The project is mounted from the lake: it stays versioned elsewhere, and the
# notebook holds no copy of it that could drift.
PROJECT = "/lakehouse/default/Files/dbt/warehouse"

runner = dbtRunner()
result: dbtRunnerResult = runner.invoke([
    "build",                       # run + test, in graph order
    "--project-dir", PROJECT,
    "--profiles-dir", PROJECT,
    "--target", "prod",
])

if not result.success:
    # No log to re-read: each node carries its own status and message.
    failures = [
        f"{r.node.name} — {r.message}"
        for r in result.result
        if r.status in ("error", "fail")
    ]
    raise RuntimeError("dbt failed:
" + "
".join(failures))
02The profile, written rather than deployed

A notebook has no stable home directory in which to place a profile file. It is therefore written at the start of the run, from secrets read from the platform’s vault. No credential is stored in the repository as a result, but the run fails without a clear message if the vault is misconfigured. The trade-off is acceptable, provided it is documented in the code.

yaml
warehouse:
  target: prod
  outputs:
    prod:
      type: fabric
      driver: "ODBC Driver 18 for SQL Server"
      server: "<sql-endpoint>.datawarehouse.fabric.microsoft.com"
      database: warehouse
      schema: marts
      # Service principal authentication: the notebook does not borrow the
      # identity of whoever started it, otherwise the scheduled run would
      # depend on one person.
      authentication: ServicePrincipal
      tenant_id: "{{ env_var('TENANT_ID') }}"
      client_id: "{{ env_var('CLIENT_ID') }}"
      client_secret: "{{ env_var('CLIENT_SECRET') }}"
      threads: 4
03What this arrangement does not replace

The platform’s scheduler triggers the notebook and does nothing else: it sees one task where there are two hundred models. This is the flaw described in another article, with no alternative this time, since the platform offers no way of making the graph visible. It can, however, be made readable afterwards. The results returned by the entry point are written to a table, and a report built on that table shows which model failed and since when.

04Native pipelines, and tests in production

I do not compare this solution with the platform’s native pipelines, which cover some of the same needs without dbt. The choice depends on a non-technical criterion. If the team already knows dbt and uses it elsewhere, keeping it is better than learning another tool; otherwise, introducing it for a single warehouse creates a debt that should be stated. Nor do I cover dbt tests in production, which we ran at the same time as the build. That is convenient but debatable, since an incorrect table is published before it is flagged as such.