dbt exposures generated from TMDL
The dependency graph stops at the last table, while the question asked in meetings is about what comes after. A single repository and a text format are enough to extend it, provided the extension is generated and never written by hand.
The question comes up with every change: if I remove this column, what will stop working? The transformation tool answers precisely for its own models, but stops at the last table. What reads that table afterwards (a report, a measure, a dashboard opened every Monday by a management team) does not appear in its graph. The accurate answer would be “I don’t know”, and the answer usually given is “I don’t think so”.
Exposures are meant for this: declaring what consumes a model without being produced by it. They have a reputation for being useless, for a simple reason: written by hand, they are out of date within a month. Someone adds a measure, nobody updates the file, and the graph then shows false dependencies. False lineage is more dangerous than missing lineage, because decisions are based on it.
As long as the semantic model is a binary file stored in someone’s personal space, there is nothing to analyse. Now that it can be written as text, it is a source like any other, and if it sits in the same repository as the transformation project, a script can read both in a single pass. That is the only condition, and it concerns how the repository is organised more than the technology.
warehouse/
├── dbt/
│ ├── models/marts/fct_sales.sql
│ └── models/marts/_marts.yml # generated exposures land here
└── semantic/
└── Sales.SemanticModel/
└── definition/tables/Sales.tmdlA measure contains everything needed: its display name, its expression, and whatever description its author chose to write. The tables it queries can be read from the expression itself.
/// Revenue excluding tax, at invoice date. Excludes cancelled orders.
measure 'Revenue ex tax' =
CALCULATE (
SUMX ( Sales, Sales[quantity] * Sales[unit_price] ),
Sales[status] <> "cancelled"
)
formatString: #,##0 €
displayFolder: Measures\SalesThe script fits on one page: read the measures, find the tables named in each expression, write an exposures file. Two schema constraints determine the shape of the result, and finding them at run time costs an hour each.
import re
from pathlib import Path
import yaml
MEASURE = re.compile(
r"(?:^///(?P<doc>.*)$
)?" # the /// description line, when there is one
r"^s*measures+'(?P<name>[^']+)'s*=s*(?P<dax>.*?)(?=^s*(?:measure|table|Z))",
re.M | re.S,
)
def slug(label: str) -> str:
# An exposure name accepts letters, digits and underscores only, so a measure
# called 'Revenue ex tax' cannot be one. The real name goes to label.
return re.sub(r"[^a-z0-9]+", "_", label.lower()).strip("_")
def exposures(tmdl_dir: Path, models: set[str]) -> list[dict]:
found = []
for path in tmdl_dir.rglob("*.tmdl"):
for m in MEASURE.finditer(path.read_text(encoding="utf-8")):
dax = m.group("dax").strip()
# Tables cited in the expression, kept only when dbt actually builds one
# of that name — anything else is a calculated table, not a dependency.
cited = {t.lower() for t in re.findall(r"([A-Za-z_]w*)s*[", dax)}
depends = sorted(cited & models)
if not depends:
continue
found.append({
"name": slug(m.group("name")),
"label": m.group("name"),
"type": "dashboard",
# description is markdown, which is the whole trick: a fenced block
# puts the DAX itself into the generated documentation site.
"description": f"{(m.group('doc') or '').strip()}
```dax
{dax}
```",
"depends_on": [f"ref('{d}')" for d in depends],
"config": {"meta": {"source_file": str(path)}},
})
return foundFirst constraint: an exposure name accepts only letters, digits and underscores. A measure called “Revenue ex tax” therefore cannot keep its name. It has to be transliterated, with the real name placed in `label`, otherwise generation fails on half the model.
The second constraint is also the main benefit of the approach. The `description` field accepts markdown, so a code block, and that is how the DAX expression reaches the generated documentation site. Nobody had to write anything: a measure’s documentation is its own definition, always up to date, shown next to the model that feeds it.
exposures:
- name: revenue_ex_tax
label: Revenue ex tax
type: dashboard
description: |
Revenue excluding tax, at invoice date. Excludes cancelled orders.
```dax
CALCULATE (
SUMX ( Sales, Sales[quantity] * Sales[unit_price] ),
Sales[status] <> "cancelled"
)
```
depends_on:
- ref('fct_sales')
config:
meta:
source_file: semantic/Sales.SemanticModel/definition/tables/Sales.tmdlGeneration runs in the integration pipeline, and the resulting file is versioned. The documentation is only a secondary benefit. The main one is that the pull request removing a column shows, in its own diff, the removal of the exposures that depended on it. The question of what will stop working no longer needs to be asked in a meeting, since the answer is in the review.
Parsing the expression with a regular expression is a crude method: it finds the tables named, without understanding DAX. A measure that calls another without naming a table produces an exposure with no dependency, which the script skips without warning. That is the flaw to watch. A proper parser exists, but it takes far more than a page of code, and I did not consider it worthwhile at this scale.
Nor do I cover the reverse direction, which would push model descriptions into the measures. It is feasible, but I am wary of it: a generated file would become the source of a file reviewed in code review, and the day the two diverge, nobody knows which one is the reference.