← All work
dbt Core spike: deciding a transform layer with evidence
A time-boxed evaluation of dbt for the silver-to-gold layer of the ETL v2 platform, run on our data shapes, our Postgres and our Celery orchestration, with every number reproducible from a runbook.
WhereCogwheel Analytics
When2026
My roleBuilt and authored
OwnershipSole author
Why
The build-versus-buy decision for the transform layer was being argued from opinions. I replaced the argument with measurements: three gold models over three normalised feeds, including a daily incremental model of 5.2 million rows on synthetic data at target scale.
What came out
- Nine distinct ways to get silently wrong data with dbt's defaults, four of them serious. Each fixed in a few dozen lines and pinned with a regression test.
- Enforced model contracts caught silent type widening twice during development.
- Clear ownership boundaries: Python owns ingestion and silver validation, dbt owns SQL and business-rule tests, Celery owns orchestration. dbt never writes to silver, Celery never contains SQL.
- A channel-taxonomy seed, one CSV in git, resolved a long-standing reporting inconsistency independently of the dbt decision.