Transformations and Templates

What it is

DeltaVault generates the code that builds and loads your warehouse. Create-table statements, staging notebooks, merge procedures into persistent staging, whole workflow definitions: all rendered from the metadata already sitting in your catalog. You describe a table once; the engine writes the platform-correct code for Databricks, which is the target platform DeltaVault generates for today. The templates themselves are written in Jinja, the templating language most data engineers already read.

Which template renders for which table isn’t hardcoded. A binding-rule engine matches on methodology, target platform, data layer, and table type, with platform-shipped defaults underneath, organization rules above them, project-scoped rules above those, and a per-table override at the top that you author inline from a table’s preview. When the default output isn’t what you want, you don’t fork the product: you override one named section of a template, or bring a template of your own and bind it.

The quotable version: metadata in, pipelines out. When the metadata changes, the pipelines follow.

In the free edition

Not included. The template engine, the generated pipelines, the builds page, data tests and the orchestration output are one surface, and it is outside the free edition. It is also outside every paying tier today: templates and builds are held back from core organizations as well, so this is not a line the free edition draws on its own.

Generation needs physical tables and columns to generate against, which is the same reason the catalog sits on the paid side. A free organization models the business; turning that model into pipelines is the step that needs a warehouse under it.

How it works

Templates, written in Jinja

DeltaVault ships a library of global templates: DDL for each platform, staging and persistent-staging load code, orchestration shells. Template authors work in a playground with a file tree, an editor with Jinja syntax highlighting, and a live preview pane that renders the in-flight template against a real table, connection, project, or workflow from the catalog. Validation runs on save, so a template with a syntax error never reaches render time silently.

Templates don’t hardcode table names. A ref() function resolves a logical reference (“this connection, this schema, this table”) into the platform-correct three-part identifier at render time, pulling the catalog or database segment from the bound render target. The same template body renders against development, staging, and production by swapping the target. Each ref() call also records a lineage edge, so DeltaVault knows which generated artifacts read which tables.

Which template renders for which table

Binding rules decide. Each rule names one of eleven output kinds: create-table, load, test, piece, and ingestion-object at table grain, plus six orchestration-family artifact kinds, orchestration (the control notebook), workflow, pipeline, bundle, deploy, and ingest-workflow (the composed per-workflow ingest notebook), at project grain. Each rule also matches on up to five dimensions: methodology, target platform, data layer, table type, and entity type (the table’s vault kind, such as hub, link, or satellite). An absent dimension is a wildcard, and an entity-specific rule outranks a layer-general one. Resolution ranks candidates by explicit priority first, then scope (a table override beats a workflow-scoped rule, which beats a project-scoped rule, which beats an organization rule, which beats a platform default), then specificity. Rules can point at the shipped global templates or at templates your organization owns. The whole mechanism mirrors how you’d expect overrides to work: the platform supplies sensible defaults, you shadow exactly the cases you care about, and one project can diverge without touching the rest. Templates can be assigned per entity type, not only per layer, and from three directions. A project’s Templates tab opens with a target platform header line, then a by-layer and entity-kind readout: pick a data layer in a tree and see the template bound for each output kind and entity kind at that layer, each row badged with where it comes from (its scope), and below that a Table overrides list of every table-scoped override in the project. A template’s own Where it applies tab shows every binding rule that routes to it and lets you add, edit, or remove them in place. A table’s own Preview tab authors a table-scoped override inline for that one table. The three surfaces describe the same rules from different angles: one project at a time, one template at a time, or one table at a time.

Orchestration is not one slot either. On a Databricks project, it splits into six independently bindable artifacts: the control notebook, the workflow definition, the pipeline resources, the bundle configuration, the deploy workflow, and the composed per-workflow ingest notebook. Each resolves its own template for the project’s target platform through the same binding-rule precedence as a table template, project rule over organization rule over platform default, so a project can override just the bundle configuration and leave the other five on the shipped default. Until a project sets its own rule, the build renders the platform’s default template for that artifact. Other target platforms currently resolve a single control notebook slot; the rest of their orchestration artifacts are not yet bindable.

See the code before you run anything

Every table’s detail pane carries a Preview tab with three views: Create table (the DDL), Load, and Test, rendered live for that exact table. The tab tells you which binding rule matched and whether the output validated. An editor can also override this table’s template for the selected output kind directly from the tab (the fuller flow is in the user guide). It also degrades honestly: no rule bound for this layer yet, it says so; a load preview that needs upstream lineage the table doesn’t have, it says that instead of rendering something wrong.

One table family, four layers, one target platform

Behind the staging templates sits a single dialect-neutral description of a table family: source, landing, staging, and persistent staging, resolvable from any member. No platform SQL lives in that description: quoting, hashing functions, and timestamp functions are applied at render time per dialect. Today that one description renders the Databricks output:

• Databricks: a complete Lakeflow Asset Bundle generated from the catalog, with a databricks.yml definition, pipeline and job resources, and declarative source code for every layer. Bronze ingestion is selected by source type (file Auto Loader, database, software-as-a-service application, or Lakeflow Connect). Silver and gold are declarative streaming tables and materialized views that apply change data capture with the current AUTO CDC INTO syntax, with keys, sequencing, delete handling, and history type derived from the table’s metadata. The same generator emits a full Data Vault (hubs, links, and satellites, the business vault, reference tables, and point-in-time and bridge views) as batch insert and merge loads rather than as streaming tables, and a file-based change-data-capture pattern (an Auto Loader raw feed plus a cleaned silver table with data-quality expectations). The combined staging load can be authored in SQL or Python; the Data Vault combined load and the individual hub, link, and satellite batch loads each ship in one format.

Current and end-dated views, generated not hand-written

History tables, persistent staging and Data Vault satellites, keep every version of a row. Reading them usually means writing the same “latest row per key” and “valid from and to” SQL by hand on every table. DeltaVault generates both as opt-in views from the catalog: a current view (one row per key, newest load wins) and an end-dated view (each version gets an effective-from and an effective-to). They are ordinary views over the base table, reconciled automatically as the table changes, and a single toggle per kind turns them on. When the generated SQL needs adjusting, customizing a view freezes its SELECT into your own editable copy, and resetting hands it back to be managed again. The generated views appear in the catalog next to their base table and carry that base table as a lineage source.

From catalog to running order: workflow generation

Pick a Databricks project, pick a topology policy, pick a parallelism degree from 1 to 32, and DeltaVault generates four workflows: an ingest workflow (source into landing), a transform workflow (landing into staging and persistent staging), a data vault workflow, and a data mart workflow. A master workflow chains the four together in medallion order, ingest then transform then data vault then data mart, each layer’s job completing before the next starts. Dependencies inside each layer workflow are wired from the lineage your catalog already holds. Three policies: topological (tier barriers computed from lineage), sequential (one chain), parallel (everything at once). Cycles in the lineage are detected and named, never silently ordered. An empty layer produces no workflow, and the master chain skips it. The unified Build action drives the same planner: a build whose scope lacks a generated workflow for one of its populated layers creates one with planner defaults, and an existing generated workflow is reused as-is so hand-shaped ordering survives.

The data vault workflow keeps to three stages that hold under every topology policy, parallel included, because the order is a correctness requirement rather than a lineage preference: reference tables load first (satellites look them up), then one control notebook per staging source (control_<source>.py), each firing that source’s hubs, links, and satellites in dependency tiers, then the point-in-time, bridge, and business vault objects, which read across the whole raw vault. Hubs, links, and satellites carry no step of their own: their loads run inside their source’s control notebook, parameterized to that source’s slice, so an object fed by two sources loads once per feeding source rather than twice over the same rows. The data mart workflow orders every dimension before the facts that reference it.

Re-running generation shows a diff preview that summarizes every verdict before you apply anything. A step you added by hand is left alone. A step you have edited is preserved exactly as you left it and marked detached, with the preview naming what regeneration would otherwise have changed on it. An edited step whose source table has since dropped out of the plan is kept exactly as edited too, flagged as missing its source. A step you deleted stays deleted: regeneration never brings it back, though the preview notes that the plan still calls for it. Reordering steps or changing a workflow’s thread count is your layout choice, and it survives regeneration independent of any per-step edits. Workflow generation currently runs for Databricks projects.

A second generation pass authors the actual code for every ingest and transform step (the ingestion pipeline definition and one notebook per transform step) and writes it as git-tracked files in your metadata repository, riding the same branch, commit, and review flow as everything else. This code pass keeps its own sentinel-header model: a generated file you have hand-edited surfaces as a conflict and is skipped unless you explicitly force an overwrite, and the sentinel header in each file is how the generator knows what it owns. Each step gets a read-only code viewer with copy and download.

Build artefacts: the Target Platform settings that shape what gets generated

The Target Platform settings carry a Build artefacts section: seventeen fields, in three groups, that govern how your generated bundle is laid out and run. The Orchestration group sets the Job Cluster, the Git Source, and the Asset Bundle Path. The Data Loading Behavior group sets the Staging Mode and the Notebook Language. The Environment group sets the Build Output Path, Repository Name, Notebook Path, Append Notebook Name, Temp Table Schema, Table Owner, Table Properties, Read Files Options, Copy Format Options, Copy Options, Display Time Zone, and Data Time Zone. Set them once for the organization, or override any of them for a single project: a project that ships to a different repository or bundle path carries its own values, and the rest fall through to the organization defaults. The values flow into the generated bundle and the per-table render context, so the same catalog produces the right artefacts for each project without forking templates.

Platform administrators author these Target Platform defaults once per platform, and organizations inherit them: the baseline shows through live while an organization has set nothing of its own, seeds an organization’s settings when it first selects the platform, and is written in by the organization-facing Apply platform defaults action.

Build once, review, then commit

One Build action on a project (or scoped to a workspace) produces the whole deployable tree for every selected layer as an immutable in-app snapshot: table DDL resolved through the same binding rules as the per-table preview, ingest pipelines routed per table (runtime override, then Lakeflow Connect where configured and supported, then a generated declarative pipeline for snapshot, watermark, or change data capture modes), orchestration jobs built from the project’s workflows with their schedules and timezones, an ordered idempotent DDL deployment notebook that bootstraps each layer’s assigned catalog and schema before creating tables, and a generated continuous integration workflow that validates and deploys the bundle. Every file carries a validation status; failures are isolated per artifact, so three bad tables still yield the other ninety-seven files with the failures flagged. A build containing any artifact that failed to render outright is blocked from commit, with no override; an artifact that rendered but failed validation can still commit through an explicit override. Committing is an explicit second step: after review, one action pushes the exact snapshot to the connected repository and stamps the run with the commit hash. Nothing reaches git without it, and DeltaVault never pushes to a Databricks workspace; the customer’s own continuous integration deploys the bundle.

Your SQL, kept honest

Two escape hatches exist for hand-written SQL, and both are validated. Macros are parameterized SQL fragments you define once and reference from any column transformation as @macro:name(arguments). Macros are also first-class in View Designer column transformations and derived expressions, with autocomplete and an insert control in the designer, and they expand at build render time, so the saved view stays readable while the generated SQL carries the expanded body. Expansion is recursive with cycle protection, and argument counts are checked.

A default macro library ships with the product: 36 cleansing, masking, and Data Vault macros (null handling, whitespace cleanup, PII masking and AES-GCM encryption, business-key and hash-key treatment) that every organization can invoke immediately, grouped by folder in the macro library. Defining an organization macro with the same name overrides the shipped default everywhere it is used, including inside other macros that reference it, so the library is a starting point, not a constraint.

Column transformations themselves are validated live against a per-dialect function allowlist: a built-in baseline per platform plus whatever functions your organization adds. Typo a function name and the column editor flags it as you type, though that one is a hint rather than a restriction: only an unresolved macro, a macro argument-count mismatch, or an undefined placeholder is durable enough to become a finding and reach the commit gate. A column with no SQL dialect at all (file-based sources) still gets the placeholder check; only the macro and function checks stand down for it, because those are the two that need a resolved dialect.

Customize without forking the product

The positioning is deliberate: you don’t fork a template, you tweak one piece of it and keep getting upstream improvements. Organization users do this in a notebook-style sectioned editor, with a left tree of the shipped global templates plus your organization’s own forks, and a live preview pane on the right that renders the in-flight template against a real table, connection, project, or workflow from your catalog.

The editor presents a template as a vertical stack of named cells. Block cells are the named sections that the template author has exposed as customization points; glue cells are the read-only Jinja text that connects them. An editor can read, override, or author each named section in isolation, without scrolling through the entire Jinja file to find the relevant passage. The live preview updates as you edit.

What each role can do depends on the template’s ownership and the user’s position:

• Organization users on a read-only global: override any block cell in place by writing a fragment that replaces just that section at render time. The rest of the template keeps tracking upstream, so later improvements to the global keep flowing. From a glue cell that imports a shared helper, an organization user can fork that helper into their own copy and change a macro or expression everywhere it is used, without touching the calling template.
• Admins and users working on an organization-owned fork: add, delete, convert, and reorder sections directly, and edit both block and glue content. The full editing surface is available because the fork is fully owned.

A fragment that overrides a section composes against the live global at render time. If a block is renamed in a later revision of the global, a fragment pointing at the old name degrades to a logged warning rather than breaking your build. Platform administrators curate which sections exist as named blocks, so the customization surface is governed rather than open-ended.

Organizations can also hold full templates of their own, namespaced apart from the globals so nothing is silently shadowed. They are git-synced metadata like tables and columns: branch-scoped, audit-logged, reviewed at commit. Binding rules, including rules scoped to a single project, can point at them.

Custom attributes drive your templates

Organizations define their own typed attributes for tables and columns (Text, Number, Boolean, or Select), set values in the catalog, and reference them from any template or workflow built on the table and column context as table.custom.<key> and column.custom.<key>. Definitions are branch scoped and version to the organization’s git repository like the rest of the metadata; values validate on write and fall back to per-attribute defaults at render time, in every rendering context including previews.

Declare tests where the data flows

A data test is a declarative assertion attached to a source table: “whenever this table feeds a load, verify this.” Five types ship: row-count reconciliation across the hop (with absolute or percentage tolerance), aggregate reconciliation (sum, minimum, maximum, or distinct count, optionally grouped), row-count threshold bands, uniqueness over a column combination, and custom SQL assertions that must return zero rows or an expected value. Each test is marked Fail or Warn. Every load step that reads the table inherits its tests, and each step carries an overlay to disable inherited tests or add step-local ones: declare once on the staging table that feeds three loads, tune per load where needed. Test definitions are git-synced metadata, versioned inside the table’s own file.

Declared tests are generated for execution: each load step with effective tests gains a generated companion step that runs after the load, writes one result row per assertion to a results table on your platform, and fails the workflow step on a hard failure. Tables and workflow steps carry a Tests tab for configuring them. Test execution currently generates for Databricks projects.

Why it matters

The plumbing in a warehouse build is the part nobody wants to own: the fortieth merge statement, the hash-key expression copy-pasted between notebooks, the workflow dependency list maintained by memory. DeltaVault’s position is that none of that should be authored per table. You manage the metadata (what the table is, where it comes from, which columns form the business key) and the code is a derived artifact, like a compiled binary.

What you stop doing by hand: writing per-table load notebooks, keeping staging and persistent-staging logic consistent across dozens of tables, re-deriving step order when lineage changes, porting the same load pattern to a second platform. What you manage instead: table metadata, a small set of binding rules, targeted section overrides where your shop genuinely differs, and test declarations that travel with the table they describe. Regeneration is a diff-previewed, idempotent operation that respects your hand edits, so adopting generation isn’t a one-way door.

Category contrast

Code-first transformation frameworks make every model a file you write and own. That’s real power, and real surface area: a thousand models is a thousand files of hand-maintained SQL, and the metadata describing them is a by-product of parsing the code. DeltaVault inverts the relationship: the catalog is primary and the code is rendered from it. We evaluated adopting a code-first framework as the generation layer and rejected it: the engine has to emit notebooks, stored procedures, and workflow definitions per platform dialect, select templates per table from governance rules, and accept customer overrides at the section level. A model-file approach fits none of that.

Standalone governance suites document a warehouse someone else builds. The catalog and the code have no mechanical link, so they drift, and the catalog loses the argument every time. Here the catalog can’t drift from the code because the code came from it.

Hand-built lakehouse pipelines are where most teams actually live: notebook twelve is a copy of notebook eleven with two edits, one of which was forgotten. The family context exists precisely so that “how we do staging” is written once and rendered everywhere.

Orchestration tools wire boxes together but have no idea what’s inside them. DeltaVault generates the boxes and the wiring from the same lineage graph.