Generate Databricks Lakeflow pipelines

DeltaVault generates a complete Databricks Lakeflow pipeline from the metadata you already hold in your catalog: your source tables and their ingestion settings, your business model, and any Data Vault objects you have modeled. The result is a Databricks Asset Bundle, produced as a reviewable build snapshot in the app; committing the reviewed snapshot to your connected Git repository is an explicit second step. See Build and deploy your project for the Build, review, and commit flow; this guide covers what gets generated.

What gets generated

DeltaVault assembles a Databricks Asset Bundle under a databricks/ folder in your project repository:

databricks.yml, the bundle definition that names the pipelines and jobs.
resources/workflows.yml, the deployable resource definitions. Every generated job and declarative pipeline is written into this one document.
Source code under src/landing, src/staging, src/persistent_staging, src/data_vault, and src/data_mart, one file per object.
src/deploy/deploy_ddl.py, an ordered, idempotent notebook that creates the catalogs, schemas, and tables before the first ingest runs.
Governed source columns carry their classification and certification status into Unity Catalog as column tags, applied by deploy_ddl.py alongside table creation.
.github/workflows/deploy-databricks-bundle.yml at the repository root: a continuous integration workflow that validates and deploys the bundle on push.

Pipelines are grouped per generated workflow rather than per layer: each workflow that carries declarative work gets its own pipeline, declared alongside its job in resources/workflows.yml. See Orchestration workflows for how those workflows are built and ordered.

Once committed, the generated continuous integration workflow validates and deploys the bundle on push; you can also run databricks bundle deploy from your repository yourself, the same way you would deploy any hand-written Asset Bundle.

The medallion layers

DeltaVault models the classic medallion shape and writes the code for each available layer.

Bronze ingests raw data. DeltaVault picks the right runtime from your table’s ingestion settings: file ingestion with Auto Loader, a database source, a software-as-a-service application source, or a Lakeflow Connect managed pipeline. Every bronze table enables the Delta change data feed so downstream layers can stream changes.

For a SQL Server source, the runtime is chosen per table in priority order: an explicit runtime override on the table wins; otherwise a source connection with Lakeflow Connect configured handles supported tables as managed ingestion entries; every other table gets a declarative ingest pipeline generated for its ingest mode. Snapshot tables are re-read in full, watermark tables read incrementally above their watermark column, and change data capture tables read the SQL Server change table and apply the changes with AUTO CDC. Credentials are referenced through Databricks secret scopes, never written into the generated code.

Silver generates your data vault: hubs, links, satellites, the business vault, and reference tables.

Gold generates your data mart: dimensions as streaming tables that apply change data capture, and facts as materialized views that join to the current rows of their dimensions.

To understand how the keys, sequencing, delete handling, and history type are chosen for each streaming table, read Change data capture with AUTO CDC.

Data Vault objects

If you have modeled a Data Vault, the same generator emits it: hubs and links as insert-only loads, satellites as versioned inserts that keep full history, plus the business vault, reference tables, and point-in-time and bridge objects. See Data Vault on Lakeflow for what each object becomes.

Choose SQL or Python for the staging load

The combined staging load can be generated in SQL or in Python. The choice is the Notebook Language setting in the Data Loading Behavior group of Target Platform settings; see Build artefacts in Target Platform settings.

SQL writes the combined staging load as SQL.
Python writes the same load as Python.

Both choices do the same work in one notebook, the staging load and the persistent staging append, so the setting changes how that notebook reads and not what it produces. A table bound to a legacy template keeps whatever that template generates. Nothing else follows this setting: data mart dimensions and facts are always generated as declarative SQL, and Data Vault objects always as Python load notebooks.

Certified metadata in Unity Catalog

If you classify and govern your source columns in DeltaVault’s business model, described in Data governance overview and Governing source data inline, that governance does not stay confined to the app. Every source column mapped to a governed business attribute carries its classification into the generated DDL as native Unity Catalog column tags, applied by deploy_ddl.py alongside CREATE TABLE for the tables it creates.

A governed column gets one tag per classification scheme applied (for example sensitivity set to confidential), a business_attribute tag naming the mapped attribute, and a governance_certified tag set to true once that attribute’s definition has been approved. An unmapped or unclassified column gets no tags at all. The result is queryable straight out of Unity Catalog itself, wherever tags are surfaced today, Catalog Explorer, information schema queries, or your own governance tooling, without anyone hand-tagging a single column in Databricks.