Skip to content

Landing families

Five families write the landing layer. Three of them are the per-table shapes a landing table routes to, and this page documents those. The other two are ingest-workflow (databricks/landing/ingest_workflow.jinja, roots workflow and dv), which renders at workflow grain rather than per table, and ingest-piece (databricks/landing/pieces/cloud_files_table.jinja and jdbc_table.jinja), the include-only per-table piece those workflow-grain bodies compose. Both are indexed on the overview page and their field-by-field sections are still to be written.

A landing table renders through one of three shapes, decided in a fixed order. First the ingest runtime is routed from the table’s ingestion settings, the source connector type, and whether the source connection carries Lakeflow Connect settings. Then the file change-data-capture mode is rejected outright (the routed runtime would be change-blind for it). What remains routes: the Lakeflow Connect runtime renders through the run-scoped pipeline resource (never a per-table file), the declarative pipeline runtime takes the ingest-pipeline shape, and every other runtime takes the ingest-notebook shape with the runtime naming which notebook.

Templates: databricks/landing/ingest_pipeline.jinja

Roots: workflow, this, connection (conditional), source (conditional), sources, ingest, dv

The declarative landing pipeline: one pipeline per source connection, replicating source tables verbatim into the landing (bronze) catalog. Because a landing table has no audit columns of its own, there is no methodology root here.

FieldTypeMeaning
workflow.namestringlanding_ plus the source connection’s raw name; the literal landing_unlinked in a preview with no resolved source connection (a Build throws first).

The landing table being written: a table reference with the resolved ingestion block and a Python-safe function name on top.

FieldTypeMeaning
this.namestringLanding table name, raw.
this.fqnstringThe stored fully qualified name, else composed from the landing catalog, schema, and name.
this.columns[]arrayThe landing table’s own columns, ordinal-sorted, system columns included. Elements carry ulid (when present), name, dataType, isPrimaryKey, isNullable.
this.parts.catalogstringDestination catalog assigned to the landing layer (sample_catalog in an unpinned preview).
this.parts.schemastringDestination schema (sample_schema in an unpinned preview).
this.parts.namestringDestination table name.
this.ingestion.ingestModestringResolved ingest mode; defaults to snapshot.
this.ingestion.watermarkColumnstringThe landing column the high-water bound is read from; empty when none. A resolvable high-water-mark filter’s target column outranks the stored setting.
this.ingestion.sourceClausestringThe restriction appended unconditionally to the source read, complete with its leading WHERE; empty when nothing restricts. Python-string escaped. Exactly one of sourceClause and watermarkClause is ever populated.
this.ingestion.watermarkClausestringThe restriction that applies only once a previous run has landed rows, complete with its leading WHERE, carrying the {lower_bound} placeholder, brace-escaped for the formatted string it lands in.
this.ingestion.cdcKeyColumnsstring[]Change-data-capture key columns; empty by default.
this.ingestion.cdcSequenceColumnstringChange-data-capture sequence column; defaults to __$start_lsn.
this.ingestion.cdcTombstoneHandlingstringTombstone policy; defaults to delete_row.
this.pythonFunctionNamestringThe table name as a Python identifier: sanitized, with every hyphen replaced by an underscore.

Present only when the source connection resolved. Always the source connector, never the destination.

FieldTypeMeaning
connection.namestringSource connection’s raw name.
connection.connectionTypestringSource connector type (for example SQL Server).

source is present only when the source connection, source table, and source schema all resolved; it is omitted rather than emitted empty, so a template’s {% if source %} guard takes the branch a real Build would. sources is always present: a one-element list holding the identical object, or empty. Beyond the table reference fields, source.schema carries the source schema name; source.parts is deliberately absent (a source-system table has no honest catalog).

Templates: databricks/landing/auto_loader.jinja, databricks/landing/pyspark_jdbc.jinja, databricks/landing/pyspark_rest.jinja

Roots: this, connection, source (conditional), sources, environment, extract, dv

The per-table landing notebooks. Unlike the pipeline shape there is no workflow root, the connection root is always present (empty strings when unresolved), and the ingestion block is passed raw rather than resolved: the notebooks read change-data-capture settings with no defaults, and resolving them would change rendered bytes.

FieldTypeMeaning
this.ulidstringLanding table identifier (supplied on this shape, unlike the pipeline shape).
this.namestringLanding table name.
this.fqnstringThe stored fully qualified name, else composed from catalog, schema, and name.
this.columns[]arrayThe landing table’s own columns, ordinal-sorted, system columns included.
this.parts.catalog / .schema / .namestringDestination (bronze) catalog, schema, and table name.
this.ingestionobjectThe raw stored ingestion block, empty object when absent. Deliberately unresolved: fields may be missing and templates read them with no defaults.
this.ingestion.ingestModestring?Ingest mode, raw.
this.ingestion.watermarkColumnstring?Watermark column, raw.
this.ingestion.runtimeOverridestring?The stored runtime override that routed this table to a notebook.
this.ingestion.cdcKeyColumnsstring[]?Raw change-data-capture key columns.
this.ingestion.cdcSequenceColumnstring?Raw sequence column.
this.ingestion.cdcTombstoneHandlingstring?Raw tombstone policy.

Always present: what the notebook actually extracts from, resolved ahead of render. When a source table is linked, its schema and name; when none is, the landing table’s own (and the builder reports an unlinked-source-self-reference diagnostic so the fallback stays visible). A linked source wins even when its fields are empty strings. Both the raw and the snake-cased forms travel: raw feeds the notebook’s headers and the JDBC table option, snake-cased feeds the physical bronze table name and the quarantine path.

FieldTypeMeaning
extract.schemastringSource schema when linked; the landing table’s own schema when not.
extract.namestringSource table name when linked; the landing table’s own name when not.
extract.schemaSnakeCasestringThe schema through the snake-case naming model.
extract.nameSnakeCasestringThe name through the snake-case naming model.

Roots: source, sources, connection, environment

Section titled “Roots: source, sources, connection, environment”
FieldTypeMeaning
sourcetable ref?Conditional: present only when a source table resolved (no connection or schema check on this shape). Identical object to sources[0]. Carries the extra source.schema; parts is absent by design.
sourcesarrayAlways present: one element or empty.
connection.namestringSource connection name; empty string when unresolved (always present on this shape).
connection.connectionTypestringSource connector type; empty string when unresolved.
environment.catalogs.bronzestringThe landing (bronze) catalog the notebook writes into.
environment.schemaPolicystringTarget connection’s schema policy; defaults to permissive.
environment.landingRootstringTarget connection’s landing storage root; empty when unset.

Templates: databricks/landing/lakeflow_ingest_object.jinja

Roots: this, source, sources, connection

One Lakeflow Connect ingestion entry. All four roots are unconditional: a table with no source link refuses to render rather than emitting a partial context. Entries carry no column data by design, so this.columns and source.columns are always empty; the entry names tables, it does not enumerate them.

FieldTypeMeaning
this.namestringDestination table name.
this.fqnstringDestination catalog.schema.table.
this.columnsarrayAlways empty by design.
this.parts.catalog / .schema / .namestringThe destination split into parts.
this.keystringThe pipeline resource key: lfc_ plus the snake-cased connection, source schema, and source table.
this.ingestion.ingestModestringResolved ingest mode; snapshot default.
this.ingestion.watermarkColumnstringThe source column a high-water-mark filter restricts (name only, never composed SQL), else the stored setting, else empty. The filter outranks the stored setting.
source.namestringSource table name.
source.fqnstringSource schema.table.
source.schemastringThe source schema name (family extra; parts is absent by design).
sourcesarrayOne element, identical object to source.
connection.namestringSource connection name.
connection.connectionTypestringSource connector type as DeltaVault stores it (for example Azure SQL Database).
connection.connectorstringPrecomputed Lakeflow Connect connector key (sqlserver, oracle, postgres, mysql, salesforce, workday, servicenow, netsuite, googleanalytics); an unmapped type lowercases through.

The bundle’s bundle-lakeflow family loops enriched entries carrying these same four roots.