Staging families

Three families write the staging layer. The staging family carries the loads themselves on all three platforms; step-notebook is the workflow-step shape the Databricks persistent staging full load renders through; fabric-with-logging is the Fabric notebook variant with run logging.

staging-family

Templates: databricks/staging/python/combined_load.jinja, databricks/staging/sql/combined_load.jinja, fabric/staging/notebook/delta_load.jinja, fabric/staging/notebook/full_load.jinja, fabric/staging/warehouse/delta_load.jinja, fabric/staging/warehouse/full_load.jinja, snowflake/staging/source_to_staging.jinja, snowflake/staging/delete_detection.jinja, snowflake/persistent_staging/load.jinja

Declared roots: this, source (conditional), target (conditional), refs (conditional), extraction, behavior, dv

Still supplied, but out of contract: sources, build, stagingPrefix, platform, fabricComputeModel. The declaration narrowed to the seven above because none of this family’s nine bodies read these five, and the declaration is what an organization author is told is available. The projection still supplies all five to every render, so a body that already reads one keeps rendering correctly; the authoring surface reports it as an out-of-contract root, which is a warning and never a broken document. Treat them as available but unsupported, and prefer a declared root in anything you write now.

The staging loads across all three platforms. The engine first resolves the table’s whole family (its source, landing, staging, and persistent staging layers with full modeling detail), then projects that family onto the loading vocabulary: which layer plays this and which plays source is decided per template path, so the same table renders differently through different templates. Your templates read this and source; they never see the internal family or pick layers themselves.

Which layer plays which role

Template	this	source	Also writes	Load model
`databricks/staging/python/combined_load.jinja`	staging	landing	persistent staging	`combinedLoad`
`databricks/staging/sql/combined_load.jinja`	staging	landing	persistent staging	`combinedLoad`
`fabric/staging/notebook/delta_load.jinja`	staging	source		`fabricLakehouse`
`fabric/staging/notebook/full_load.jinja`	staging	source		`fabricLakehouse`
`fabric/staging/warehouse/delta_load.jinja`	staging	source		`fabricWarehouse`
`fabric/staging/warehouse/full_load.jinja`	staging	source		`fabricWarehouse`
`snowflake/staging/source_to_staging.jinja`	staging	nearest upstream (landing when present, else source)		`snowflakeStaging`
`snowflake/persistent_staging/load.jinja`	persistent staging	staging		`snowflakeStaging` (columns from source)
`snowflake/staging/delete_detection.jinja`	persistent staging	staging		`snowflakeStaging`

Roles are registered per exact template path. An unregistered path refuses to render, and a load-model row whose this or source layer the family could not resolve refuses too: no silent hollow loads.

Roots: this, source, sources, target

Each is the full layer node: every family column, the key partitions, the system-column role resolution, and naming, plus name, fqn, and ulid on top. source is the role table above’s source layer and never carries a load model; target is present only on the two Databricks combined loads and only when the family resolved a persistent staging layer.

Field	Type	Meaning
`this.name`	string	The layer’s table name (parts.name).
`this.fqn`	string	catalog.schema.name; two-part schema.name when the catalog is empty.
`this.ulid`	string?	The layer’s table identifier.
`this.layer`	string	Which member layer this node is: source, landing, staging, or persistentStaging.
`this.parts.catalog` / `.schema` / `.name`	string	The name parts, dialect-neutral.
`this.columns[]`	family column[]	Every column of this layer.
`this.businessKeys[]`	family column[]	Data columns flagged primary or business key. Column objects here, not name strings: the staging filters consume them whole.
`this.diffAttributes[]`	family column[]	Non-key data columns.
`this.systemColumns[]`	family column[]	System-managed columns.
`this.systemColumnRoles`	object?	Conditional: which column carries which system-column role. Absent on source layers; readers treat absence as empty.
`this.systemColumnRoles.definitionNameByRole`	object	Role to the platform catalog’s expected column name, whether or not the table has it.
`this.systemColumnRoles.columnNameByRole`	object	Role to this table’s actual column name, for the roles it has. What renders.
`this.systemColumnRoles.roleByColumnName`	object	Inverse lookup, column name to role.
`this.hashNullValue`	string?	Conditional: Fabric layers only: wraps each hash-input cast in COALESCE(…, ‘<value>’).
`this.businessKeyNormalisation`	object?	Shared business-key normalization (trim, upper, collapse whitespace); absent means none.
`this.businessKeyAddRecordSource`	boolean?	Import automation flag: include the record source in the business key.
`this.businessKeyPadToMaxIdentifier`	boolean?	Import automation flag: pad key constituents to the maximum identifier length.
`this.maxIdentifierLength`	number?	The pad width for the flag above.
`this.primaryKeyColumnUlids`	string[]?	Ordered primary-key definition; orders the undesignated business-key cast list.
`this.businessKeyColumnUlid`	string?	The designated business-key column.
`this.extractionFilters[]`	array?	This layer’s raw row-selection predicates, uncomposed (composition happens under extraction.filter).
`this.tableType`	string?	This layer’s own table type (for example Reference).
`this.namingName`	string?	Resolved naming name; set on the source node only.
`this.keyEmissionGroups[]`	array?	Conditional: hub and link hash-key emission groups, on the staging node only, and only when the source has two or more distinct far ends. Elements: name (pattern-derived column, for example customer_hk), columnNames (ordered source columns composing the hash), kind (hub or link), farEndBaseName (hub groups only), farEndSlotSql (precomposed key slot; absent means the raw cast list renders identically).
`this.custom`	object?	Org-defined custom attribute values for this layer’s table.
`this.load`	object?	Conditional: the precomputed load model (shapes below). Present on load-model rows when no system-role requirement gap blocked it.
`target.load`	object?	Conditional: the persistent staging insert model: insert[], select[], keys[] (aligned with this.load.keys), and roles.loadTimestamp / recordSource / auditId / hashKey / hashDiff, each {name, quoted} when resolved.

The four load models under this.load

combinedLoad (the two Databricks combined loads):

Field	Type	Meaning
`load.insert[]`	{name, quoted}[]	Work-table and staging-insert columns, emitted order; quoted is the sanitized, backtick-quoted form.
`load.select[]`	string[]	SELECT expressions reading from the landing table, aligned index-for-index with insert.
`load.keys[]`	{name, quoted}[]	Join and window key columns, aligned with target.load.keys.
`load.roles.loadTimestamp` / `.recordSource` / `.auditId` / `.hashKey` / `.hashDiff`	{name, quoted}?	The staging system-role columns, each present when the role resolved.
`load.work` / `.workCurrent` / `.workDelta` / `.workUnchanged`	{name, ref}	The work table and its current, delta, and unchanged companions: bare name plus the delimited reference (the same strings as the `refs` root). These four are session temporary tables, so `ref` is the backtick-delimited name alone and never an `IDENTIFIER()` composition: a temporary table is not a catalog object, so there is no catalog to bind.
`load.parameters.declarations[]`	{name, default}[]	Run-time bindings the notebook declares, in order (order is load-bearing): catalog_landing, catalog_staging, catalog_persistent_staging (only when that layer resolved), row_audit_id. Defaults are pre-escaped design-time fallbacks.
`load.parameters.displayTimeZone`	string	Pre-escaped display time zone feeding `SET TIME ZONE` (a literal, not a declaration).

snowflakeStaging:

Field	Type	Meaning
`load.insert[]`	{name, quoted}[]	Emitted columns, ordinal order; names double-quoted verbatim, never sanitized (sanitizing would change bytes).
`load.select[]`	string[]	Per-column SELECT expressions, aligned with insert.
`load.businessKeys[]` / `load.nonKeys[]`	{name, quoted}[]	The `MERGE` match columns and `MERGE` update columns.
`load.businessKeyCount` / `load.nonKeyCount`	number	Precomputed sizes (an empty array is truthy in the template language, so counts are what you branch on).
`load.safeName` / `load.sourceSafeName`	string	This table’s and the upstream table’s sanitized names.
`load.literalName`	string	This table’s name with single quotes doubled, for the stored procedure’s `RETURN` message literal.

On snowflake/persistent_staging/load.jinja the column model is built from the source (staging) layer rather than this: the load merges staging into persistent staging.

fabricWarehouse: identical shape to snowflakeStaging minus literalName; quoting is bracket delimiters over sanitized names.

fabricLakehouse:

Field	Type	Meaning
`load.ddl[]`	aligned column[]	One finished `CREATE TABLE` line per column (see the aligned DDL column), over every column of this, in context order.
`load.alignment.nameWidth` / `.dataTypeWidth`	number	The pad budgets the lines were built with.
`load.insert[]`	{name, safe, quoted}[]	Emitted columns, ordinal order. The `CREATE TABLE` block emits `safe`, because Spark SQL takes undelimited identifiers there; expression positions take `quoted`.
`load.select[]` / `load.selectEscaped[]`	string[]	Per-column SELECT expressions, and the same with every brace doubled for the notebook’s Python formatted string.
`load.businessKeys[]` / `load.nonKeys[]`	{name, safe, quoted}[]	`MERGE` match and update columns.
`load.businessKeyCount` / `load.nonKeyCount`	number	Precomputed sizes.
`load.safeName`	string	This table’s sanitized name.

Root: refs (conditional)

Present only on combined-load rows when the model built. The three catalog members are fully composed IDENTIFIER() expressions binding the catalog at run time. The four work members are not, and the difference is deliberate: the work tables are session temporary tables rather than catalog objects, so each of those is the backtick-delimited name alone. The IDENTIFIER() splice exists only because a catalog name cannot be parameterized in a definition statement, and a reference with no catalog in it has nothing to parameterize.

Field	Type	Meaning
`refs.landing`	string	Reference to the landing table, via the catalog_landing binding.
`refs.staging`	string	Reference to the staging table, via catalog_staging.
`refs.persistentStaging`	string?	Conditional: reference to the persistent staging table, when that layer resolved.
`refs.work` / `.workCurrent` / `.workDelta` / `.workUnchanged`	string	The work table and its companions: the delimited name alone, with no catalog or schema qualification, because these four are session temporary tables. The names carry a `temp_` prefix.

Roots: extraction, behavior, stagingPrefix, platform, fabricComputeModel

Field	Type	Meaning
`extraction.sourceSystemType`	string	The source system’s type.
`extraction.isPushdownExtraction`	boolean	Whether extraction pushes down to the source system.
`extraction.recordSource`	string	The record-source value stamped on loaded rows.
`extraction.recordSourceName`	string?	The resolved record-source name (catalog, then connection, then source connection name). Read sites default to empty.
`extraction.fileFormat`	string	The source file format.
`extraction.filter.sql`	string	The composed row-selection predicate for the layer being read as source, without a leading `WHERE`, relaxed on first run (a high-water-mark reference with no floor wraps the predicate in an `IS NULL OR`). Recomposed against the layer actually playing source.
`extraction.filter.isEmpty`	boolean	True exactly when the composer produced nothing. Branch on this, not on string emptiness: the first-run relaxation applies to sql only.
`extraction.filterEscaped`	string	filter.sql with every brace doubled, for the Fabric Lakehouse notebooks’ Python formatted strings.
`behavior.displayTimeZone`	string	Resolved display time zone.
`behavior.stagingMode`	`all` or `delta`	What the truncate-and-load staging table holds after a run: delta (default) keeps only genuine new versions versus persistent staging, all keeps every landed row. Persistent staging is always delta-guarded regardless.
`behavior.notebookLanguage`	`sql` or `python`	The generated load notebook’s language; sql default. Per-table override, then organization or project default.
`stagingPrefix`	string	The literal prefix of your staging table-name pattern (for example stg_), used when deriving source temp-table names.
`platform`	string	The target platform token: databricks, fabric, or snowflake.
`fabricComputeModel`	`lakehouse` or `warehouse`	Derived from the target connection’s system type; meaningful only when platform is fabric. Selects the Lakehouse Spark path versus the Warehouse Transact-SQL path.

The build root is the shared build settings record.

step-notebook

Templates: databricks/persistent_staging/full_load.jinja

Roots: workflow, step (conditional), source (conditional), sources (conditional), this, methodology (conditional)

The workflow-step notebook shape: the Databricks persistent staging full load renders through it, and it is the base the data-tests family extends. Its rule is strict opt-in: every group beyond workflow and this appears only when the producer supplied it, and the builder never invents placeholder values.

Field	Type	Meaning
`workflow.name`	string	Workflow name. Always present.
`workflow.ulid` / `.kind` / `.fqn`	string?	Conditional: identifier, kind (ingest or transform), and fully qualified name, each only when supplied.
`this`	table ref	The table being written: a table reference, plus the resolved ingestion block and pythonFunctionName when the producing family reads them (see ingest-pipeline).
`step.name`	string	Conditional: the workflow step, when supplied; with optional ulid, layer, stepType (lakeflow-pipeline, notebook, stored-procedure), dependsOn.
`sources[]`	table ref[]	Conditional: upstream feeds, when supplied.
`source`	table ref	Conditional: present only when sources is non-empty; the identical object to sources[0].
`methodology.auditColumns[]`	array	Conditional: audit columns the persistent staging load emits: name, expression (the Spark SQL that produces it), and dataType where known.
`methodology.key` / `.layerPatterns` / `.surrogateKeyStyle`	varies	Conditional: the active methodology key, per-layer naming patterns, and surrogate key style, when the producer passes them.
`connection.catalog` / `.schema`	string?	Conditional: destination catalog and schema (most producers); the ingest shapes carry the source connector’s name and connectionType here instead.
`project.ulid` / `.name` / `.slug`	string	Conditional: the owning project, when supplied.
`generation.generatedAt` / `.generatedBy` / `.templatePath`	varies	Conditional: artifact header stamps: epoch milliseconds, producer sentinel, and the template path that produced the artifact.

fabric-with-logging

Templates: fabric/staging/notebook/with_logging.jinja

Roots: this, source, sources, pipeline

The Fabric staging notebook with run logging. It sits outside the staging role registry on purpose: pipeline.loadType is a build-time parameter (the Build chooses between the full and delta notebook with it), so the staging projector cannot produce this context. Its documented design exception: the upstream is a string the author typed, not a resolvable catalog table, so source carries only the typed reference.

Field	Type	Meaning
`this.name`	string	Staging table name.
`this.parts.catalog` / `.schema`	string?	Destination lakehouse and schema; when absent, the widget defaults below substitute and an info diagnostic says so.
`this.columns[]`	array	Every column: name, dataType, length, precision, scale, isNullable, isPrimaryKey, and transformation (a SQL expression selected instead of the column name when set).
`this.load.lakehouseName` / `.schemaName`	string	Widget values, always populated: the parts when present, else the literals lakehouse and dbo.
`this.load.safeName`	string	The table name, sanitized.
`this.load.ddl[]`	aligned column[]	The finished `CREATE TABLE` block (see the aligned DDL column).
`this.load.alignment`	{nameWidth, dataTypeWidth}	The two pad budgets the block’s lines were built with. One object, not an array.
`this.load.columns[]`	{name, safe, quoted}[]	Every column, raw plus sanitized plus delimited, in context order.
`this.load.primaryKeys[]` / `.nonPrimaryKeys[]`	{name, safe, quoted}[]	`MERGE` match and update columns.
`this.load.primaryKeyCount` / `.nonPrimaryKeyCount`	number	Precomputed sizes.
`this.load.selectEscaped[]`	string[]	One SELECT expression per column for the full-load branch (the transformation when set, else the sanitized name), braces doubled for the Python formatted string.
`source.fqn`	string	The upstream reference exactly as the author typed it.
`source.fqnForFString`	string	The same with braces doubled, for the notebook’s formatted string. Named separately so the Context pane still shows what was typed.
`source.name` / `source.columns`	string? / array	The upstream name when supplied; columns empty by design (the upstream is unresolvable).
`sources`	array	One element, identical object to source.
`pipeline.loadType`	string	The build-time load type: delta selects the notebook’s delta branch, anything else the full-load branch.