Mapping and Lineage

What it is

Every column in a DeltaVault catalog knows where it came from. Whether metadata arrives through an import, a file discovery, or a hand edit, each derived column records which source columns feed it and what transform sits between them. That record isn’t a report generated after the fact; it’s part of the metadata itself, stored on the column, drawn as a diagram, and committed to Git alongside everything else.

Lineage in DeltaVault is column-grained and traversable. Click a column in the Lineage tab and its entire path lights up: every ancestor upstream, every descendant downstream, across as many hops as the pipeline has. Tables map to business entities, so the technical catalog and the business model stay linked rather than living in two separate tools.

All of it rides on branch-based editing. You change mappings on a branch, review the exact differences on a commit screen, and the result lands in GitHub as human-readable YAML with one file per column. The lineage history is a Git history.

In the free edition

Included: branches, review, commit, and the git history. Branch-based editing is granted to the free edition whole, with git sync and GitHub push included, and so is the per-row version history that the branch model produces. The YAML in the repository is the same YAML.

Not included: everything that needs a column to point at. Column mappings, column-level lineage tracing, impact analysis across hops, entity-relationship diagrams over the catalog, relationship discovery and mapping the catalog to the business model all rest on physical tables and columns. The free edition has no catalog, so it has no source side to map from and no column path to trace.

What survives is the half this page treats as the mechanism rather than the subject: the model is versioned, reviewed and committed like code, and that is true on every tier.

How it works

Column mappings, captured and editable

Imports populate column-to-column mappings automatically: each derived column carries a list of the source columns that feed it, with an optional transform expression per mapping. The columns grid can show a “Source columns” column, hidden until you add it, summarizing how many mappings a column carries; the column editor’s “Derived from” section is where those resolve to readable schema.table.column labels. You can also maintain mappings by hand: the column editor includes a source-column picker that lets you add, remove, or change mappings and their transforms, picking the table and column yourself from anything in scope.

Tracing a column across the whole pipeline

The Lineage tab on every table renders a column-level diagram: each table is a box listing its columns (with primary-key, business-key, and identity markers), its header tinted by the table’s functional layer, and rounded orthogonal edges connect individual source columns to individual target columns, each anchored to its column row and pointing downstream. Each edge carries its transform as a label rather than hiding it behind a hover. Columns that feed nothing and come from nothing render dimmed, so gaps are visible at a glance.

Selection is transitive. Click a column and the graph walks the full dependency chain in both directions: all ancestors, all descendants, and only the edges actually on that path. An edge that bypasses the selected column stays dim, so what lights up is genuinely that column’s lineage and nothing else. The graph is bounded at 500 tables with a truncation notice, which in practice means whole pipelines fit.

You can act from where you’re reading. Right-click a table header for Edit, Map and match, Add or edit a business key, and Delete. The per-column Ask AI opener now lives in the Ask AI pane in the top navigation rather than on the column’s own menu. The same impact question works from the table’s header too: the Analyze Impact skill assesses the downstream consequences of changing a column before you change it.

Lineage on the canvas

The workspace and project canvas derives table-to-table flow edges from the same column mappings: one edge per table pair, a rounded orthogonal connector that can carry its column count as a label, off by default and turned on per canvas type, with every node tinted by its functional layer. Derivation chains render as chains, not fan-outs: a CSV import that produces source, landing, staging, and persistent-staging tables draws Source → Landing → Stage → Persistent Staging, each layer pointing at its immediate predecessor. Layout is automatic, left to right along the dependency graph.

The project canvas is not read-only: it shows every table the project uses and runs the same actions as a workspace canvas (Data Vault and the rest) at project scope. Project-scope actions write to a Default workspace DeltaVault maintains for each project, so a generated set of tables lands at once on the project canvas, the project’s Tables tab, and the Default workspace. Deleting a table on the project canvas removes it from the project (cascading into the workspaces that use it), never deleting the underlying metadata.

A canvas you can configure

The canvas reads cleanly by default: tables are color-only nodes tinted by their functional layer, the object kind shows on hover instead of as a badge, and connectors are drawn as clean lines without text labels or endpoint dots, so the shape of a pipeline is legible at a glance. This canvas answers one question, how data flows, and stays on it: foreign key relationships are drawn on the catalog’s entity relationship diagram instead, so neither reading crowds the other out of the picture.

The appearance itself is configurable, per kind of canvas. A platform administrator works from one Canvas appearance page: universal layer and object-kind colors (from a fixed palette of named swatches), the default visibility of each connector layer, and a settings tab for each of the six canvas types (Lineage, Column Lineage, Data Vault, Dimensional, Entity-Relationship, and View Designer), so table-level and column-level lineage are styled independently. Each canvas type carries its own node decorations, connector styling (marker style including crow’s foot notation, connector style with Bezier, Smooth step, Chamfered, or Straight lines, marker size and line width as separate dials, so an arrowhead can grow without thickening the line it sits on, line style, each overridable per connector type, and a color per connector type picked from an expanded gallery of hues), and layout algorithm (grid, layered, orthogonal, radial, or organic, plus direction and spacing), with a live per-type preview rendered from the same resolvers the real canvases use. Out of the box the types ship distinct looks: lineage canvases stay layered left to right, data vault and entity-relationship canvases route orthogonally, dimensional models lay out radially, and entity-relationship diagrams draw crow’s foot markers with relationship labels. The colors restore a high-contrast layer palette so Source, Landing, Staging, Persistent Staging, Data Warehouse, Data Vault, and Data Mart each read as a distinct band, and the saved appearance applies across organizations.

Entity-relationship diagrams on the catalog

Selecting a connection, catalog, or schema in the catalog opens a Canvas tab: an entity relationship diagram of that scope’s tables drawn with crow’s foot connectors, each node showing the table’s name and its key columns (primary key, business key, and the columns on either side of a relationship). Relationships that cross the scope boundary keep their far table visible as a greyed neighbor, toggleable off, so a cross-schema foreign key never dangles. A table selector prunes the diagram to the tables that matter, the pruned view persists per browser, newly imported tables default to visible, and very large scopes ask before first render instead of drawing a thousand-table diagram unprompted.

The diagram is also a launch surface. Discover keys and relationships runs scoped to exactly the visible tables, or to a node selection, with a layer-aware scope selector in the launch form and a choice of which passes to run; the review opens in the docked Ask AI pane beside the diagram, so the canvas stays visible while pending relationship candidates render on it as dashed edges (and proposed keys as ghost chips) that restyle the moment you accept or reject, and clear once you apply; and a right-click records a relationship by hand, kept within a single catalog by construction.

Relationships are authored, not only discovered. You draw one by dragging from one table’s connect handle onto another, by right-clicking a table and choosing Add Relationship, from the Relationships tab on a table’s detail pane, or from the foreign key section of a single column’s editor, and imports and AI discovery write the same records through the same door. Each relationship carries a stable identity of its own, so two foreign keys from one table to the same parent, a ship-to and a bill-to both pointing at an address table, stay two relationships rather than collapsing into one, and re-importing a connection reuses them instead of renumbering them. Where the source system names its constraints, that name rides through the import and is shown beside the relationship it belongs to. Downstream the same records drive the view designer’s join suggestions and the Data Vault accelerator’s hub and link classification, so the work of getting relationships right is repaid in generation rather than spent on documentation.

AI relationship discovery

DeltaVault can propose foreign-key-style relationships between tables without you drawing them by hand. The relationships pass of the Discover keys and relationships skill examines each table’s columns, names, types, and profiling signals (value distributions, how many distinct values a column holds, and where sample data overlaps between two columns) to judge whether one table’s column plausibly references another. A deterministic pre-filter narrows the field first: it rules out table pairs that cannot be a parent and child relationship on structural grounds alone, so the model spends its reasoning only on pairs worth considering. The pre-filter works before any keys exist: on a freshly imported schema with no primary or business keys designated yet, it falls back to key-shaped columns (a column named id, a name ending in id, or endings like code, key, number, and reference) matched by name and type, so discovery produces candidates on day one instead of waiting for key design. You can run it across every table on a canvas for a first pass, or narrow the scope before launching to a single functional layer or a hand-picked set of tables. Landing-layer tables are listed but never scanned, since Landing mirrors raw arrivals before any keys or relationships exist to find.

Each proposed relationship arrives with a confidence score and a short written reason (the naming, types, or profiling signals that led the skill to suggest it), and the results form a candidate list you review before anything changes. Nothing writes to the catalog until you accept a candidate, individually or in bulk, and acceptance happens on a branch like every other change. Accepting a relationship records it on the source tables and, by choice at apply time, promotes it forward to the same tables’ staging and persistent staging counterparts. Each of those two layers is its own toggle beside the apply button, ticked by default, so you can skip either layer or both and record on the source layer only. Each layer also gets its own preview showing exactly what will be created there: a layer with no matching table is simply left out, and a layer whose columns cannot be carried forward shows a notice naming what it skipped. The landing layer is never a promotion target: landing tables carry no relationships, though they still anchor the trace that connects a staging table back to its source.

Two refinements keep relationship work fast and safe on the canvas. Creating a relationship by hand is catalog-scoped: a relationship stays within one catalog and its owning connection (schemas within a catalog may differ), enforced in the picker, on drag, and where relationships are saved, so a cross-catalog pair cannot be recorded. And discovery is one click from the relationship dialog itself: Suggest relationships runs the skill scoped to the table you are editing and surfaces its proposals as reviewable dashed edges on the canvas, without a trip to a separate launcher.

Building a view on the canvas

Views are built where the data already lives: on the canvas. Select the tables you want, right-click Create View, and DeltaVault works out the joins for you, including history and temporal overlap (effective-dated from/to ranges) and an as-of setting. Related tables surface as one-click additions. The join type, key pairs, and temporal columns are all editable when the suggestion needs a human correction, each carrying a confidence signal (derived, heuristic, or manual). The canvas reads as a schema diagram: each source table is a column-level node that collapses to its key and join columns with primary-key and foreign-key markers, or expands to every column with an in-node add control on each row, and the columns you pick build up an always-visible view node on the canvas (the output, distinct from the source tables it draws from). Grey connectors (solid for an inner join, dashed for a left one) carry a single arrowhead toward the referenced key, re-anchoring as you drag a table. Columns are projected with explicit source-to-target mappings, renamed and reordered inline, and the saved view flows through the same branch, review, and commit path as the rest of your metadata. Beyond one-to-one column picks, you can add computed (derived) columns written as SQL expressions: DeltaVault resolves each expression back to the exact source columns it reads, so the view records column-level lineage even for calculated fields, and the same holds for macro-bearing expressions: macro arguments contribute their column references to lineage before the macro is ever expanded. Selecting a view column lights up the exact source columns it reads on the diagram (rows on expanded tables, the table header when collapsed), so column-level lineage is visible while the view is being built, and solid blue chamfered flow connectors trace every source table into the view node. Transforms are authored inline on each projected column with macro-aware SQL autocomplete, so reusable macro logic drops into a column expression without leaving the designer. A structured WHERE builder narrows the rows, combining conditions with AND and OR, and any join or filter can drop to hand-authored SQL and back when the structured form is not enough. The designer generates the full CREATE VIEW statement and shows it live as you build, so what you save is what runs. Views can also start from the canvas chat: describe the view in natural language and Ask AI proposes it, opening this same designer pre-populated for review (see the AI architecture reference).

A view’s dependency link survives a rename

A view’s saved configuration stores each referenced table’s stable identifier alongside its fully qualified name: the table the view starts from, every table it draws in as an input, and each side of every join. Renaming a source table changes its fully qualified name but never its identifier, so the dependency link between the view and its source tables is never lost. Those same identifiers are what DeltaVault checks to link a table to the views built directly on top of it: a view counts as one of those dependents as soon as it references the table this way, whether the table contributes a projected column, only a join key, or only feeds a filter.

Mapping the catalog to the business model

A table links to a business entity directly from its Details tab. The picker searches entities by name and technical name; once linked, the entity’s technical name becomes the table’s effective business name for downstream consumers such as the Data Vault accelerator, with a free-text business name as the fallback when no entity fits. The Enrich Table Metadata skill proposes a comment, a business name, and a matching entity as a preview you apply or discard; nothing changes until you apply it. At the column level, the metadata model carries business-attribute references on each column, serialized into the same reviewable YAML as the rest of the catalog.

Tables also anchor directly to projects, and one table can belong to several projects at once: a shared source feeding two delivery projects shows up in both, and reorganizing or deleting a workspace never silently drops a table out of its project. Editors can add tables straight to a project from its Tables tab (Add tables), without going through a workspace; workspace membership remains a separate, narrower scope on top of the project anchor.

Branches, review, and commit

Metadata edits happen on branches. When GitHub is connected, the main branch is protected: you create a feature branch, work in isolation, and two people on different branches never see each other’s in-progress edits. Switching to a branch loads that branch’s actual Git state. Pulling main into your branch performs a real three-way merge, and genuine conflicts surface in a resolution screen instead of being silently rebased away. Teams without source control connected still get the same editing and history machinery against a single main line.

The Review & Commit screen shows every pending change as a file tree with per-entity before-and-after diffs, loads instantly from a cached copy of the committed state, and lets you revert individual changes before committing. The preview paths match the committed paths byte for byte: what you review is what lands. Committing pushes the YAML to GitHub and opens a pull request.

History is first-class. Every change records who made it, when, on which branch, with a structured before-and-after snapshot. Each entity has its own history timeline, and an organization-wide Activity feed groups changes by day with filters for time range and action type. Expand any row to see the field-level diff.

A Git history a human can read

The repository layout mirrors the catalog: each table is a folder under its connection, catalog, and schema, and every column is its own YAML file inside it. Filenames are fully qualified names, not opaque identifier strings, so a pull request reads like a map of what changed. This matters on wide tables: a 200-column import used to rewrite one giant table file whenever any single column changed; with per-column files, the diff for a one-column edit is one small file, and Git blame for a column points at that column. References inside the YAML carry explicit, labeled identity fields (stable identifier, fully qualified name, source-system identifier) rather than encoded strings, so a reviewer can tell at a glance what a mapping points at. Renaming a connection, catalog, schema, or table moves every descendant file in the same commit. From a table’s Details tab you can also move it to a schema under a different connection, re-homing the table and its columns under that connection in the catalog.

Why it matters

Lineage is usually the thing you reconstruct during an incident: a column changed three pipelines upstream, a report broke, and someone spends an afternoon grepping notebooks to find out why. The plumbing is exactly the work DeltaVault removes: keeping a separate lineage tool synchronized, hand-drawing mapping documents, writing the “where does this field come from” wiki page that’s stale by Friday.

Here the mapping is the metadata. You manage the catalog, and lineage, impact tracing, the canvas, the business-model links, and the Git history all fall out of the same records. Before you rename a column, you click it and watch its whole blast radius light up; you ask the impact question from the same screen. And because every mapping edit goes through a branch and a reviewed commit, the lineage record carries the same trust as code: you know who changed it, when, and what it looked like before.

Category contrast

Catalog-first tools that added a chat panel typically harvest lineage after the fact, parsing query logs or warehouse history into a read-only graph. It’s an observation of what already ran, often table-grained, and there’s no path from the diagram back to editing the metadata it describes. DeltaVault’s lineage is declared in the metadata that drives generation, column-grained, and editable from the same surface.

Code-first transformation frameworks get table-level dependency graphs from model references, which is genuinely useful, but column-level lineage is an add-on, and the metadata itself has no review surface beyond raw text diffs. DeltaVault gives the metadata its own review-and-commit workflow with structured before-and-after diffs, while still landing everything in Git.

Standalone governance suites treat lineage and business glossaries as documentation maintained beside the pipeline, with stewards doing the synchronizing. Hand-built lakehouse pipelines mostly keep lineage in people’s heads. In both cases the lineage drifts from reality because nothing forces them to move together; in DeltaVault they can’t drift, because they’re the same record.