Discovery and Imports

What it is

DeltaVault gets metadata out of source systems and into a governed catalog without forcing one acquisition path on everyone. A paired local agent inside your own network does the reaching, so DeltaVault’s cloud never connects to your database itself and you never open an inbound port. Where an agent cannot be installed, a one-shot command-line probe covers discovery and import, and if nothing can connect at all you run a provided SQL query or a generated script where the database lives and upload the result. And if your source is files in Azure Data Lake Storage rather than a database, you drop a sample file in the browser and DeltaVault discovers the schema from its contents.

Every path converges on the same place: a review screen that shows exactly what this import adds, changes, or removes before anything is committed. Imports aren’t silent catalog edits. They land as reviewed, branch-scoped changes. The review grid groups every object by its functional layer (source tables first, then the Landing, Stage, and Persistent Stage objects each one generates, each carrying the system columns the project’s per-layer settings resolve), and a companion canvas shows the same import as a diagram, source tables linked to their generated layers by directional grey-dashed lineage edges.

Re-running an import is safe by design. Tables and columns match by their fully qualified names and update in place instead of duplicating, and for file imports your curated types, key flags, and descriptions survive. A heuristic re-discovery never clobbers a deliberate edit.

In the free edition

Not included, and this page is the clearest statement of what the free edition trades away. Connections, verification, import from files or a live database, file-based discovery against cloud storage, the local agent and column profiling all create and read physical tables and columns. They are the catalog, and the catalog is the line the free edition is drawn on. A paying core organization also has projects and workspaces and the AI administration pages, but the catalog is the half of the product that touches a warehouse.

A free organization models the business without connecting a warehouse to it. The model it builds is real metadata, versioned in git the same way, and it is the same model a paying organization maps its sources onto later. Nothing has to be rebuilt to cross that line.

How it works

Connect and verify

Creating a connection starts from a connector catalog: a searchable grid of tiles, the way modern cloud data platforms present their own “add data” flows. The SQL Server connector covers three flavors in one form (on-premises SQL Server, Azure SQL Database, Amazon RDS for SQL Server) with SQL authentication or a Microsoft Entra service principal.

A saved connection carries a verification status the whole organization can see: Unverified, Verifying, Verified, or Failed. Clicking Verify runs a real probe against the target and persists the outcome, so “does this connection actually work” is answered once, not rediscovered by every engineer who touches it.

Verification runs through a local agent: a small Windows service paired into your organization with a code that expires after 15 minutes. The agent makes outbound HTTPS calls only, so you never open an inbound port. Once it shows online (about a minute: up to 30 seconds for the pairing to be picked up, then a heartbeat every 30), verify, discovery, and import all run from the browser through that agent, which reaches the database from inside your own network. DeltaVault never connects to the database directly.

Verification needs an online agent. If none is connected, the connection screen shows “No DeltaVault Agent is online” with a link to agent management. When a verify fails with a network-shaped error, a remediation card points you at the local-agent setup. Authentication failures (a wrong password, for example) are reported as such, because no infrastructure change fixes a credential problem.

Live discovery from source databases

Against a verified connection, discovery is two-phase. The cheap pass lists schemas and objects (seconds, even on a 2,000-object database) and you pick what matters in the browser. The heavy pass then pulls full-fidelity metadata for just that selection: tables, views, columns with complete type detail, nullability, identity and computed flags, defaults, primary keys, foreign keys, indexes, check constraints, and source descriptions.

Both phases run through the command-line tool or, on an agent-paired connection, through the local agent with no terminal at all. For machines that can’t reach DeltaVault (air-gapped networks, hosts where installing a runtime isn’t an option), there’s an offline variant: download a generated script, run it next to the database, and upload the compressed archive of results it produces. All three transports feed the identical pipeline downstream. On a project whose source is a verified database connection, the workspace Import Metadata dialog offers Agent discover-and-import as the default option, alongside delimited upload.

When using agent live discovery, import is selective: after the agent discovers the source, the user picks which schemas and objects to bring in using a filterable, checkable tree before the import run begins. For large catalogs, agent discovery lists the schemas first and presents the import screen immediately, then loads each schema’s tables on demand when you open or select it, with server-side table-name search across the whole database.

CSV and SQL metadata import

The lowest-friction path needs no connectivity at all: run one SQL query against your source’s information schema, export it as a CSV, upload it. The query is provided in-product: pick a SQL Server-family connection and a “Need the SQL script?” panel offers the exact extraction query with copy and download buttons.

The CSV contract is a single flat sheet (20 columns) covering tables, columns, primary keys, and foreign keys, including composite keys. Source data types are kept verbatim alongside a canonical type vocabulary of recognizable SQL names, so round-tripping to target platform DDL stays unambiguous. A raw type DeltaVault doesn’t recognize doesn’t abort the import: it’s flagged as unknown, recoverable in-app, and gates the commit until resolved.

The same path also reverse-engineers existing Databricks tables: a Databricks-typed connection resolves its own dialect, with the complex types (array, map, struct, variant) mapped first-class, so you can pull a deployed Unity Catalog schema back into the governed catalog.

File-based discovery against Azure Data Lake Storage

For file sources, DeltaVault discovers schemas from the data itself. Drop sample files (CSV, Parquet, JSON, or any text-based format, up to one gigabyte each, several at a time) and a four-step wizard walks Files, Structure, Columns, Review:

• Structure exposes what discovery guessed: delimiter, quote character, header row, rows to sample (default 200), and a minimum string length for when a small sample understates production width. A raw first-lines preview lets you check the guesses against reality. Parquet skips all of this; its schema comes from the file’s own metadata.
• Columns is an editable grid: types, lengths, precision and scale, nullability, primary-key and business-key flags, include toggles. Discovery’s inferred type stays visible as a recommendation with a one-click revert, and a table with no key selected gets a warning chip, not a blocker.

Discovery runs entirely in the browser. The file’s data never leaves your machine; only the schema metadata you approved is sent. On a project whose source is Azure Data Lake Storage Gen2, the workspace’s Import Metadata dialog offers only the file options that apply: a delimited upload, or “Discover from file”, which opens this wizard with the source connection already selected.

Re-importing a file whose table already exists is explicitly safe: existing columns are badged, their curated values are retained, and only structural changes (new columns, removed columns, moved ordinals) flow through.

Data profiling on the target catalog

Once metadata is imported and layers are generated, DeltaVault can capture the actual statistics of the data sitting in the physical tables. A user picks any lineage layer (landing, staging, or persistent staging), then chooses an execution path: run live through the paired local agent, which connects to the target warehouse and executes the profiling statements directly, or download a generated script, run it in the target environment, and upload the result grid as a CSV.

Either way, the result lands as a pending run. An editor reviews the captured statistics in a per-column grid (null percent, distinct count, min/max, mean, standard deviation, length statistics, top values) and accepts or discards them. Accepting replaces the table’s prior profile; nothing is written to the catalog until an editor commits.

Once accepted, statistics flow along the existing column lineage chain. Profiling landing once is enough: any downstream staging or persistent staging column that inherits through sourceColumns resolves the nearest upstream accepted profile at read time, annotated with where the statistics were measured, when, and whether a transformation was crossed (in which case they are flagged approximate). The Profiling tab on the table-detail page shows the resolved statistics for every column together with their provenance.

Statistics also reach Ask AI and AI skills automatically. The table family context includes a compact one-line summary per profiled column (null rate, distinct count, min/max, measured-on provenance). Unprofiled columns contribute nothing, so the token cost is zero until a profile exists. Chat and skill runs consume this context without any new wiring.

Review before commit

Every import ends at the same review surface: a tree grid that groups every object by its functional layer (Source, Landing, Stage, and Persistent Stage), with tables nested under their layer and each table’s columns flat beneath it (indexes and check constraints keep their own grouping). Every row is chipped Added, Changed, Removed, or Unchanged, and each layer’s chip rolls up the strongest change beneath it. Column rows also carry their data type and key flags: a primary-key and a business-key indicator. Changed rows expand to exact before-and-after field differences. Filter chips, a quick filter, and expand-and-collapse controls keep a several-hundred-row import scannable.

The diff is scoped to the import in front of you. File imports never claim a table was removed just because it wasn’t in this upload; database imports report removals only for their own source connection. The generated Landing, Stage, and Persistent Stage objects each occupy their own layer group, with a real status computed against the catalog and drillable to columns, so the whole pipeline an import will produce is visible before you commit it.

Committing is one action and all-or-nothing: every layer of every table lands on your current branch, or none of it does. The catalog assignments configured per layer on the project decide where each generated table goes, and the system-column settings configured per layer decide what system columns each generated table carries, resolved the way the project’s Settings page shows them, from the Global default through the platform and organization scopes to the project, with the most specific scope winning, so an import emits the system columns you configured rather than a fixed factory set. A layer’s system columns can also be cleared entirely (for example, when an extract method cannot add derived transformation logic); the generated tables for that layer then carry only their source columns. After the commit, a propagated system column that simply carries down the layers, such as a load timestamp or a record-source marker, is linked on the table’s Lineage tab to the same-named column on the layer above it; the per-layer change hashes each layer recomputes stay unlinked by design. Because the result is branch-scoped, an import is reviewable, auditable, and revertible through the same branch workflow as any hand-made edit.

Catalog structure can also be hand-authored directly, not only discovered or imported. A tree-aware New control on the catalog header adds the most likely child of the selected node (a catalog under a connection, a schema under a catalog, a table under a schema, a column under a table), and those hand-made additions land on your current branch and flow through the same commit review as any import.

Hand editing is not limited to forms. The catalog’s tables and columns grids are directly editable, in the spreadsheet idiom a data engineer already has in their fingers: cells are always live, every change saves on its own with no Save step, and ranges, fill, and clipboard work at any time. Each grid exposes every field DeltaVault tracks for that object through a column chooser rather than a fixed handful, and each user’s chosen columns, order, widths, and sort persist to their account, so a layout follows them between devices rather than living in one browser. Pasting a block of rows copied from a spreadsheet updates the cells it lands on, and on the columns grid, and on a schema’s tables grid, a paste that runs past the last row offers to create the extra rows, previewing them with per-row validation and creating all of them or none. Pasted data types resolve through the canonical type catalog and its aliases for the rows a paste creates, so a spreadsheet full of int and varchar lands as the canonical types there rather than as new spellings. Every one of those edits is a branch-scoped change that flows through the same commit review as an import.

Integration keys on import

An import does not have to stop at column lists. Turn on integration keys and DeltaVault designates the business key for you: it infers each source table’s key from its primary key, or from the business-key columns you flagged, and sets it through the same designation model a hand-built key uses. A single-column key is designated directly on the existing column; a key that spans several columns gets one derived business-key column built at the Staging layer, joined with the organization’s separator default, with an optional record-source prefix that keeps keys from different sources distinct. When integration keys are on, the Staging change hash is computed from that one designated column rather than from the separate raw columns, so the model loads and historizes on one clean key. Re-importing preserves the designation byte-identically: a table that already carries one keeps it exactly as set, so re-running the extraction never clobbers a designation you made or the import itself created.

Integration keys also line up across related tables. With apply-to-relationships on, every child table whose parent has an integration key gets a matching foreign integration-key column built from its foreign-key columns in the parent’s key order, and the relationship is repointed onto integration keys. Multiple foreign keys to the same parent and self-references are disambiguated with role-prefixed names, so a child that ships and bills to the same customer carries both keys without collision. The defaults are organization-wide and project-overridable, and any single import can override the key-building choice for that run. The integration-key SQL generates for Databricks and Microsoft Fabric.

Why it matters

Bootstrapping a catalog is where most metadata initiatives stall. Someone gets assigned to hand-type column lists into a tool, or to maintain a pile of one-off introspection scripts, and six weeks later the catalog describes the database as it was, not as it is. DeltaVault’s position: acquisition is the product’s job. You pick the transport that matches your network reality; the extraction queries, the type mapping, the inference heuristics, and the diff against what’s already cataloged are all handled.

The firewall fight disappears too. “Our security team won’t allowlist rotating cloud addresses” stops being a project blocker and becomes a dropdown choice: one stable address, a ten-minute token on a jump box, or an agent that needs nothing inbound at all.

And what you manage afterwards is the metadata, not the plumbing. Re-import after a source schema change and you review a handful of Changed rows instead of reconciling duplicates. The staging layers you’d otherwise hand-build are generated, previewed, and committed alongside their source, so the catalog starts ahead of the pipeline work instead of behind it.

Category contrast

Catalog-first governance suites typically acquire metadata with a cloud crawler that assumes direct network access to every source; when the firewall says no, the answer is a support ticket or a feature gap. Their imports also tend to write straight into the catalog: the crawl runs, the catalog changes, and nobody reviewed the delta.

Code-first transformation frameworks go the other way: you hand-author schema definitions in configuration files, which is precise but means a human retypes what the database already knows, and re-typing it again after every source change.

Hand-built lakehouse pipelines usually skip cataloging entirely and embed schema knowledge in notebook code, where it can’t be reviewed, diffed, or reused.

DeltaVault treats acquisition as its own workflow (live crawl, agent, command-line, script-and-upload, file sampling) and treats every import as a proposed change with a mandatory review gate, scoped diffs, and a branch-based commit. The category norm is either “automatic but unreviewed” or “reviewed but manual”. DeltaVault is automatic and reviewed.