Integration by exception
Many definitions of the silver layer charge integration as an entry fee. Every source is matched and merged into one Customer before gold may read a row, and everyone waits for it.
Our post Show Me Your Silver Layer charged that fee too. It asked silver to hold “one governed Customer fed by every system that knows one,” and we still want that Customer, just not first.
So make silver layer integration the last step, taken only where a business requirement needs it, and build it beside the tables, never instead of them. That’s AIMS, short for AI-ready, Meaning-first Silver. Model the meaning, preserve the source and integrate by exception.
Keeping each source’s shape isn’t the failure described in our post No silver layer, no lakehouse. That failure is silver with no decisions written down. AIMS writes every decision down as metadata.
A silver layer can be finished with nothing integrated
In a medallion architecture, data moves from raw bronze through cleaned silver to report-ready gold. AIMS keeps one silver table per source table, in that source’s shape, and calls it a source-aligned table. Take four of them: an order system’s customers, orders and order lines, and a sales system’s accounts.
Every row carries the identifier the business itself uses, like a customer number. That’s its business key. But customer 123 in the order system and customer 123 in the sales system can be two different customers. A key only means something inside the system that issued it, and that system is its record source. So every key starts with its record source’s code: awlt for the order system, wb for sales.
Some columns identify a record, relate it to another, hold a code or hold sensitive data. Those are the critical columns, and each is tied to an attribute the business model defines once. That tie is a mapping. Gold can read each table once it passes its checks.
Is anything here waiting for integration? Nothing is.
Matching says two keys are one customer account
Now a business requirement arrives. Finance needs one customer list across both systems. But Customer Account has two business keys, one per record source, and they never join until something says they describe the same customer.
That statement is a match. An exact-match rule on a tax number finds that customer 00001042 under company code AU in the order system and account A-937 in the sales system are one customer account. Both rows get one shared key, customer_account||tax_number||48123456789, and each says the tax number rule matched it.
The tax number rule is a match rule, and a person approves each one. Where no rule can decide, a person types a manual match. An entity’s business keys already sit on one list, its key register. Each run adds each key’s shared key and what matched it.
Why only exact rules and manual matches? A guessed identity is wrong in every join downstream of it, and nobody downstream can see which joins it touched. So AIMS leaves probabilistic entity resolution out on purpose. AI may suggest a rule, never a match, and no rule counts until a person approves it.
The golden record sits beside what it summarizes
Finance’s list needs one row per customer. So Customer Account gets one combined record per shared key. That’s a golden record.
But both systems hold a name. Which one wins? A rule per attribute decides, by precedence of record source, by the most recent change, or by the first value that isn’t null in a stated order. Those are survivorship rules.
- Name survives from the sales system, with precedence
wb, thenawlt. - Status survives from the order system, the only source that has it.
- Country survives from the sales system.
Survivorship works per business attribute. The rule can rank wb above awlt for Name only because both name columns are mapped to the same attribute, Customer Account Name. Our post The Lakehouse Ended One Satellite Per Source made the same point about Data Vault: “Conform the business attribute first, or you never get a common satellite at all.”
What makes this integration by exception, not integration by rebuild? Sales Order keeps pointing at awlt||AU||00001042. Gold joins through the key register when it needs the shared key, and through the business key when it doesn’t. Nothing about the four source-aligned tables changed.
Which business requirement calls for more?
AIMS names each business requirement and what it calls for, so every build follows a requirement you can point to, not a preference nobody wrote down.
| Business requirement | Build |
|---|---|
| One record source per entity, analytics consumers | Source-aligned tables only |
| Two or more record sources and a consumer who needs one record, or an entity many gold products reference | A shared key, then a golden record |
| Every source version kept for audit | Nothing new: bronze already keeps every version |
| A stable interface independent of any source | A golden record, as a view or table |
| Consumption | A data mart, always |
Audit needs nothing new. An auditor asks what each source sent and when, and a layer in bronze already keeps every version a source delivered. That layer is persistent staging.
What about the things Data Vault was built for besides audit? They need nothing new either. Many volatile sources and parallel loading are met because each new source adds its own table in bronze and silver. Organizing a team around Data Vault is a preference, not a requirement.
Find your business requirements before you build
Before you integrate another entity, write down why.
- List the entities your gold layer reads.
- Count the record sources behind each one.
- For each entity with two or more, name the consumer who needs one record.
- Find the natural key those systems share, and the person who approves a match rule on it.
- Leave every other entity source-aligned, and note what would change that.
Every entity without a business requirement is integration you don’t have to build yet.