We built AI into every corner of DeltaVault, and we still recorded it losing to a data engineer. It didn’t lose on a hard problem. It lost on a single view over two staging tables, the kind of thing you build before lunch.

Every AI demo rests on the same premise, that the engineer is optional. You type what you want, and the panel does the rest. We spent months building that panel. This post is about the day we watched it lose, because that recording is the best argument for your job we have.

Will AI replace data engineers? Watch it try to build one view

The task was ordinary. A Data Vault satellite needs to carry a customer’s delivery address. The source for that satellite is a view over the persistent staging layer. The view joins the customer address table to the address table with full history, so every version of the address lands on its own row. Ask AI built this view without the history for an earlier post, and that was the easy case.

Ask AI opened well. It took the request “create a full history”, found the staging tables, listed the customer and address tables, and asked what came next. Then the conversation started. The business key is defined on the staging table, not on the view, so we asked it to pick the key up from there. It couldn’t see it. We pointed at the table, and it still saw nothing. We told it the table was on the canvas. Let me check, it said, and then: found it.

You’ve had this conversation. Everyone doing AI and data engineering together has had it, and every turn of it spends credits. To be fair, what came back at the end was decent. There was a proposal to join three tables, a filter, an as-of setting, and a card offering to open it in the view designer. So we accepted, added a couple of columns, confirmed the proposed joins, and read the SQL.

So what went wrong? One join was wrong, and one join should never have been there. The customer address and the address were joined, which we wanted, but as a plain join rather than a temporal overlap. The history the view exists to keep would have been flattened. A third table had also worked its way in, as an inner join to the staging customer table. That join crosses two staging layers, and it filters rows out of a full-history view that must not lose any. A data engineer spots both in a second. We’ve run this request many ways over many sessions. Each run returns a different answer, with the joins wrong in a different place.

Watch this in action:

How DeltaVault lets a data engineer build the same view by hand

Here’s the other take, with no AI in it.

Start in business modeling, where the intent lives. The model says a customer is the customer, a home address, and a delivery address. So the entity the view serves is settled before anyone touches a table.

From there, move to the workspace canvas. The persistent staging table already has an end-dated view generated for it, and that’s the timeline a full history needs. Right-click the table and choose Create View. Name it for the delivery address entity, set the layer to persistent staging, and pull in the address table. The designer proposes the join and marks it as a temporal overlap, because that’s how two end-dated tables join and the tool knows it. Add the customer identifier and the columns the satellite needs, then rename the projected columns so the generated code reads well.

Now open the preview. The full CREATE VIEW statement is on screen, left join and all, carrying the overlap condition the designer promised. It’s the view you wanted, so save it. On the saved view, designate the business key derived from the customer identifier and set the record source. Then add the relationship to the customer table so the hub has something to key on.

That took less time than the conversation did, and nothing had to be undone.

Why the human was faster

The AI answered the question it was asked. The data engineer knew which question to ask, and brought three things nobody had written into the request:

  • Two staging layers must not be inner-joined in a history view.
  • Two end-dated tables join on overlap.
  • This view serves the delivery address entity and will feed a satellite hanging off the customer hub.

None of that is prompt engineering. Those are data engineering skills, and they’re the job.

Without that mindset behind it, an AI panel never converges on the right answer. It converges on a different plausible answer each time, and you pay for every one of them. Put a data engineer in front of the same panel instead. Someone who knows the model, the layers, and when not to press the button turns the panel into a real accelerator. Is that human in the loop AI? We’d put it the other way around: the human starts the loop. Elsewhere, AI drafts the tedious part and you keep the decision. When the work is judgment over a model you hold in your head, the right-click is faster.

We need data engineers now more than ever

The industry says AI makes the engineer optional. Our recording says the opposite. The more AI a platform carries, the more it needs someone who can tell a wrong join from a right one before it ships. The platform needs the same person to judge in a second that this one is faster by hand. So keep doing what you’re doing.

  1. Open the canvas, right-click, and build the view.
  2. Ask the AI when it can see something you cannot.

If you’re the data engineer who knows what they’re doing, the panel hasn’t made you replaceable. It has made you worth more than you were before it arrived.