At Data + AI Summit in June, Databricks announced Lakeflow Designer — a drag-and-drop, natural-language interface that builds production ETL pipelines without writing code — along with Genie Code, which writes your ingestion connectors and jobs for you, and Genie ZeroOps, a background agent that watches your pipelines in production, does root-cause analysis on failures using quality metrics and lineage, then proposes and sandbox-tests a fix before handing it to you for approval.
Let me be straight about my standing here: I haven't shipped anything on Lakeflow Designer. I use Databricks at work, I'm studying for the certification, and I read the announcements like everyone else. This isn't a review — I haven't earned the right to write one.
But I do have one data point, from my own pipeline, that I think complicates the story these announcements tell. And it's been sitting in my head since I read them.
The bug that has no symptom
Last month I found that my Australian labour market dashboard had been quietly showing six of Australia's eight states and territories. The ACT and the Northern Territory were missing. I wrote about it in detail here, but the mechanism is what matters now.
My extract asked the ABS API for all eight regions, explicitly, by number. The API returned 200 OK — with six of them. It turns out the ABS doesn't publish a seasonally adjusted series for the NT or the ACT, because those samples are too small to seasonally adjust. Request one on its own and you correctly get a 404. Request all eight at once and the API hands back the intersection of what you asked for and what exists, says nothing, and returns success.
Now hold that bug up against the announcements.
Lakeflow Designer would have built that pipeline beautifully. I could have described what I wanted in plain English and it would have generated a clean, well-governed, correctly-orchestrated pipeline that ingests six states. It would not have been wrong. My pipeline was never wrong. The request was wrong, and the request looked completely reasonable.
Genie ZeroOps monitors production and detects failures. So: what failure? There wasn't one. The API returned success. Every row that arrived was valid, correctly typed, and landed in the right table. The job ran green. Lineage was pristine — you could trace every one of those rows from the ABS all the way to a bar in a Power BI chart, and every hop was correct. There is no error log to analyse, because nothing errored. There is no anomaly in the data, because the data that arrived was perfectly normal data. It was just missing two states, and nothing in the system had any idea that mattered.
By every operational measure available to an agent watching that pipeline, it was healthy. It was also, for months, producing a ranked chart of Australian states that omitted two jurisdictions and told you Tasmania was the smallest labour market in the country.
To be fair to the tool
I want to steelman this properly, because the lazy version of this post is a cheap shot and I don't believe the cheap shot.
Genie ZeroOps doesn't only watch for crashes — the announcement is explicit that root-cause analysis draws on data quality metrics, and Lakeflow pipelines support declared expectations. So if someone had written the rule "this table must contain exactly 8 distinct regions," then yes: the platform would have caught this, enforced it forever, told me exactly which upstream hop dropped them, and quite possibly proposed the fix. That's real, it's valuable, and it's better than what I had, which was nothing.
But look closely at what had to happen first. Someone had to write the rule.
And writing that rule requires knowing that Australia has eight states and territories. That the ABS codes the Northern Territory as 7 and the ACT as 8. That seasonal adjustment is a series type, not a data quality attribute. That a labour force survey can't seasonally adjust a sample that small. That a chart which ranks states cannot mix a seasonally adjusted series against an unadjusted one, because part of the gap you'd be reading as signal is just what month it is.
None of that is in my data. None of it is derivable from my data. It is not in the logs, not in the lineage, not in the schema, not in the row counts. An agent with total observability over my entire stack could stare at it forever and never conclude that two states were missing, because the pipeline is a perfect and faithful representation of a request that was wrong. The knowledge that makes the bug visible lives outside the system entirely. It lives in the world.
That's the line I keep coming back to. These tools are automating the enforcement of expectations. They are not — and I don't think they can be — automating the authorship of them.
Which half of the job is actually getting automated
Here's what I take from Summit, as someone about to walk into this industry.
The mechanical half of data engineering is going away, and honestly it should. Hand-writing an ingestion connector for the four-hundredth REST API is not craft, it's typing. If Lakeflow Designer builds it from a sentence and Genie Code wires up the orchestration, good — that work was tedious and it was never where the value was. I'll use it the moment I can.
But that means the value doesn't disappear; it concentrates. It concentrates in exactly the part that a model watching my logs cannot do for me: knowing what the number is supposed to say. And that part isn't a technical skill you can bootstrap from a certification. It's domain knowledge, plus the specific paranoid instinct to write it down as an assertion before the pipeline lies to you.
Which is, when I look at it honestly, the part I skipped. I built a clean pipeline, on a real cloud warehouse, with a staging and mart layer and a report generated as code — the parts that look impressive on a portfolio — and I never once wrote down what I expected to come out of it. The engineering was fine. The thinking is what was missing, and no agent was going to supply it.
So here's what I actually changed
Not my stack. My first step.
Now, for every source, before I pull a single row, I write down what I expect: how many members each dimension should have, what date range it should cover, whether any dimension already contains a pre-aggregated total, and what the latest available period actually is. Then the pipeline fails loudly when reality disagrees:
"state_summary": {
"key": "M3+M13.3.1599.30.1+2+3+4+5+6+7+8.M",
"expect": {"REGION": 8},
},
That's four lines. It is by far the cheapest code in the repo and by an enormous margin the most valuable, and it's the one artifact in this entire project that an AI could not have written for me — because 8 isn't a fact about my data. It's a fact about Australia.
I'm not bearish on any of this. I think Lakeflow Designer and ZeroOps are genuinely good, I'm going to use them, and I'd rather work in an industry where the boring half is handled. But I'd caution anyone entering this field against reading the announcements as "the job is being automated." The job is being relocated — out of the pipeline and into the expectations. Away from "can you build it" and toward "do you know what should come out of it."
That's a better job than the one it's replacing. It's also a much harder one to fake, which is probably the point.
— Melvin
Sources: Lakeflow: A new era of agentic data engineering · Databricks Data + AI Summit 2026 announcement recap
— Melvin