A System Nobody Chose

Engineers have always built upon things we do not understand. The compiler, the kernel, the database, the cloud, the orchestrator, the fifty libraries in the lockfile — nobody on your team understands them all the way down, and nothing about that is a crisis. It is how the profession works. So when someone says AI is letting us build things we don’t understand, the correct first response is: we always have.

But the old bargains had a shape. Abstraction let us stop understanding what was below the boundary. AI lets us cross boundaries before we know which ones matter — or notice we crossed them. A compiler gives you a stable layer to stand on. An agent notices a production symptom, reads the application code, the Terraform, the IAM policy, the CI configuration and the logs, infers a relationship across them, changes two layers, and verifies the fix — moving through boundaries as though they were terrain. You no longer need to know which layer contained the problem, which abstraction applied — or, sometimes, what the problem even was.

So the new thing is not that we can use systems we do not understand. We have always done that. The new thing is that we can now change them without ever learning what changing them used to require. Software has always outlived the understanding of its builders. Now it can be born without it.

Before deciding how to feel about that, be honest about the upside. Suppose an AI builds a service. Nobody deeply understands its deployment machinery. It runs for ten years. Agents upgrade it, repair its incidents, tune its costs, patch its vulnerabilities. Ask the uncomfortable question: did anyone actually need to understand it?

Maybe no. And that answer should not be waved away, because it suggests something engineers rarely say out loud: much of the knowledge we have traditionally carried was simply the cost of getting things done — knowledge we were forced to hold because the tools were not capable enough to hold it for us. If the tools can hold it now, putting it down is not a loss. For the generated glue code, the disposable dashboard, the migration tool that runs once — it may be pure gain.

Then where is the problem?

Understanding matters when “it works” stops answering the important question. A service returning 200s is easy. A test suite passing is easy. An agent confirming its change fixed the incident is easy. None of them answers: why does this service own this responsibility now? Why does this dependency exist? Why did the fix cross four architectural boundaries? Is this still the system we meant to build? Every individual change can pass every check it was given, and the checks can all be right.

Every change can be correct. The system they produce can still be wrong. A thousand correct changes can build a system nobody chose.

Understanding used to be the price of action: you could not make the change without paying it. While it was required, there was little reason to ask what else it was for. Now the machine pays the price for us — and for the first time, we get to ask.

You don’t preserve comprehension for its own sake. You preserve the possibility of judgment.

Here is the moment that matters. An agent reports its fix is correct: service A now publishes directly to queue C. The tests pass. The incident closes. And someone on the team says: service A was never supposed to know queue C exists.

Nothing in the toolchain could have produced that sentence. Not more tests — they check what the system does, not what it should be. Not a second agent — it would confirm the first. The machine is correct about what the system does. The person disagrees with what the system has become.

Every team carries knowledge like this: the boundary nobody crosses because crossing it once cost two teams a quarter of untangling; the dependency everyone treats as temporary; the module you may rewrite freely so long as it never touches the money path. Almost none of it is written down — until now, it never had to be, because the person who knew it was also the person making the change. When changes start arriving from a machine, that knowledge has to live somewhere outside the people who happen to hold it. Not as documentation of what was decided — as standing reasons to reject what works. Yes, it works. And we still shouldn’t ship it. Whoever can say why is exercising the judgment the machine did not replace.

I am not describing this from above the transition. I am inside it. Some time ago I began accepting AI-proposed changes whose implementation I did not fully understand before accepting them — surface-tested, interrogated when they seemed consequential, but accepted. My bargain was simple: I will give up continuous understanding in exchange for dramatically greater capability, provided I keep a route back into understanding when it matters. So I built the route: machinery that wired together our production MCP servers, Slack, the documentation, the codebases, so that the reasoning behind any change could be reconstructed on demand. I wrote the how-to channel. I posted another company’s success story as encouragement. Then I released it, and the response was one offhand question — “What can you do?” — a concern that it was too expensive, a suggestion that everyone just use their own connectors, and then quiet. The non-developers I invited never tested it. Nobody asked.

For a while I read this as a team not seeing what was coming. The likelier reading is less flattering to me: the tool solved the consequence of a bargain the organization had not agreed to make. Their implicit contract was still the old one — the person changing the thing should understand the thing — and under that contract, a machine for recovering understanding is an elaborate answer to a question nobody asked. And there is a worse reading still: perhaps AI had simply made it cheap enough for me to overengineer my own anxiety. Both can be true. I built infrastructure for a future that has not arrived, at a cost that would once have forced me to think harder about whether I should.

This separation does not stop at tools. It is coming for seniority. AI doesn’t remove implementation from senior engineering. It removes difficult implementation as proof of senior engineering. Taking on the hardest problem used to bundle everything — depth, experience, architecture, failure intuition, judgment — into one visible act. Implementation difficulty made senior judgment visible, and AI is removing the lens. If an agent can produce the implementation, the difficulty stops hiding everything else we were calling seniority. For some, that will be uncomfortable. For others it will be overdue recognition: the parts of the work that never showed up in a diff — the cliff steered around before anyone saw it, the architecture carried in one head for years, the knowledge of what must remain true after the change — finally stand in plain view, with nothing in front of them. Ask yourself, without an audience: if producing the hard solution became easy tomorrow, what would still make you senior?

Perhaps seniority becomes clearest in the moment when everything works and you are still the person who can say: we shouldn’t do this. The senior engineer still writes code. But increasingly, their harder job is making sure hundreds of individually sensible changes continue to describe one intentional system.

There is a hole in all of this, and it is fair to name it rather than solve it. Today’s senior engineers know when understanding matters partly because they spent years being forced to understand. Tomorrow’s engineers may not get that training. We may be teaching people to skip the very experiences that taught us which things are safe to skip. That is not an argument for making juniors reproduce the old world out of educational purity. It is an admission that “load understanding only when it matters” contains a hidden problem: someone still has to learn what matters.

What comes next is not a rule. It is a decision we now have to make deliberately. AI extends the distance we can travel before understanding becomes necessary. How far should we go? Sometimes all the way — some generated implementation may never need to be understood by a human, because understanding it contributes nothing. Sometimes we must stop and load: before a change crosses a domain boundary, when an agent invents a persistent dependency, when a subsystem starts acquiring responsibilities, when the architecture changes shape without anyone deciding it should. There is no universal threshold, and this essay will not pretend to supply one. The hard part is no longer understanding everything. It is knowing what you can safely leave unexplained.

If understanding is no longer required to act, what is it for?

Perhaps, increasingly, for the moments when success itself needs to be questioned. Understanding is what lets us disagree with success — what lets someone say: it works, and it’s wrong.

Soon, changing a system may require almost no understanding of it. Engineering will increasingly be about knowing when that is perfectly fine — and recognizing the moment it isn’t.

We are about to find out what engineering expertise means when you can change a system correctly without understanding it. Knowing when understanding still matters is about to become the discipline itself.