The one great principle of the English law is, to make business for itself.
The recurring theme running through my mind the last few months has been
complexity within a software application. Forget coding. Sales is using AI to
write all new code, so for us engineers there's not a hell of a lot to do
besides think (and be there to hold the bag).
Last week I generated a CSV of some internal company metrics. With only a
sentence or two of prompt, generative AI extrapolated meaningful signals,
correlated changes in the data with external signals that were not explicitly
expressed (e.g. interest rate hikes), and built a polished interactive
dashboard with relevant visualizations. Nevermind the fetishization of
dark-mode or the tell-tale slop signs (what is it with that fucking font?) -
most people would never notice these, it's coded to look "modern" and it looks
the part. I didn't even ask for the dashboard or any visualizations. Results
like these seem magical. I believe this is how most people experience
generative AI.
Around the same time, I ran another AI coding experiment on one of my smaller
open-source libraries, scout, and the
process was so riddled with flaws and subtle failures that I know I lost
time (and sanity) by even attempting to let AI write code. You see, scout is
just a dead-simple RESTful search server written as a flask
app. This is not frontiers of engineering shit, it's about as mechanical as it
gets in terms of implementation. As in my previous experiments
with AI, the strength of the tool in coding tasks was that it could trace logic
bugs and find inconsistencies precisely and accurately. The weakness is that as
soon as it began to write code it produced tangles of weeds that had to be
aggressively hand-pruned, because each iteration the weeds had a tendency to
spread...and spread.
Claude and his bros sit down to write me some code.
This is why I'm stuck. I'm stuck between competing narratives, each of which is
exerting real business pressure. To push-back when people's daily experience of
AI is of the magical variety is seen as almost perverse. I find myself
constantly wanting to say "No! I embrace these tools! This is not thinly-veiled
self-preservation! Just hear me out..." But how do I express this when, at
every turn, a new silver bullet for agent orchestration, automatic coding,
automatic review, automatic thinking is being announced? Going further, as one
concerned with code as ground-truth for a system, how do I take the leap of
faith and relinquish control to a swarm of agents and markdown files?
Intelligent Machines, 1935.
These dynamics, the rise of agentic coding loops, and some unrelated UFO stuff
had me thinking about cybernetics (of all things). Cybernetics emerged after
WWII as a framework for studying control mechanisms in complex systems. The
canonical example is a thermostat that kicks on heating or cooling when the
temperature falls outside the specified range, and then returns to passive mode
when back within the acceptable range. The central idea is feedback.
The "first law" of cybernetics, Ashby's Law of Requisite Variety, states that
in order to control a system, the regulating function (feedback) must be able
to match the state-space complexity of the operating environment. The idea is
that without adaptive control, the environment dominates the system and
eventually leads to failure. In software engineering, I see a two-layered
system where at the surface you have the software artifact itself, the
application that users interact with. It must be able to encode and handle the
complexity of it's intended usages. And then beneath that you have the actual
code, the primary source of truth, where it is the programmer who is the
control function for the overall system. The programmer's job, then, is
two-fold: to manage the state of the code so that it can produce an artifact
which, in turn, correctly handles its designed use-case.
The framing also explains to me why I've found the greatest utility in AI
tooling in analysis tasks. When directed to do deep analyses on existing
code-bases, reason about design tradeoffs, trace deadlocks or diagnose memory
leaks AI has been amazing. In cybernetic terms, AI extends the amount of
variety I'm able to cope with, and allows me to better regulate the code-base.
Yet when directed top-down with specs, no matter how detailed, AI replaces the
regulator with its own loop, made from the same substrate as the thing being
regulated - the model watching the code and the model producing the code
are now the same kind of process, and control dissolves.
According to that first law, the programmer must be able to match the
state-space complexity of the code itself, in order to be able to effectively
wield it and adapt it over time. Over the years, approaches like Agile, YAGNI,
KISS all tend towards optimizing for this kind of adaptability. The core idea
is to keep the system simple and minimal enough that both the programmer and
the software artifact can adapt as things unfold. On the other end of the
spectrum, domain-driven design and spec-driven development emphasize explicit
front-loading of complexity modeling. This way the operating modes of the
system are well-understood beforehand and the programmer's role becomes more
mechanical. Formal methods, meanwhile, are in their own special corner. They
front-load, too, but are anchored to machine-verifiable proofs and are the
opposite of a vibed-out markdown file.
Those readers who are familiar with my open-source work can probably guess
which camp I belong to. I prefer smaller tools, built bottom-up,
where the design, behavior and invariants can reasonably be held in your head.
Designing software from the bottom-up means building the lower-level component
pieces to be clean and orthogonal, so that they can be composed into larger
structures. When done correctly, new features tend to write themselves as new
patterns emerge. For instance, working on huey,
things like retry delays, revocation, rescheduling, ETAs, rate-limiting, chords
-- all these features came out as natural consequences from a core set of
building blocks. They are robust because the underlying structures are robust
and compose well.
So where does AI-written code live in this framework? To me it lands very
firmly in the top-down world. There's been a recent wave of hype around
"spec-driven" AI development, where you front-load all design requirements into
markdown beforehand (see prompting converges with coding).
But more importantly, in the two-layered model of control, AI tools eliminate
the programmer-as-mediator of the system. All that exists is the artifact,
produced by AI, and the specification - some of which exists in markdown, some
of which is nothing but a dim spectre haunting a long-forgotten context window.
Quicksand Supercedes Brooks' Tarpit
When the programmer is removed as the control system for managing software
complexity, what happens? AI evangelists would argue that control is retained,
it has simply shifted to the network of prompts, code, tests, and agents. I would
argue, based on my own experience, that this is actually where things begin to
break down. An AI-modulated feedback loop inevitably becomes self-referential
and at some point the loop closes, because the thing anchoring it to reality -
the programmer - can no longer keep up. Code gets written, reviewed, tested and
modified using the same system that produced it. Because the speed at which AI
produces code far exceeds what any person can reasonably review and fully
understand, there's a kind of event horizon that gets crossed. The break occurs
and beyond that point the ownership of the code is implicitly transferred.
The consequences of an AI feedback loop go beyond the loss of that lower-layer of
control (the system can be understood by human programmers). Errors in design
have a way of compounding in AI-written code, so that you end up with many
islands which are internally consistent, but do not compose well with one
another, much less produce a coherent whole. Even in my tiny 1,000 LOC image
viewer prototype, AI produced two completely independent thumbnail caching
mechanisms, redundant image display widgets, and three nearly-identical
implementations of a context-menu. When prompted to refactor, the result ended
up being worse - skeletal remnants of the old APIs calling into "refactored"
functions that held the same old logic grafted into new (redundant) functions.
Worse still, this drift leads to real costs. Every iteration consumes tokens,
so the code-base is not merely accumulating noise but paying to accumulate
it. The feedback loop becomes an epistemically unstable, economically
self-reinforcing pit of quicksand. Maybe just a few more hours of token-spend
will fix it... But the tens- or hundreds-of-thousands of lines of code sitting
behind the software artifact resist attempts to refactor, because the refactor
requires the same tools which introduced the problems in the first place.
I recently had a call with an AI-native developer where, with refreshing
candor, he showed me his AI development process. Several times he mentioned,
almost apologetically and without a trace of defensiveness, that he was not a
"real developer", yet he had vibe-coded a real product. He expressed
frustration with opaque token spend, noted he had paid several thousand dollars
over his normal usage just to get his agents "un-stuck" and the looms spinning
again. One of the features in his application was a complex visualization of a
knowledge-graph, each node brightly illuminated against a background web of
connections. But for reasons which remain obscure, the graph had a tendency to
wiggle around and reconfigure itself so that one was forced to mechanically
mouse over nodes at random until the tooltip informed you that you'd arrived at
the node you were interested in. How many more thousand dollar re-ups would it
take to get the graph to sit still and behave, I wondered?
Cocytus
Look how thou steppest!
Take heed thou do not trample with thy feet
The heads of the tired, miserable brothers!
Ashby's Law gives us a few ways a thermostat can fail: it doesn't sample the
environment frequently enough, it models the wrong system, or the environment
changes too quickly to enable it to respond effectively. AI coding tools manage
to hammer at all three of these at once, and the programmer can no longer be an
effective regulator of the system. Iterating becomes a closed circuit of AI
driving AI, while code bloats, errors compound, and prompts drift. The artifact
may appear correct, but the underlying code is such a mess that no one can be
sure.
Anthropic, OpenAI,
Google all want us to believe that these tools will speed up and simplify the
process of developing software. And in a way they do... right up until the
event-horizon is crossed and the loop closes. Beyond there is nothing but
iteration upon iteration, token burn, loss of grounding and increased spend. As
the software system evolves and code grows, there is an almost addictive sense
of making progress - something the AI-native developer spoke about to me -
but towards whose goal? In the end the system may run, the dashboard may
continue to render, and nobody will be able to say why.
