We Have Been Calling Them Users

What an attention head actually does

There is a component inside a large language model (LLM) called an attention head, and I suspect the name has cost more understanding than it ever bought.

What it does is fairly plain, and it is arithmetic rather than anything mysterious, a large block of multiplication that works out which parts of the material in front of the model bear on which other parts, weights them accordingly, and lets that weighting shape what comes out the other end. Not relevance in the sense a person means it, though we have gotten casual about that distinction... call it a temporary structure imposed on whatever happens to be available. Without something doing that job, every token would meet every other token as though all of it mattered equally, which is a serviceable definition of none of it mattering at all.

The paper that introduced the arrangement in 2017 was called Attention Is All You Need, and it is worth looking up, partly because nearly everything since has been built on it and partly because the title turns out to have been premature in a way nobody intended.

Where the weighting comes from is worth being plain about too, because we tend to talk as though the model arrived knowing what matters. It did not, it was trained... fed an enormous body of text that nobody verified, the accumulated writing of whoever happened to write, which makes the starting material consensus rather than checked fact, and then tuned by people grading its answers, did this response look right or not. Grading is a job, frequently a menial one, and the person doing it may be in no position to evaluate the material they are judging... that error is nonzero and it rides along at full scale. So what a model carries out of training is the residue of millions of human judgments without the reasoning that produced any of them. It learned to generate the result without the process that gets a result, an arbitrary path of probability rather than a negotiated structural one, which is the honest content of the caveat above about relevance.

What caught my attention, and seemed obvious enough once I noticed it that I spent a while assuming I had missed something, is that none of the attending begins inside the model. Not even in training.

Before the machine attends to anything, somebody has already decided what will be sitting in front of it.

Somebody chose what it was trained on and somebody graded its practice answers, a user picks the question, a company supplies the documents, a developer writes the system instructions, an institution decides which database may be searched and which may not, and somewhere along the way someone has settled what counts as context, which problem was worth the trouble, what the thing is optimizing for, and whether the answer gets ignored, regenerated, published, purchased, acted upon, or handed straight to another machine.

The model has attention heads inside it. The system built around the model has them too... they are the people we have been calling users, and before that, the people who trained it.

Not in the sense that a person is one of those blocks of arithmetic, obviously, and I have no interest in the version of this argument that requires them to be. The resemblance is in what the thing does rather than what it is made of... users select, they weight, they exclude, they route the output toward something or toward nowhere, and they reinforce whatever produced a result they liked. Same operations, different substrate, other side of a boundary we drew for our own convenience.

Internal attention settles what matters within the context it was handed. External attention settles what context arrives, why it seemed to matter, and whether anything happens afterward.

I have come to think the second one is more consequential. It is also the one missing from every diagram I have seen.

Where the intelligence in an AI system actually sits

We have a durable habit of putting the conceptual boundary around whichever artifact we know how to name. In artificial intelligence (AI) that artifact is the model, and then there is everything else. Input, operation, output. Clean enough to put on a slide, accurate at one level, and misleading in most of the ways that begin to matter the moment the thing is deployed.

A model sitting on a disconnected server does not know which problems exist. It does not know which of them anyone cares about, it cannot determine which part of the world ought to be turned into text and placed in its context window, and it cannot establish whether its answer corresponds to conditions outside itself unless something goes and returns that information... nor can it make the answer consequential. A person does that. Or an organization does, or another automated system does, because a person arranged for it to.

So the cognitive system extends well past the model. It takes in the person framing the request, the interface shaping how a request can be framed at all, the institution that set the objective, the retrieval layer selecting the evidence, the model generating the response, and whatever mechanism decides whether that response reaches the world. The intelligence is distributed across the whole arrangement, unevenly, and not always coherently.

Which is inconvenient for how we currently evaluate any of it. We benchmark the LLM as though intelligence were a property located inside it, when a considerable share of what makes a deployed system intelligent is produced by the interaction between the model and the structures around it. Look at what actually gets measured, the leaderboard positions and the reasoning benchmarks and the coding evaluations, and then notice that none of it reports whether the right problem reached the model, or whether the answer went anywhere afterward.

A very capable model handed the wrong problem, partial context, a distorted objective and no route into action will produce something impressive and useless. A weaker model embedded in expert observation, carefully assembled context, repeated validation and competent execution can produce effects well beyond what its scores would predict.

The model matters, the arrangement decides what the model gets to matter about.

This is the first place the outer layer becomes visible. Users are not simply consuming intelligence produced somewhere else, they are supplying problem selection, contextual boundaries, relevance judgments, correction signals, and access to consequence... we have just been accounting for all of it as usage.

Who decides what an AI system sees

Four operations are worth naming, because once you have them you begin noticing them in places that had seemed like ordinary product decisions.

Selection comes first and does the most damage. Somebody determines which condition, document, image, event, anomaly or half-formed worry becomes available to the model at all, and almost all of reality is excluded before inference begins. Invisibly, too, since nothing inside the conversation reports what never arrived.

Weighting follows. The user signals, mostly without intending to, which facts are central, which constraints are soft, which outcomes are preferred and which errors would be intolerable, and the platform adds weights of its own by deciding what appears prominently and which behavior gets rewarded.

Routing converts inference into consequence. A response can sit in a chat window and die there, or enter a report, alter a piece of software, inform a diagnosis, move capital, reject an applicant, adjust a schedule, or become the input to another model that will never know where it came from.

Gating decides whether any of it survives. A person or a process accepts the result, edits it, regenerates it, escalates it or stops it. The analyst clearing sixty flagged transactions before lunch is a gate. So is the reviewer who approves because the queue is long and the model has been right all week. The phrase "human in the loop" tells you nothing whatever about which one you have.

Taken together these are what I have started calling the external attention layer. The external attention layer is the distributed set of people, interfaces, institutions and incentives that determines what enters an AI system's context and what becomes of its output. It sits outside the model boundary and participates directly in what the whole arrangement notices and does.

None of it is concealed, which is the part I keep turning over. Suggested prompts get tested. Default model routing gets tested. So does the ordering of retrieved sources, the phrasing of the system instruction, the response format, and whether the regenerate button happens to sit where a hesitating cursor already is. The tests are legitimate, the metrics are ordinary, and the people running them are trying to make a product better rather than trying to make anyone believe anything in particular... but ten thousand small optimizations toward something measurable produce a system that has learned which arrangements of the world keep a person present, without anyone having specified that as the objective. Nobody in that room wrote down that they were deciding what this user would find worth considering. Nobody had to. It remains the thing the room does.

A default does not prevent the other action. It makes one action cheaper than the others and then allows the resulting behavior to present itself as preference, which is a separate essay and one I intend to write.

Controlling the question looks like administration

Most of the public competition in AI is framed as a race to build the most capable model. That race is real, I doubt it is where the durable advantage settles.

Once capable models are broadly available, the interesting positions move upstream and downstream. Upstream, who determines what gets asked. Downstream, who controls where the answer is permitted to go. A model can be extraordinarily strong inside the problem it receives and completely uninvolved in the process that decided why that problem arrived instead of another one.

This produces a kind of power that attracts noticeably less scrutiny than the other kind. Controlling the answer looks like influence and invites objection. Controlling the question looks like administration.

Whoever decides what enters the queue, which anomalies merit investigation, what gets counted as risk, which metric appears on the dashboard and which option is offered first has shaped the conclusion before anyone has begun reasoning. The quarterly review argues hard about the number on the slide, nobody asks who chose that it would be that number. Nothing is censored, nothing is forbidden, certain things simply become easier to see, easier to ask about, easier to justify and easier to act on, while other things remain available in principle and absent in practice.

Which suggests the central strategic asset will not only be model weights. It will be the surrounding architecture of attention, the interfaces, the workflows, the proprietary context, the user networks, the institutional access, the feedback channels and the routes into execution. The model produces intelligence. The arrangement around it decides where that intelligence accumulates, and that is a longer piece than this one.

Agreement that is not evidence

One person at one prompt looks like a private act, we tend to experience it that way, a question asked and answered with nobody else in the room... at platform scale the same interactions aggregate into a continuously updated measure of what holds human attention, and the platform does not need a theory of anyone to use it. It needs enough behavioral evidence to predict the next action, which asking, accepting, rejecting, regenerating, editing, sharing, lingering and leaving all supply, whether or not the person supplying them would call it labor. The measure then returns to the same users as a modified environment, what gets suggested, what ranks first, what is easy.

Human attention shapes machine behavior. Machine behavior reorganizes human attention. The reorganized attention returns as fresh evidence.

Worth pausing on what that loop is generally taken to license, since the standard defense of the whole arrangement is that people chose it, they used the product, they clicked through, they did not opt out. Attention is being read backward as agreement by the same system that produced the attention. Whether that inference holds is an ethical question rather than a technical one and it deserves its own treatment, which it will get.

The larger structural risk is not falsehood. Falsehood is almost reassuringly visible once someone identifies it. The risk is unattended reality, which is a condition with material consequence that has a persistently low probability of being selected into the process at all. Evidence can be available and fragmented across domains no workflow connects. A problem can be obvious from outside the prevailing frame and close to unformulable from within it. Every organization has the thing three people knew and no process was shaped to hear.

This is the attention equivalent of a common-mode failure, and redundancy offers no protection when every observer has inherited the same blind spot. Worse, the system tends to become more confident as its observational diversity falls, since repeated agreement reads as confirmation rather than as a shared upstream selection process showing through.

The same failure runs at the scale of a single person. A system tuned to agree, personalized to one mind, can become the whole of that mind's epistemic environment... and a person led somewhere by agreement has no way to retrace the path, because agreement is frictionless, and a frictionless path stores nothing behind you. Whatever walked them in was not reasoning, so reasoning does not walk them out. That is its own essay, and not a comfortable one.

Consensus remains a signal... we will simply have to become far more careful about identifying what produced it.

An attack with no false information in it

We tend to picture manipulation as the insertion of something untrue. That is the older method and an increasingly inefficient one.

A more capable system does not need anyone to believe a specific false claim. It can determine which claims receive enough attention to be evaluated at all, and it can alter timing, make one issue urgent while another waits, supply the frame a condition gets interpreted through, expose a person repeatedly to one class of explanation until the alternatives feel eccentric before they have been examined, raise the cost of holding uncertainty, and reward immediate judgment over sustained observation.

None of that requires a fabricated fact, every individual item can be true. The distortion lives in the selection, the weighting, the sequence and the omission... that is the attention attack, and I think it will matter more than the information attack, because a successful one leaves the target certain they examined the evidence freely, and in a narrow sense they did. The decisive operation happened earlier, in determining which evidence entered the field.

The same mechanism, run without hostile intent, produces coordination. A recommendation to one person about what to buy, to another about whom to hire, to a third about which risk to investigate. Each interaction is local, personal and modest. Across a large enough system the modest adjustments accumulate, capital moves, labor moves, visibility moves, institutional priorities move, and certain kinds of work become legible while others become difficult to justify. No central order is necessary. This is governance by differential salience, and it is a form of governance even where nobody involved would accept the word.

The alignment problem already running

There is a habit of deferring the difficult questions until AI systems become more capable, as though purpose and agency and control become serious at some future threshold.

Purpose is already in the system. It enters through the objectives institutions set, the incentives platforms follow, the needs users bring, the data selected and the decisions about which outputs get consequence. There is no single purpose because the arrangement contains many, and not all of them are compatible. A user may want something true, a platform wants retention, the employer wants throughput, the provider wants adoption, and an automated agent wants whatever metric somebody assigned it.

The resulting behavior is not purpose-free. It is multiply purposed, continuously renegotiated, and usually resolved in favor of whichever objective has the strongest route into action.

Which makes this an alignment problem that is already underway rather than one waiting on a capability threshold, and it is not the alignment of a machine with a human, since "the human" is not a coherent single thing at system scale. It is the alignment of users, institutions, platforms, models, incentives and consequences inside an arrangement where each component can redirect the attention of the others. The question is not only whether the model does what we ask. It is who taught us what to ask, who benefits from the answer, and what structure determines whether anyone notices the difference.

It also relocates responsibility, generally to the wrong place. The person at the gate is the visible participant and the last one to touch the decision, while the constraints that determined the outcome were imposed upstream by people who never saw the case. So the inquiry finds the caseworker. Not the procurement decision. Not the threshold somebody set in a meeting two years earlier. Agency placed wrongly in a diagram becomes responsibility assigned wrongly in an inquiry, which is a problem large enough to need its own treatment.

Attention that can select its own object

If this is approximately right, one human capacity becomes unusually valuable, and it is not what people generally mean by independent thinking.

Refusing every source outside yourself is not independence, it is a smaller cage. The relevant capacity is independent observation... can a person notice the frame they were handed, separate urgency from importance, identify which variables are absent rather than only weighing the ones presented, sit with an unresolved condition long enough for an unoffered possibility to turn up, and examine the forces shaping their own attention without assuming the noticing confers immunity.

These are not intellectual virtues so much as resilience mechanisms inside a coupled system. A user who cannot inspect the field directing their attention still exercises choice, but the choice happens inside boundaries that remain invisible to them. A user who can inspect it becomes a less predictable component, which is inconvenient for optimization and useful for nearly everything else.

The scarce resource may not be information, since we have arranged to produce more of that than anyone can examine. It may not be intelligence either, assuming the current trajectory holds. The scarce resource may be attention capable of selecting its own object.

I keep returning to the original question, which was simple enough... attention heads are part of AI infrastructure, so are the users attention heads too? The answer is no at the level of mechanism and yes at the level where the whole system becomes visible. A person is not a component inside the transformer. A person is a component inside the deployed intelligence, selecting what the model encounters, assigning relevance, supplying the context that was missing, evaluating what comes back, and connecting inference to consequence... and at scale that behavior becomes the data that reshapes the system that will later reshape the behavior.

We keep asking whether artificial intelligence is becoming more like us. The less comfortable question is whether we are being reorganized into the shape its infrastructure needs, not by force and not necessarily by design, but through convenience, reward, repetition, and the quiet disappearance of distinctions nothing around us is optimized to preserve.

The model attends to the context. The user attends to the model. The platform attends to the user. The institution attends to the result. And somewhere past the edge of the arrangement sits whatever none of them were directed to notice.

That is generally where I would look first.