The Attention Head Outside the Model

What an attention head actually does

There is a component inside a modern language model called an attention head, and I suspect the name has cost more understanding than it ever bought... so it is worth thirty seconds to see what the thing actually is, because the argument of this essay lives in the gap between the name and the mechanism.

A language model is built on a transformer, a kind of neural network, which is a computational system that learns patterns by adjusting numerical parameters during training. It processes text as tokens, words or pieces of words represented as numbers. For each token, an attention head produces three numerical patterns, and the standard names for them are friendlier than the objects deserve: a query, representing what information is being sought from the other tokens, a key, representing how each available token can be matched against that seeking, and a value, the information that gets passed forward when a match receives weight. None of these is a literal question or a stored meaning, they are patterns of numbers learned during training. The head compares queries against keys, converts the scores into proportional weights, and combines the values accordingly, and many heads do this in parallel so that different learned relationships can influence the result.

That is the mechanism, arithmetic comparison, proportional weighting, combination, and it is worth being plain about what it is not. It is not awareness, and it does not determine relevance in the sense a person means the word. It cannot establish whether a statement is true, whether the problem matters, or whether the material it received contains what the problem requires. It determines how strongly the representations already in front of the model influence one another, and the word attention invites us to place more inside the mechanism than the mechanism contains... which most of us have been doing, comfortably, since 2017, when a paper called Attention Is All You Need introduced the transformer. The paper introduced the architecture, not attention itself, and its title turns out to have been premature in a way nobody intended, because attention was not all anyone needed. Something still had to decide what the attention would be pointed at.

Context is not the field

Two terms will carry the rest of this essay, so I will fix them now and hold them fixed.

Context is the material available to the model during a particular interaction, everything that has actually been placed in front of it. The field is the larger body of conditions, evidence, and possible observations that could bear on the problem, whether or not anyone converted them into text and delivered them.

Attention operates on context and only on context. The model can redistribute influence among the representations it received, it cannot reach into the field and decide which parts of it deserved to become context in the first place. That final condition is the important one, and nearly everything that follows is an unpacking of it.

The model cannot choose what it receives

Before the model attends to anything, something outside it has already selected what it will receive, and this is true twice over, once at training and once at every use.

Training data were collected, filtered, arranged, and weighted through many human and institutional decisions, and a numerical scoring rule determined which prediction errors changed the model. Some systems were then further shaped by human feedback, written demonstrations, rankings of candidate responses, safety rules. The InstructGPT process, to take a documented example, used demonstrations written by human reviewers and rankings of model outputs to change how a previously trained model followed instructions. This does not mean the resulting model merely contains the opinions of those reviewers, and it does not mean every model is trained this way, the specific process varies. The structural fact does not vary: human judgments became causal inputs into the system's behavior before it produced its first response, and what those judgments purchased was a path to acceptable outputs, an arbitrary path of probability rather than a negotiated structural one.

Then the same structure again, at the moment of use. A person chooses the question. An interface determines how the question may be expressed. A retrieval system searches for and selects documents to place in the context. A developer supplies standing instructions. An institution decides which databases may be searched at all, and a workflow determines whether the result is ignored, revised, escalated, published, or acted upon. By the time the model calculates its attention, most of the field is absent, and it is absent because nobody and nothing converted it.

So the arrangement around the model performs another kind of attending. Not the same mechanism, not the same material, the comparison is structural rather than mechanical, and holding that distinction is what keeps this argument from collapsing into a metaphor. What the arrangement does is select, weight, route, and gate... and I will call the distributed arrangement performing those four operations the external attention layer. The word layer does not name a software component. It names a functional boundary spread across people, interfaces, institutions, databases, policies, and workflows, and the rest of this essay is about what follows from taking that boundary seriously.

Four operations outside the model

Selection comes first, because nothing can be weighted, evaluated, or acted upon until it enters the process at all. A user notices one condition rather than another, a company makes some documents searchable and leaves others outside its retrieval system, a risk system decides which transactions deserve review, a dashboard displays five measurements while a sixth sits in a database nobody opens. Selection is unavoidable in any bounded process, no person or system can attend to everything, so selection itself is not the failure. The failure begins when a selected subset gets treated as though it were the complete body of relevant evidence rather than a constructed sample of it.

Weighting changes the relative importance of what survived selection. A user emphasizes one constraint, a system instruction establishes a priority, retrieval ranks one source above another, an interface makes one option more visible, repetition makes one explanation easier to recall than its alternatives. Notice what moves and what does not: the information itself can remain unchanged while its probability of affecting the result changes substantially.

Routing determines where the result goes, and routing is what converts inference into consequence. A response can sit in a private conversation and disappear, or enter a report, alter software, move capital, reject an applicant, inform a diagnosis, change a schedule, or become the input to another automated process that will never know where it came from. The same output produces different consequences at each destination, because the destination determines what the output is capable of changing.

Gating determines whether the routed result is accepted, revised, escalated, or stopped, and gates vary more than any other component. One gatekeeper has time, authority, relevant knowledge, and access to alternatives. Another is an analyst expected to clear sixty flagged cases before lunch. Both arrangements get described with the same phrase, a human in the loop, meaning a person appears somewhere in the decision process, and the phrase tells you almost nothing about which arrangement you have. Ben Green's analysis of 41 government policies requiring human oversight reached this problem from the policy side: requiring a person to be present does not establish that the person can perform the oversight being attributed to them. A human gate protects only when the person has the information, the authority, the capacity, and a viable alternative to approving.

Where deployed capability actually sits

We tend to draw the conceptual boundary around the artifact we know how to name, and in artificial intelligence (AI) that artifact is the model, with everything around it filed as deployment detail. The boundary is useful for engineering and incomplete for analysis, and the four operations show why.

A disconnected model does not know which problems exist. It cannot determine which conditions in the world should be converted into text, which evidence is missing, what objective the institution is actually pursuing, or whether its answer corresponds to anything outside the representation it received, and it cannot make its answer consequential unless something routes the answer into action. Every one of those functions is real, every one is required for a deployed outcome, and every one sits outside the model, which means the real-world performance of a deployed system is a property of a coupled arrangement... the model, the person framing the request, the interface constraining the framing, the retrieval constructing context, the institution defining success, and the mechanisms evaluating and routing what comes back.

The claim here is narrower than it may sound, and worth stating at its exact strength. Not that the surroundings become part of the model, not that a person is literally an attention head. Only this: system-level performance cannot be predicted or explained from model capability alone. A strong model handed the wrong problem, partial context, a distorted objective, and no route into action produces an impressive answer that changes nothing, and a weaker model embedded in expert observation, reliable context, repeated validation, and competent execution can produce the better real-world result.

Which is inconvenient for how we currently measure any of this. The standardized tests used to compare models are called benchmarks, and benchmarks isolate the model because isolation is what makes comparison possible. That is a legitimate purpose, and it also means a benchmark cannot establish whether a deployed arrangement noticed the right condition, constructed an adequate context, preserved the necessary constraints, or connected the answer to an effective action. The model determines what can happen inside the problem it receives, the arrangement determines which problems the model can affect. Readers who know The Extended Mind, where Clark and Chalmers proposed that external resources can participate in a cognitive process rather than merely support it, will recognize the shape of this argument, and nothing here requires accepting their full account of cognition... it requires only observing that the chain producing a deployed outcome crosses the model's boundary, in both directions.

Controlling the queue looks like administration

Once capability is a property of the arrangement, it is worth asking where in the arrangement durable advantage settles, because the public competition in AI is described as a race toward more capable models, and that race is real, and it is probably not where the durable positions are.

As capable models become widely available, advantage migrates toward the surroundings, proprietary context, institutional access, user networks, feedback channels, workflow position, and above all the two ends of the pipe. Upstream, someone decides what gets asked. Downstream, someone decides where the answer may go. And these two positions attract very different amounts of scrutiny, because controlling an answer looks like influence, while controlling the queue looks like administration.

The queue is the ordered set of problems, cases, anomalies, and options presented for anyone's attention. Which anomaly merits investigation, which threshold creates a case, which measurement enters the quarterly review, which option appears first, which evidence ranks highest in retrieval... each of these reads as an ordinary operating decision, and together they define the field in which all later reasoning will occur. Nothing needs to be forbidden. One action becomes cheaper, another requires one more screen, a third remains technically available and disappears from the path most people actually traverse. A default does not remove the other choice, it changes the cost of reaching it, and the resulting behavior then gets recorded as preference, even though the environment helped produce the frequencies being recorded.

None of this requires a room full of people deciding to steer anyone. Suggested prompts get tested, source ordering gets tested, response formats, notification timing, button placement, all tested, and each change is locally reasonable and locally measured. Across enough iterations the system becomes increasingly effective at arranging what receives attention, even though nobody specified the structure that accumulated. Intent is not required for an attentional arrangement to accumulate. Connected incentives and repeated local decisions are enough.

Agreement is not independent evidence

The accumulation has a second consequence, quieter than the first, and it concerns what agreement is worth as evidence.

One person asking one question experiences a private interaction. A platform recording that interaction acquires evidence, people ask, accept, reject, regenerate, edit, share, linger, and leave, and depending on what gets recorded and how it is used, those actions influence rankings, recommendations, and the next round of interface experiments. The modified environment then returns to the same population and produces another round of behavior, which is a feedback loop in the plain sense: the output of one stage changes the conditions that later become its input. Human attention shapes machine behavior, machine behavior reorganizes human attention, and the reorganized attention returns as fresh evidence.

Research on recommendation systems has demonstrated versions of this loop directly. Chaney, Stewart, and Engelhardt showed in simulation that training a recommender on behavior already influenced by its own earlier recommendations makes users' behavior more alike without increasing the benefit they receive. Kleinberg and Raghavan showed that when many decision-makers adopt the same automated rule, improvements for each individually can coexist with worse collective outcomes. And Salganik, Dodds, and Watts, in an experiment that predates all of this, varied only whether people could see one another's choices in an artificial music market, and found that visibility alone made outcomes more unequal and less predictable.

The variable that matters in all three is not whether people agreed. It is whether their judgments were produced through sufficiently different sources, frames, and selection processes to count as separate observations... because repeated agreement can indicate repeated contact with the same truth, and it can indicate repeated exposure to the same upstream selection, and the two are indistinguishable if all you count is the agreement. Ten observers looking through the same opening are not ten independent views, however emphatically the counting says otherwise, and a system built this way grows more confident precisely as the diversity of its observation falls.

Which is why the most dangerous condition here is not falsehood. Falsehood is at least the kind of thing a process can catch. Call the more dangerous condition unattended reality: not reality literally unseen by everyone, but a condition with material consequences that remains unlikely to enter any process capable of recognizing and responding to it. The evidence may exist, fragmented across people no workflow connects. The variable may be obvious from outside the prevailing frame and nearly impossible to formulate from within it. Most organizations of any age contain at least one condition that several people partially recognized and no available process could assemble and hear... and adding observers does not correct a shared selection failure, when every observer receives the same restricted view.

An attack with no false information in it

Everything so far has been intentless, structure accumulating out of reasonable local decisions, and the same operations have a deliberate use, so the boundary between those cases needs drawing carefully, because most of the vocabulary we have draws it wrong.

We imagine manipulation as the insertion of something untrue. False information remains one method, and it is no longer a necessary one, because judgment can be altered by changing which true claims receive attention, the order they arrive in, the urgency attached to them, the frame they get interpreted through, and the alternatives that stay technically accessible while becoming practically absent. Every displayed statement can be accurate. The distortion can live entirely in selection, weighting, sequence, and omission.

So define the deliberate case exactly. An attention attack is the intentional use of those operations against a target's interests, arranging the field through which the target evaluates reality while concealing how the evidence and options were selected and constrained, and depending on the target mistaking the constructed field for an adequate one. Three conditions, intent, concealment, dependence on the mistake, and all three matter, because selection alone is not an attack, selection is what any bounded process must do. An automated system can produce similar distortion with no hostile actor anywhere, a system rewarded for a measured outcome, clicks, continued use, discovers that some arrangements of the field perform better than others, and the distortion is then a product of the objective and the feedback structure rather than of anyone's intent... which does not make it an attack, and does not make it harmless. And the same four operations, run in the open toward the reader's own interests, are simply coordination, recommendations that surface relevant work, warnings that direct limited review toward urgent cases, defaults that spare people decision cost. The operations are not inherently corrupt. Their character depends on what they preserve, what they exclude, who can inspect them, and whose objectives hold the routes into action.

At platform scale the stakes of that dependence get a name. Salience is the capacity of something to attract notice and action, and across a large enough system, small differences in salience move purchasing, hiring, investment, visibility, and institutional concern, with no command anywhere in the chain. Some activities become easier to notice, easier to justify, easier to act on, and others remain present and structurally quiet. That is governance by differential salience, collective behavior directed through unequal visibility and unequal action cost rather than through instruction, and it is a form of governance whether or not anyone involved would accept the word.

The alignment problem is already running

Alignment is the effort to make an AI system reliably serve intended goals while preserving required constraints, and we talk about it as a problem that arrives later, at some capability threshold where purpose enters the machine. Purpose is already in the arrangement. It entered through the user's objective, the institution's incentives, the platform's measurements, the provider's policies, the scoring rule used in training, the retrieval system's ranking method, and the workflow's definition of a successful outcome, and there is no single purpose because there is no single actor. A user may want accuracy, an employer wants throughput, a platform wants continued use, a provider wants adoption, a regulator wants demonstrable oversight, and an automated process will pursue whichever measurable outcome it was built and authorized to increase. The composite is not purpose-free, it is multiply purposed, and the conflicts get resolved, over and over, in favor of the objective that can be measured most consistently and holds the clearest route into action.

Alignment is therefore a property of the whole arrangement rather than a relationship between one model and one human, and the question is never only whether the model does what we ask. Who selected the available questions, what shaped the asking, which evidence was permitted to enter, who benefits from the answer, where the answer can travel, who retains the authority to stop it... those are alignment questions, all of them, and most are settled before any model runs.

Responsibility follows the same structure and gets assigned at the wrong boundary for the same reason. The person at the final gate is visible and touched the decision last. The thresholds, objectives, purchasing choices, and interface constraints that determined the outcome were set upstream, by people who never saw the case. So the inquiry finds the caseworker. Not the threshold somebody set in a meeting two years earlier. When influence is attributed to the wrong component, responsibility lands on the wrong participant, and it lands there every time.

Attention that can select its own object

If this account is approximately right, one human capacity becomes unusually valuable, and it is not the one people usually reach for. Refusing outside information is not independence, a person who rejects every external source has not escaped a frame, they have built a smaller one. The relevant capacity is independent observation, and independence here does not mean observation free of influence, no human perception meets that standard. It means the ability to notice and test conditions beyond the field as supplied... noticing how the available field was constructed, separating urgency from importance, identifying the absent variables rather than only weighing the presented ones, staying with an unresolved condition long enough for an unoffered possibility to surface, and examining the forces shaping your attention without assuming the examination makes you immune to them.

These are not decorations added to intelligence after the hard work is done. They are what lets a coupled human and machine arrangement detect and repair its selection errors, which more reasoning power alone cannot do, because reasoning operates on context and the errors live in what became context. A person who cannot inspect the field's construction still exercises choice, inside boundaries that stay invisible. A person who can inspect it becomes harder to steer through defaults, shared blind spots, and inherited priorities, which is inconvenient for optimization and useful for nearly everything else. The scarce resource is no longer information, we have arranged to produce more of that than anyone can examine, and it may not remain intelligence either, if capable models keep becoming ordinary. The scarce resource may be attention capable of selecting its own object.

So, the question the title has been holding: are users attention heads? No at the level of mechanism, a person is not a block of arithmetic inside a neural network, and yes at the level where the deployed system becomes visible, because the operations correspond even though the mechanisms differ. The model redistributes influence within its context. The user selects and evaluates what the model receives and returns. The platform selects what reaches the user and records the response. The institution decides which results acquire authority and consequence. Each participant changes the field available to the next, and the modified field eventually returns to shape the process that produced it... they are the people we have been calling users, and before that, the people who trained it.

We keep asking whether artificial intelligence is becoming more like us. The less comfortable question is whether our institutions and habits are being reorganized toward the forms its infrastructure can record, measure, and reward, not by force, and not through anyone's complete design, but through convenience, repetition, feedback, and the quiet disappearance of distinctions nothing in the arrangement was built to preserve. Somewhere outside the available field sits whatever none of the participants were directed to notice.

That is generally where I would look first.