Attention Is All You Need

A visual research note on Vaswani et al. (2017). Read this as a reconstruction of the paper's argument, not as a generic introduction to modern LLMs.

The paper's central claim: a sequence-to-sequence model can dispense with recurrence and convolution entirely, using attention to connect positions globally while making training much more parallelizable.

The engineering trade: self-attention reduces the sequential path between distant tokens to one layer, but a full attention matrix costs quadratic memory and compute in sequence length.

The short version: the Transformer is not one trick. It is a coordinated system:

1. represent tokens as vectors and inject position;
2. use queries and keys to compute pairwise compatibility;
3. use the resulting weights to mix values;
4. run several learned attention projections in parallel;
5. wrap attention and position-wise feed-forward layers in residual and normalization paths;
6. use a causal mask in the decoder so generation remains autoregressive.

The original paper studied English-German and English-French translation, plus English constituency parsing. It did not propose a decoder-only chat model, and it did not claim that attention has no computational cost.

How to use this note

The static prose gives the derivation. The widgets are small experiments:

• change a query token and inspect the attention distribution;
• turn the square-root scaling on and off;
• drag query and key vectors to see a dot product become a weight;
• move through sinusoidal position encodings;
• rotate a 3D multi-head routing sketch;
• drag value tokens and watch their weighted average move;
• compare the asymptotic cost and path length of recurrent, convolutional, and self-attention layers.

The attention maps, routing examples, and physical analogy are explanatory simulations, not weights extracted from the paper's trained model. The reported BLEU scores, parameter counts, schedules, and ablations are transcribed or paraphrased from the paper.

1. What changed in 2017?

Before the Transformer, strong sequence-to-sequence systems usually processed a sequence through recurrence or convolution. Both approaches can model order, but their computational shape differs:

• An RNN has a natural sequential dependency: position \(t\) waits for the state at \(t-1\).
• A convolution can process positions in parallel, but distant positions require multiple layers or dilation.
• Self-attention lets every position directly access every other position in one layer.

The paper's argument is easiest to see as a graph problem: how many sequential operations separate two tokens?

flowchart TB
    A[Input tokens] --> B[Token embeddings plus positional encoding]
    B --> C[Encoder stack: self-attention plus FFN]
    C --> D[Decoder stack: masked self-attention plus cross-attention plus FFN]
    D --> E[Linear projection and softmax]
    E --> F[Next output token]

In the published base model, the encoder and decoder each contain \(N=6\) identical layers. The common hidden width is \(d_{\mathrm{model}}=512\). Every sub-layer is wrapped as

$$ \mathrm{LayerNorm}\bigl(x+\mathrm{Sublayer}(x)\bigr). $$

The decoder has one extra sub-layer per layer: attention whose queries come from the decoder and whose keys and values come from the encoder output. Its self-attention is masked so position \(i\) cannot read positions to its right.

Base configuration from the paper

2. The entire mechanism in one equation

For one attention layer, queries \(Q\), keys \(K\), and values \(V\) are matrices. The output is

$$ \mathrm{Attention}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^\mathsf{T}}{\sqrt{d_k}}\right)V. $$

Read this left to right:

1. \(QK^\mathsf{T}\) computes every query-key compatibility score.
2. Division by \(\sqrt{d_k}\) keeps the score scale stable as the key dimension grows.
3. The row-wise softmax turns scores into non-negative weights summing to one.
4. Multiplication by \(V\) forms a weighted sum of value vectors.

For one query \(q\), key \(k_j\), and value \(v_j\),

$$ \alpha_j = \frac{\exp(q\cdot k_j/\sqrt{d_k})} {\sum_\ell \exp(q\cdot k_\ell/\sqrt{d_k})}, \qquad o=\sum_j\alpha_jv_j. $$

The important conceptual separation is that keys decide relevance while values carry content. A token can be highly relevant because its key matches the query, while the vector that gets mixed into the output is its value.

3. Attention matrix explorer

This first widget uses a deliberately constructed compatibility pattern for the sentence:

The animal did not cross the street because it was tired.

The pattern is designed to make a few relationships visible, including the pronoun link between “it” and “animal”. It is a teaching model, not a claim about a particular trained head.

Try these experiments:

• select “it” and observe the high-weight candidate;
• increase \(d_k\) with scaling disabled;
• turn on the causal mask and notice that future tokens disappear;
• change temperature to make the distribution flatter or sharper;
• click a cell to inspect one query-key pair.

<div style="max-width:1000px;margin:0 auto;color:var(--color-text);font-family:system-ui,sans-serif">
  <div style="display:flex;flex-wrap:wrap;gap:10px;align-items:end;margin-bottom:12px">
    <label style="display:flex;flex-direction:column;gap:4px;font-size:12px;min-width:145px">
      Query token
      <select id="aml-query" style="padding:6px;background:var(--color-bg);color:var(--color-text);border:1px solid var(--color-border)"></select>
    </label>
    <label style="display:flex;flex-direction:column;gap:4px;font-size:12px;min-width:145px">
      Key dimension \(d_k\): <span id="aml-dk-value">64</span>
      <input id="aml-dk" type="range" min="4" max="256" step="4" value="64">
    </label>
    <label style="display:flex;flex-direction:column;gap:4px;font-size:12px;min-width:145px">
      Temperature: <span id="aml-temp-value">1.0</span>
      <input id="aml-temp" type="range" min="0.25" max="3" step="0.05" value="1">
    </label>
    <label style="display:flex;align-items:center;gap:6px;font-size:12px;padding-bottom:7px">
      <input id="aml-scale" type="checkbox" checked>
      divide by \(\sqrt{d_k}\)
    </label>
    <label style="display:flex;align-items:center;gap:6px;font-size:12px;padding-bottom:7px">
      <input id="aml-causal" type="checkbox">
      causal mask
    </label>
  </div>
  <div id="aml-chart" style="width:100%;min-height:385px;border:1px solid var(--color-border);background:var(--color-bg-alt);overflow:auto"></div>
  <div id="aml-readout" style="margin-top:10px;padding:10px 12px;border-left:3px solid var(--color-accent);background:var(--color-bg-alt);font-size:13px;line-height:1.5">
    Select a cell to inspect its score.
  </div>
  <p style="margin:10px 0 0;color:var(--color-text-secondary);font-size:12px">
    Cell color is attention weight. Rows are queries; columns are keys. A row sums to one after softmax.
  </p>
</div>
<script>
(function(){
  const d3 = Solaris.libs.d3;
  const host = document.getElementById('aml-chart');
  const query = document.getElementById('aml-query');
  const dkInput = document.getElementById('aml-dk');
  const tempInput = document.getElementById('aml-temp');
  const scaleInput = document.getElementById('aml-scale');
  const causalInput = document.getElementById('aml-causal');
  const dkValue = document.getElementById('aml-dk-value');
  const tempValue = document.getElementById('aml-temp-value');
  const readout = document.getElementById('aml-readout');
  const tokens = ['The','animal','did','not','cross','the','street','because','it','was','tired'];
  const relations = {
    '8-1':2.7, '8-6':0.35, '8-7':0.75,
    '9-8':2.1, '9-10':0.7,
    '10-8':1.1, '10-9':1.0,
    '4-6':2.0, '6-4':1.1,
    '7-4':0.75, '7-8':0.45,
    '3-4':0.5, '1-0':0.4
  };
  tokens.forEach(function(t,i){
    const o=document.createElement('option');
    o.value=i; o.textContent=i+' · '+t; query.appendChild(o);
  });
  query.value='8';

  function rawScore(i,j){
    const distance = Math.abs(i-j);
    const local = -0.24*distance;
    const relation = relations[i+'-'+j] || 0;
    const lexical = tokens[i].toLowerCase()===tokens[j].toLowerCase() ? 0.25 : 0;
    return local + relation + lexical + 0.08*Math.sin((i+1)*(j+2));
  }
  function softmax(values){
    const mx=d3.max(values);
    const e=values.map(function(v){return Math.exp(v-mx);});
    const z=d3.sum(e);
    return e.map(function(v){return v/z;});
  }
  function render(){
    const q=+query.value, dk=+dkInput.value, temperature=+tempInput.value;
    const scale=scaleInput.checked, causal=causalInput.checked;
    dkValue.textContent=dk;
    tempValue.textContent=temperature.toFixed(2);
    const rows=[];
    tokens.forEach(function(rowToken,i){
      const logits=tokens.map(function(keyToken,j){
        if(causal && j>i) return -Infinity;
        const base=rawScore(i,j);
        return base*Math.sqrt(dk);
      });
      const transformed=logits.map(function(v){
        if(v===-Infinity) return v;
        return (scale ? v/Math.sqrt(dk) : v)/temperature;
      });
      const probs=softmax(transformed.map(function(v){return v===-Infinity?-1e9:v;}));
      tokens.forEach(function(colToken,j){
        rows.push({row:i,col:j,token:rowToken,key:colToken,raw:logits[j],logit:transformed[j],weight:causal&&j>i?0:probs[j]});
      });
    });
    const width=Math.max(700,tokens.length*58+90), cell=48, top=82, left=76;
    host.innerHTML='';
    const svg=d3.select(host).append('svg').attr('width',width).attr('height',top+tokens.length*cell+36);
    const color=d3.scaleSequential(d3.interpolateYlOrRd).domain([0,0.55]);
    svg.append('text').attr('x',left).attr('y',20).attr('font-size',13).attr('font-weight','600')
      .text('Attention weights for every query row');
    svg.append('text').attr('x',left).attr('y',39).attr('font-size',11).attr('fill','currentColor').attr('opacity',.7)
      .text((scale?'scaled':'unscaled')+' logits, temperature '+temperature.toFixed(2)+(causal?' · causal mask on':''));
    svg.append('g').selectAll('text.col').data(tokens).join('text').attr('class','col')
      .attr('x',function(d,i){return left+i*cell+cell/2;}).attr('y',top-13).attr('text-anchor','middle')
      .attr('font-size',11).attr('transform',function(d,i){return 'rotate(-42 '+(left+i*cell+cell/2)+' '+(top-13)+')';})
      .text(function(d,i){return i+' '+d;});
    svg.append('g').selectAll('text.row').data(tokens).join('text').attr('class','row')
      .attr('x',left-10).attr('y',function(d,i){return top+i*cell+cell/2+4;}).attr('text-anchor','end')
      .attr('font-size',11).attr('font-weight',function(d,i){return i===q?'700':'400';})
      .text(function(d,i){return i+' '+d;});
    const cells=svg.append('g').selectAll('g.cell').data(rows).join('g').attr('class','cell')
      .attr('transform',function(d){return 'translate('+(left+d.col*cell)+','+(top+d.row*cell)+')';})
      .style('cursor','pointer')
      .on('click',function(event,d){
        const raw=d.raw===-Infinity?'masked':d.raw.toFixed(3);
        const logit=d.logit===-Infinity?'masked':d.logit.toFixed(3);
        readout.innerHTML='<strong>'+d.row+' '+d.token+'</strong> attending to <strong>'+d.col+' '+d.key+
          '</strong>: raw score <code>'+raw+'</code>, post-scale/temperature logit <code>'+logit+
          '</code>, weight <strong>'+d.weight.toFixed(3)+'</strong>.';
      });
    cells.append('rect').attr('width',cell-2).attr('height',cell-2).attr('rx',3)
      .attr('fill',function(d){return d.weight===0?'transparent':color(d.weight);})
      .attr('stroke',function(d){return d.row===q?'var(--color-text)':'var(--color-border)';})
      .attr('stroke-width',function(d){return d.row===q?1.7:.7;});
    cells.append('text').attr('x',(cell-2)/2).attr('y',cell/2+4).attr('text-anchor','middle')
      .attr('font-size',10).attr('fill',function(d){return d.weight>.28?'#2b2114':'var(--color-text)';})
      .text(function(d){return d.weight===0?'—':d.weight.toFixed(2);});
  }
  [query,dkInput,tempInput,scaleInput,causalInput].forEach(function(el){el.addEventListener('input',render);el.addEventListener('change',render);});
  render();
})();
</script>

The matrix makes two facts visible:

• Attention is a distribution, not a hard pointer. Several keys can receive nonzero weight.
• A causal mask changes the support of the distribution before softmax by replacing illegal logits with \(-\infty\).

4. Why divide by $\sqrt{d_k}$?

Assume the components of \(q\) and \(k\) are independent, zero-mean, unit-variance random variables. Then

$$ q\cdot k=\sum_{i=1}^{d_k}q_i k_i, \qquad \operatorname{Var}(q\cdot k)=d_k. $$

So the standard deviation of the dot product grows like \(\sqrt{d_k}\). Large logits push softmax toward a nearly one-hot distribution, where most derivatives are tiny. Scaling brings the score variance back toward a dimension-independent range.

This is not a decorative normalization. It changes the optimization regime.

5. Softmax microscope: scale, temperature, and entropy

This widget isolates the numerical reason for the scale factor. The underlying score pattern is fixed, while \(d_k\), temperature, and the scaling switch change the logits. The plot is a controlled experiment, not a trained attention head.

• Scaling on: score magnitude stays roughly stable as \(d_k\) grows.
• Scaling off: the largest key quickly monopolizes the probability mass.
• Higher temperature: flatter distribution.
• Lower temperature: sharper distribution.

<div style="max-width:980px;margin:0 auto;font-family:system-ui,sans-serif;color:var(--color-text)">
  <div style="display:flex;flex-wrap:wrap;gap:12px;align-items:end;margin-bottom:12px">
    <label style="display:flex;flex-direction:column;gap:4px;font-size:12px;min-width:150px">
      \(d_k\): <span id="sm-dk-value">64</span>
      <input id="sm-dk" type="range" min="4" max="512" step="4" value="64">
    </label>
    <label style="display:flex;flex-direction:column;gap:4px;font-size:12px;min-width:150px">
      temperature: <span id="sm-temp-value">1.00</span>
      <input id="sm-temp" type="range" min="0.2" max="3" step="0.05" value="1">
    </label>
    <label style="display:flex;align-items:center;gap:6px;padding-bottom:7px;font-size:12px">
      <input id="sm-scale" type="checkbox" checked>
      use \(1/\sqrt{d_k}\)
    </label>
  </div>
  <div style="height:310px;position:relative"><canvas id="sm-chart"></canvas></div>
  <div id="sm-summary" style="padding:10px 12px;margin-top:8px;background:var(--color-bg-alt);border-left:3px solid var(--color-accent);font-size:13px;line-height:1.5"></div>
</div>
<script>
(function(){
  const Chart = Solaris.libs.Chart;
  const labels=['k1','k2','k3','k4','k5','k6','k7','k8'];
  const base=[0.35,0.55,-0.10,0.18,0.40,0.02,-0.24,0.12];
  const dk=document.getElementById('sm-dk');
  const temp=document.getElementById('sm-temp');
  const scale=document.getElementById('sm-scale');
  const chartCanvas=document.getElementById('sm-chart');
  const summary=document.getElementById('sm-summary');
  let chart=null;
  function softmax(a){
    const m=Math.max.apply(null,a), e=a.map(function(x){return Math.exp(x-m);});
    const z=e.reduce(function(s,x){return s+x;},0);
    return e.map(function(x){return x/z;});
  }
  function entropy(p){
    return -p.reduce(function(s,x){return s+(x>0?x*Math.log2(x):0);},0);
  }
  function draw(){
    const d=+dk.value,t=+temp.value,scaled=scale.checked;
    const raw=base.map(function(x){return x*Math.sqrt(d);});
    const logits=raw.map(function(x){return (scaled?x/Math.sqrt(d):x)/t;});
    const p=softmax(logits);
    document.getElementById('sm-dk-value').textContent=d;
    document.getElementById('sm-temp-value').textContent=t.toFixed(2);
    if(chart) chart.destroy();
    chart=new Chart(chartCanvas,{
      type:'bar',
      data:{labels:labels,datasets:[{label:'attention probability',data:p,backgroundColor:p.map(function(x,i){return i===p.indexOf(Math.max.apply(null,p))?'#c48628':'#7597b8';}),borderRadius:4}]},
      options:{responsive:true,maintainAspectRatio:false,plugins:{legend:{display:false},tooltip:{callbacks:{label:function(c){return ' '+(100*c.raw).toFixed(1)+'%';}}}},
        scales:{x:{grid:{display:false}},y:{beginAtZero:true,max:1,ticks:{callback:function(v){return (100*v)+'%';}},grid:{color:'rgba(128,128,128,.16)'}}}}
    });
    const top=p.indexOf(Math.max.apply(null,p));
    summary.innerHTML='<strong>'+(scaled?'Scaled':'Unscaled')+' logits</strong> at \(d_k='+d+'\), temperature '+t.toFixed(2)+
      '. Largest weight: <strong>'+labels[top]+' = '+(100*p[top]).toFixed(1)+'%</strong>. Distribution entropy: <strong>'+
      entropy(p).toFixed(2)+' bits</strong>.';
  }
  [dk,temp,scale].forEach(function(el){el.addEventListener('input',draw);el.addEventListener('change',draw);});
  draw();
})();
</script>

A useful mental model is: the scale factor controls the logit variance, while temperature controls the deliberate sharpness of the final distribution. They are mathematically similar operations, but their roles in the architecture are different.

6. Query, key, and value as geometry

A dot product is easiest to understand geometrically:

$$ q\cdot k = \lVert q\rVert\lVert k\rVert\cos\theta. $$

Alignment matters, but vector magnitude matters too. Learned projections \(W^Q\) and \(W^K\) can rotate and rescale the representation so that a head's notion of compatibility is useful for its task.

The next widget lets you drag a query and key in a 2D plane. It shows:

• angle \(\theta\);
• dot product;
• cosine similarity;
• the projection of \(q\) onto the direction of \(k\).

This is a 2D analogy for the same algebra in a 64-dimensional head.

<div style="max-width:980px;margin:0 auto;font-family:system-ui,sans-serif;color:var(--color-text)">
  <div id="qkg-board" style="width:100%;height:390px;border:1px solid var(--color-border);background:var(--color-bg-alt)"></div>
  <div id="qkg-readout" style="margin-top:10px;padding:10px 12px;border-left:3px solid var(--color-accent);background:var(--color-bg-alt);font-size:13px;line-height:1.55"></div>
  <p style="font-size:12px;color:var(--color-text-secondary);margin:9px 0 0">Drag the arrow endpoints. Positive dot products indicate acute alignment; negative values indicate opposing directions.</p>
</div>
<script>
(function(){
  const host=document.getElementById('qkg-board');
  const readout=document.getElementById('qkg-readout');
  const lib=Solaris.libs.JXG || Solaris.libs.JSXGraph || Solaris.libs.jsxgraph;
  if(!lib || !lib.JSXGraph){
    host.innerHTML='<div style="padding:18px;color:var(--color-text-secondary)">JSXGraph is not available in this runtime.</div>';
    return;
  }
  const board=lib.JSXGraph.initBoard(host,{boundingbox:[-5,5,5,-5],axis:true,grid:true,showCopyright:false,showNavigation:false});
  const origin=board.create('point',[0,0],{fixed:true,visible:false});
  const q=board.create('point',[3.0,1.1],{name:'q',size:5,color:'#c48628',strokeColor:'#c48628',fillColor:'#c48628'});
  const k=board.create('point',[1.1,3.1],{name:'k',size:5,color:'#4f8d75',strokeColor:'#4f8d75',fillColor:'#4f8d75'});
  board.create('arrow',[origin,q],{strokeColor:'#c48628',strokeWidth:3});
  board.create('arrow',[origin,k],{strokeColor:'#4f8d75',strokeWidth:3});
  const projection=board.create('point',[
    function(){const kx=k.X(),ky=k.Y(),den=kx*kx+ky*ky;return den?(q.X()*kx+q.Y()*ky)*kx/den:0;},
    function(){const kx=k.X(),ky=k.Y(),den=kx*kx+ky*ky;return den?(q.X()*kx+q.Y()*ky)*ky/den:0;}
  ],{name:'projection',size:3,color:'#7997b8'});
  board.create('segment',[q,projection],{dash:2,strokeColor:'#7997b8',strokeWidth:2});
  board.create('text',[function(){return q.X()+0.12;},function(){return q.Y()+0.14;},'q'],{fontSize:15,color:'#c48628'});
  board.create('text',[function(){return k.X()+0.12;},function(){return k.Y()+0.14;},'k'],{fontSize:15,color:'#4f8d75'});
  function update(){
    const qx=q.X(),qy=q.Y(),kx=k.X(),ky=k.Y();
    const qn=Math.hypot(qx,qy),kn=Math.hypot(kx,ky);
    const dot=qx*kx+qy*ky, cos=qn&&kn?dot/(qn*kn):0;
    const angle=Math.acos(Math.max(-1,Math.min(1,cos)))*180/Math.PI;
    const proj=kn?dot/kn:0;
    readout.innerHTML='<strong>q · k = '+dot.toFixed(3)+'</strong> · |q| = '+qn.toFixed(3)+' · |k| = '+kn.toFixed(3)+
      '<br>cosine similarity = '+cos.toFixed(3)+' · angle = '+angle.toFixed(1)+'° · signed projection of q onto k = '+proj.toFixed(3);
  }
  board.on('update',update);
  update();
})();
</script>

7. Multi-head attention is a set of learned views

One full-width attention operation can average away distinctions. Multi-head attention instead learns \(h\) separate projections:

$$ \mathrm{head}_i = \mathrm{Attention}(QW_i^Q,KW_i^K,VW_i^V), $$

$$ \mathrm{MultiHead}(Q,K,V) = \mathrm{Concat}(\mathrm{head}_1,\ldots,\mathrm{head}_h)W^O. $$

In the base Transformer, \(h=8\) and each head uses \(d_k=d_v=64\). The total attention cost remains comparable to a single 512-dimensional attention operation, but the model can learn different compatibility geometries at once.

A head is not literally a human-readable “feature detector”. The paper's visualizations show that some heads appear to follow syntactic or anaphoric relationships, but the interpretation is empirical and local. A head's function can be distributed across layers and projections.

A 3D sketch of head diversity

This is a conceptual 3D model. The colored lines do not come from the paper's weights. They visualize how several heads can connect the same query token to different positions or subspaces.

<div style="max-width:980px;margin:0 auto;font-family:system-ui,sans-serif;color:var(--color-text)">
  <div id="mhr-scene" style="height:370px;position:relative;overflow:hidden;border:1px solid var(--color-border);background:var(--color-bg-alt);touch-action:none">
    <div id="mhr-labels" style="position:absolute;inset:0;pointer-events:none;z-index:2"></div>
  </div>
  <div style="display:flex;gap:8px;flex-wrap:wrap;margin:10px 0" id="mhr-buttons"></div>
  <div id="mhr-detail" style="padding:10px 12px;border-left:3px solid #c48628;background:var(--color-bg-alt);font-size:13px;line-height:1.5"></div>
  <p style="font-size:12px;color:var(--color-text-secondary);margin:9px 0 0">Drag to rotate. Click head buttons to hide or show a route. This illustrates diversity of projections, not learned attention weights.</p>
</div>
<script>
(function(){
  const THREE=Solaris.libs.THREE;
  const host=document.getElementById('mhr-scene'), labelLayer=document.getElementById('mhr-labels');
  const detail=document.getElementById('mhr-detail'), buttons=document.getElementById('mhr-buttons');
  if(!THREE){host.innerHTML='<div style="padding:18px;color:var(--color-text-secondary)">Three.js is not available in this runtime.</div>';return;}
  const scene=new THREE.Scene(), camera=new THREE.PerspectiveCamera(42,1,.1,100);
  const renderer=new THREE.WebGLRenderer({antialias:true,alpha:true});
  renderer.setPixelRatio(Math.min(window.devicePixelRatio||1,2)); renderer.setClearColor(0x000000,0);
  renderer.domElement.style.cssText='display:block;width:100%;height:100%;cursor:grab';
  host.insertBefore(renderer.domElement,labelLayer);
  camera.position.set(6,4.2,8); camera.lookAt(0,0,0);
  scene.add(new THREE.HemisphereLight(0xffffff,0x333333,1.5));
  const group=new THREE.Group(); scene.add(group);
  const palette=['#c48628','#4f8d75','#7997b8','#ca6b50','#b06d96','#6e8da3'];
  const tokens=[
    {name:'The',x:-3.1},{name:'animal',x:-2.0},{name:'did',x:-1.0},{name:'not',x:0},
    {name:'cross',x:1.0},{name:'the',x:2.0},{name:'street',x:3.0}
  ];
  tokens.forEach(function(t){
    const mesh=new THREE.Mesh(new THREE.SphereGeometry(.16,18,12),new THREE.MeshStandardMaterial({color:0x9a9a9a,roughness:.4}));
    mesh.position.set(t.x,0,0); group.add(mesh);
    const label=document.createElement('div');label.textContent=t.name;label.style.cssText='position:absolute;transform:translate(-50%,-50%);font-size:11px;padding:3px 5px;background:var(--color-bg);border:1px solid var(--color-border);white-space:nowrap';labelLayer.appendChild(label);t.mesh=mesh;t.label=label;
  });
  const qIndex=3, qPoint=new THREE.Vector3(tokens[qIndex].x,.35,0);
  const qMesh=new THREE.Mesh(new THREE.IcosahedronGeometry(.25,2),new THREE.MeshStandardMaterial({color:0xc48628,emissive:0xc48628,emissiveIntensity:.2}));
  qMesh.position.copy(qPoint);group.add(qMesh);
  const heads=[
    {name:'head 1 · local syntax',target:2,y:.85,z:.2,color:palette[0],text:'A local-syntax head can prefer the immediately preceding function word.'},
    {name:'head 2 · subject link',target:1,y:1.15,z:-.4,color:palette[1],text:'A subject-oriented head can connect a query to the noun that supplies context.'},
    {name:'head 3 · object link',target:4,y:1.45,z:.55,color:palette[2],text:'Another head can focus on the object or predicate neighborhood.'},
    {name:'head 4 · boundary/context',target:6,y:1.75,z:-.6,color:palette[3],text:'A longer-range head can allocate weight to a distant contextual token.'}
  ];
  const routes=[];
  function makeRoute(h){
    const target=new THREE.Vector3(tokens[h.target].x,h.y,h.z), start=new THREE.Vector3(qPoint.x,h.y*.18,qPoint.z);
    const geo=new THREE.BufferGeometry().setFromPoints([start,target]);
    const mat=new THREE.LineBasicMaterial({color:h.color,transparent:true,opacity:.9});
    const line=new THREE.Line(geo,mat);group.add(line);
    const ball=new THREE.Mesh(new THREE.SphereGeometry(.12,14,10),new THREE.MeshStandardMaterial({color:h.color,emissive:new THREE.Color(h.color),emissiveIntensity:.16}));
    ball.position.copy(target);group.add(ball);h.line=line;h.ball=ball;
  }
  heads.forEach(function(h){
    makeRoute(h);
    const b=document.createElement('button');b.type='button';b.textContent=h.name;b.style.cssText='padding:6px 8px;border:1px solid '+h.color+';background:transparent;color:var(--color-text);border-radius:4px;font-size:12px';
    b.onclick=function(){h.line.visible=!h.line.visible;h.ball.visible=h.line.visible;b.style.opacity=h.line.visible?'1':'.4';detail.style.borderLeftColor=h.color;detail.innerHTML='<strong>'+h.name+'</strong><br>'+h.text;};buttons.appendChild(b);h.button=b;
  });
  let dragging=false,last={x:0,y:0},rot={x:-.22,y:.42};
  function render(){
    group.rotation.set(rot.x,rot.y,0);group.updateMatrixWorld(true);
    const w=Math.max(1,host.clientWidth),h=Math.max(1,host.clientHeight);
    camera.aspect=w/h;camera.updateProjectionMatrix();renderer.setSize(w,h,false);renderer.render(scene,camera);
    tokens.forEach(function(t){const v=t.mesh.getWorldPosition(new THREE.Vector3());v.project(camera);t.label.style.left=((v.x*.5+.5)*w)+'px';t.label.style.top=((-v.y*.5+.5)*h)+'px';});
  }
  renderer.domElement.addEventListener('pointerdown',function(e){dragging=true;last={x:e.clientX,y:e.clientY};renderer.domElement.setPointerCapture(e.pointerId);renderer.domElement.style.cursor='grabbing';});
  renderer.domElement.addEventListener('pointermove',function(e){if(!dragging)return;rot.y+=(e.clientX-last.x)*.008;rot.x=Math.max(-.8,Math.min(.5,rot.x+(e.clientY-last.y)*.008));last={x:e.clientX,y:e.clientY};render();});
  renderer.domElement.addEventListener('pointerup',function(e){dragging=false;renderer.domElement.releasePointerCapture(e.pointerId);renderer.domElement.style.cursor='grab';});
  detail.innerHTML='<strong>Query token: “not”</strong><br>Each colored route is a different learned projection and attention pattern. The output is formed after these head-specific mixtures are concatenated and projected again.';
  new ResizeObserver(render).observe(host);render();
})();
</script>

8. Position is not implicit

Self-attention is permutation-equivariant: if the same set of token vectors is permuted in the same way, attention alone has no built-in reason to know the original order. Translation needs order, so the paper adds a positional vector to each token embedding:

$$ x_{\mathrm{input}}(pos)=\mathrm{Embedding}(token)+PE(pos). $$

For position \(pos\) and dimension index \(i\),

$$ PE(pos,2i)=\sin\left(\frac{pos}{10000^{2i/d_{\mathrm{model}}}}\right), \qquad PE(pos,2i+1)=\cos\left(\frac{pos}{10000^{2i/d_{\mathrm{model}}}}\right). $$

The dimensions have geometrically increasing wavelengths. Low-index dimensions vary quickly with position; high-index dimensions vary slowly. The paper hypothesized that relative offsets could be represented by linear transformations of these encodings and reported nearly identical development performance when learned positional embeddings were used instead.

Positional encoding explorer

This widget draws a position-by-dimension heatmap and the sinusoid traces for selected channels. Move the position marker, change the base, and watch how the encoding changes. It is showing the fixed sinusoidal formula, not an embedding learned from data.

<div style="max-width:980px;margin:0 auto;font-family:system-ui,sans-serif;color:var(--color-text)">
  <div style="display:flex;flex-wrap:wrap;gap:12px;align-items:end;margin-bottom:10px">
    <label style="display:flex;flex-direction:column;gap:4px;font-size:12px;min-width:170px">
      selected position: <span id="pel-pos-value">18</span>
      <input id="pel-pos" type="range" min="0" max="63" step="1" value="18">
    </label>
    <label style="display:flex;flex-direction:column;gap:4px;font-size:12px;min-width:170px">
      sinusoid base: <span id="pel-base-value">10000</span>
      <input id="pel-base" type="range" min="1000" max="20000" step="100" value="10000">
    </label>
  </div>
  <div id="pel-canvas" style="width:100%;height:390px;border:1px solid var(--color-border);background:var(--color-bg-alt);overflow:hidden"></div>
  <div id="pel-readout" style="padding:10px 12px;margin-top:9px;background:var(--color-bg-alt);border-left:3px solid var(--color-accent);font-size:13px;line-height:1.5"></div>
  <p style="font-size:12px;color:var(--color-text-secondary);margin:9px 0 0">Top: encoding value by position and dimension. Bottom: the first eight dimensions at the selected position.</p>
</div>
<script>
(function(){
  const P5=Solaris.libs.p5, host=document.getElementById('pel-canvas');
  const posInput=document.getElementById('pel-pos'), baseInput=document.getElementById('pel-base');
  const posValue=document.getElementById('pel-pos-value'), baseValue=document.getElementById('pel-base-value'), readout=document.getElementById('pel-readout');
  if(!P5){host.innerHTML='<div style="padding:18px;color:var(--color-text-secondary)">p5.js is not available in this runtime.</div>';return;}
  const dims=32, positions=64, cells=16;
  function pe(pos,dim,base){
    const i=Math.floor(dim/2), angle=pos/Math.pow(base,(2*i)/dims);
    return dim%2===0?Math.sin(angle):Math.cos(angle);
  }
  const sketch=function(p){
    function drawLabel(txt,x,y,size,fill){p.noStroke();p.fill(fill||'#777');p.textSize(size||11);p.text(txt,x,y);}
    p.setup=function(){const c=p.createCanvas(760,390);c.parent(host);p.pixelDensity(1);p.noLoop();drawAll();};
    function drawAll(){
      const w=p.width,h=p.height,selected=+posInput.value,base=+baseInput.value;
      p.background(getComputedStyle(document.documentElement).getPropertyValue('--color-bg-alt')||'#f5f5f2');
      const left=50,top=35,heatW=Math.min(w-72,positions*cells),heatH=positions?dims*7:220;
      const cellW=heatW/positions,cellH=heatH/dims;
      for(let r=0;r<dims;r++){
        for(let c=0;c<positions;c++){
          const v=pe(c,r,base), amt=Math.round(127+127*v);
          p.noStroke();p.fill(amt,Math.max(0,amt-35),Math.max(0,255-amt),210);p.rect(left+c*cellW,top+r*cellH,cellW+0.25,cellH+0.25);
        }
      }
      p.stroke('#c48628');p.strokeWeight(2);p.line(left+(selected+.5)*cellW,top,left+(selected+.5)*cellW,top+heatH);
      drawLabel('position →',left+heatW/2,top+heatH+19,11);
      drawLabel('dimension',8,top+heatH/2,11);
      drawLabel('sinusoidal encoding: red/green/blue indicates value across dimensions',left,17,12);
      const barsTop=top+heatH+47,barW=Math.max(18,(heatW-20)/8);
      p.stroke('#777');p.line(left,barsTop,left+barW*8,barsTop);
      for(let d=0;d<8;d++){
        const v=pe(selected,d,base), bh=v*48;
        p.stroke(d%2===0?'#c48628':'#4f8d75');p.strokeWeight(3);p.line(left+d*barW+barW/2,barsTop,left+d*barW+barW/2,barsTop-bh);
        drawLabel(String(d),left+d*barW+barW/2-3,barsTop+17,10);
      }
      drawLabel('first eight dimensions at position '+selected,left,barsTop+36,11);
      posValue.textContent=selected;baseValue.textContent=base;
      const vals=[];for(let d=0;d<8;d++)vals.push(pe(selected,d,base).toFixed(3));
      readout.innerHTML='<strong>PE('+selected+', ·)</strong> = ['+vals.join(', ')+', …] · Even dimensions use sine; odd dimensions use cosine.';
    }
    posInput.addEventListener('input',function(){p.redraw();});
    baseInput.addEventListener('input',function(){p.redraw();});
    p.draw=drawAll;
  };
  new P5(sketch);
})();
</script>

A useful property follows from the sine addition identity. For a fixed offset \(k\), the pair at position \(pos+k\) can be written as a linear transformation of the pair at \(pos\). That gives the network a route to represent relative displacement without a separate lookup table.

9. What the value mixing actually does

Suppose a query attends to three values:

$$ v_1=-2,\quad v_2=0.5,\quad v_3=3, \qquad \alpha=[0.2,0.3,0.5]. $$

Then the output is

$$ o=0.2(-2)+0.3(0.5)+0.5(3)=1.1. $$

The output is not “the token with the maximum attention”. It is a learned weighted combination in representation space. This distinction matters when interpreting attention maps: a large weight tells us which value contributes strongly in that head, but the value vector itself may encode many features.

A physical analogy for weighted sums

The following widget uses Matter.js as a deliberately imperfect but concrete analogy. Each token is a draggable point on a value axis. Its attention weight controls the size of the point. The vertical marker is the weighted average:

$$ \bar v=\frac{\sum_i\alpha_i v_i}{\sum_i\alpha_i}. $$

Drag points left or right. The output moves according to both content position and attention weight. This is not how Transformer tensors are implemented; it is a visualization of the weighted-sum operation.

<div style="max-width:980px;margin:0 auto;font-family:system-ui,sans-serif;color:var(--color-text)">
  <div id="wvm-stage" style="height:330px;border:1px solid var(--color-border);background:var(--color-bg-alt);position:relative;overflow:hidden"></div>
  <div style="display:flex;flex-wrap:wrap;gap:8px;margin:9px 0" id="wvm-controls"></div>
  <div id="wvm-readout" style="padding:10px 12px;background:var(--color-bg-alt);border-left:3px solid var(--color-accent);font-size:13px;line-height:1.5"></div>
  <p style="font-size:12px;color:var(--color-text-secondary);margin:9px 0 0">Preset buttons change attention weights. Drag the colored tokens to change their scalar values.</p>
</div>
<script>
(function(){
  const M=Solaris.libs.Matter, host=document.getElementById('wvm-stage');
  const controls=document.getElementById('wvm-controls'), readout=document.getElementById('wvm-readout');
  if(!M){host.innerHTML='<div style="padding:18px;color:var(--color-text-secondary)">Matter.js is not available in this runtime.</div>';return;}
  const Engine=M.Engine,Render=M.Render,Runner=M.Runner,Bodies=M.Bodies,Composite=M.Composite,Mouse=M.Mouse,MouseConstraint=M.MouseConstraint,Events=M.Events;
  const W=760,H=330,left=58,right=702,axisY=230;
  const engine=Engine.create();engine.gravity.y=0;engine.gravity.x=0;
  const render=Render.create({element:host,engine:engine,options:{width:W,height:H,wireframes:false,background:'transparent',pixelRatio:1}});
  const wallTop=Bodies.rectangle(W/2,80,W,12,{isStatic:true,render:{fillStyle:'transparent'}});
  const wallBottom=Bodies.rectangle(W/2,286,W,12,{isStatic:true,render:{fillStyle:'transparent'}});
  Composite.add(engine.world,[wallTop,wallBottom]);
  const colors=['#c48628','#4f8d75','#7997b8','#ca6b50','#b06d96'];
  const names=['subject','verb','object','cause','state'];
  const weights=[.14,.22,.16,.30,.18];
  const bodies=[];
  function xToValue(x){return ((x-left)/(right-left))*6-3;}
  function valueToX(v){return left+((v+3)/6)*(right-left);}
  for(let i=0;i<5;i++){
    const body=Bodies.circle(valueToX([-1.8,-.8,.2,1.1,2.1][i]),axisY,22,{restitution:.4,frictionAir:.08,render:{fillStyle:colors[i],strokeStyle:colors[i],lineWidth:2}});
    body.plugin={token:names[i],weight:weights[i],color:colors[i]};bodies.push(body);Composite.add(engine.world,body);
  }
  const mouse=Mouse.create(render.canvas),mc=MouseConstraint.create(engine,{mouse:mouse,constraint:{stiffness:.18,render:{visible:false}}});
  Composite.add(engine.world,mc);render.mouse=mouse;
  const presets=[
    {name:'balanced',values:[.2,.2,.2,.2,.2]},
    {name:'focus subject',values:[.62,.10,.10,.08,.10]},
    {name:'focus cause',values:[.08,.10,.10,.62,.10]},
    {name:'focus state',values:[.08,.10,.10,.10,.62]}
  ];
  presets.forEach(function(preset){
    const b=document.createElement('button');b.type='button';b.textContent=preset.name;b.style.cssText='padding:6px 9px;border:1px solid var(--color-border);background:transparent;color:var(--color-text);border-radius:4px;font-size:12px';
    b.onclick=function(){preset.values.forEach(function(v,i){bodies[i].plugin.weight=v;});update();};controls.appendChild(b);
  });
  function update(){
    let num=0,den=0; bodies.forEach(function(b){const v=xToValue(b.position.x),w=b.plugin.weight;num+=v*w;den+=w;});
    const mean=num/den;
    readout.innerHTML='<strong>weighted output = '+mean.toFixed(3)+'</strong> · '+bodies.map(function(b){return b.plugin.token+' weight '+b.plugin.weight.toFixed(2)+' at value '+xToValue(b.position.x).toFixed(2);}).join(' · ');
  }
  Events.on(render,'afterRender',function(){
    const ctx=render.context;
    ctx.save();
    ctx.strokeStyle='#777';ctx.lineWidth=1;ctx.beginPath();ctx.moveTo(left,axisY);ctx.lineTo(right,axisY);ctx.stroke();
    for(let i=-3;i<=3;i++){const x=valueToX(i);ctx.beginPath();ctx.moveTo(x,axisY-6);ctx.lineTo(x,axisY+6);ctx.stroke();ctx.fillStyle='#777';ctx.font='11px sans-serif';ctx.fillText(String(i),x-3,axisY+23);}
    let num=0,den=0;bodies.forEach(function(b){const v=xToValue(b.position.x),w=b.plugin.weight;num+=v*w;den+=w;});
    const mean=num/den,mx=valueToX(mean);
    ctx.strokeStyle='#c48628';ctx.lineWidth=3;ctx.beginPath();ctx.moveTo(mx,100);ctx.lineTo(mx,axisY+13);ctx.stroke();
    ctx.fillStyle='#c48628';ctx.font='bold 12px sans-serif';ctx.fillText('weighted output '+mean.toFixed(2),Math.min(mx+7,right-120),95);
    bodies.forEach(function(b){ctx.fillStyle='#555';ctx.font='11px sans-serif';ctx.fillText(b.plugin.token,b.position.x-25,b.position.y+37);});
    ctx.restore();
    update();
  });
  Events.on(mc,'startdrag',update);Events.on(mc,'enddrag',update);
  Render.run(render);Runner.run(Runner.create(),engine);update();
})();
</script>

10. The feed-forward network is not optional decoration

Each Transformer layer contains attention and a position-wise feed-forward network:

$$ \mathrm{FFN}(x) = \max(0,xW_1+b_1)W_2+b_2. $$

The same \(W_1\) and \(W_2\) are applied independently at every sequence position, but parameters differ between layers. In the base model \(d_{\mathrm{model}}=512\) and \(d_{\mathrm{ff}}=2048\).

This separation is useful:

• attention mixes information across positions;
• the FFN performs a richer nonlinear transformation within each position;
• residual connections let each block add an update to the existing representation.

The Transformer alternates global communication and local computation. Saying “it is just attention” hides half of the layer.

11. Masks and the direction of information flow

The encoder's self-attention is bidirectional: every input token can attend to every input token.

The decoder has two distinct attention operations:

1. masked self-attention over already-generated target tokens;
2. encoder-decoder attention over the complete encoded source sequence.

For a target position \(i\), future target positions are masked:

$$ M_{ij}= \begin{cases} 0 & j\le i,\\ -\infty & j>i. \end{cases} $$

The mask is added to the pre-softmax scores. Because \(\exp(-\infty)=0\), illegal positions receive exactly zero probability. This is why the model can train all target positions in parallel with teacher forcing while still representing an autoregressive factorization during inference.

12. Residuals, normalization, and optimization

The published formulation is post-norm:

$$ x\longmapsto\mathrm{LayerNorm}\bigl(x+\mathrm{Sublayer}(x)\bigr). $$

The residual branch gives each sub-layer a stable route to preserve or incrementally modify information. Layer normalization normalizes features within a token representation rather than across a batch.

The paper trained with Adam and a warmup/inverse-square-root schedule:

$$ \mathrm{lrate} = d_{\mathrm{model}}^{-0.5} \min\left( \mathrm{step}^{-0.5}, \mathrm{step}\cdot\mathrm{warmup\_steps}^{-1.5} \right). $$

The schedule rises linearly during the first 4000 steps, then decays proportional to the inverse square root of the step number. The point of the warmup is practical: early updates should not be too large while the randomly initialized attention projections are still uncalibrated.

flowchart LR
    X[residual stream x] --> A[attention or FFN]
    A --> P[dropout]
    X --> R[add]
    P --> R
    R --> N[LayerNorm]
    N --> Y[updated representation]

13. Why self-attention won the path-length argument

The paper compares three quantities per layer: complexity, sequential operations, and maximum path length.

Self-attention minimizes the sequential path but pays for all pairwise interactions. The quadratic term becomes the limiting factor for very long sequences, which is why the paper suggested restricted attention as a future direction.

Complexity and path-length explorer

The curves below are normalized asymptotic quantities, not wall-clock timings. Constants, GPU kernels, memory bandwidth, batching, and implementation details are intentionally excluded.

<div style="max-width:1000px;margin:0 auto;font-family:system-ui,sans-serif;color:var(--color-text)">
  <div style="display:flex;flex-wrap:wrap;gap:12px;align-items:end;margin-bottom:10px">
    <label style="display:flex;flex-direction:column;gap:4px;font-size:12px">view
      <select id="cpl-mode" style="padding:6px;background:var(--color-bg);color:var(--color-text);border:1px solid var(--color-border)">
        <option value="cost">per-layer complexity</option>
        <option value="path">maximum path length</option>
      </select>
    </label>
    <label style="display:flex;flex-direction:column;gap:4px;font-size:12px;min-width:145px">
      representation \(d\): <span id="cpl-d-value">512</span>
      <input id="cpl-d" type="range" min="64" max="2048" step="64" value="512">
    </label>
    <label style="display:flex;flex-direction:column;gap:4px;font-size:12px;min-width:145px">
      conv kernel \(k\): <span id="cpl-k-value">3</span>
      <input id="cpl-k" type="range" min="2" max="16" step="1" value="3">
    </label>
    <label style="display:flex;flex-direction:column;gap:4px;font-size:12px;min-width:145px">
      restricted neighborhood \(r\): <span id="cpl-r-value">64</span>
      <input id="cpl-r" type="range" min="8" max="256" step="8" value="64">
    </label>
  </div>
  <div style="height:355px;position:relative"><canvas id="cpl-chart"></canvas></div>
  <div id="cpl-summary" style="padding:10px 12px;margin-top:9px;background:var(--color-bg-alt);border-left:3px solid var(--color-accent);font-size:13px;line-height:1.5"></div>
</div>
<script>
(function(){
  const Chart=Solaris.libs.Chart, canvas=document.getElementById('cpl-chart'), mode=document.getElementById('cpl-mode');
  const dInput=document.getElementById('cpl-d'), kInput=document.getElementById('cpl-k'), rInput=document.getElementById('cpl-r'), summary=document.getElementById('cpl-summary');
  const nVals=[16,32,64,128,256,512,1024,2048,4096]; let chart=null;
  function draw(){
    const d=+dInput.value,k=+kInput.value,r=+rInput.value,isCost=mode.value==='cost';
    document.getElementById('cpl-d-value').textContent=d;document.getElementById('cpl-k-value').textContent=k;document.getElementById('cpl-r-value').textContent=r;
    const data=[
      {label:'self-attention',color:'#c48628',values:nVals.map(function(n){return isCost?n*n*d:1;})},
      {label:'recurrent',color:'#4f8d75',values:nVals.map(function(n){return isCost?n*d*d:n;})},
      {label:'convolutional',color:'#7997b8',values:nVals.map(function(n){return isCost?k*n*d*d:Math.log(n)/Math.log(k);})},
      {label:'restricted attention',color:'#ca6b50',values:nVals.map(function(n){return isCost?r*n*d:n/r;})}
    ];
    if(chart)chart.destroy();
    chart=new Chart(canvas,{type:'line',data:{labels:nVals,datasets:data.map(function(s){return {label:s.label,data:s.values,borderColor:s.color,backgroundColor:s.color,pointRadius:3,tension:.15};})},
      options:{responsive:true,maintainAspectRatio:false,interaction:{mode:'index',intersect:false},plugins:{legend:{position:'bottom'}},scales:{x:{title:{display:true,text:'sequence length n'},grid:{display:false}},y:{type:isCost?'logarithmic':'linear',title:{display:true,text:isCost?'relative operation count':'relative path length'},grid:{color:'rgba(128,128,128,.16)'}}}}});
    const idx=nVals.indexOf(512),values=data.map(function(s){return s.values[idx];});
    summary.innerHTML='<strong>At n=512</strong>: '+data.map(function(s,i){return s.label+' '+(isCost?values[i].toExponential(2):values[i].toFixed(2));}).join(' · ')+
      '. '+(isCost?'Full attention grows quadratically in n; restricted attention replaces n² with r·n.':'Full attention keeps a constant path while recurrence grows linearly; restricted attention pays a path-length cost of n/r.');
  }
  [dInput,kInput,rInput,mode].forEach(function(el){el.addEventListener('input',draw);el.addEventListener('change',draw);});draw();
})();
</script>

14. The original evidence

Translation results

The headline result was the Transformer big model:

• WMT 2014 English-German: 28.4 BLEU.
• WMT 2014 English-French: Table 2 reports 41.8 BLEU.
• Training: 8 NVIDIA P100 GPUs for 3.5 days for the big model.
• Base model: 100,000 steps, about 12 hours on the same 8-GPU machine.

The paper's prose in Section 6.1 says 41.0 for English-French, while Table 2 reports 41.8. That is an internal inconsistency in the paper text. This note preserves the table value as the main reported comparison and flags the narrative value instead of silently choosing one.

<div style="max-width:980px;margin:0 auto;font-family:system-ui,sans-serif;color:var(--color-text)">
  <div style="height:330px;position:relative"><canvas id="orc-chart"></canvas></div>
  <div style="padding:10px 12px;background:var(--color-bg-alt);border-left:3px solid var(--color-accent);font-size:13px;line-height:1.5">
    These are the paper's reported BLEU values. BLEU is a corpus-level translation metric and should not be treated as a universal measure of language-model quality.
  </div>
</div>
<script>
(function(){
  const Chart=Solaris.libs.Chart, canvas=document.getElementById('orc-chart');
  new Chart(canvas,{type:'bar',data:{labels:['Transformer base','Transformer big'],datasets:[
    {label:'WMT14 EN→DE BLEU',data:[27.3,28.4],backgroundColor:'#c48628'},
    {label:'WMT14 EN→FR BLEU',data:[38.1,41.8],backgroundColor:'#4f8d75'}
  ]},options:{responsive:true,maintainAspectRatio:false,plugins:{legend:{position:'bottom'},tooltip:{callbacks:{label:function(c){return ' '+c.dataset.label+': '+c.raw;}}}},scales:{x:{grid:{display:false}},y:{beginAtZero:true,max:50,title:{display:true,text:'BLEU'}}}}});
})();
</script>

The comparison was historically important, but it is not a clean modern benchmark:

• earlier systems used different architectures, data processing, and search procedures;
• the paper compares against published literature rather than a single controlled re-run;
• BLEU scores depend on tokenization, test set, decoding, and reference conventions;
• “state of the art” is bounded by the 2017 task and data setup.

Constituency parsing

The Transformer also generalized beyond translation. With a four-layer Transformer:

• WSJ-only discriminative setting: 91.3 F1;
• semi-supervised setting: 92.7 F1.

The paper notes that the model exceeded many reported baselines despite limited task-specific tuning, although some recurrent and generative parsers remained higher.

15. Ablations: what mattered?

The base model's English-German development BLEU was 25.8. The paper's ablations show patterns rather than a single magic hyperparameter:

The ablations support a systems interpretation:

• multi-head attention is useful, but head count is not monotonic;
• positional encoding choice was not the main determinant of quality in this setup;
• model capacity and regularization mattered;
• the FFN width was a meaningful part of the model, not an afterthought.

16. Training setup and reproducibility details

The original experiments used:

At inference time, the paper used beam search with beam size 4 and length penalty \(\alpha=0.6\) for translation. It averaged the last five checkpoints for base models and the last twenty checkpoints for big models.

Those details matter because the result is not produced by architecture alone. Data scale, batch construction, optimizer schedule, checkpoint averaging, and decoding are part of the experimental object.

17. What the paper means by “interpretable”

The authors wrote that self-attention might yield more interpretable models and inspected attention distributions. Their appendix shows examples where heads appear to:

• follow a long-distance dependency;
• connect a verb to a relevant subject or complement;
• resolve anaphora such as “its” to “Law”.

This is valuable as a mechanistic probe, but it should not be inflated into a general explanation guarantee.

An attention weight answers a narrow question:

For this head, layer, query position, key position, and forward pass, how much weight was assigned to that value before the next transformations?

It does not by itself answer:

• which input caused the final output;
• whether removing that token changes the output;
• whether another head or residual path would compensate;
• whether the model would make the same decision under a counterfactual;
• how the value vectors and output projection transform the mixture.

A disciplined interpretation therefore combines attention visualization with interventions, ablations, activation patching, gradients, causal tracing, or behavioral tests. The original paper's visualizations are evidence of learned structure, not a complete causal explanation.

18. What survives in modern language models?

The Transformer paper introduced an encoder-decoder sequence-to-sequence architecture. Many current language models use a decoder-only variant:

The through-line is not “all modern LLMs are exactly the 2017 Transformer.” It is that the paper established a compositional pattern: token representations, position information, learned pairwise mixing, nonlinear per-token transformation, residual depth, and autoregressive masking.

19. Failure modes and limits

The paper's own computational analysis points toward several limits:

1. Quadratic attention: full self-attention materializes \(n\times n\) interactions per layer.
2. Autoregressive decoding: even if training is parallel, decoder generation still emits one token at a time.
3. Memory pressure: storing attention scores and activations can dominate long-context workloads.
4. Interpretation ambiguity: a readable heatmap is not a causal explanation.
5. Benchmark dependence: BLEU and the paper's baselines measure a specific historical task.
6. Architecture dependence: the paper's conclusions do not automatically transfer to every sparse, recurrent, state-space, or retrieval-augmented model.
7. Position extrapolation: the paper's motivation for sinusoidal encodings is a hypothesis, not a guarantee of unlimited useful context.

20. A compact mental simulation

When reading an attention layer, ask five questions:

1. What is the query asking for?
2. Which keys are compatible with that request?
3. How does scaling and masking change the logits before softmax?
4. What content lives in the corresponding values?
5. What does the residual and FFN do with the mixture afterward?

That sequence prevents the common mistake of equating an attention map with the model's complete reasoning.

21. Key takeaways

• The Transformer replaced sequential recurrence with globally connected attention.
• \(QK^\mathsf{T}\) selects relevance; \(V\) supplies the mixed content.
• The \(\sqrt{d_k}\) scale prevents high-dimensional dot products from saturating softmax.
• Multi-head attention creates several learned compatibility spaces.
• Position must be injected because attention alone does not encode order.
• The FFN performs per-position nonlinear computation after cross-token mixing.
• Residual connections and normalization make deep stacking trainable.
• The central computational benefit is constant sequential path length, not zero cost.
• The central computational limitation is the quadratic attention matrix.
• The original interpretability claim is best read as “attention can expose learned structure,” not “attention is a complete causal explanation.”
• The paper's enduring idea is a modular architecture, not one frozen implementation.

22. Primary source and further reading

• Vaswani et al., “Attention Is All You Need,” arXiv:1706.03762
• Direct PDF of the cited paper version
• Tensor2Tensor code referenced by the paper
• The original paper's appendix attention visualizations
• Related notes in this vault: technical-architecture and projects/attention-lab/project-brief

Provenance note

This note was derived from the supplied file 1706.03762v7.pdf and cross-checked against the paper's equations, architecture description, complexity table, training details, results tables, ablations, conclusion, and appendix. Equations are typeset for explanation; interactive widgets marked as simulations use synthetic values unless explicitly labeled as paper-reported measurements.