Attention

What Attention means

Attention is the part of a language model that, for each word it produces, scores every piece of the input for how much it should count right now, and lets the highest scores dominate. For the notice period, the words near "notice" and "days" won; for the follow-up, a different set won, and page four scored low. Owning this changes how we work with long inputs. We stop trusting that "it has read the document" and start putting the thing that matters where it will score: named plainly, near the question, repeated if it must be. We stop burying the key clause in the middle. Before: paste everything and hope. After: point the model at what counts, because that is the only reading it does.

Not to be confused with

Attention in AI is not attention in psychology. The human kind is a limited spotlight. The model kind is a way of weighing every word against every other word at once, and it has no limit of the same sort.

Attention examples: where it shows up

• At work: the summary of the long thread was accurate about the loud messages and missed the one quiet line that held the decision.
• At home: when we ask about a recipe halfway through a long chat, the answer is shaped by whatever in the conversation scored highest, not by what we meant.
• In the news: each jump in how much text a model can take in at once is a story about attention, and about the cost of weighing every word against every other.

Where it sits

Concept 2 of 3 in How AI Thinks, a journey in the course Understand AI.

See where Attention sits on the sky →

Nearest on the sky

• Abstraction
• The Context Window
• Embeddings
• Flow
• Gradient Descent
• Mindfulness

A filled dot marks a star that is already open to walk.