Tokenisation: what this star will be
A model never sees words, only the pieces it was taught to cut.
By Constellate · 9 September 2026 · the short form
Tokenisation is a star on the Constellate sky: placed by meaning, named, and connected to the stars around it. This is the short form of its post: the line it carries, the area we colour Computing, and what stands beside it today.
The line it carries on the sky: A model never sees words, only the pieces it was taught to cut.
What stands beside it
The stars nearest to it are Attention, The Context Window, Embeddings and The Generation Loop, and Attention, Embeddings and The Generation Loop are already open to walk. When Tokenisation opens, the conversation will land it among them, with a line drawn to each idea it touches.
Ask for this star
Every star is lit by demand. Ask for Tokenisation and it moves up the list; when it opens, this post becomes the full essay. The open stars beside it can be walked today, and the first concept of every course is free.