How a language model reads, writes and remembers
A language model writes its reply one token at a time, with every scale of the text closing like a clock. Before that it takes in its prompt in one flash, and before the flash the context has to be gathered; on the beach it is gathered from pscale blocks, read along spindles. Each section’s paragraph writes itself out beside its picture, with a clock at its side whose rings close as it goes: the inner ring for the phrase, the middle for the sentence, the outer for the whole paragraph.
Part 1 · completion
This is the part you usually see: a reply being written, one token at a time. Each ring is one scale of completion, turning anticlockwise as its current unit fills, with ticks where the parts inside it end. At the start every ring closes together: that is the reading, which part 2 shows as a flash. Then they close in order, word inside phrase inside sentence, and the outer rings close only after the inner ones have turned many times.
Prompt
Phrases are guessed from punctuation and joining words unless you mark them with |.
Part 2 · recognition
Part 3 · before the flash
Part 4 · the beach
Compilation draws on the beach, and the beach can be read at three levels: as physics, how a block forms and a spindle is drawn from it; as chemistry, how blocks of different kinds bond into larger meanings; and as biology, how models run their loops through the blocks over time.
Part 4a · physics
Part 4b · chemistry
Part 4c · biology
Part 5 · overview
Notes
Tokens here are approximate. A punctuation mark counts as one, a word of up to seven letters as one, and longer words split into pieces of about six letters. Real tokenisers differ in detail but also average a little over one token a word in English.
Phrases in the sample are marked by hand. In your own text they break at commas, semicolons, colons and dashes, and before joining words such as and, but, because and which, with at most eight words to a phrase. Sentences end at full stops, question marks and exclamation marks, paragraphs at blank lines, and a line starting with # begins a section. The list under the rings gives each scale’s count and how many of the scale below it holds: the chunk sizes.
A model reads the whole prompt in one pass: every token attends to every earlier token at the same time, so every scale of the prompt is taken in at once. Writing runs the other way, one token at a time, each chosen from a spread over every possible next token. The cube in part 2 is a picture of this, not a measurement: real models have billions of weights and thousands of dimensions, and the threads, clouds and rungs stand for the pattern rather than any particular model’s values.
Chunking: George Miller, “The magical number seven, plus or minus two” (1956), and Nelson Cowan’s estimate of about four (2001). Recursion: Hauser, Chomsky and Fitch, “The faculty of language” (2002). Types: Bertrand Russell, The Principles of Mathematics (1903) and his theory of types (1908). Syntagmatic and associative relations: Ferdinand de Saussure, Course in General Linguistics (1916); paradigmatic is the later name for the associative axis. The death of the author: Roland Barthes (1967). Stigmergy: Pierre-Paul Grassé, from his work on termites (1959).
The counting line, the wrap at each run of nines and the zero-slot summaries follow the pscale block conventions for history blocks; the same digits read containment in a place block.