The Machines That Learned to Talk: Large Language Models

The pretraining objective is next-token prediction and nothing else. The model is shown a stretch of text with the continuation hidden, it produces a probability distribution over what comes next, and the gap between its guess and the actual next token is used to adjust its weights. Repeat that across a corpus large enough and the system that emerges can hold an argument, translate a contract, or write a working function, none of which it was separately taught. Our own research file draws the consequence that follows, and it is the hinge of everything below: fluent falsehood 'arises from the fundamental training objective (next-token prediction)', because 'the model optimizes for plausibility, not truth'. A machine built to produce likely continuations will produce a likely continuation whether or not a true one exists. Hold that sentence; sections 07 and 09 are both arguments about how far it reaches.

The reflex to read a mind into fluent text is older than any of this. In January 1966 Joseph Weizenbaum, at MIT, published a program called ELIZA in Communications of the ACM. It matched patterns in whatever the user typed and reflected them back as questions. It had no model of language, no memory and no knowledge of anything at all, and people talked to it as though it had all three. None of our four research files carries ELIZA, which is a gap worth naming, because the central psychological fact of this subject was documented in a peer-reviewed journal by the program's own author six decades ago. Nothing since has removed the reflex. The machines simply got better at triggering it, and section 09 returns to what that does and does not license anyone to conclude.

Start with what the model reads, because it is not words. Text is first cut into sub-word units by a learned merge procedure, byte pair encoding, brought into neural machine translation by Rico Sennrich, Barry Haddow and Alexandra Birch at ACL 2016. Common words survive as single units; rare ones are split into fragments, and the fragments cut across morphology, because the procedure is a compression algorithm repurposed and nothing in the paper claims the resulting units mean anything. That engineering convenience is the reason these systems are mocked for spelling and arithmetic: they are manipulating units that were never designed to line up with letters or digits. The consequence is public and measurable. Anthropic's own model documentation, read on 28 August 2026, states that a one-million-token context window is 'roughly 555k words or 2.5M Unicode characters on the current tokenizer (introduced with Claude Opus 4.7)', and that 'models before it fit about 750k words in 1M tokens'. The same advertised window holds a different quantity of English depending on which tokenizer sits underneath it.

Attention did not arrive in 2017. Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio introduced additive attention for neural machine translation at the Universite de Montreal in 2014, in 'Neural Machine Translation by Jointly Learning to Align and Translate', published at ICLR 2015, letting a decoder focus on the relevant parts of a source sentence instead of forcing the whole of it through one fixed vector. Minh-Thang Luong proposed a simpler multiplicative form at Stanford in 2015. Our own architecture file states the 2017 contribution precisely, and it is subtractive rather than additive: 'The transformer's contribution was eliminating recurrence entirely and building an architecture from attention alone.' A note on sourcing: the identifier our file carries for the Bahdanau paper could not be independently resolved for this page, so that work is named here by title, authors and venue and is not linked below.

The architecture itself was posted on 12 June 2017 as 'Attention Is All You Need', by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin, and published at NIPS 2017. The paper's own footnotes give three affiliations, Google Brain, Google Research and the University of Toronto, where our file gives two. The abstract proposes 'a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.' That sentence is the whole move. Everything the field had built on processing a sentence in order was taken out, and what remained was a stack of layers in which every position can look at every other position at once.

Self-attention is one operation and it is simple enough to say in a sentence. Every position in the sequence produces three vectors by linear projection: a query, a key and a value. Each query is compared against every key to produce a score, the scores are turned into weights that sum to one, and the position's output is the weighted average of all the values. Every token asks every other token how relevant it is, and the answer is an average. Two engineering details matter. The comparison is a dot product divided by the square root of the key dimension, and that divisor exists for a mundane and decisive reason: without it the scores grow large enough to push the weighting function into a region where the gradients vanish and the network stops learning. And the whole thing is a matrix operation, so all positions are computed together rather than in order. Our own primary file states the effect plainly: the mechanism 'allows the model to weigh relationships between all tokens in a sequence simultaneously, replacing the sequential processing of RNNs and LSTMs'.

A transformer does not run one attention function but several in parallel, called heads, whose outputs are concatenated and projected back down. The 2017 model used eight of them, each working in 64 dimensions, that being the model dimension divided by the number of heads. The paper's own justification is about representational room rather than division of labour: 'Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions.' Our own architecture file adds the familiar gloss that one head handles syntax, another semantics, another position. That gloss is an interpretation, not a finding, and it is the first place this article should show its hand: section 08 carries the work showing that where a model attended is not a reliable account of what it used.

Attention has no idea what order anything is in. The operation is permutation invariant: shuffle the tokens and exactly the same set of comparisons is made. Word order, which is a great deal of what a sentence is, has to be added by hand, and the 2017 paper adds it as a set of sinusoidal position signals mixed into the input. Later models replaced those with learned position embeddings, with relative position representations (Shaw, Uszkoreit and Vaswani, 'Self-Attention with Relative Position Representations', NAACL 2018) and with rotary position embeddings (Su and colleagues, 2021). The identifier our file carries for the Shaw paper could not be independently resolved for this page either, so it too is named here and not linked below.

Attention is also not the only thing in a transformer layer, and our own architecture file says so in a refusal box. Each layer carries a two-layer feed-forward network applied at every position independently, typically four times as wide as the model dimension, plus layer normalisation (Ba and colleagues, University of Toronto, 2016) and residual connections (He and colleagues, Microsoft Research, 2015). Our file marks the proposition that attention is a complete theory of neural computation as unsupported, at Tier 4, on the grounds that transformers 'rely heavily on feed-forward layers (which some available evidence suggests store the majority of factual knowledge), normalization, and positional information'. The famous title is a claim about what was removed, not a description of what remains.

Comparing every position against every other position costs time and memory in proportion to the square of the sequence length. Double the input and the work quadruples. Our own architecture file gives the consequence at scale in its own framing: processing a document of a million tokens takes on the order of 10^12 attention computations per layer. Engineering has attacked the constant rather than the exponent, and our file records the outcome: FlashAttention (Dao and colleagues, 2022) cut the memory overhead by reorganising the computation around the memory hierarchy, but 'the asymptotic complexity remains quadratic'. Our file also reads Tay, Dehghani, Bahri and Metzler's 2022 survey of efficient transformer variants as finding that no sub-quadratic method consistently matches full attention quality across tasks. That negative verdict is our file's reading; the survey itself was not opened for this page, and its own stated shape is a taxonomy of variants rather than a head-to-head ranking.

It is worth being exact about what that paper claimed, because it is not what the architecture became. It was a machine translation paper with a table of BLEU scores. Its abstract reports: 'Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature.' Two corrections to our own files sit inside that quotation. The base model in the paper had 65 million parameters, which is the figure our primary file prints as the parameter count, but the result quoted above came from the big model at 213 million. And our architecture file attaches the three and a half days on eight GPUs to the German result; the abstract attaches it to the French one. What the authors were reporting was a two-point improvement on a translation benchmark and a large saving in training time.

Within about a year the architecture had split into the three families above, and that split is why the abbreviation now names one specific shape of machine. Our file gives the verdict: 'The decoder-only architecture dominates current frontier models due to its simplicity and effectiveness for generative tasks.' Note what that means for section 01. The family that won is the one whose training objective is the single sentence this article opened with, and our file's stated reason for its winning is simplicity and effectiveness for generative tasks, not any argument that predicting the next token is the right thing to be doing.

One further change matters for a reason that arrives in the next section. A mixture-of-experts layer routes each token to a small subset of the network instead of all of it, so a model can hold an enormous number of total parameters while using only a fraction of them on any given token. Shazeer, Mirhoseini, Maziarz, Davis, Le, Hinton and Dean introduced the sparsely-gated version at Google in January 2017, reporting 'greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters'. In January 2021 Fedus, Zoph and Shazeer simplified the routing in the Switch Transformer and reported that they 'advance the current scale of language models by pre-training up to trillion parameter models on the "Colossal Clean Crawled Corpus"'. The consequence is that a trillion-parameter sparse model and a trillion-parameter dense model are not the same object, so a published parameter count would mean less than a reader expects even in the years when one was published.

The architecture did not stay in language. Our machine learning file states at Tier 1 that the transformer 'underpins all major LLMs (GPT, BERT, T5, LLaMA, Claude, Gemini) and has been adapted for computer vision (Vision Transformer, ViT), protein folding (AlphaFold), weather prediction (GraphCast), and other domains', and that 'its impact across ML subfields is comparable to the invention of the convolutional layer'. Our architecture file grades the same generalisation one step lower, at Tier 2, and this page carries the more conservative of the two. What those systems do is not this article's subject, and our article on the alignment problem tells the protein-folding story at length. That a translation architecture from 2017 ended up in all of them is the point here, and then this article returns to language.

BERT was posted by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova at Google on 11 October 2018 and published at NAACL 2019. It came in two sizes and our own file prints only the larger: BERT-BASE at 110 million parameters and BERT-LARGE at 340 million. Its pretraining recipe is worth one concrete sentence because it is checkable: 'we simply mask some percentage of the input tokens at random, and then predict those masked tokens', with a masked token replaced by a placeholder 80 percent of the time, by a random token 10 percent, and left unchanged 10 percent. The paper reports that the model 'obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement)'. Those are October 2018 numbers on October 2018 benchmarks, and they are recorded here as history rather than as a current standing.

GPT-2 arrived from OpenAI in February 2019 at 1.5 billion parameters and was initially withheld over misuse concerns. Our own file calls it 'the first major AI "staged release"', keeping the inner quotation marks it puts around the term. OpenAI's own peer-facing account of the decision, Solaiman and colleagues in August 2019, describes 'staged release, which allows time between model releases to conduct risk and benefit analyses as model sizes increased'. The full model was released in November 2019. This page does not quote the announcement that accompanied that release, because openai.com returns an error to non-browser clients and the page could not be read for this article. What can be said is the shape of the episode: an organisation treated its own output as hazardous and then undid that decision itself within nine months.

The GPT-3 paper, 'Language Models are Few-Shot Learners', was posted by Tom B. Brown and thirty co-authors on 28 May 2020, and the API private beta followed in June 2020; our own file gives one date for those two events. The paper reports that 'we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting', and that all its models 'were trained for a total of 300 billion tokens'. The result that changed the field was procedural rather than numerical: 'For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model.' The same abstract names its own failures, and they belong here too: 'we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora.' Deeper in, the paper discloses that its own contamination filtering broke: 'a bug in the filtering caused us to ignore some overlaps, and due to the cost of training it was not feasible to retrain the model.' The founding paper of the modern era says in its own pages that it could not afford to fix a flaw in its evaluation, and that admission is almost never quoted. Our own file also carries a launch-day price per thousand tokens for this model. No primary record of it could be obtained, the June 2020 launch was a free private beta, and this page therefore does not print the figure.

PaLM arrived from Google in April 2022 at 540 billion parameters and demonstrated, in our own file's phrasing, 'breakthrough performance on reasoning tasks, including explaining jokes'. The joke result is real and appears in the PaLM paper; that paper was not opened for this page, so it is attributed and not quoted. The checkable reasoning result at that scale comes from a different paper and is carried in section 04, where it belongs to a method rather than to a model.

Meta released LLaMA in February 2023: a family from 7 billion to 65 billion parameters trained only on publicly available data. Our own file gives one training-set size for the whole family. The paper gives two: the 7B and 13B models were trained on 1.0 trillion tokens and the 33B and 65B on 1.4 trillion. The headline is the one that mattered, and it is the strongest single piece of evidence for the compute-optimal argument three claims below: 'LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B.' The paper says 'We release all our models to the research community.' The weights then reached the wider public, which is widely reported and was not verified against a primary source for this page; our own file credits the release with catalysing the open-weight ecosystem, and section 05 returns to what that does and does not settle.

GPT-4 was released by OpenAI in March 2023 and accepted image as well as text input. Its size is not public and never has been. The technical report says so in as many words: 'Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar.' Our own file prints an estimate above one trillion parameters, marked unconfirmed, in a claim it grades Tier 1. There is no primary source for it and this page does not print a number. This is the hinge of the article's middle: for five years the history could be told in parameter counts, and then it stopped being tellable that way, not because the numbers got bigger but because they stopped being published. The exam results announced alongside this model are not retold here either. Our article on the alignment problem already carries the bar exam claim together with its peer-reviewed re-evaluation, and our own file's Tier 1 grade for a vendor's own percentile is not one this page would repeat.

The reason anyone kept building bigger ones is a measurement. Jared Kaplan and colleagues at Johns Hopkins and OpenAI posted 'Scaling Laws for Neural Language Models' on 23 January 2020, finding that a model's loss falls as a power law in model size, in dataset size and in compute. The paper gives the exponents: about 0.076 for non-embedding parameters, about 0.095 for training tokens, and about 0.050 for optimally allocated compute. They are worth stating only with their meaning attached, which is that a larger exponent means loss falls faster as you spend more of that particular thing. The abstract says the trends span 'more than seven orders of magnitude', and draws the conclusion the industry acted on: 'Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.' Our article on the alignment problem states this paper in one sentence as a safety backdrop and carries the compute-growth figures; this page carries the result as a machine learning finding and leaves the growth curves there.

In March 2022 Jordan Hoffmann and colleagues at DeepMind showed that the models of the GPT-3 generation were undertrained rather than merely large. Training over 400 models from 70 million to more than 16 billion parameters on 5 to 500 billion tokens, they concluded 'that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled.' They then built Chinchilla at 70 billion parameters on the same compute budget as the 280-billion-parameter Gopher, and report that 'Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks.' A smaller model beat a model four times its size on the same budget, which is also what the LLaMA result two claims above is showing.

The claim that made scaling feel like more than economics is emergence. Jason Wei and fifteen colleagues argued in June 2022, in a paper later published in Transactions on Machine Learning Research, that some abilities are simply absent below a scale threshold and present above it. Their definition is the whole of the claim: 'We consider an ability to be emergent if it is not present in smaller models but is present in larger models. Thus, emergent abilities cannot be predicted simply by extrapolating the performance of smaller models.' The examples our own file names are multi-step arithmetic, chain-of-thought reasoning, and translation for low-resource languages. The abstract's closing sentence is the one that moved money: 'The existence of such emergence implies that additional scaling could further expand the range of capabilities of language models.' Note the conditional verb. It is doing a great deal of work.

In April 2023 Rylan Schaeffer, Brando Miranda and Sanmi Koyejo argued that the cliff is in the scoring rather than in the model. Their claim: 'for a particular task and model family, when analyzing fixed model outputs, emergent abilities appear due to the researcher's choice of metric rather than due to fundamental changes in model behavior with scale. Specifically, nonlinear or discontinuous metrics produce apparent emergent abilities, whereas linear or continuous metrics produce smooth, continuous predictable changes in model performance.' The strongest part is a demonstration our own file omits: they chose metrics that produced 'never-before-seen seemingly emergent abilities in multiple vision tasks across diverse deep networks'. They did not merely argue that the cliff was an artefact; they manufactured cliffs to order. And their conclusion is narrower than the way it is usually cited: 'alleged emergent abilities evaporate with different metrics or with better statistics, and may not be a fundamental property of scaling AI models.' That is a claim about the shape of the acquisition, not a claim that the abilities are absent. A reader who takes it as a debunking has misread it.

Our two files reach the same verdict independently, and this page carries it as written. The primary file states the standoff: 'The debate remains unresolved: whether LLM capabilities represent genuine phase transitions or gradual improvements perceived as sudden due to measurement choices'. The machine learning file puts it more briefly: 'the nature and reliability of emergence is debated'. Neither side has been retired by the other. What can be said is narrower and more useful than either: whether a capability arrives or grows depends on how you agreed to score it before you looked, which is a fact about measurement that would be worth knowing even if the dispute were settled tomorrow.

A pretrained language model predicts text. It does not follow instructions, and nothing in the objective asks it to. The bridge is reinforcement learning from human feedback: supervised fine-tuning on human-written examples, then a reward model trained on human rankings of model outputs, then optimisation of the language model against that reward model. Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg and Dario Amodei published the method in June 2017, reporting that it works while 'providing feedback on less than one percent of our agent's interactions with the environment' and can teach 'complex novel behaviors with about an hour of human time'. An accident of history worth one clause: that paper is about Atari games and simulated robots, not about language at all. OpenAI applied it at scale to language in InstructGPT, announced in January 2022 and published as a paper on 4 March 2022; our own file gives one date for those two events. What this method does to a model's safety, and where it fails, belongs to our article on the alignment problem. Here it is simply the step that turned a text predictor into something a person could talk to.

In December 2022 Yuntao Bai and colleagues at Anthropic published a way to cut the human labelling of harmful outputs almost entirely, replacing it with a written list of principles. The abstract states: 'The only human oversight is provided through a list of rules or principles, and so we refer to the method as "Constitutional AI".' The reinforcement stage uses preferences generated by a model rather than by people, and the stated result is 'a harmless but non-evasive AI assistant that engages with harmful queries by explaining its objections to them.' The disclosure: the model that drafted this article was trained with a later version of that method. That is not a reason to leave the paper out, and it is a reason to say so on the page where it appears. Whether any of these methods make a system safe is a different question and belongs to the sibling article named above.

Training GPT-3 took roughly 3,640 petaflop per second days of compute. The figure follows from the paper's own total of approximately 3.14 x 10^23 floating-point operations, and the paper supplies its own scale marker, which is more informative than the headline: 'Although GPT-3 3B is almost 10x larger than RoBERTa-Large (355M params), both models took roughly 50 petaflop/s-days of compute during pre-training.' Ten times the parameters for the same compute, because the compute is a product of size and data and not of size alone. That is a single measurement for a single 2020 model and this page does not put it on a curve; our article on the singularity is where growth curves are argued about.

The same run has been costed in electricity. Patterson, Gonzalez, Le, Liang, Munguia, Rothchild, So, Texier and Dean published carbon and energy figures for five large models in April 2021, including GPT-3, and put its training at about 1,287 megawatt hours and about 552 tonnes of carbon dioxide equivalent. Two disclosures travel with those numbers. First, they could not be read out of the paper's own table for this page, because its text layer would not extract, so they are high-confidence and second-hand. Second, and more important, the authors' own abstract undercuts any confident use of them: the choice of model, datacenter and processor 'can reduce the carbon footprint up to ~100-1000X', and 'These large factors also make retroactive estimates of energy cost difficult.' A retroactive estimate of a model the authors did not train is precisely what the GPT-3 figure is. Our own architecture file cites Strubell, Ganesh and McCallum's 2019 ACL paper for the comparison that training one large model can emit as much carbon dioxide as five cars over their lifetimes. The honest version of this paragraph is not a tonnage. It is that a careful estimate exists, its authors warn that estimates like it can be wrong by two or three orders of magnitude, and the organisations who could settle it by publishing their own meter readings do not.

The same run has been costed in money, and our own file's dollar figures carry no citation, so this page replaces them with a sourced measurement. In May 2024 Ben Cottier, Robi Rahman, Loredana Fattorini, Nestor Maslej, Tamay Besiroglu and David Owen built a cost model covering hardware, energy, cloud rental and staff, and report that 'the amortized cost to train the most compute-intensive models has grown precipitously at a rate of 2.4x per year since 2016 (90% CI: 2.0x to 2.9x)'. Their breakdown puts accelerator chips and research staff each in the tens of millions of dollars for models such as GPT-4 and Gemini, server components at 15 to 22 percent of the total, cluster interconnect at 9 to 13 percent, and energy at 2 to 6 percent. Their conclusion is a conditional and is reproduced as one: 'If the trend of growing development costs continues, the largest training runs will cost more than a billion dollars by 2027, meaning that only the most well-funded organizations will be able to finance frontier AI models.' That is an estimate built from hardware prices and headcount, not an invoice, which is why its authors give an interval that wide. This page does not restate their conditional as a prediction and does not extend their trend by a single year.

The first of those three ways past has a peer-reviewed problem attached to it. Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson and Yarin Gal published 'AI models collapse when trained on recursively generated data' in Nature on 24 July 2024. They report that indiscriminate use of model-generated content in training causes irreversible defects, in which the tails of the original content distribution disappear, and they show the effect in language models, in variational autoencoders and in Gaussian mixture models. Two things about that sentence. Nothing in it is quoted, because the article page sits behind a cookie wall and the wording was read from the registration record rather than from the paper. And the word indiscriminate is load-bearing: the finding is about replacing human data with model output, not about synthetic data as such, and subsequent work arguing that accumulating synthetic data alongside the original avoids the collapse was not reviewed for this page. So: the first proposed route past the data wall is the thing this paper says destroys the model if you do it carelessly, and that tension is real without either half being stretched.

One frontier model in this subject went through referees before it was published. On 17 September 2025 Nature published 'DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning' by Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song and 198 further authors, at Nature 645:633-638. The paper reports that reasoning ability can be developed by reinforcement learning without human-labelled reasoning examples, that behaviours including self-reflection, verification and dynamic strategy adaptation emerged during that training, and that the result surpassed models trained conventionally on human demonstrations. Nothing there is quoted, for the same cookie-wall reason as the Nature paper in section 05. This is described elsewhere as the first frontier model published after independent peer review; that superlative comes from Nature's own news coverage, which could not be read at source for this page, so what this page says is that it is among the first, which loses nothing. Eight years after the architecture was posted, a laboratory sending a frontier model to referees was still news. That this is notable at all is the finding.

Language models produce fluent text that is not true. The standard survey, Ji and nine colleagues in ACM Computing Surveys in March 2023, characterises the failure as 'the generated content that is nonsensical or unfaithful to the provided source content'. Our own file says that survey coined the word in 2023. It did not, and the survey itself is the source that refutes the claim: it records that 'The term "hallucination" first appeared in Computer Vision (CV) in Baker and Kanade 2000', where it described something desirable, and that researchers began applying it to unwanted generation later. Our own file also carries a specific hallucination rate for GPT-4, attributed to two named sources. Neither of them contains it: the GPT-4 technical report carries no such figure and the second attributed paper could not be located at all. This page does not print that number in any form, and says so here rather than quietly omitting it. What can be said with a source is narrower. The GPT-4 technical report puts the model 19 percentage points above GPT-3.5 on OpenAI's own internal, adversarially designed factuality evaluations, which is a vendor's measurement of its own product on its own test, and the same report admits in its own words that the model 'still is not fully reliable (it "hallucinates" facts and makes reasoning errors)'.

The engineering answer is retrieval. Patrick Lewis and eleven colleagues at Facebook AI Research proposed retrieval-augmented generation in May 2020, pairing a pretrained language model with a searchable index, in their case a dense vector index of Wikipedia reached through a neural retriever, so that an answer can be grounded in a retrieved document rather than in the weights alone. Nothing is quoted from that paper here, because the abstract came back paraphrased rather than verbatim. Our own file records the motivation exactly: retrieval 'mitigates hallucination by grounding responses in retrieved documents'. And our file supplies its own closing objection, which survives into this page unchanged: 'Researchers argue hallucination is an inherent feature of generative models and cannot be fully eliminated without external verification systems.' Retrieval moves the failure rather than removing it. A model can still misread a document it correctly retrieved, over-generalise from it, or contradict it in the next sentence.

The useful result about failure is not a list of anecdotes but a measurement of where the training objective shows through. R. Thomas McCoy, Shunyu Yao, Dan Friedman, Mathew D. Hardy and Thomas L. Griffiths tested five models on eleven tasks and found that accuracy depends on how probable the task is, how probable the output is, and how probable the input is, even on tasks that are entirely deterministic. Their sharpest result: GPT-4's accuracy at decoding a cipher fell from 51 percent to 13 percent purely as a function of how probable the correct output text was, on a task where the probability of the answer should be irrelevant to whether it can be computed. Those two figures come from the paper's abstract as recorded by the registration agency rather than from the paper itself, which was not opened for this page, and nothing here is quoted. The authors' conclusion is the frame this whole article has been building: these systems should be understood as shaped by the specific problem they were trained to solve, next-word prediction over internet text, rather than measured against expectations formed by watching humans. The objective from section 01 leaves fingerprints on the finished machine, on tasks where it has no business being visible at all.

Nobody can trace how a particular input produces a particular output inside a frontier model. Our explainability file states it flatly at Tier 1: deep networks, ensembles and large language models 'achieve state-of-the-art accuracy but are opaque', and 'with millions to trillions of parameters, no human can trace how a specific input leads to a specific output through internal computations'. Our machine learning file puts the consequence in its counter-arguments: 'we often cannot explain why a model makes a specific prediction, which is problematic for high-stakes applications (medicine, criminal justice, autonomous vehicles); interpretability research is active but far from solving this'. Two files, written separately, both at the highest tier. This is as solid as anything in this article, and it sits underneath everything section 09 argues about.

The obvious way to look inside a transformer is to look at where it attended, and it does not work. Our explainability file states at Tier 1 that 'Jain and Wallace (2019) showed attention weights do not reliably indicate feature importance'. The multi-head gloss from section 02, one head for syntax and another for semantics, is exactly the kind of tidy story this finding complicates: the weights show what the mechanism looked at, not what the computation used. The same file summarises Cynthia Rudin's more general 2019 argument in Nature Machine Intelligence, that post-hoc explanations of black boxes 'explain an approximation of the model, not the model itself', and that high-stakes decisions should use models that are interpretable to begin with. Those words are our file's rendering of Rudin's position and not a quotation from her paper, which was not opened for this page.

The serious alternative is mechanistic interpretability, which tries to reverse-engineer the computations themselves, identifying circuits built out of attention heads and neurons. Our two files agree on where it stands and it is not where its press coverage puts it. Our explainability file places the entire programme at Tier 3, speculative: 'whether complete mechanistic understanding of billion-parameter models is achievable remains open'. Our architecture file adds that 'current interpretability covers only a small fraction of model behavior'. The findings the corpus does record are real and specific: induction heads, which implement learning from the context window, and the result that factual knowledge in GPT-style models sits in particular feed-forward layers and can be edited there surgically. Note that the second of those confirms the Tier 4 refusal in section 02: the facts live in the layers the famous title left out.

The criticism that stuck was published at the ACM Conference on Fairness, Accountability, and Transparency in March 2021, pages 610 to 623, by Emily M. Bender, Timnit Gebru, Angelina McMillan-Major and Shmargaret Shmitchell. Our own file attributes the whole author line to the University of Washington; the conference record does not. Bender and McMillan-Major are recorded there at the University of Washington, Gebru at Black in AI, and the fourth name is a pseudonym, with an affiliation given as The Aether, which our file never mentions. That detail is worth one clause because it says something true about the conditions the paper was written under, without this page having to litigate the employment dispute around it, which is out of scope and was not verified here. The definition the phrase comes from: 'an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot.' A provenance note this page owes the reader: the publisher's page returns an error to non-browser clients, so that sentence was not read from the paper's own text layer. It is reproduced from a sourced secondary reproduction that places it on pages 616 to 617, it matches the wording quoted throughout the literature, and this page does not claim to have read it in the original.

The empirical evidence this page can offer on that side is a game. Kenneth Li, Aspen K. Hopkins, David Bau, Fernanda Viegas, Hanspeter Pfister and Martin Wattenberg trained a GPT variant on nothing but sequences of legal Othello moves, with no knowledge of the game or its rules, and report: 'Although the network has no a priori knowledge of the game or its rules, we uncover evidence of an emergent nonlinear internal representation of the board state.' They could intervene on that representation and change the model's output accordingly, which is the part that makes it evidence rather than correlation. It was an ICLR 2023 oral. The following year Neel Nanda, Andrew Lee and Martin Wattenberg revisited it: 'Prior work suggests that Othello-playing neural network learned nonlinear models of the board state (Li et al., 2023). In this work, we provide evidence of a closely related linear representation of the board.' The difference was how the probe was framed, asking about the board relative to the player to move rather than in fixed colours. Two limits travel with all of it, and our own file states the first: 'Whether this extends to real-world knowledge (physics, causality) is unknown.' The second is that the original finding was refined within a year by a different instrument pointed at the same model, which is a caution about how much weight any probing result can carry. Othello has sixty-four squares and two colours. The world does not.

A prominent technical objection is Yann LeCun's, that the architecture is fundamentally insufficient whatever is done with it. An attribution correction first, because it is exactly the defect class this page exists to catch: our own file puts quotation marks around a sentence attributed to LeCun, and that string could not be located in any primary source for this article, so it is not repeated here. What can be sourced is an essay he co-wrote with Jacob Browning, published in Noema on 23 August 2022, which states the position without hedging: 'A system trained on language alone will never approximate human intelligence, even if trained from now until the heat death of the universe.' Attribute that to Browning and LeCun jointly, in an opinion essay rather than a result. Our own files record the argument behind it, that autoregressive language models lack world models and causal reasoning, and that LeCun proposes a different architecture in their place, the Joint Embedding Predictive Architecture. His technical proposal for it was not consulted for this page.

On the other side sits a preprint that made a much larger claim. Sebastien Bubeck and thirteen colleagues at Microsoft Research published 'Sparks of Artificial General Intelligence: Early experiments with GPT-4' on 22 March 2023, arguing that 'Given the breadth and depth of GPT-4's capabilities, we believe that it could reasonably be viewed as an early (yet still incomplete) version of an artificial general intelligence (AGI) system.' Its own abstract carries three hedges that citing articles routinely drop. It studied 'an early version of GPT-4, when it was still in active development by OpenAI', which the public could not test. It says 'we put special emphasis on discovering its limitations'. And it names 'the possible need for pursuing a new paradigm that moves beyond next-word prediction', which sits oddly beside the way the paper is usually cited. Three further facts are checkable and none of them is an insult: the authors worked at the company that had invested most heavily in the model's developer, the version they tested was not available to anyone outside, and the paper was not peer reviewed. State them and let the reader weigh it.

Now the separation this page will not blur, because our own primary file blurs it. That file puts consciousness and understanding into the same Tier 4 sentence: 'Claims that LLMs possess consciousness, sentience, or understanding are not supported by current evidence', noting that the models 'process tokens through mathematical operations (matrix multiplications, softmax attention) without demonstrated subjective experience'. Our architecture file is more careful about the second half and this page follows it: whether the computation constitutes understanding 'depends on one's definition and remains a philosophical debate, not a settled scientific question', and 'Performance on benchmarks does not demonstrate comprehension in the human sense'. Those are two different claims. Sentience: refused at Tier 4, with no live scientific dispute for this class of system on this evidence, and the general question of machine minds belongs to our article on machine consciousness. Understanding: genuinely contested, argued at strength above by serious people on both sides, and left open here. Collapsing the second into the first would be this page taking a side in a fight it promised to hold.

Our own file records that in June 2022 the Google engineer Blake Lemoine publicly claimed the company's LaMDA chatbot was sentient, and that the claim 'was widely rejected by AI researchers'. Nothing is quoted here from Lemoine or from Google: the primary reporting could not be read for this page, and the transcript that circulated was not checked for provenance, so this page does not reproduce it or characterise the man. The system itself is documented. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha and colleagues describe a family of transformer-based dialogue models of up to 137 billion parameters, pretrained on 1.56 trillion words of public dialogue data and web text. The useful content is the pattern rather than the person, and section 01 named it fifty-six years earlier: a program that produces fluent, responsive text recruits the reader into supplying a mind for it. ELIZA did it with pattern matching and no memory. The reflex did not need the parameters.

The figure most often quoted about work comes from a bank. In a report dated March 2023, Goldman Sachs estimated that generative AI could expose the equivalent of 300 million full-time jobs worldwide to automation, that around two thirds of current jobs in the United States and Europe are exposed to some degree of AI automation, and that up to a quarter of current work tasks could be done by AI. Two framing corrections, and both matter because both push in the same direction. Our own file writes that the report says these jobs could be affected; the report's own word is exposed, and exposure to some degree of automation is not replacement. And our file omits the report's own offsetting observation, that the jobs historically lost to automation have been offset by new kinds of work that account for the bulk of long-run employment growth; the same report projected a possible 7 percent rise in global GDP over a ten-year period. Nothing here is quoted, because the report itself was not opened and its contents were established from the publisher's own summary. When a number loses its qualifier and its counterweight in the same retelling, that is the signature of a claim that has drifted.

There is a careful measurement of what deployment actually did, and it covers one industry and one company. Erik Brynjolfsson, Danielle Li and Lindsey Raymond studied customer-support agents given access to a generative AI assistant, and the peer-reviewed version published in the Quarterly Journal of Economics in 2025 reports that access to the assistant increased productivity, measured as issues resolved per hour, by 15 percent on average, with substantial variation between workers. Our own file gives 14 percent, which is the figure in the earlier working paper. The variation is the finding our file drops entirely, and it is the more interesting half: the gains concentrated among less experienced and lower-skilled workers, who improved in both speed and quality, while experienced and high-performing workers saw small speed gains and slight declines in quality. Nothing here is quoted; the wording was read from the registration record rather than the article page, which rendered without an abstract. One study, one industry, one company, and it should not be made to stand in for an economy.

Our own file refuses, at Tier 4, the claim that these systems will replace all human cognitive work within five years, and it refuses it with reasons rather than with confidence: 'While LLMs excel at text manipulation, summarization, and pattern matching, they remain unreliable for tasks requiring real-time factual accuracy, physical reasoning, or long-horizon planning without supervision', and 'Current deployment patterns emphasize human-AI collaboration (copilot models) rather than full automation.' The measurement above supports that refusal better than assertion does, and so does everything in section 07. A system whose accuracy on a deterministic cipher moves with the probability of the answer, and which loses ground when an irrelevant clause is added to a school arithmetic problem, is not a system that quietly absorbs cognitive work while nobody checks. Note what this refusal is not. It is not a forecast in the other direction, and this page makes none.