Shannon and the Birth of the Bit

Claude Shannon was a quiet, playful mathematician at Bell Labs when, in 1948, he published the paper that made information measurable. His central move was to strip away meaning and ask a narrower, sharper question: how much uncertainty does a message resolve? A message tells you something only to the extent that you did not already know it; the more it could have been otherwise, the more it carries when it arrives. Shannon measured this in a unit a colleague, the statistician John Tukey, had coined the year before, in a Bell Labs memo dated January 1947: the bit, a contraction of 'binary digit,' the amount of information in a single yes-or-no answer when the two answers are equally likely. Shannon credited Tukey by name in a footnote of the paper itself. It is worth pausing on how strange and bold this was. Information had always seemed inseparable from significance, from what a message was about. Shannon's insight was that the engineering problem of communication, getting a message reliably from here to there, does not care what the message is about at all; it cares only that the message is one selection out of a set of possible messages. From that deliberately austere starting point, everything followed.

To turn surprise into arithmetic, Shannon borrowed both a formula and a name from physics: entropy. The entropy of a source, written H, is the average number of bits its messages carry, computed as the negative sum, over all possible outcomes, of each outcome's probability multiplied by the logarithm of that probability. The formula looks forbidding and says something simple. Entropy is largest when every outcome is equally likely, when you are maximally unsure what comes next, and it falls to zero when one outcome is certain and there is no surprise left to have. A fair coin flip carries exactly one bit; a two-headed coin carries none, because the answer is already known. The same measure applies to anything with an element of chance: the roll of a die, the next letter in a sentence, the next pixel in a photograph. And here Shannon proved that this number is not merely a description but a hard physical limit. The entropy of a source is the absolute floor on how far its messages can be compressed. You can squeeze out redundancy, which is exactly what a ZIP file does, but you can never get below H bits per symbol without discarding information. Ordinary English, as it happens, is roughly half redundant, which is why u cn stll rd ths sentence with its vowels gone.

Shannon's 1948 paper contained two theorems that, between them, built the digital age. The first, the source-coding theorem, is the compression result just seen: data can be squeezed down to its entropy, and no further. Every lossless format you use, ZIP, PNG, the FLAC in a music library, is an attempt to reach that Shannon limit; lossy formats like JPEG and MP3 deliberately go below it, discarding detail your eyes and ears are unlikely to miss, a trade-off Shannon also made rigorous in his 1959 theory of rate and distortion. The second theorem is deeper, and at the time genuinely shocking. Shannon's model of communication, drawn at the top of this page, runs from an information source through a transmitter, across a noisy channel, to a receiver and a destination, with noise corrupting the signal on the way. Common sense says noise must always degrade a message: push faster, or accept a noisier line, and you must accept more errors. Shannon proved common sense wrong. Every channel has a capacity, a maximum rate in bits per second (for a simple channel, the bandwidth times the logarithm of one plus the signal-to-noise ratio), and at any rate below that capacity you can communicate with an error rate as close to zero as you wish, provided you encode cleverly enough. He proved that such near-perfect codes exist without showing how to build them; engineers spent the next half-century constructing codes that approach his bound, and today's 5G phones and deep-space probes run within about one decibel of the limit Shannon set in 1948. The error-correcting codes that make it possible, Reed-Solomon and its relatives, are why a scratched CD still plays, a smudged QR code still scans, and the Voyager spacecraft can still whisper data across billions of miles of noise. Shannon returned to communication's other half the next year: in a 1949 companion paper he proved that perfect secrecy is genuinely achievable, a 'one-time pad,' a truly random key as long as the message and never reused, is provably impossible to break.

Now the single most important thing to understand about Shannon's information, and the thing most often gotten wrong. Shannon's theory says nothing whatsoever about what a message means. This was not an oversight; it was the founding move of the entire field, and Shannon stated it plainly in the 1948 paper itself: 'Frequently the messages have meaning... These semantic aspects of communication are irrelevant to the engineering problem. The significant aspect is that the actual message is one selected from a set of possible messages.' A bit measures choice, the resolution of uncertainty among possibilities, and nothing else. The sentence 'the meeting is at noon' and a random string of the same length can carry exactly the same number of Shannon bits, even though one means something and the other means nothing at all. This is why it is a mistake, however common, to speak of Shannon information as though it measured meaning, understanding, knowledge, or content. It does not, and it was carefully built not to. The strange power of the theory flows directly from this austerity: by refusing to ask what messages mean, Shannon uncovered laws that govern all of them equally, the profound message and the meaningless one alike. Any claim that the bit captures significance is not an extension of Shannon's theory but a misreading of its very first sentence.

There is one place where Shannon's abstract bits touch the physical world in a genuinely deep way, and it is worth stating exactly, because it is so often overstated. Shannon's entropy has the identical mathematical form as the entropy of thermodynamics, the quantity that measures disorder and drives the second law, the reason heat flows from hot to cold and time seems to run one way. For a long while this looked like nothing more than a suggestive coincidence of formulas. Then, in 1961, Rolf Landauer proved a real bridge between them: erasing a single bit of information must release a tiny but definite amount of heat, about 3 x 10^-21 joules at room temperature, no matter how the erasure is performed. This is Landauer's principle, and it was confirmed in the laboratory in 2012. It also resolves a puzzle a century old, Maxwell's 'demon,' an imagined being that seemed able to break the second law by sorting fast and slow molecules; the demon must eventually erase the records it keeps, and paying Landauer's price restores the balance. So information and thermodynamic entropy are genuinely linked, at exactly this one proven point. But that narrow, real bridge is not a license to declare the two 'the same thing.' Shannon entropy counts uncertainty among possible messages; thermodynamic entropy counts the microscopic states available to a physical system. They share a formula and one rigorous connection, and casually equating them beyond that is a popular overreach, not a result. The honest statement is the precise one: analogous mathematics, distinct concepts, and a single proven link between them.

From these foundations, information theory spread into nearly every science. The genetic code is now read as information, roughly two bits per rung of the DNA ladder, complete with its own error-correcting redundancy. Neuroscientists count the bits a nerve carries; statisticians use 'mutual information' to measure what one variable reveals about another; and modern machine learning is threaded through with information-theoretic ideas about compressing away the irrelevant. Shannon himself loved to build as much as to prove: in 1950 he made Theseus, a mechanical mouse that learned its way through a maze by trial and error and then ran it flawlessly, an early, tangible machine that learns. At the field's speculative edge, information reaches toward the deepest questions there are. Quantum information replaces the bit with the 'qubit,' which can be a blend of 0 and 1 at once. The entropy of a black hole turns out to be set by the area of its horizon rather than its volume, hinting that a region's capacity for information is somehow written on its boundary, and the fate of information that falls into a black hole remains one of physics' genuinely unsolved problems. And the physicist John Wheeler distilled a lifetime's wonder into a phrase, 'it from bit,' the radical proposal that physical reality itself might, at bottom, be made of information. It is essential to keep the line clear. Shannon's own theory is proven, exact, and world-building. The leap from 'information is a superb way to describe reality' to 'reality is made of information' is a real and serious idea, but a separate and unproven one, taken up directly in this wing's own article on information as fundamental reality. Here it is enough, and more than enough, that one man, asking a deliberately narrow question in 1948, found a law obeyed by everything that communicates, from a phone call to a strand of DNA to a signal from the edge of the solar system.