Trigpoint
“Entropy is disorder”: that is a misreading.

Entropy

Entropy · Shannon (1948)
Bibliography checked ✓ Information theory · core measures Read first: Self-information Tell the editor privately ▸
In three sentences Entropy is not disorder. It is the quantification of unpredictability. It is the number you get when the information of each possible outcome is averaged, each weighted by its own probability. The more even the distribution, the harder to guess and the higher the entropy. And the same number carries a second identity: it is the floor of lossless compression.

↳ Seen a counter-example?

Entropy Shannon (1948) It measures the shape of the distribution Almost certain Skewed Even Entropy The second identity average lossless code length can never fall below entropy the floor is here if it will not compress, it may not be technique Attribution has three parts; do not merge them combinatorial form 1877 · modern closed form 1901 · the naming anecdote usually cites a 1971 recollection, denied in 1982 The three steps show relative height only: no numerical scale, no units, no coordinates. The formal isomorphism is agreed; whether the two are ontologically the same is still argued. Figure: trigpoint.pub
Figure 1 · A ladder showing how entropy moves with the shape of the distribution. The more concentrated the distribution, the easier it is to guess and the lower the entropy; the more even it is, the harder to guess and the higher the entropy. The panel on the right marks the second identity of the same ruler: the average code length of lossless compression can never fall below it. This figure deliberately carries no numerical scale and no units. View full size ▸

1. What happened

The page Self-Information dealt with the information of a single event: the less likely the event, the more it carries when it happens. But the quantity for a single event cannot describe a system, because the system has not resolved yet. The quantity Shannon's 1948 paper gives amounts to one simple and decisive move: weight the information of each possible outcome by its own probability, and average[1]Shannon, C. E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal, 27(3), 379–423; 27(4), 623–656. DOI 10.1002/j.1538-7305.1948.tb01338.x. Bibliographic details checked against the official record. Claims matched against the full text. Defines H = −Σ p log p, that is, the probability-weighted average of log(1/p) over the outcomes, and in section 9 (the fundamental theorem for a noiseless channel, theorem 9) sets the limit of lossless coding.. The number that comes out is entropy, and it answers how much uncertainty I face on average before the resolution.

The construction has one property you can feel at once. The more concentrated the distribution, the lower the entropy, because you can nearly guess it. The more even, the higher, because every outcome is equally possible. At the extreme, a certain event has entropy zero.

Note that what is measured is the shape of the distribution, not how messy the outcome itself looks. That is exactly where the intuition entropy equals disorder goes wrong: it drops the probability structure out of the account.

The same paper gave entropy a second identity, and this one turns it from a descriptive quantity into a hard boundary: the average code length of lossless compression can never fall below the entropy[1]Shannon, C. E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal, 27(3), 379–423; 27(4), 623–656. DOI 10.1002/j.1538-7305.1948.tb01338.x. Bibliographic details checked against the official record. Claims matched against the full text. Defines H = −Σ p log p, that is, the probability-weighted average of log(1/p) over the outcomes, and in section 9 (the fundamental theorem for a noiseless channel, theorem 9) sets the limit of lossless coding.. How small you can compress a body of data depends not on how clever your algorithm is but on the probability structure of the data. That lower bound is the premise of the whole section “How to shrink it: the three limits of compression” on this shelf.

The formal kinship was pointed out in the same paper. In section 6 Shannon wrote, word for word, that this H is the H in Boltzmann's famous H theorem[2]Shannon 1948, section 6, in the original: “H is then, for example, the H in Boltzmann's famous H theorem”.. That sentence opened an argument still running today, which the verification layer below handles.

One thing commonly got wrong should be corrected first. The famous thermodynamic formula with the constant in it does not come from Boltzmann's 1877 paper, which contains the combinatorial form only. The modern form and that constant are generally credited to Planck, written down in 1900 to 1901, and by secondary accounts Planck himself said in his 1920 Nobel lecture that Boltzmann never introduced the constant[3]Boltzmann, L. (1877). Wiener Berichte, 76, 373–435. Bibliographic details confirmed by two sources. That this paper lacks the closed form with k follows secondary accounts that agree with one another. Builds the probabilistic basis of entropy on the number of permutations, a combinatorial measure of probability. The closed form with the constant k does not appear in this paper and is generally credited to Planck.[4]Planck, M. (1901). Ueber das Gesetz der Energieverteilung im Normalspectrum. Annalen der Physik, 309(3), 553–563. DOI 10.1002/andp.19013090310. Bibliographic details checked against the official record. Planck's remark in his 1920 lecture follows secondary accounts that agree with one another. The modern closed form with the constant k, and the introduction of the constant, are generally credited to this paper. By secondary accounts, Planck said in his 1920 Nobel lecture that the constant is often called Boltzmann's constant, but that to his knowledge Boltzmann himself never introduced it..

Attribution where due: Entropy as the expectation of information, and its identity as the lower bound of lossless compression, belong to Shannon (1948)[1]Shannon, C. E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal, 27(3), 379–423; 27(4), 623–656. DOI 10.1002/j.1538-7305.1948.tb01338.x. Bibliographic details checked against the official record. Claims matched against the full text. Defines H = −Σ p log p, that is, the probability-weighted average of log(1/p) over the outcomes, and in section 9 (the fundamental theorem for a noiseless channel, theorem 9) sets the limit of lossless coding.. The combinatorial quantity in statistical mechanics belongs to Boltzmann (1877), while the modern closed form with the constant is generally credited to Planck (1900 to 1901). The two cannot be compressed into one[3]Boltzmann, L. (1877). Wiener Berichte, 76, 373–435. Bibliographic details confirmed by two sources. That this paper lacks the closed form with k follows secondary accounts that agree with one another. Builds the probabilistic basis of entropy on the number of permutations, a combinatorial measure of probability. The closed form with the constant k does not appear in this paper and is generally credited to Planck.[4]Planck, M. (1901). Ueber das Gesetz der Energieverteilung im Normalspectrum. Annalen der Physik, 309(3), 553–563. DOI 10.1002/andp.19013090310. Bibliographic details checked against the official record. Planck's remark in his 1920 lecture follows secondary accounts that agree with one another. The modern closed form with the constant k, and the introduction of the constant, are generally credited to this paper. By secondary accounts, Planck said in his 1920 Nobel lecture that the constant is often called Boltzmann's constant, but that to his knowledge Boltzmann himself never introduced it.. Establishing maximum entropy as a principle of inference, and treating statistical mechanics as a form of statistical inference, belongs to Jaynes (1957)[5]Jaynes, E. T. (1957). Information Theory and Statistical Mechanics. Physical Review, 106(4), 620–630; Information Theory and Statistical Mechanics. II. Physical Review, 108(2), 171–190. Bibliographic details checked against the official record. Consistent with the abstract. Treats statistical mechanics as a form of statistical inference: probability distributions are set up by the principle of maximum entropy, committing to nothing beyond the information available, so the usual rules of statistical mechanics need no further physical assumption. The author calls the result subjective statistical mechanics.; for a contemporary argument that the measure of information and thermodynamic entropy cannot simply be equated, see Ben-Naim (2017)[6]Ben-Naim, A. (2017). Entropy, Shannon's Measure of Information and Boltzmann's H-Theorem. Entropy, 19(2), 48. DOI 10.3390/e19020048. Bibliographic details checked against the official record. Consistent with the abstract. Argues that Shannon's measure of information applies to any probability distribution, while thermodynamic entropy is defined only on one special class of distributions, so the two cannot simply be equated; on that basis, Boltzmann's H-function is a measure of information and not entropy.. As for the anecdote about von Neumann suggesting the name, its most often cited source is a personal recollection from 1971[7]Tribus, M. & McIrvine, E. C. (1971). Energy and Information. Scientific American, 225(3), 179–188. DOI 10.1038/scientificamerican0971-179. Bibliographic details checked against the official record. The account of the anecdote and its date follows secondary accounts that agree with one another. The most often cited source for the von Neumann naming anecdote: a personal recollection set down about ten years later. (another account places an earlier telling), and one of the parties to it denied it in a recorded interview in 1982[8]Price, R. (1982). Claude E. Shannon, an oral history conducted by Robert Price, 28 July 1982. IEEE History Center, Interview #423. Published as: A Conversation with Claude Shannon, IEEE Communications Magazine, 22(5), 123–126 (1984). Bibliographic details confirmed by two sources. Claims matched against the full text. Shannon denies the story in the interview: “I'm quite sure that it didn't happen between von Neumann and me”. The commonly repeated 1984 interview should be read as interviewed in 1982, published in 1984.. This site does not list it as historical fact.

2. Why this matters to you

The first thing entropy gives you is a diagnostic for this will not compress any further.

You trim a piece of material over and over, and past a certain point every change feels like a loss. The usual reaction is to blame your own writing. This page says there is another possibility: you have hit the floor. The probability structure of that material fixes its shortest average length, and every cut past that point stops removing redundancy and starts removing information.

The difference can be asked out loud: can the reader reconstruct the sentence I deleted from what remains? If yes, it was redundancy. If no, it was information.

The second thing is more counterintuitive. Since entropy depends on the distribution, the same content is compressible to different degrees for different readers. For someone who already holds the context, most of your groundwork is high-probability, predictable material and can go. For someone unfamiliar, that same material carries the real reduction in uncertainty, and cutting it breaks the chain.

So how long should this document be has no general answer, only an answer for a given reader. It also explains why the context you give a model and the explanation you give a person cannot be one and the same document.

The third thing concerns the word itself. Fighting entropy, in everyday usage, is mostly rhetorical: leave a system alone and it gets worse. The entropy on this page does not mean that. It is a quantity about a distribution. The two are not the same thing, and they are not unrelated either, and how substantial that relation actually is happens to be the question the page Information Theory and Thermodynamics takes up. Until then, treat them as two words.

Verification layer · Members
The guided layer of this node is open to all readers.
Verification details and failure boundaries are for members.
US$8.25 a month on average · or subscribe monthly ▸

Lab cases Written only where there is evidence

No field record for this node yet.

Common misreadings

Page verification: Bibliography checked ✓ · Content last revised 2026-11-14

  • A common account says that Boltzmann wrote S = k log W in 1877. His 1877 paper contains only a combinatorial measure of probability and does not write the closed form with the constant k. The modern form and the constant are generally credited to Planck, written down in 1900 to 1901. The tombstone at the Vienna Central Cemetery that carries the formula was unveiled in 1933 (per secondary accounts), long after Boltzmann's death, and the form is Planck's. #
  • A common account says that von Neumann suggested Shannon call the quantity entropy, on the grounds that nobody really knows what entropy is. The most often cited source is a personal recollection in a 1971 popular-science article, about ten years after the conversation, and von Neumann had died in 1957; another account says the same person had already told it in a 1964 memorial volume. More decisively, Shannon himself denied it in a 1982 oral-history interview. The anecdote circulates very widely; this site lists it as unconfirmed. #
  • A common account says that information entropy and thermodynamic entropy are the same thing. That the mathematical forms are isomorphic is agreed; in what sense the two are the same thing is still argued. This page sets out the layers side by side and does not crown one side. #
References

How to read the source notes “Bibliographic details checked” means author, year, title and venue were confirmed. It does not mean the source's claims were checked. “Consistent with the abstract” means the claim agrees with the paper's abstract. It does not include sentence-by-sentence checking against the full text. “Matched against the full text” means the claim was compared with the text of the source itself. “Follows secondary accounts” means the claim rests on sources that report it, not on the original document. Definitions as of 2026-09-20.

  1. [1] Shannon, C. E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal, 27(3), 379–423; 27(4), 623–656. ↩
  2. [2] Shannon, C. E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal, 27(3), 379–423; 27(4), 623–656.
  3. [3] Boltzmann, L. (1877). Wiener Berichte, 76, 373–435. ↩
  4. [4] Planck, M. (1901). Ueber das Gesetz der Energieverteilung im Normalspectrum. Annalen der Physik, 309(3), 553–563. ↩
  5. [5] Jaynes, E. T. (1957). Information Theory and Statistical Mechanics. Physical Review, 106(4), 620–630; 108(2), 171–190. ↩
  6. [6] Ben-Naim, A. (2017). Entropy, Shannon's Measure of Information and Boltzmann's H-Theorem. Entropy, 19(2), 48. ↩
  7. [7] Tribus, M. & McIrvine, E. C. (1971). Energy and Information. Scientific American, 225(3), 179–188. ↩
  8. [8] Price, R. (1982/1984). Claude E. Shannon oral history. IEEE History Center Interview 423; IEEE Communications Magazine, 22(5), 123–126. ↩

↳ Is a citation wrong?

Tell me when a new shelf opens

Tell me when a new shelf opens

Shelf notices are sent only when a new shelf opens or when the reading rules change.

Trial

Tell the editor privately: a correction, a counter-example or where it applies

Submissions are read only by the editor, and nothing is published before they are handled. Once a note is handled, a summary and response written by the editor are published. No sign-up is needed, and the nickname may be left blank.

What are you submitting?
Set a public name to be credited by name ▸
Submit anonymously, without signing in

Signing in lets you follow the status on your page. What you have written is kept.

You are signed in: this note is linked to your account, visible only to you and the editor, so you can follow its status on your page. The text is never published. Full privacy policy ▸

Submit anonymously instead, not linked to your account ▸

No email is collected. Your original text and nickname are read only by the editor and are never published; the site publishes only a summary the editor writes. The human check is provided by Cloudflare Turnstile. To limit abuse, the system keeps a hash of your IP address, not the address itself. Your original text is stored with our cloud provider and can be deleted on request. Full privacy policy ▸