Members' shelf
All 9 nodes on Naming & Classification are for members only.
Pay with the email you sign in with here, so the Pass is linked to this account.
Members' shelf
All 16 nodes on System Protection & Threat Modeling are for members only.
Pay with the email you sign in with here, so the Pass is linked to this account.
Entropy
1. What happened
The page Self-Information dealt with the information of a single event: the less likely the event, the more it carries when it happens. But the quantity for a single event cannot describe a system, because the system has not resolved yet. The quantity Shannon's 1948 paper gives amounts to one simple and decisive move: weight the information of each possible outcome by its own probability, and average[1]Shannon, C. E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal, 27(3), 379–423; 27(4), 623–656. DOI 10.1002/j.1538-7305.1948.tb01338.x. Bibliographic details checked against the official record. Claims matched against the full text. Defines H = −Σ p log p, that is, the probability-weighted average of log(1/p) over the outcomes, and in section 9 (the fundamental theorem for a noiseless channel, theorem 9) sets the limit of lossless coding.. The number that comes out is entropy, and it answers how much uncertainty I face on average before the resolution.
The construction has one property you can feel at once. The more concentrated the distribution, the lower the entropy, because you can nearly guess it. The more even, the higher, because every outcome is equally possible. At the extreme, a certain event has entropy zero.
Note that what is measured is the shape of the distribution, not how messy the outcome itself looks. That is exactly where the intuition entropy equals disorder goes wrong: it drops the probability structure out of the account.
The same paper gave entropy a second identity, and this one turns it from a descriptive quantity into a hard boundary: the average code length of lossless compression can never fall below the entropy[1]Shannon, C. E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal, 27(3), 379–423; 27(4), 623–656. DOI 10.1002/j.1538-7305.1948.tb01338.x. Bibliographic details checked against the official record. Claims matched against the full text. Defines H = −Σ p log p, that is, the probability-weighted average of log(1/p) over the outcomes, and in section 9 (the fundamental theorem for a noiseless channel, theorem 9) sets the limit of lossless coding.. How small you can compress a body of data depends not on how clever your algorithm is but on the probability structure of the data. That lower bound is the premise of the whole section “How to shrink it: the three limits of compression” on this shelf.
The formal kinship was pointed out in the same paper. In section 6 Shannon wrote, word for word, that this H is the H in Boltzmann's famous H theorem[2]Shannon 1948, section 6, in the original: “H is then, for example, the H in Boltzmann's famous H theorem”.. That sentence opened an argument still running today, which the verification layer below handles.
One thing commonly got wrong should be corrected first. The famous thermodynamic formula with the constant in it does not come from Boltzmann's 1877 paper, which contains the combinatorial form only. The modern form and that constant are generally credited to Planck, written down in 1900 to 1901, and by secondary accounts Planck himself said in his 1920 Nobel lecture that Boltzmann never introduced the constant[3]Boltzmann, L. (1877). Wiener Berichte, 76, 373–435. Bibliographic details confirmed by two sources. That this paper lacks the closed form with k follows secondary accounts that agree with one another. Builds the probabilistic basis of entropy on the number of permutations, a combinatorial measure of probability. The closed form with the constant k does not appear in this paper and is generally credited to Planck.[4]Planck, M. (1901). Ueber das Gesetz der Energieverteilung im Normalspectrum. Annalen der Physik, 309(3), 553–563. DOI 10.1002/andp.19013090310. Bibliographic details checked against the official record. Planck's remark in his 1920 lecture follows secondary accounts that agree with one another. The modern closed form with the constant k, and the introduction of the constant, are generally credited to this paper. By secondary accounts, Planck said in his 1920 Nobel lecture that the constant is often called Boltzmann's constant, but that to his knowledge Boltzmann himself never introduced it..
2. Why this matters to you
The first thing entropy gives you is a diagnostic for this will not compress any further.
You trim a piece of material over and over, and past a certain point every change feels like a loss. The usual reaction is to blame your own writing. This page says there is another possibility: you have hit the floor. The probability structure of that material fixes its shortest average length, and every cut past that point stops removing redundancy and starts removing information.
The difference can be asked out loud: can the reader reconstruct the sentence I deleted from what remains? If yes, it was redundancy. If no, it was information.
The second thing is more counterintuitive. Since entropy depends on the distribution, the same content is compressible to different degrees for different readers. For someone who already holds the context, most of your groundwork is high-probability, predictable material and can go. For someone unfamiliar, that same material carries the real reduction in uncertainty, and cutting it breaks the chain.
So how long should this document be has no general answer, only an answer for a given reader. It also explains why the context you give a model and the explanation you give a person cannot be one and the same document.
The third thing concerns the word itself. Fighting entropy, in everyday usage, is mostly rhetorical: leave a system alone and it gets worse. The entropy on this page does not mean that. It is a quantity about a distribution. The two are not the same thing, and they are not unrelated either, and how substantial that relation actually is happens to be the question the page Information Theory and Thermodynamics takes up. Until then, treat them as two words.
Lab cases Written only where there is evidence
No field record for this node yet.
- A common account says that Boltzmann wrote S = k log W in 1877. His 1877 paper contains only a combinatorial measure of probability and does not write the closed form with the constant k. The modern form and the constant are generally credited to Planck, written down in 1900 to 1901. The tombstone at the Vienna Central Cemetery that carries the formula was unveiled in 1933 (per secondary accounts), long after Boltzmann's death, and the form is Planck's. #
- A common account says that von Neumann suggested Shannon call the quantity entropy, on the grounds that nobody really knows what entropy is. The most often cited source is a personal recollection in a 1971 popular-science article, about ten years after the conversation, and von Neumann had died in 1957; another account says the same person had already told it in a 1964 memorial volume. More decisively, Shannon himself denied it in a 1982 oral-history interview. The anecdote circulates very widely; this site lists it as unconfirmed. #
- A common account says that information entropy and thermodynamic entropy are the same thing. That the mathematical forms are isomorphic is agreed; in what sense the two are the same thing is still argued. This page sets out the layers side by side and does not crown one side. #
How to read the source notes “Bibliographic details checked” means author, year, title and venue were confirmed. It does not mean the source's claims were checked. “Consistent with the abstract” means the claim agrees with the paper's abstract. It does not include sentence-by-sentence checking against the full text. “Matched against the full text” means the claim was compared with the text of the source itself. “Follows secondary accounts” means the claim rests on sources that report it, not on the original document. Definitions as of 2026-09-20.
- [1] Shannon, C. E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal, 27(3), 379–423; 27(4), 623–656. ↩
- [2] Shannon, C. E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal, 27(3), 379–423; 27(4), 623–656.
- [3] Boltzmann, L. (1877). Wiener Berichte, 76, 373–435. ↩
- [4] Planck, M. (1901). Ueber das Gesetz der Energieverteilung im Normalspectrum. Annalen der Physik, 309(3), 553–563. ↩
- [5] Jaynes, E. T. (1957). Information Theory and Statistical Mechanics. Physical Review, 106(4), 620–630; 108(2), 171–190. ↩
- [6] Ben-Naim, A. (2017). Entropy, Shannon's Measure of Information and Boltzmann's H-Theorem. Entropy, 19(2), 48. ↩
- [7] Tribus, M. & McIrvine, E. C. (1971). Energy and Information. Scientific American, 225(3), 179–188. ↩
- [8] Price, R. (1982/1984). Claude E. Shannon oral history. IEEE History Center Interview 423; IEEE Communications Magazine, 22(5), 123–126. ↩
Tell me when a new shelf opens
Tell me when a new shelf opens
Shelf notices are sent only when a new shelf opens or when the reading rules change.
Tell the editor privately: a correction, a counter-example or where it applies
Submissions are read only by the editor, and nothing is published before they are handled. Once a note is handled, a summary and response written by the editor are published. No sign-up is needed, and the nickname may be left blank.
We have your note