Information and uncertainty
Information is not content. It is the amount by which uncertainty is reduced.
Members' shelf
All 9 nodes on Naming & Classification are for members only.
Pay with the email you sign in with here, so the Pass is linked to this account.
Members' shelf
All 16 nodes on System Protection & Threat Modeling are for members only.
Pay with the email you sign in with here, so the Pass is linked to this account.
You work every day on a channel that is finite and noisy: saying something so the other side can reconstruct it. You have intuitions but no ruler. This is the ruler.
Information is not content. It is the amount by which uncertainty is reduced.
How much information something carries depends on how unlikely it was in the first place.
Entropy is not disorder. It is the quantification of unpredictability.
Signal is the part of the input that reliably lowers uncertainty. The rest is noise.
A channel is anything that lets information flow, not wires and radio only but also language, attention and memory.
Compression is not deleting content. It is representing the same content in a shorter way.
Allow distortion and you can compress below the entropy, but how far below is fixed by a curve that cannot be crossed.
Besides Shannon's route, information has a second definition: the amount of information in something is the length of the shortest program that generates it.
Mutual information measures how much of the uncertainty in one thing goes away once you know the other.
This quantity answers a practical question: if you encode real data with the wrong model, how much extra do you pay per item on average.
The mind is an information-processing system working under capacity limits and noise.
Information is not purely abstract.
Set the twelve pages side by side and this shelf draws out three further points. None of the three is a statement made by any single source.
First, the numbers these measures produce have content only once a reference has been specified. Eight of the twelve pages each describe one such reference. Whose probabilities: see "Information and uncertainty", "Self-information" and "Entropy". Which judgement the input is being used to answer: see "Signal and noise". What the receiver already knows: see "Coding and compression". Which kind of loss is acceptable: see "Rate-distortion theory". Under what noise conditions: see "Channel and capacity". Seen from which direction: see "Kullback-Leibler divergence". When the reference is left unstated, comparisons such as "this material carries a lot of information", "all of that is noise", "this compresses better" or "this channel has more capacity" have, on these pages' own terms, no content, and there is nothing to settle them by.
Second, five pages describe the same shape of failure. Transmitting above capacity, compressing too far, diluting the context, and memory changing as it is rebuilt all produce no error message; the receiving side still gives a response that looks reasonable. See "Channel and capacity", "Coding and compression", "Rate-distortion theory", "Signal and noise" and "Information theory and cognitive science".
Third, four pages state that what can be achieved in practice sits some distance from the theoretical limit. Capacity and the rate-distortion curve are approached only with very long code blocks; short messages and real-time settings generally fall short of them. The length of the shortest program cannot be computed, only estimated from above. The lower bound on the heat cost of erasure is far below what real hardware consumes. See "Channel and capacity", "Rate-distortion theory", "Algorithmic information theory" and "Information theory and thermodynamics".
From these the shelf draws three working practices. Before using any of these measures, write down the reference: for whom, for which judgement, and what loss is allowed. Failures of this kind do not show themselves afterwards, so the safeguard has to come beforehand; for example, decide in advance which half must survive if the other side can recover only half. These limits are suited to checking whether a claim goes past what is possible; they are not suited to serving as targets that ought to be reached.
Whether a piece of content is right or wrong is something the measures on this shelf cannot answer. "Information and uncertainty" and "Self-information" both state that high information content means only that something was unexpected, and that judging correctness takes a different tool. "Mutual information" states that it says only that two things are related, not which one influences the other; the family of time-ordered measures that deals with direction is not held on this shelf.
In what sense information entropy and thermodynamic entropy are the same thing, "Entropy" and "Information theory and thermodynamics" both set out two positions without ruling between them. Whether the general proof of the principle that erasure must dissipate heat holds, "Information theory and thermodynamics" likewise sets out both sides and does not rule.
How much a person can reliably handle at one time is a question this shelf gives no number for. "Information theory and cognitive science" states that the familiar figure is not settled and deliberately adopts no specific value; research cited on that page still lists, as of 2024, how the capacity limit should be assessed as a disputed question.
The note on the group "Who bears these limits" already says that its two pages extrapolate across domains. Four pages in the other groups likewise mark as analogy the parts where they are applied to human communication, judgement or governance: "Channel and capacity", "Rate-distortion theory", "Algorithmic information theory" and "Kullback-Leibler divergence". The six pages agree: the structure may be borrowed to ask questions; the numerical conclusions may not.
The pages were not all checked to the same depth, and each page states its own. On two of them, "Information and uncertainty" and "Signal and noise", the framing statements were checked only at the level of the discipline, not sentence by sentence against primary sources.
Tell me when a new shelf opens
Shelf notices are sent only when a new shelf opens or when the reading rules change.