Trigpoint
Collections › Information Theory
SHELF

Information Theory

The ruler for measuring your own workflow
Discipline: Applied Mathematics
“Information theory is mathematics for communications engineers”: the fault is structural.
12 volumes · 8 lab cases Every source: bibliography checked ✓
↓ Jump to the node list

You work every day on a channel that is finite and noisy: saying something so the other side can reconstruct it. You have intuitions but no ruler. This is the ruler.

Node sequence

Suggested reading order · pin the word down, why it degrades, how to shrink it, what measures relatedness and error, who bears these limits
№ 01§ First pin down the word “information”Order within the group: the definition (not the content) → how to measure a single event → how to measure a whole system
01

Information and uncertainty

Shannon (1948) / Hartley (1928)

Information is not content. It is the amount by which uncertainty is reduced.

4 min read
02

Self-information

Shannon (1948) / Samson (1951)

How much information something carries depends on how unlikely it was in the first place.

4 min read
03

Entropy

Shannon (1948)

Entropy is not disorder. It is the quantification of unpredictability.

5 min read
№ 02§ Why it degrades: noise and capacityOrder within the group: tell what counts as noise first → then meet the hard ceiling of the channel itself
04

Signal and noise

Shannon (1948)

Signal is the part of the input that reliably lowers uncertainty. The rest is noise.

Lab cases ×24 min read
05

Channel and capacity

Shannon (1948, 1949)

A channel is anything that lets information flow, not wires and radio only but also language, attention and memory.

Lab cases ×25 min read
№ 03§ How to shrink it: the three limits of compressionOrder within the group: lossless (where the floor is) → lossy (how much distortion you will pay) → individual (a second route that does not use probability)
06

Coding and compression

Shannon (1948) / Fano (1949) / Huffman (1952)

Compression is not deleting content. It is representing the same content in a shorter way.

Lab cases ×25 min read
07

Rate-distortion theory

Shannon (1959) / Berger (1971)

Allow distortion and you can compress below the entropy, but how far below is fixed by a curve that cannot be crossed.

5 min read
08

Algorithmic information theory

Solomonoff (1964) / Kolmogorov (1965) / Chaitin (1966)

Besides Shannon's route, information has a second definition: the amount of information in something is the length of the shortest program that generates it.

5 min read
№ 04§ What measures relatedness and errorOrder within the group: how much the two share → what a wrong assumption costs
09

Mutual information

Shannon (1948) / Fano (1961)

Mutual information measures how much of the uncertainty in one thing goes away once you know the other.

4 min read
10

Kullback-Leibler divergence

Kullback & Leibler (1951)

This quantity answers a practical question: if you encode real data with the wrong model, how much extra do you pay per item on average.

5 min read
№ 05§ Who bears these limitsOrder within the group: people → physics. This group is also the honesty anchor for the shelf: what we do here is exactly the cross-domain extrapolation the founder of the theory warned against, so the warning is printed alongside
11
12

After reading this shelf

Set the twelve pages side by side and this shelf draws out three further points. None of the three is a statement made by any single source.

First, the numbers these measures produce have content only once a reference has been specified. Eight of the twelve pages each describe one such reference. Whose probabilities: see "Information and uncertainty", "Self-information" and "Entropy". Which judgement the input is being used to answer: see "Signal and noise". What the receiver already knows: see "Coding and compression". Which kind of loss is acceptable: see "Rate-distortion theory". Under what noise conditions: see "Channel and capacity". Seen from which direction: see "Kullback-Leibler divergence". When the reference is left unstated, comparisons such as "this material carries a lot of information", "all of that is noise", "this compresses better" or "this channel has more capacity" have, on these pages' own terms, no content, and there is nothing to settle them by.

Second, five pages describe the same shape of failure. Transmitting above capacity, compressing too far, diluting the context, and memory changing as it is rebuilt all produce no error message; the receiving side still gives a response that looks reasonable. See "Channel and capacity", "Coding and compression", "Rate-distortion theory", "Signal and noise" and "Information theory and cognitive science".

Third, four pages state that what can be achieved in practice sits some distance from the theoretical limit. Capacity and the rate-distortion curve are approached only with very long code blocks; short messages and real-time settings generally fall short of them. The length of the shortest program cannot be computed, only estimated from above. The lower bound on the heat cost of erasure is far below what real hardware consumes. See "Channel and capacity", "Rate-distortion theory", "Algorithmic information theory" and "Information theory and thermodynamics".

From these the shelf draws three working practices. Before using any of these measures, write down the reference: for whom, for which judgement, and what loss is allowed. Failures of this kind do not show themselves afterwards, so the safeguard has to come beforehand; for example, decide in advance which half must survive if the other side can recover only half. These limits are suited to checking whether a claim goes past what is possible; they are not suited to serving as targets that ought to be reached.

Whether a piece of content is right or wrong is something the measures on this shelf cannot answer. "Information and uncertainty" and "Self-information" both state that high information content means only that something was unexpected, and that judging correctness takes a different tool. "Mutual information" states that it says only that two things are related, not which one influences the other; the family of time-ordered measures that deals with direction is not held on this shelf.

In what sense information entropy and thermodynamic entropy are the same thing, "Entropy" and "Information theory and thermodynamics" both set out two positions without ruling between them. Whether the general proof of the principle that erasure must dissipate heat holds, "Information theory and thermodynamics" likewise sets out both sides and does not rule.

How much a person can reliably handle at one time is a question this shelf gives no number for. "Information theory and cognitive science" states that the familiar figure is not settled and deliberately adopts no specific value; research cited on that page still lists, as of 2024, how the capacity limit should be assessed as a disputed question.

The note on the group "Who bears these limits" already says that its two pages extrapolate across domains. Four pages in the other groups likewise mark as analogy the parts where they are applied to human communication, judgement or governance: "Channel and capacity", "Rate-distortion theory", "Algorithmic information theory" and "Kullback-Leibler divergence". The six pages agree: the structure may be borrowed to ask questions; the numerical conclusions may not.

The pages were not all checked to the same depth, and each page states its own. On two of them, "Information and uncertainty" and "Signal and noise", the framing statements were checked only at the level of the discipline, not sentence by sentence against primary sources.

Read it yourself · Theory layer

The whole theory layer is here, with no account needed. Read it and go.

Trigpoint Pass · Application layer

The case layer (our own tests of capacity, signal-to-noise and compression discipline built into governance rules, including the record of one that overloaded first) and the full verification layer of most nodes (how far each source was checked, counter-evidence and failure boundaries).

Trigpoint Pass ▸

Tell me when a new shelf opens

Tell me when a new shelf opens

Shelf notices are sent only when a new shelf opens or when the reading rules change.