Trigpoint
Collections › Information Theory › Self-information
Should the information in an event be counted with the insider's probabilities or the outsider's?

Self-information

Self-Information · Shannon (1948) / Samson (1951)
Bibliography checked ✓ Information theory · core measures Read first: Information and uncertainty Tell the editor privately ▸
In three sentences How much information something carries depends on how unlikely it was in the first place. Something certain happens and it has told you nothing. Something all but impossible happens and it carries the most information there is. So what information measures is how much of your prior expectation was overturned.

↳ Seen a counter-example?

Self-information Shannon (1948) · Samson (1951) The less likely it was, the more it says certain: no information almost impossible: most information Information low probability high probability Why take the logarithm two independent events multiply but their information should add the log turns product into sum Whose probability is it the same sentence tells an expert less and an outsider a great deal say whose expectation first Quantity: 1948, unnamed · term coined in a 1951 report · popularised 1961 · other name: first use unknown The curve shows shape and direction only: no numerical scale, no units, no coordinates. The 1951 report was not read here, so the wording could not be checked; attribution rests on later sources. Figure: trigpoint.pub
Figure 1 · How information relates to probability. The lower the probability, the more information the event carries when it occurs; when something is all but certain, its occurrence tells you almost nothing. The panel on the right shows why the logarithm is taken: it is what makes the information of two independent events add up. This figure deliberately carries no numerical scale and no units. View full size ▸

1. What happened

Before the information in an event can be counted, a choice has to be made: the probabilities of someone with inside knowledge, or those of an outsider. Once information is defined as the reduction of uncertainty, the next step is to fix a number for a single event. The answer is already contained in Shannon's 1948 paper: entropy is the probability-weighted average of log(1/p) over the outcomes, and the paper uses the same quantity to bound the code length a single message needs[1]Shannon, C. E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal, 27(3), 379–423; 27(4), 623–656. DOI 10.1002/j.1538-7305.1948.tb01338.x. Bibliographic details checked against the official record. Claims matched against the full text. The paper uses the quantity: entropy is the probability-weighted average of log(1/p) over the outcomes, and the bound on the code length a single message needs is also stated in terms of log(1/p). But the paper does not define it separately as the information of a single event, and gives it no name; the words self-information, surprisal and mutual information appear nowhere in it.. Taken on its own, it gives a result simple to the point of being counterintuitive: the information of an event is determined by its probability alone, and the lower the probability the higher the information. For something certain, the information is zero.

Why take a logarithm rather than the reciprocal of the probability directly? Because information has to add up. When two unrelated events both occur, what they carry together should intuitively be the sum of what each carries, and the probability of two independent events both occurring is the product of the two probabilities. To turn multiplication into addition, the logarithm is the only natural choice.

That apparently technical decision is what makes information a quantity you can accumulate, compare and use as a unit, and the whole practical value of the theory rests on it.

One detail slips by easily: whose probability. For the same event, someone with inside knowledge assigns a high probability and gets low information, while an outsider assigns a low probability and gets high information. The theory does not prescribe where the probabilities come from. It only requires you to say whose expectation is in play. That is why how much information is in this sentence has no absolute answer, only an answer for a given person.

As for the name, two things are booked separately. The quantity comes out of the framework of Shannon's 1948 paper, which gave it no name[1]Shannon, C. E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal, 27(3), 379–423; 27(4), 623–656. DOI 10.1002/j.1538-7305.1948.tb01338.x. Bibliographic details checked against the official record. Claims matched against the full text. The paper uses the quantity: entropy is the probability-weighted average of log(1/p) over the outcomes, and the bound on the code length a single message needs is also stated in terms of log(1/p). But the paper does not define it separately as the information of a single event, and gives it no name; the words self-information, surprisal and mutual information appear nowhere in it.. Priority for coining the word surprisal is assigned by the later literature to a 1951 air force research report, with Tribus's 1961 book putting it into circulation rather than originating it[2]Samson, E. W. (1951). Fundamental Natural Concepts of Information Theory. AFCRC Report E5079. Bibliographic details confirmed by two sources. The attribution of the coinage of surprisal follows secondary accounts that agree with one another. The later literature assigns priority for coining the word surprisal to this report. This site has not read the report itself, so whether the word appears in it could not be checked word for word.[3]Tribus, M. (1961). Thermostatics and Thermodynamics. Van Nostrand. Bibliographic details confirmed by two sources. The account of this book as populariser of the word follows secondary accounts that agree with one another. Put the word surprisal into circulation: popularisation rather than origination.. The term self-information does not appear in Shannon's 1948 paper, and it should be credited to nobody[4]The term self-information does not appear in Shannon's 1948 paper; this site has found no reliable source for its first use and so credits it to nobody..

Attribution where due: The quantity comes out of the framework of Shannon (1948), though the paper neither defines it separately nor names it[1]Shannon, C. E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal, 27(3), 379–423; 27(4), 623–656. DOI 10.1002/j.1538-7305.1948.tb01338.x. Bibliographic details checked against the official record. Claims matched against the full text. The paper uses the quantity: entropy is the probability-weighted average of log(1/p) over the outcomes, and the bound on the code length a single message needs is also stated in terms of log(1/p). But the paper does not define it separately as the information of a single event, and gives it no name; the words self-information, surprisal and mutual information appear nowhere in it.. Priority for coining surprisal is assigned by the later literature to Samson (1951)[2]Samson, E. W. (1951). Fundamental Natural Concepts of Information Theory. AFCRC Report E5079. Bibliographic details confirmed by two sources. The attribution of the coinage of surprisal follows secondary accounts that agree with one another. The later literature assigns priority for coining the word surprisal to this report. This site has not read the report itself, so whether the word appears in it could not be checked word for word., with Tribus (1961) as populariser and not originator[3]Tribus, M. (1961). Thermostatics and Thermodynamics. Van Nostrand. Bibliographic details confirmed by two sources. The account of this book as populariser of the word follows secondary accounts that agree with one another. Put the word surprisal into circulation: popularisation rather than origination.. Book the quantity and the word separately. Attributing both to Shannon is a common simplification, and it erases an author with a real priority claim along with a real piece of terminological history.

2. Why this matters to you

What this page gives you is a reading action you can apply directly: for any passage, ask first which of your expectations it overturned.

The action tells apart two things that read very much alike. One retells something you already believe, more completely and in better order. The other tells you that something you took to be so is not. The first is comfortable to read and carries information close to zero. The second is uncomfortable and carries a lot.

Comfort and quantity of information are negatively correlated here, which makes a sharp criterion for choosing what to read.

On model output it applies more directly still. A response that falls entirely inside your expected distribution reduces none of your uncertainty, however long and however complete it is. Useful output usually shares one feature: at least one item in it is something you would have bet against beforehand.

So rather than asking whether this answer is good, ask whether anything in it surprised you. If nothing did, the yield of that call is close to zero, whatever the word count.

The reverse holds too, and it makes a practical self-check: when everything you write has everyone nodding, you may have said nothing. Consensus statements have high probability and low information. That does not make them worthless, since laying groundwork and aligning have their functions, but be clear that that part is not you transmitting information. It is you aligning probability distributions. Book the two separately.

Verification layer · Members
The guided layer of this node is open to all readers.
Verification details and failure boundaries are for members.
US$8.25 a month on average · or subscribe monthly ▸

Lab cases Written only where there is evidence

No field record for this node yet.

Common misreadings

Page verification: Bibliography checked ✓ · Content last revised 2026-11-17

  • A common account says that Shannon proposed both the concept and the name surprisal. Shannon's 1948 paper uses the quantity but coins no word for it. Priority for the word is assigned by the later literature to a 1951 air force research report, and a 1961 book by Tribus put it into circulation. As for the other common term, self-information, it does not appear in Shannon's 1948 paper, and this site has found no reliable source for its first use. #
  • A common account says that information depends on whether an event is important. It does not depend on importance, only on how improbable the event was. That is hard to accept intuitively, but it is exactly the premise that lets the whole theory quantify. #
References

How to read the source notes “Bibliographic details checked” means author, year, title and venue were confirmed. It does not mean the source's claims were checked. “Matched against the full text” means the claim was compared with the text of the source itself. “Follows secondary accounts” means the claim rests on sources that report it, not on the original document. Definitions as of 2026-09-20.

  1. [1] Shannon, C. E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal, 27(3), 379–423; 27(4), 623–656. ↩
  2. [2] Samson, E. W. (1951). Fundamental Natural Concepts of Information Theory. AFCRC Report E5079. ↩
  3. [3] Tribus, M. (1961). Thermostatics and Thermodynamics. Van Nostrand. ↩
  4. [4] Editorial note with no external source; the full text is in note [4] in the body.

↳ Is a citation wrong?

Tell me when a new shelf opens

Tell me when a new shelf opens

Shelf notices are sent only when a new shelf opens or when the reading rules change.

Trial

Tell the editor privately: a correction, a counter-example or where it applies

Submissions are read only by the editor, and nothing is published before they are handled. Once a note is handled, a summary and response written by the editor are published. No sign-up is needed, and the nickname may be left blank.

What are you submitting?
Set a public name to be credited by name ▸
Submit anonymously, without signing in

Signing in lets you follow the status on your page. What you have written is kept.

You are signed in: this note is linked to your account, visible only to you and the editor, so you can follow its status on your page. The text is never published. Full privacy policy ▸

Submit anonymously instead, not linked to your account ▸

No email is collected. Your original text and nickname are read only by the editor and are never published; the site publishes only a summary the editor writes. The human check is provided by Cloudflare Turnstile. To limit abuse, the system keeps a hash of your IP address, not the address itself. Your original text is stored with our cloud provider and can be deleted on request. Full privacy policy ▸