This is not a Wikipedia post. I won't lecture you on history or theorems. Instead, I'm exploring what Shannon's framework actually means—as a way of seeing the world.
After reading A Mathematical Theory of Communication by Claude Shannon, I found 4 ideas that felt like unlocking new ways to think. This is my attempt to map them out.
Uncertainty is Maximized When Outcomes are Equally Likely
A system feels most uncertain when you have no reason to prefer one outcome over another. That sounds obvious, but it's actually a deep mathematical constraint.
If you flip a biased coin that lands heads 90% of the time, you stop thinking. You just bet heads. But if the coin is fair, you're stuck. No shortcut exists. Every outcome is equally plausible.
This is the core of Shannon's information theory: uncertainty peaks when everything is symmetric. For a discrete system:
H(X) = −∑p(x)log₂p(x)
When all outcomes are equally likely (p(x) = 1/n):
H(X) = log₂(n)
This is the maximum uncertainty you can get for n outcomes.
The insight: Systems feel most unpredictable not when they're chaotic, but when they're balanced.
Information is the Number of Bits Required to Resolve Uncertainty
People treat "information" like it's abstract and philosophical. But in Shannon's world, it's mechanical.
Information is simply: how many binary questions do you need to figure out what happened?
Say you have 8 possible outcomes. You ask:
- Is it in the first half?
- Is it in the first quarter?
- Is it this exact one?
That's 3 yes/no questions, which equals:
log₂(8) = 3 bits
So information is the cost of removing uncertainty as efficiently as possible.
This is why good explanations feel compressed—they don't pile on details. They collapse possibilities. They answer the right questions in the right order.
In Adversarial Systems, Inject Uncertainty Into Their Decision Tree
I once heard a chess story (forgotten the source) where a coach said: the foundation of tactics is the double attack.
That sounds like just chess. But it's really about something broader: forcing your opponent into too many branches.
Chess is two players constantly mangling each other's decision trees. Every move creates branches. Every threat forces new calculations.
Here's the trick: you don't need perfect moves. You just need to make the position messy enough that your opponent can't keep up.
Mikhail Tal's sacrifices are considered "incorrect" by modern engines. But against humans, they were lethal. Why? Because he was brilliant at one thing: making the position too complicated to calculate in real time.
Humans aren't calculators. When there are too many equally plausible threats, we guess. We prune the tree badly. We hallucinate danger. We miss simple stuff.
So in adversarial situations, being right isn't enough. You want the other person overwhelmed by options. You want to inject uncertainty into their head.
This is the practical spine of information theory: it's not just about communication and compression. It's a lens for seeing how certainty, clarity, and power distribute in any system.
When Learning, Add Noise on Purpose
Most people learn by making things as clean as possible: same problem type, same format, same context.
That feels efficient. But it creates shallow understanding.
You only truly understand something when you can recognize it in different disguises. Different contexts. Different framings.
So instead of repeating the same pattern, mix things up:
- Same idea, different context
- Same problem type, different framing
- Re-derive instead of memorize
- Mix problem sets instead of batch them
What you're doing is forcing your brain to rebuild the structure, not just memorize a single path through it. That's how understanding becomes robust.
There is a balance, though. Too much noise and nothing sticks. Too little and you overfit.
The sweet spot is where it feels slightly uncomfortable—like you almost understand it, but not quite automatically yet. That's usually where learning is actually happening.