Borges already knew about entropy

In 1941, Jorge Luis Borges (an Argentine poet) wrote a short story called The Library of Babel. In this story, "The universe (which others call the Library) is composed of an indefinite, perhaps infinite number of hexagonal galleries." He introduces a library that contains every single combination of books that can ever possibly exist in the universe. Each book has 410 pages, 40 lines per page, 80 characters per line, sampled from an alphabet of 25 symbols.

The librarians celebrate at first. The library must contain the proof of every conjecture, the biography of every person who will ever live, the cure for every disease. But they quickly realize that it also contains every false proof, every wrong biography, every cure that's actually a poison. And since there's no index, you can't actually tell a true book from a false one without already knowing the truth.

The number of books in this library is $25^{1{,}312{,}000}$. So its not truly infinite (Borges calls it "unlimited and periodic"). Its just $|\Sigma|^n$, all strings of fixed length over a finite alphabet. A combinatorialist would call this unremarkable.

Claude Shannon would call it something worse: uninformative.

Information entropy measures average surprise per symbol. If you draw uniformly over all possible books, entropy is maximized:

$$ H = \log_2(25^{1{,}312{,}000}) = 1{,}312{,}000 \cdot \log_2 25 \approx 6{,}090{,}816 \text{ bits per book} $$

Therefore 6,090,816 bits per book is the theoretical maximum. Since we're sampling from a uniform distribution, where every book is equally likely, scanning through a book doesn't actually tell you anything since you knew it was in there before you scanned it.

And this is also why the librarians can't conduct search for interesting books. A meaningful book --- a correct proof of the Riemann hypothesis, say --- has low Kolmogorov complexity relative to its length. Structure, redundancy, the kind of thing you can compress. But it sits on a shelf next to $25^{1{,}312{,}000} - 1$ other books, almost all of which are just incompressible gibberish. The ratio of meaningful to meaningless is, however you want to define "meaningful," essentially zero.

And the cost of finding a meaningful book in the library is the same as just writing it yourself. And so The Library of Babel gives you nothing!

Borges intuited in 1941 what Shannon and Kolmogorov would formalize decades later: the complexity of a string is the length of the shortest program that produces it. A random string has complexity roughly equal to its own length --- you can't compress it, there's no structure, it doesn't mean anything in a very precise sense. The meaningful strings are the ones whose complexity is much less than their length.