19. Etymology of Entropy#

19.1. Information Theory#

In information theory [Shannon and Weaver, 1949], entropy is a measure of the unpredictability of a random variable.

To illustrate things, let $X$ be a discrete random variable taking values $x_1,\dots ,x_n$ with probabilities $p_i=\text{Prob}(X=x_i)\ge 0,\sum _i{p}_i=1$.

Claude Shannon’s [Shannon and Weaver, 1949] definition of entropy is

where ${\log }_b$ denotes the log function with base $b$.

Inspired by the limit

we set $p\log p=0$ in equation (19.1).

Typical bases for the logarithm are $2$, $e$, and $10$.

In the information theory literature, logarithms of base $2$, $e$, and $10$ are associated with units of information called bits, nats, and dits, respectively.

Shannon typically used base $2$.

19.2. A Measure of Unpredictability#

For a discrete random variable $X$ with probability density $p=\{p_i{\}}_{i=1}^n$, the surprisal for state $i$ is $s_i=\log \left(\frac{1}{p_i}\right)$.

The quantity $\log \left(\frac{1}{p_i}\right)$ is called the surprisal because it is inversely related to the likelihood that state $i$ will occur.

Note that entropy $H(p)$ equals the expected surprisal

19.2.1. Example#

Take a possibly unfair coin, so $X=\{0,1\}$ with $p=\mathrm{P}\mathrm{r}\mathrm{o}\mathrm{b}(X=1)=p\in [0,1]$.

Then

$$H(p)=-(1-p)\log (1-p)-p\log p.$$

Evidently,

$$H^{\prime }(p)=\log (1-p)-\log p=0$$

at $p=.5$ and $H^{″}(p)=-\frac{1}{1-p}-\frac{1}{p}<0$ for $p\in (0,1)$.

So $p=.5$ maximizes entropy, while entropy is minimized at $p=0$ and $p=1$.

Thus, among all coins, a fair coin is the most unpredictable.

See Fig. 19.1

19.3. Mathematical Properties of Entropy#

For a discrete random variable with probability vector $p$, entropy $H(p)$ is a function that satisfies

• $H$ is continuous.

• $H$ is symmetric: $H(p_1,p_2,\dots ,p_n)=H(p_{r_1},\dots ,p_{r_n})$ for any permutation $r_1,\dots ,r_n$ of $1,\dots ,n$.

• A uniform distribution maximizes $H(p)$: $H(p_1,\dots ,p_n)\le H(\frac{1}{n},\dots ,\frac{1}{n}).$

• Maximum entropy increases with the number of states: $H(\frac{1}{n},\dots ,\frac{1}{n})\le H(\frac{1}{n+1},\dots ,\frac{1}{n+1})$.

• Entropy is not affected by events zero probability.

19.4. Conditional Entropy#

Let $(X,Y)$ be a bivariate discrete random vector with outcomes $x_1,\dots ,x_n$ and $y_1,\dots ,y_m$, respectively, occurring with probability density $p(x_i,y_i)$.

Conditional entropy $H(X|Y)$ is defined as

Here $\frac{p(y_j)}{p(x_i,y_j)}$, the reciprocal of the conditional probability of $x_i$ given $y_j$, can be defined as the conditional surprisal.

19.5. Independence as Maximum Conditional Entropy#

Let $m=n$ and $[x_1,\dots ,x_n]=[y_1,\dots ,y_n]$.

Let $\sum _{j}p(x_i,y_j)=\sum _{j}p(x_j,y_i)$ for all $i$, so that the marginal distributions of $x$ and $y$ are identical.

Thus, $x$ and $y$ are identically distributed, but they are not necessarily independent.

Consider the following problem: choose a joint distribution $p(x_i,y_j)$ to maximize conditional entropy (19.2) subject to the restriction that $x$ and $y$ are identically distributed.

The conditional-entropy-maximizing $p(x_i,y_j)$ sets

Thus, among all joint distributions with identical marginal distributions, the conditional entropy maximizing joint distribution makes $x$ and $y$ be independent.

19.6. Thermodynamics#

Josiah Willard Gibbs (see https://en.wikipedia.org/wiki/Josiah_Willard_Gibbs) defined entropy as

where $p_i$ is the probability of a micro state and $k_B$ is Boltzmann’s constant.

• The Boltzmann constant $k_b$ relates energy at the micro particle level with the temperature observed at the macro level. It equals what is called a gas constant divided by an Avogadro constant.

The second law of thermodynamics states that the entropy of a closed physical system increases until $S$ defined in (19.3) attains a maximum.

19.7. Statistical Divergence#

Let $X$ be a discrete state space $x_1,\dots ,x_n$ and let $p$ and $q$ be two discrete probability distributions on $X$.

Assume that $\frac{p_i}{q_t}\in (0,\mathrm{\infty })$ for all $i$ for which $p_i>0$.

Then the Kullback-Leibler statistical divergence, also called relative entropy, is defined as

Evidently,

where $H(p,q)=\sum _i{p}_i\log q_i$ is the cross-entropy.

It is easy to verify, as we have done above, that $D(p|q)\ge 0$ and that $D(p|q)=0$ implies that $p_i=q_i$ when $q_i>0$.

19.8. Continuous distributions#

For a continuous random variable, Kullback-Leibler divergence between two densities $p$ and $q$ is defined as

19.9. Relative entropy and Gaussian distributions#

We want to compute relative entropy for two continuous densities $\varphi$ and $\hat{\varphi }$ when $\varphi$ is $\mathcal{N}(0,I)$ and $\hat{\varphi }$ is $\mathcal{N}(w,\mathrm{\Sigma })$, where the covariance matrix $\mathrm{\Sigma }$ is nonsingular.

We seek a formula for

Claim

Proof

The log likelihood ratio is

Observe that

Applying the identity $\epsilon =w+(\epsilon -w)$ gives

Taking mathematical expectations

Combining terms gives

which agrees with equation (19.5). Notice the separate appearances of the mean distortion $w$ and the covariance distortion $\mathrm{\Sigma }-I$ in equation (19.7).

Extension

Let $N_0=\mathcal{N}({\mu }_0,{\mathrm{\Sigma }}_0)$ and $N_1=\mathcal{N}({\mu }_1,{\mathrm{\Sigma }}_1)$ be two multivariate Gaussian distributions.

Then

19.10. Von Neumann Entropy#

Let $P$ and $Q$ be two positive-definite symmetric matrices.

A measure of the divergence between two $P$ and $Q$ is

where the log of a matrix is defined here (https://en.wikipedia.org/wiki/Logarithm_of_a_matrix).

A density matrix $P$ from quantum mechanics is a positive definite matrix with trace $1$.

The von Neumann entropy of a density matrix $P$ is

19.11. Backus-Chernov-Zin Entropy#

After flipping signs, [Backus et al., 2014] use Kullback-Leibler relative entropy as a measure of volatility of stochastic discount factors that they assert is useful for characterizing features of both the data and various theoretical models of stochastic discount factors.

Where $p_{t+1}$ is the physical or true measure, $p_{t+1}^{\ast }$ is the risk-neutral measure, and $E_t$ denotes conditional expectation under the $p_{t+1}$ measure, [Backus et al., 2014] define entropy as

Evidently, by virtue of the minus sign in equation (19.9),

where $D_{KL,t}$ denotes conditional relative entropy.

Let $m_{t+1}$ be a stochastic discount factor, $r_{t+1}$ a gross one-period return on a risky security, and $(r_{t+1}^1{)}^{-1}\equiv q_t^1=E_t{m}_{t+1}$ be the reciprocal of a risk-free one-period gross rate of return. Then

[Backus et al., 2014] note that a stochastic discount factor satisfies

They derive the following entropy bound

which they propose as a complement to a Hansen-Jagannathan [Hansen and Jagannathan, 1991] bound.

19.12. Wiener-Kolmogorov Prediction Error Formula as Entropy#

Let $\{x_t{\}}_{t=-\mathrm{\infty }}^{\mathrm{\infty }}$ be a covariance stationary stochastic process with mean zero and spectral density $S_x(\omega )$.

The variance of $x$ is

As described in chapter XIV of [Sargent, 1987], the Wiener-Kolmogorov formula for the one-period ahead prediction error is

Occasionally the logarithm of the one-step-ahead prediction error ${\sigma }_{ϵ}^2$ is called entropy because it measures unpredictability.

Consider the following problem reminiscent of one described earlier.

Problem:

Among all covariance stationary univariate processes with unconditional variance ${\sigma }_x^2$, find a process with maximal one-step-ahead prediction error.

The maximizer is a process with spectral density

Thus, among all univariate covariance stationary processes with variance ${\sigma }_x^2$, a process with a flat spectral density is the most uncertain, in the sense of one-step-ahead prediction error variance.

This no-patterns-across-time outcome for a temporally dependent process resembles the no-pattern-across-states outcome for the static entropy maximizing coin or die in the classic information theoretic analysis described above.

19.13. Multivariate Processes#

Let $y_t$ be an $n\times 1$ covariance stationary stochastic process with mean $0$ with matrix covariogram $C_y(j)=E{y}_t{y}_{t-j}^{\prime }$ and spectral density matrix

Let

be a Wold representation for $y$, where $D(0){ϵ}_t$ is a vector of one-step-ahead errors in predicting $y_t$ conditional on the infinite history $y^{t-1}=[y_{t-1},y_{t-2},\dots ]$ and ${ϵ}_t$ is an $n\times 1$ vector of serially uncorrelated random disturbances with mean zero and identity contemporaneous covariance matrix $E{ϵ}_t{ϵ}_t^{\prime }=I$.

Linear-least-squares predictors have one-step-ahead prediction error $D(0)D(0{)}^{\prime }$ that satisfies

Being a measure of the unpredictability of an $n\times 1$ vector covariance stationary stochastic process, the left side of (19.12) is sometimes called entropy.

19.14. Frequency Domain Robust Control#

Chapter 8 of [Hansen and Sargent, 2008] adapts work in the control theory literature to define a frequency domain entropy criterion for robust control as

where $\theta \in (\underset{―}{\theta },+\mathrm{\infty })$ is a positive robustness parameter and $G_F(\zeta )$ is a $\zeta$-transform of the objective function.

Hansen and Sargent [Hansen and Sargent, 2008] show that criterion (19.13) can be represented as

for an appropriate covariance stationary stochastic process derived from $\theta ,G_F(\zeta )$.

This explains the moniker maximum entropy robust control for decision rules $F$ designed to maximize criterion (19.13).

19.15. Relative Entropy for a Continuous Random Variable#

Let $x$ be a continuous random variable with density $\varphi (x)$, and let $g(x)$ be a nonnegative random variable satisfying $\int g(x)\varphi (x)dx=1$.

The relative entropy of the distorted density $\hat{\varphi }(x)=g(x)\varphi (x)$ is defined as

Fig. 19.2 plots the functions $g\log g$ and $g-1$ over the interval $g\ge 0$.

That relative entropy $\text{ent}(g)\ge 0$ can be established by noting (a) that $g\log g\ge g-1$ (see Fig. 19.2) and (b) that under $\varphi$, $Eg=1$.

Fig. 19.3 and Fig. 19.4 display aspects of relative entropy visually for a continuous random variable $x$ for two densities with likelihood ratio $g\ge 0$.

Where the numerator density is $\mathcal{N}(0,1)$, for two denominator Gaussian densities $\mathcal{N}(0,1.5)$ and $\mathcal{N}(0,.95)$, respectively, Fig. 19.3 and Fig. 19.4 display the functions $g\log g$ and $g-1$ as functions of $x$.