Following yesterday’s post about measuing information and uncertainty, reference Frankel and Kamenica (2019) AER paper.

Definitions

Decision Problem

A decision problem $\mathcal D = (A, u)$ is defined as, an action set $A$ and a utility function $u: A\times \Omega\to \R\cup {-\infty}$. Assume that there exists some action $a$ such that $u(a, \omega)$ is finite for every $\omega$. Assume that the decision problem is well-defined, by letting $\arg\max_{a\in A} E_q[u(a, \omega)]$ be nonempty for all beliefs $q$.

Value/Cost of Information/Uncertainty

The value of information is defined as $\nu_\mathcal D: \Delta(\Omega)\times \Delta(\Omega) \to \R$, given by $$ \nu_\mathcal D(p, q) = E_p[u(a^(p), w)] - E_p[u(a^(q), w)] $$ where $a^*(q) \in \arg\max_{a\in A} E_q[u(a, w)]$.

The cost of uncertainty is defined as $\mathcal C_\mathcal D:\Delta(\Omega)\to \R$, given by $$ \mathcal C_\mathcal D = E_q[\max_au(a, w)] - \max_a E_q [u(a, w)] $$

Defining Validity

A measure of information $d$ is valid iff.

  • (Null-information) $d(q, q) = 0, \forall q$.

  • (Positivity) $d(p, q)\ge 0,\forall (p, q)$.

  • (Order-invariance) Given any prior and pair of signals, the expected sum of information generated by them is indepdent of the order of their observation, i.e. $$ E[d(q(\alpha), q) + d(q(\alpha\cap \beta), q(\alpha))] = E[d(q(\beta), q) + d(q(\alpha\cap \beta), q(\beta))] $$

A measure of uncertainty $H$ is valid iff.

  • (Null-uncertainty) $H(\delta_w) = 0$ for all $w$.
  • (Concavity) $H$ is concave.

Notice that these two properties jointly imply that $H$ is positive everywhere, and concavity implies that observing any signal necessarily reduces uncertainty, i.e. $\forall q$ and $\pi_s$, $E[H(q(s))] \le H(q)$.

Notice that in the above, we consider ex post value/cost of information/uncertainty - $d$ and $H$ are calculated w.r.t. realized value of signal and then take expectation.

Jointly Valid vs. Coupled

  • Jointly Valid

    $d$ and $H$ are called jointly valid if they are defined on the same decision problem $\mathcal D$.

  • They are coupled if, for every prior $q$ and signal $\pi_s$, we have $$ E[d(q(s), q)] = E[H(q) - H(q(s))]. $$ In other words, for the signal $\pi_s$, the expected amount of information generated equals the reduction in uncertainty measured.

Consistency

Because $H$ is concave, we have $H(q) + \nabla H(q) \cdot(p -q)\ge H(p)$. A Bregman divergence of it be, some function mapping from $(p, q)$ to real numbers which equal to $$ g(p, q) = H(q) - H(p) + \nabla H(q)\cdot (p -q) $$ [Theorem] Given a valid measure of information $d$ and a valid measure of uncertainty $H$, these following conditions are equivalent to each other

  • $d$ and $H$ are jointly valid.
  • $d$ and $H$ are coupled.
  • $H(q) = \sum_{w\in \Omega} q^w d(\delta_w, q)$
  • $d$ is a Bregman divergence of $H$.

Checking for Validity

It’s easy to check the validity of a given measure of uncertainty (positivity, null-info and concavity). But checking the validity of some measure of information seems not so straightforward, especially its order-invariance condition.

But first, notice that, if a measure of information if smooth, order-invariance can be confirmed by inspecting its derivatives:

[Proposition 1] For $d$ - a measure of information - be twice differentiable in $p$ for all $q$ and satisfies Null-info. Then, $d$ satisfies Order-invariance iff. $\partial^2 d(p, q) / \partial p^2$ is independent of $q$.

And, we have an even cooloer proposition for general $d$:

[Proposition 2] For $d$ - a measure of information - satisfies Null-info and Positivity, it is Order-invariance iff. $d$ is a Bregman divergence of $\sum_{w\in \Omega} q^w d(\delta_w, q)$.

Invalidity of Metrics

A Bergman divergence cannot be a metric, cause if it’s a metric $$ g(p, q) + g(q, t) \ge g(p, t) $$ which implies $$ H(q) - H(p) + H(t) - H(q) + \nabla H\cdot (q) (p - q) + \nabla H(t) \cdot (q - t)\ge H(t) - H(p) + \nabla H(t) (p - t) $$ and further implies $\nabla H(q) <= \nabla H(t)$ If this measure is nontrivial, that sometimes the inequality holds strictly, then it yields a contradiction.

[Corollary] If a measure of information is a metric on $\Delta (\Omega)$, it does not satisfy Order-invariance and thus cannot be valid.

For example, as noted by Augenblick and Rabin (2018), no measure of uncertainty that can be coupled with Euclidean distance.