数学代写|EE276/Stats376a Information Theory

Two fair dice are thrown. The outcome of the experiment is taken as the sum of the resulting outcomes of the two dice. What is the entropy of this experiment?

Suppose there are 3 urns, each containing two balls. The first urn contains two black balls; the second contains one black ball and one white ball; the third contains two white balls. An urn is selected at random (prob. $1 / 3$ of selecting each urn), and one ball is removed from this urn. Let $X={1,2,3}$ represent the sample space of the urn selected, and let $Y={B, W}$ be the sample space of the color ball selected. Calculate the entropies $H(X), H(Y), H(X Y), H(X \mid Y), H(Y \mid X)$, and the mutual information $I(X ; Y)$

Consider a horse race in which there are a total of 7 horses, 4 black horses and 3 grey horses; hence the sample spacce of the horse race can be represented by $X=$ $\left{b_1, b_2, b_3, b_4, g_5, g_6, g_7\right}$. Subdivide the horse race such that we have two experiments, $Y=\left{b_1, \ldots, b_4\right}$, the outcome among the black horses, and $Z=\left{g_5, \ldots, g_7\right}$ the outcome among the grey horses. Define the event $B$ as the event that a black horse wins. Assuming that in the original experiment, the a priori probabilities of any particular horse winning is equal, verify that
$$
H(X)=H(B)+P(B) H(Y)+P(\bar{B}) H(Z)
$$

Let $X$ be a random variable that only assumes non-negative integer values. Suppose $\mathrm{E}[X]=M$, where $M$ is some fixed number. In terms of $M$, how large can $H(X)$ be? Describe the extremal $X$ as completely as you can.

In class, we stated the following theorem for convex functions:
Theorem. If $\lambda_1, \ldots, \lambda_N$ are non-negative numbers whose sum is unity, then for every set of points $\left{P_1, \ldots, P_N\right}$ in the domain of the convex $\cap$ function $f(P)$, the following inequality is valid:
$$
f\left(\sum_{n=1}^N \lambda_n P_n\right) \geq \sum_{n=1}^N \lambda_n f\left(P_n\right) .
$$
Prove this theorem, and using this result, prove Jensen’s inequality:
Jensen’s Inequality. Assume $X$ is a random variable taking on values from the set $R_X=\left{x_1, \ldots, x_N\right}$ with probabilities $p_1, \ldots, p_N$. If $f(x)$ be a convex $\cap$ function whose domain includes $R_X$, then
$$
f(\mathrm{E}[X]) \geq \mathrm{E}[f(X)]
$$
Furthermore, if $f(x)$ is strictly convex, equality holds if and only if one particular element of $R_X$ is certain.
(Hint: The initial theorem is most easily proven by induction.)