Consider a random variable $X$ taking values in the sample space $\Omega_{X}={0,1,2,3, \ldots}$. Denote by $p_{j}=\operatorname{Prob}[X=j]$ with $j \in \Omega_{X} .$ Let $Y$ be a Poisson random variable with parameter $\lambda$, defined over the same sample space $\Omega_{X}$
$$
q_{j}=\operatorname{Prob}[Y=j]=\frac{\lambda^{j} \mathrm{e}^{-\lambda}}{j !}
$$
(a) [5 points] Show that the distribution $\left{q_{j}\right}$ is correctly normalised over $\Omega_{X}$.
(b) [ 5 points] Show that the average of $Y$ is equal to $\lambda$.
(c) [ 7 points] Compute the KL-divergence $D_{\mathrm{KL}}(p | q)$ between the two distributions as a function of $\lambda$. The result should include the average $\langle X\rangle=\sum_{j=0}^{\infty} j p_{j}$ and the entropy $H(X)$ of $X-$ among other term(s).
(d) [3 points] Find the minimum of $D_{\mathrm{KL}}(p | q)$ by differentiating with respect to $\lambda$. Deduce which value of the Poisson parameter $\lambda$ must be chosen to best approximate any distribution over the non-negative integers with a Poisson distribution.

Consider a random variable $X$ taking values in the sample space $\Omega_{X}={0,1,2,3, \ldots}$. Denote by $p_{j}=\operatorname{Prob}[X=j]$ with $j \in \Omega_{X} .$ Let $Y$ be a Poisson random variable with parameter $\lambda$, defined over the same sample space $\Omega_{X}$
$$
q_{j}=\operatorname{Prob}[Y=j]=\frac{\lambda^{j} \mathrm{e}^{-\lambda}}{j !}
$$
(a) [5 points] Show that the distribution $\left{q_{j}\right}$ is correctly normalised over $\Omega_{X}$.
(b) [ 5 points] Show that the average of $Y$ is equal to $\lambda$.
(c) [ 7 points] Compute the KL-divergence $D_{\mathrm{KL}}(p | q)$ between the two distributions as a function of $

(a) [8 points] State and prove the inequality part of Jensen’s theorem for the case of an alphabet of two outcomes, $\mathcal{A}{X}=\left{a{1}, a_{2}\right}$, with probabilities $\mathbb{P}\left(X=a_{1}\right)=p_{1}$ and $\mathbb{P}\left(X=a_{2}\right)=p_{2} .$ Do not assume any specific form for $f(x)$ in the proof.
(b) [ 6 points] Use Jensen’s inequality with the convex function $f(x)=-\ln x$ to prove the inequality of arithmetic and geometric means, namely for any $n$ non-negative real numbers $x_{1}, \ldots, x_{n}$
$$
\frac{x_{1}+x_{2}+\ldots+x_{n}}{n} \geq\left(x_{1} x_{2} \cdots x_{n}\right)^{1 / n}
$$
(c) [6 points] Use Jensen’s inequality with the convex function $f(x)=\sqrt{x^{2}+1}$ to prove that for all $n \in \mathbb{N}$
$$
\sqrt{1^{2}+1}+\sqrt{2^{2}+1}+\ldots+\sqrt{n^{2}+1} \geq \frac{n}{2} \sqrt{n^{2}+2 n+5}
$$

Consider a random variable $X$ taking values in the sample space $\Omega_{X}={0,1,2,3, \ldots}$. Denote by

A charitable organisation is organising one raffle per day over a period of $N$ days to raise money for the environment. Assume that before the $j$ -th draw, a given person can buy $\sigma_{j}$ tickets, with $\sigma_{j}$ a random variable defined on the sample space $\Omega={0,1,2, \ldots, 8}$. Let $\sigma=\left(\sigma_{1}, \ldots, \sigma_{N}\right)$ be the sequence of tickets purchased by this person prior to each of the $N$ draws. Assume that $N$ is even. Let
$$
M_{O}(\boldsymbol{\sigma})=\sum_{\text {odd } j} \sigma_{j} \quad M_{E}(\boldsymbol{\sigma})=\sum_{\text {even } j} \sigma_{j}
$$
be the total number of tickets purchased during odd-numbered and even-numbered days, respectively. Assume that we can reliably estimate the average values $\left\langle M_{O}(\boldsymbol{\sigma})\right\rangle$ and $\left\langle M_{E}(\boldsymbol{\sigma})\right\rangle$ over many raffle seasons.
In the following, you may express the results of calculations using the shorthands $\varphi(\lambda)=\sum_{n=0}^{8} \mathrm{e}^{\lambda n}$ and $\theta(\lambda)=\sum_{n=0}^{8} n \mathrm{e}^{\lambda n}$, without evaluating the sums
explicitly.
(a) $[6$ points $]$ By finding the minimum of a suitably defined Lagrangian $\mathcal{L}$, determine the probability distribution $P(\sigma)$ (depending on two Lagrange multipliers $\left.\lambda_{1}, \lambda_{2}\right)$ that maximises the entropy $S[P]=-\sum_{\sigma} P(\sigma) \ln P(\sigma)$ subject to the constraints on $\left\langle M_{O}(\boldsymbol{\sigma})\right\rangle$ and $\left\langle M_{E}(\boldsymbol{\sigma})\right\rangle$. Determine the partition function explicitly.
(b) [2 points] Express the free energy $F$ in terms of the Lagrange multipliers $\lambda_{1}, \lambda_{2}$
(c) [2 points] Using the free energy $F$, determine $\left\langle M_{O}(\boldsymbol{\sigma})\right\rangle$ and $\left\langle M_{E}(\boldsymbol{\sigma})\right\rangle$ as functions of the Lagrange multipliers $\lambda_{1}, \lambda_{2}$
(d) [4 points] Using the results in (b) and (c), compute $S[P]$ as a function of $\lambda_{1}, \lambda_{2}$
(e) $[6$ points $]$ Setting $\lambda_{1}=\lambda_{2}=\lambda$, discuss what happens to the MaxEnt distribution $P(\sigma)$ in the cases $\lambda \rightarrow-\infty, \lambda \rightarrow+\infty$ and $\lambda \rightarrow 0$, relating your answer to the value taken by $S[P]$ in each of these limits.

Consider a random variable $X$ taking values in the sample space $\Omega_{X}={0,1,2,3, \ldots}$. Denote by $p_{j}=\operatorname{Prob}[X=j]$ with $j \in \Omega_{X} .$ Let $Y$ be a Poisson random variable with parameter $\lambda$, defined over the same sample space $\Omega_{X}$

Consider a system of $N$ voters $\sigma_{i} \in{-1,0,1} .$ Denote the voting pattern by $\sigma=\left(\sigma_{1}, \ldots, \sigma_{N}\right)$, and assume that the distribution of patterns is
$$
P(\boldsymbol{\sigma})=\exp (h M(\boldsymbol{\sigma})) / Z \text { , }
$$
where $M(\sigma)=\sum_{i=1}^{N} \sigma_{i}$ is the magnetisation, $h$ the associated Lagrange multiplier, and $Z$ the partition function. Define the number of negative votes as $M_{-}(\boldsymbol{\sigma})=\frac{1}{2} \sum_{i=1}^{N}\left(\sigma_{i}^{2}-\sigma_{i}\right)$. In the following, you may use without proof:

• the binomial expansion formula $(x+y)^{n}=\sum_{k=0}^{n}\left(\begin{array}{l}n \ k\end{array}\right) x^{k} y^{n-k}$, for $n \in \mathbb{N}$.
• the symmetry of binomial coefficients $\left(\begin{array}{c}n \ k\end{array}\right)=\left(\begin{array}{c}n \ n-k\end{array}\right)$, for $0 \leq k \leq n$.
• the identity $\sum_{k=0}^{n} k a^{k}\left(\begin{array}{l}n \ k\end{array}\right)=a n(a+1)^{n-1}$.
  (a) [4 points] Compute the partition function $Z$ and the free energy $F$ as functions of $h$ and $N$.
  (b) [ 17 points] Using the integral representation of the Kronecker delta
  $$
  \delta_{a, b}=\int_{0}^{2 \pi} \frac{\mathrm{d} \xi}{2 \pi} e^{\mathrm{i} \xi(a-b)}
  $$
  for $a, b$ integers, compute the probability distribution of the number of negative votes
  $$
  P\left(M_{-}\right)=\operatorname{Prob}\left[M_{-}(\boldsymbol{\sigma})=M_{-}\right]=\sum_{\sigma} P(\boldsymbol{\sigma}) \delta_{M_{-}, M_{-}(\sigma)}
  $$
  and check that it is correctly normalised on the alphabet $M_{-} \in{0,1, \ldots, N}$.
  (c) $[6$ points $]$ Compute the average number of negative votes
  $$
  \left\langle M_{-}(\sigma)\right\rangle=\sum_{M_{-}=0}^{N} P\left(M_{-}\right) M_{-}
  $$
  (d) [3 points] Compute the limits $h \rightarrow 0$ and $h \rightarrow \pm \infty$ of $\left\langle M_{-}(\boldsymbol{\sigma})\right\rangle$ from (c), and provide an intuitive explanation of the results.

Consider a one-dimensional Ising chain with $N$ spins taking values $\sigma_{i} \in{-1,1}$. The probability of a configuration $\sigma=\left(\sigma_{1}, \ldots, \sigma_{N}\right)$ is given by the Boltzmann weight
$$
P(\sigma)=\exp (-\beta H(\sigma)) / Z
$$
where $\beta$ is the inverse temperature and $Z$ is the partition function. The total energy $H(\sigma)$ is given by
$$
H(\sigma)=-J \sum_{i=1}^{N} \sigma_{i} \sigma_{i+1}-h \sum_{i=1}^{N} \sigma_{i}
$$
where $h \in \mathbb{R}$ is the external field and $J>0$ is the coupling constant. We assume periodic boundary conditions, so $\sigma_{N+1} \equiv \sigma_{1}$. The partition function $Z$ can be written as $Z=\lambda_{1}^{N}+\lambda_{2}^{N}$, where
$$
\lambda_{1,2}=e^{\beta J} \cosh (\beta h) \pm \sqrt{e^{2 \beta J} \sinh ^{2}(\beta h)+e^{-2 \beta J}}
$$
are the eigenvalues of the transfer matrix.
(a) [3 points] Compute the free energy per spin $f=\lim {N \rightarrow \infty} F / N$, where $F=-(1 / \beta) \ln Z$ (b) [ 6 points] Show that the average total energy can be written as $\langle H(\sigma)\rangle=$ $\frac{\partial}{\partial \beta}(\beta F)$ (c) [14 points] Show that $$ s(\beta, h, J)=\lim {N \rightarrow \infty} \frac{S[P]}{N}=-\frac{\beta}{\lambda_{1}} \frac{\partial}{\partial \beta} \lambda_{1}+\ln \lambda_{1}
$$
where $S[P]$ is the entropy in natural log units of $P(\sigma)$, and $\lambda_{1}$ is the largest of the two eigenvalues of the transfer matrix. You may use without proof the relation between $F$ and $S[P]$.
(d) [ 7 points] Find the limits of $s(\beta, 0,1)$ for $\beta \rightarrow 0$ and $\beta \rightarrow \infty$. What happens to $P(\sigma)$ in these extreme cases? Discuss whether the system undergoes a phase transition, and if it does, determine at which temperature.