ベルヌイモデルのエントロピーやKLダイバージェンスを考える

$P_{Ber}$の$H_n(x)$や$D_n(x\parallel y)$を考える

$n$回の試行結果である$x^n$の1の発生回数を$m$とすると$P_{Ber}$の$\theta$の最尤推定値$\hat{\theta }$は$\frac{m}{n}$となる． $$\begin{array}{rcl}{\displaystyle \hat{\theta }}& =& {\displaystyle \frac{m}{n}}\\ H_n(P)& \stackrel{\mathrm{def}}{=}& E_P^n[-{\log }_{2}P(X^n)]\\ {\displaystyle H(\theta )}& \stackrel{\mathrm{def}}{=}& {\displaystyle -\theta {\log }_2\left(\theta \right)-(1-\theta ){\log }_2(1-\theta )\dots P=P_{Ber}\mathrm{で}\theta \mathrm{を}\mathrm{引}\mathrm{数}\mathrm{と}\mathrm{し}\mathrm{て}\mathrm{括}\mathrm{弧}\mathrm{の}\mathrm{内}\mathrm{側}\mathrm{に}\mathrm{記}\mathrm{載}\mathrm{す}\mathrm{る}．}\\ {\displaystyle H(\hat{\theta })}& =& {\displaystyle -\hat{\theta }{\log }_2\left(\hat{\theta }\right)-(1-\hat{\theta }){\log }_2(1-\hat{\theta })}\\ & =& {\displaystyle -\frac{m}{n}{\log }_2\left(\frac{m}{n}\right)-(1-\frac{m}{n}){\log }_2(1-\frac{m}{n})}\\ & =& {\displaystyle \frac{1}{n}\{-m{\log }_2\left(\frac{m}{n}\right)-(n-m){\log }_2\left(\frac{n-m}{n}\right)\}}\\ & =& {\displaystyle \frac{1}{n}\left\{{\displaystyle -m{\log }_2\left(m\right){\displaystyle +m{\log }_2\left(n\right){\displaystyle -(n-m){\log }_2(n-m){\displaystyle +(n-m){\log }_2\left(n\right){\displaystyle }}}}}\right\}}\\ {\displaystyle D(\hat{\theta }\parallel \theta )}& =& {\displaystyle \hat{\theta }{\log }_2\left(\frac{\hat{\theta }}{\theta }\right)+(1-\hat{\theta }){\log }_2\left(\frac{1-\hat{\theta }}{1-\theta }\right)}\\ & =& {\displaystyle \frac{m}{n}{\log }_2\left(\frac{\frac{m}{n}}{\theta }\right)+(1-\frac{m}{n}){\log }_2\left(\frac{1-\frac{m}{n}}{1-\theta }\right)}\\ & =& {\displaystyle \frac{1}{n}\left\{{\displaystyle m{\log }_2\left(\frac{\frac{m}{n}}{\theta }\right){\displaystyle +(n-m){\log }_2\left(\frac{1-\frac{m}{n}}{1-\theta }\right){\displaystyle }}}\right\}}\\ & =& {\displaystyle \frac{1}{n}\left\{{\displaystyle m{\log }_2\left(\frac{m}{n}\right){\displaystyle -m{\log }_2\left(\theta \right){\displaystyle +(n-m){\log }_2(1-\frac{m}{n}){\displaystyle {\displaystyle -(n-m){\log }_2(1-\theta ){\displaystyle }}}}}}\right\}}\\ & =& {\displaystyle \frac{1}{n}\left\{{\displaystyle m{\log }_2\left(m\right){\displaystyle -m{\log }_2\left(n\right){\displaystyle -m{\log }_2\left(\theta \right){\displaystyle +(n-m){\log }_2(n-m){\displaystyle -(n-m){\log }_2\left(n\right){\displaystyle -(n-m){\log }_2(1-\theta ){\displaystyle }}}}}}}\right\}}\\ {\displaystyle H(\hat{\theta })+D(\hat{\theta }\parallel \theta )}& =& {\displaystyle \frac{1}{n}\left\{{\displaystyle -m{\log }_2\left(\theta \right)-(n-m){\log }_2(1-\theta ){\displaystyle }}\right\}}\\ {\displaystyle n\{H(\hat{\theta })+D(\hat{\theta }\parallel \theta )\}}& =& {\displaystyle -m{\log }_2\left(\theta \right)-(n-m){\log }_2(1-\theta )}\end{array}$$ $$\begin{array}{rcl}-{\log }_2(L(\theta |x^n))& =& -m{\log }_2\left(\theta \right)-(n-m){\log }_2(1-\theta )\\ n\{H\left(\hat{\theta }\right)+D(\hat{\theta }\parallel \theta )\}& =& -m{\log }_2\left(\theta \right)-(n-m){\log }_2(1-\theta )\\ -{\log }_2\left(L(\theta |x^n)\right)& =& n\{H\left(\hat{\theta }\right)+D(\hat{\theta }\parallel \theta )\}\\ & =& nH\left(\hat{\theta }\right)+nD(\hat{\theta }\parallel \theta )\\ -{\log }_2\left(L(\hat{\theta }|x^n)\right)& =& nH\left(\hat{\theta }\right)+nD(\hat{\theta }\parallel \hat{\theta })\dots \theta =\hat{\theta }\\ & =& nH\left(\hat{\theta }\right)+n0\dots D(\hat{\theta }\parallel \hat{\theta })=0\\ & =& nH\left(\frac{m}{n}\right)\dots \hat{\theta }=\frac{m}{n}\end{array}$$