ベルヌイモデルのエントロピーやKLダイバージェンスを考える

\(P_{Ber}\)の\(H_n(x)\)や\(D_n(x\parallel y)\)を考える

\(n\)回の試行結果である\(x^n\)の1の発生回数を\(m\)とすると\(P_{Ber}\)の\(\theta\)の最尤推定値\(\hat{\theta}\)は\(\frac{m}{n}\)となる． $$\begin{array}{rcl} \displaystyle \hat{\theta} &=&\displaystyle \frac{m}{n}\\ H_n(P)&\overset{\mathrm{def}}{=}& E^{n}_{P}\left[-\log_2{P(X^n)}\right] \\ \displaystyle H(\theta) &\overset{\mathrm{def}}{=}& \displaystyle -\theta\log_{2}{\left(\theta\right)}-\left(1-\theta\right)\log_{2}{\left(1-\theta\right)} \quad\dotso P=P_{Ber}で\thetaを引数として括弧の内側に記載する． \\ \displaystyle H(\hat{\theta}) &=&\displaystyle -\hat{\theta}\log_{2}{\left(\hat{\theta}\right)}-\left(1-\hat{\theta}\right)\log_{2}{\left(1-\hat{\theta}\right)}\\ &=&\displaystyle -\frac{m}{n}\log_{2}{\left(\frac{m}{n}\right)}-\left(1-\frac{m}{n}\right)\log_{2}{\left(1-\frac{m}{n}\right)}\\ &=&\displaystyle \frac{1}{n}\left\{-m\log_{2}{\left(\frac{m}{n}\right)}-\left(n-m\right)\log_{2}{\left(\frac{n-m}{n}\right)}\right\}\\ &=&\displaystyle \frac{1}{n}\left\{ \displaystyle -m\log_{2}{ \left(m\right) } \displaystyle +m\log_{2}{ \left(n\right) } \displaystyle -\left(n-m\right)\log_{2}{ \left(n-m\right) } \displaystyle +\left(n-m\right)\log_{2}{ \left(n\right) } \displaystyle \right\}\\ \displaystyle D(\hat{\theta}\parallel \theta) &=&\displaystyle \hat{\theta}\log_{2}{\left(\frac{\hat{\theta}}{\theta}\right)}+\left(1-\hat{\theta}\right)\log_{2}{\left(\frac{1-\hat{\theta}}{1-\theta}\right)}\\ &=&\displaystyle \frac{m}{n}\log_{2}{\left(\frac{\frac{m}{n}}{\theta}\right)}+\left(1-\frac{m}{n}\right)\log_{2}{\left(\frac{1-\frac{m}{n}}{1-\theta}\right)}\\ &=&\displaystyle \frac{1}{n}\left\{ \displaystyle m\log_{2}{\left(\frac{\frac{m}{n}}{\theta}\right)} \displaystyle +\left(n-m\right)\log_{2}{\left(\frac{1-\frac{m}{n}}{1-\theta}\right)} \displaystyle \right\}\\ &=&\displaystyle \frac{1}{n}\left\{ \displaystyle m\log_{2}{\left(\frac{m}{n}\right)} \displaystyle -m\log_{2}{\left(\theta\right)} \displaystyle +\left(n-m\right)\log_{2}{\left(1-\frac{m}{n}\right)} \displaystyle \displaystyle -\left(n-m\right)\log_{2}{\left(1-\theta\right)} \displaystyle \right\}\\ &=&\displaystyle \frac{1}{n}\left\{ \displaystyle m\log_{2}{\left(m\right)} \displaystyle -m\log_{2}{\left(n\right)} \displaystyle -m\log_{2}{\left(\theta\right)} \displaystyle +\left(n-m\right)\log_{2}{\left(n-m\right)} \displaystyle -\left(n-m\right)\log_{2}{\left(n\right)} \displaystyle -\left(n-m\right)\log_{2}{\left(1-\theta\right)} \displaystyle \right\}\\ \displaystyle H(\hat{\theta})+D(\hat{\theta}\parallel \theta) &=&\displaystyle \frac{1}{n}\left\{ \displaystyle -m\log_{2}{\left(\theta\right)}-\left(n-m\right)\log_{2}{\left(1-\theta\right)} \displaystyle \right\}\\ \displaystyle n\left\{H(\hat{\theta})+D(\hat{\theta}\parallel \theta)\right\} &=&\displaystyle -m\log_{2}{\left(\theta\right)}-\left(n-m\right)\log_2{\left(1-\theta\right)}\\ \end{array}$$ $$\begin{array}{rcl} -\log_2{\left( L(\theta|x^n) \right)} &=&-m\log_2{\left( \theta \right)}-(n-m)\log_2{\left(1-\theta\right)}\\ n\left\{H\left( \hat{\theta} \right)+D(\hat{\theta}\parallel \theta)\right\} &=&-m\log_2{\left( \theta \right)}-\left( n-m \right)\log_2{\left( 1-\theta \right)}\\ -\log_2{\left( L\left(\theta|x^n\right) \right)} &=&n\left\{H\left( \hat{\theta} \right)+D\left( \hat{\theta}\parallel \theta \right)\right\}\\ &=&nH\left( \hat{\theta} \right)+nD\left( \hat{\theta}\parallel \theta \right)\\ -\log_2{\left( L\left(\hat{\theta}|x^n\right) \right)} &=&nH\left( \hat{\theta} \right)+nD\left( \hat{\theta}\parallel \hat{\theta} \right)\quad\dotso\theta=\hat{\theta}\\ &=&nH\left( \hat{\theta} \right)+n0\quad\dotso D\left( \hat{\theta}\parallel \hat{\theta} \right)=0\\ &=&nH\left( \frac{m}{n} \right)\quad\dotso \hat{\theta}=\frac{m}{n}\\ \end{array}$$

ベルヌイモデルとベルヌイモデルの尤度に対する情報量

ベルヌイモデル \(P_{Ber}\)

$$\begin{array}{rcl} P(X|\theta) = \begin{cases} \displaystyle \theta & \,: \left(X=1\right)\\ \displaystyle 1-\theta & \,: \left(X=0\right)\\ \end{cases}\\ \displaystyle P_{Ber}=\left\{P(X|\theta)\,:(0\leq\theta\leq1)\right\}\\ \end{array}$$

\(x^n\)を\(P_{Ber}\)で最短長の符号化をすることを考える

データ列\(x^n\)が与えられた時の生成確率\(P(x^n|\theta)\)を\(\theta\)の凾数とみなすと $$\begin{array}{rcl} \displaystyle L(\theta|x^n)&=&P(x^n|\theta) \end{array}$$ としてこれを尤度(likelihood)と呼ぶ．

ベルヌイモデルの尤度に対する情報量

\(x^n\)の1の発生回数を\(m\)とすると0の発生回数は\(n-m\)となるので最尤推定値(maximum likelihood estimator)は $$\begin{array}{rcl} \displaystyle L(\theta|x^n)&=&\theta^{m}(1-\theta)^{(n-m)}\\ \end{array}$$ 情報量(=符号長)を考えると $$\begin{array}{rcl} \displaystyle -\log_{2}{\left(L(\theta|x^n)\right)}&=&-\log_{2}{\left(\theta^{m}(1-\theta)^{(n-m)}\right)}\\ &=&\displaystyle -\log_{2}{\left(\theta^{m}\right)}-\log_{2}{\left((1-\theta)^{(n-m)}\right)}\\ &=&\displaystyle -m\log_{2}{\left(\theta\right)}-(n-m)\log_{2}{\left(1-\theta\right)}\\ \end{array}$$

確率Pで発生しているデータ系列を確率Qに基づく符号化した際のKullback-Leiblerダイバージェンス

例:確率Pで発生しているデータ系列を確率Qに基づく符号化した際のKullback-Leiblerダイバージェンス\(D_n\)

\(y_1\)	\(y_2\)	\(x^2\) \(=y_1y_2\)	\(P(x^2)\)	\(Q(x^2)\)	\(Q'(x^2)\)	\(Q''(x^2)\)
0	0	00	\(\frac{1}{8}\)	\(\frac{1}{8}\)	\(\frac{1}{2}\)	\(\frac{1}{4}\)
0	1	01	\(\frac{1}{8}\)	\(\frac{1}{8}\)	\(\frac{1}{4}\)	\(\frac{1}{4}\)
1	0	10	\(\frac{1}{4}\)	\(\frac{1}{4}\)	\(\frac{1}{8}\)	\(\frac{1}{4}\)
1	1	11	\(\frac{1}{2}\)	\(\frac{1}{2}\)	\(\frac{1}{8}\)	\(\frac{1}{4}\)

\(Q(x^2)\)でのダイバージェンス

$$\begin{array}{rcl} D_n(P\parallel Q)&=&E^{n}_{P}\left[\log_2{\frac{P(X^n)}{Q(X^n)}}\right]\\ &=&\displaystyle \sum_{x^2\in\chi^2} P(x^2)\log_2{\frac{P(x^2)}{Q(x^2)}}\\ &=&\frac{1}{8} \times \log_2{\frac{\frac{1}{8}}{\frac{1}{8}}} +\frac{1}{8} \times \log_2{\frac{\frac{1}{8}}{\frac{1}{8}}} +\frac{1}{4} \times \log_2{\frac{\frac{1}{4}}{\frac{1}{4}}} +\frac{1}{2} \times \log_2{\frac{\frac{1}{2}}{\frac{1}{2}}}\\ &=&\frac{1}{8} \times \log_2{1} +\frac{1}{8} \times \log_2{1} +\frac{1}{4} \times \log_2{1} +\frac{1}{2} \times \log_2{1}\\ &=&\frac{1}{8} \times 0 +\frac{1}{8} \times 0 +\frac{1}{4} \times 0 +\frac{1}{2} \times 0\\ &=&0+0+0+0\\ &=&0 \end{array}$$

\(Q'(x^2)\)でのダイバージェンス

$$\begin{array}{rcl} D_n(P\parallel Q')&=&E^{n}_{P}\left[\log_2{\frac{P(X^n)}{Q'(X^n)}}\right]\\ &=&\displaystyle \sum_{x^2\in\chi^2} P(x^2)\log_2{\frac{P(x^2)}{Q'(x^2)}}\\ &=&\frac{1}{8} \times \log_2{\frac{\frac{1}{8}}{\frac{1}{2}}} +\frac{1}{8} \times \log_2{\frac{\frac{1}{8}}{\frac{1}{4}}} +\frac{1}{4} \times \log_2{\frac{\frac{1}{4}}{\frac{1}{8}}} +\frac{1}{2} \times \log_2{\frac{\frac{1}{2}}{\frac{1}{8}}} \\ &=&\frac{1}{8} \times \log_2{\frac{1}{4}} +\frac{1}{8} \times \log_2{\frac{1}{2}} +\frac{1}{4} \times \log_2{2} +\frac{1}{2} \times \log_2{4} \\ &=&\frac{1}{8} \times -2 +\frac{1}{8} \times -1 +\frac{1}{4} \times 1 +\frac{1}{2} \times 2 \\ &=&-\frac{1}{4}-\frac{1}{8}+\frac{1}{4}+1\\ &=&\frac{7}{8}=0.875 \end{array}$$

\(Q''(x^2)\)でのダイバージェンス

$$\begin{array}{rcl} D_n(P\parallel Q'')&=&E^{n}_{P}\left[\log_2{\frac{P(X^n)}{Q''(X^n)}}\right]\\ &=&\displaystyle \sum_{x^2\in\chi^2} P(x^2)\log_2{\frac{P(x^2)}{Q''(x^2)}}\\ &=&\frac{1}{8} \times \log_2{\frac{\frac{1}{8}}{\frac{1}{4}}} +\frac{1}{8} \times \log_2{\frac{\frac{1}{8}}{\frac{1}{4}}} +\frac{1}{4} \times \log_2{\frac{\frac{1}{4}}{\frac{1}{4}}} +\frac{1}{2} \times \log_2{\frac{\frac{1}{2}}{\frac{1}{4}}} \\ &=&\frac{1}{8} \times \log_2{\frac{1}{2}} +\frac{1}{8} \times \log_2{\frac{1}{2}} +\frac{1}{4} \times \log_2{1} +\frac{1}{2} \times \log_2{2} \\ &=&\frac{1}{8} \times -1 +\frac{1}{8} \times -1 +\frac{1}{4} \times 0 +\frac{1}{2} \times 1 \\ &=&-\frac{1}{8}-\frac{1}{8}+0+\frac{1}{2}\\ &=&\frac{1}{4}=0.25 \end{array}$$