確率Pで発生しているデータ系列を確率Qに基づく符号化した際のKullback-Leiblerダイバージェンス

例:確率Pで発生しているデータ系列を確率Qに基づく符号化した際のKullback-Leiblerダイバージェンス\(D_n\)

\(y_1\)	\(y_2\)	\(x^2\) \(=y_1y_2\)	\(P(x^2)\)	\(Q(x^2)\)	\(Q'(x^2)\)	\(Q''(x^2)\)
0	0	00	\(\frac{1}{8}\)	\(\frac{1}{8}\)	\(\frac{1}{2}\)	\(\frac{1}{4}\)
0	1	01	\(\frac{1}{8}\)	\(\frac{1}{8}\)	\(\frac{1}{4}\)	\(\frac{1}{4}\)
1	0	10	\(\frac{1}{4}\)	\(\frac{1}{4}\)	\(\frac{1}{8}\)	\(\frac{1}{4}\)
1	1	11	\(\frac{1}{2}\)	\(\frac{1}{2}\)	\(\frac{1}{8}\)	\(\frac{1}{4}\)

\(Q(x^2)\)でのダイバージェンス

$$\begin{array}{rcl} D_n(P\parallel Q)&=&E^{n}_{P}\left[\log_2{\frac{P(X^n)}{Q(X^n)}}\right]\\ &=&\displaystyle \sum_{x^2\in\chi^2} P(x^2)\log_2{\frac{P(x^2)}{Q(x^2)}}\\ &=&\frac{1}{8} \times \log_2{\frac{\frac{1}{8}}{\frac{1}{8}}} +\frac{1}{8} \times \log_2{\frac{\frac{1}{8}}{\frac{1}{8}}} +\frac{1}{4} \times \log_2{\frac{\frac{1}{4}}{\frac{1}{4}}} +\frac{1}{2} \times \log_2{\frac{\frac{1}{2}}{\frac{1}{2}}}\\ &=&\frac{1}{8} \times \log_2{1} +\frac{1}{8} \times \log_2{1} +\frac{1}{4} \times \log_2{1} +\frac{1}{2} \times \log_2{1}\\ &=&\frac{1}{8} \times 0 +\frac{1}{8} \times 0 +\frac{1}{4} \times 0 +\frac{1}{2} \times 0\\ &=&0+0+0+0\\ &=&0 \end{array}$$

\(Q'(x^2)\)でのダイバージェンス

$$\begin{array}{rcl} D_n(P\parallel Q')&=&E^{n}_{P}\left[\log_2{\frac{P(X^n)}{Q'(X^n)}}\right]\\ &=&\displaystyle \sum_{x^2\in\chi^2} P(x^2)\log_2{\frac{P(x^2)}{Q'(x^2)}}\\ &=&\frac{1}{8} \times \log_2{\frac{\frac{1}{8}}{\frac{1}{2}}} +\frac{1}{8} \times \log_2{\frac{\frac{1}{8}}{\frac{1}{4}}} +\frac{1}{4} \times \log_2{\frac{\frac{1}{4}}{\frac{1}{8}}} +\frac{1}{2} \times \log_2{\frac{\frac{1}{2}}{\frac{1}{8}}} \\ &=&\frac{1}{8} \times \log_2{\frac{1}{4}} +\frac{1}{8} \times \log_2{\frac{1}{2}} +\frac{1}{4} \times \log_2{2} +\frac{1}{2} \times \log_2{4} \\ &=&\frac{1}{8} \times -2 +\frac{1}{8} \times -1 +\frac{1}{4} \times 1 +\frac{1}{2} \times 2 \\ &=&-\frac{1}{4}-\frac{1}{8}+\frac{1}{4}+1\\ &=&\frac{7}{8}=0.875 \end{array}$$

\(Q''(x^2)\)でのダイバージェンス

$$\begin{array}{rcl} D_n(P\parallel Q'')&=&E^{n}_{P}\left[\log_2{\frac{P(X^n)}{Q''(X^n)}}\right]\\ &=&\displaystyle \sum_{x^2\in\chi^2} P(x^2)\log_2{\frac{P(x^2)}{Q''(x^2)}}\\ &=&\frac{1}{8} \times \log_2{\frac{\frac{1}{8}}{\frac{1}{4}}} +\frac{1}{8} \times \log_2{\frac{\frac{1}{8}}{\frac{1}{4}}} +\frac{1}{4} \times \log_2{\frac{\frac{1}{4}}{\frac{1}{4}}} +\frac{1}{2} \times \log_2{\frac{\frac{1}{2}}{\frac{1}{4}}} \\ &=&\frac{1}{8} \times \log_2{\frac{1}{2}} +\frac{1}{8} \times \log_2{\frac{1}{2}} +\frac{1}{4} \times \log_2{1} +\frac{1}{2} \times \log_2{2} \\ &=&\frac{1}{8} \times -1 +\frac{1}{8} \times -1 +\frac{1}{4} \times 0 +\frac{1}{2} \times 1 \\ &=&-\frac{1}{8}-\frac{1}{8}+0+\frac{1}{2}\\ &=&\frac{1}{4}=0.25 \end{array}$$