確率Pで発生しているデータ系列を確率Qに基づく符号化した際のKullback-Leiblerダイバージェンス

例:確率Pで発生しているデータ系列を確率Qに基づく符号化した際のKullback-Leiblerダイバージェンス$D_n$

$y_1$	$y_2$	$x^2$ $=y_1{y}_2$	$P(x^2)$	$Q(x^2)$	$Q^{\prime }(x^2)$	$Q^{″}(x^2)$
0	0	00	$\frac{1}{8}$	$\frac{1}{8}$	$\frac{1}{2}$	$\frac{1}{4}$
0	1	01	$\frac{1}{8}$	$\frac{1}{8}$	$\frac{1}{4}$	$\frac{1}{4}$
1	0	10	$\frac{1}{4}$	$\frac{1}{4}$	$\frac{1}{8}$	$\frac{1}{4}$
1	1	11	$\frac{1}{2}$	$\frac{1}{2}$	$\frac{1}{8}$	$\frac{1}{4}$

$Q(x^2)$でのダイバージェンス

$$\begin{array}{rcl}{D}_n(P\parallel Q)& =& E_P^n[{\log }_2\frac{P(X^n)}{Q(X^n)}]\\ & =& {\displaystyle \sum _{x^2\in {\chi }^2}P(x^2){\log }_2\frac{P(x^2)}{Q(x^2)}}\\ & =& \frac{1}{8}\times {\log }_2\frac{\frac{1}{8}}{\frac{1}{8}}+\frac{1}{8}\times {\log }_2\frac{\frac{1}{8}}{\frac{1}{8}}+\frac{1}{4}\times {\log }_2\frac{\frac{1}{4}}{\frac{1}{4}}+\frac{1}{2}\times {\log }_2\frac{\frac{1}{2}}{\frac{1}{2}}\\ & =& \frac{1}{8}\times {\log }_{2}1+\frac{1}{8}\times {\log }_{2}1+\frac{1}{4}\times {\log }_{2}1+\frac{1}{2}\times {\log }_{2}1\\ & =& \frac{1}{8}\times 0+\frac{1}{8}\times 0+\frac{1}{4}\times 0+\frac{1}{2}\times 0\\ & =& 0+0+0+0\\ & =& 0\end{array}$$

$Q^{\prime }(x^2)$でのダイバージェンス

$$\begin{array}{rcl}{D}_n(P\parallel Q^{\prime })& =& E_P^n[{\log }_2\frac{P(X^n)}{Q^{\prime }(X^n)}]\\ & =& {\displaystyle \sum _{x^2\in {\chi }^2}P(x^2){\log }_2\frac{P(x^2)}{Q^{\prime }(x^2)}}\\ & =& \frac{1}{8}\times {\log }_2\frac{\frac{1}{8}}{\frac{1}{2}}+\frac{1}{8}\times {\log }_2\frac{\frac{1}{8}}{\frac{1}{4}}+\frac{1}{4}\times {\log }_2\frac{\frac{1}{4}}{\frac{1}{8}}+\frac{1}{2}\times {\log }_2\frac{\frac{1}{2}}{\frac{1}{8}}\\ & =& \frac{1}{8}\times {\log }_2\frac{1}{4}+\frac{1}{8}\times {\log }_2\frac{1}{2}+\frac{1}{4}\times {\log }_{2}2+\frac{1}{2}\times {\log }_{2}4\\ & =& \frac{1}{8}\times -2+\frac{1}{8}\times -1+\frac{1}{4}\times 1+\frac{1}{2}\times 2\\ & =& -\frac{1}{4}-\frac{1}{8}+\frac{1}{4}+1\\ & =& \frac{7}{8}=0.875\end{array}$$

$Q^{″}(x^2)$でのダイバージェンス

$$\begin{array}{rcl}{D}_n(P\parallel Q^{″})& =& E_P^n[{\log }_2\frac{P(X^n)}{Q^{″}(X^n)}]\\ & =& {\displaystyle \sum _{x^2\in {\chi }^2}P(x^2){\log }_2\frac{P(x^2)}{Q^{″}(x^2)}}\\ & =& \frac{1}{8}\times {\log }_2\frac{\frac{1}{8}}{\frac{1}{4}}+\frac{1}{8}\times {\log }_2\frac{\frac{1}{8}}{\frac{1}{4}}+\frac{1}{4}\times {\log }_2\frac{\frac{1}{4}}{\frac{1}{4}}+\frac{1}{2}\times {\log }_2\frac{\frac{1}{2}}{\frac{1}{4}}\\ & =& \frac{1}{8}\times {\log }_2\frac{1}{2}+\frac{1}{8}\times {\log }_2\frac{1}{2}+\frac{1}{4}\times {\log }_{2}1+\frac{1}{2}\times {\log }_{2}2\\ & =& \frac{1}{8}\times -1+\frac{1}{8}\times -1+\frac{1}{4}\times 0+\frac{1}{2}\times 1\\ & =& -\frac{1}{8}-\frac{1}{8}+0+\frac{1}{2}\\ & =& \frac{1}{4}=0.25\end{array}$$