KLダイバージェンスの下限

\(\log_{\mathrm{e}}{x}\leq(x-1)\)の証明

$$\begin{array}{rcl} f(x)&=&x-1-\log_{\mathrm{e}}{x}\\ f'(x)&=&\frac{x-1}{x}\\ f''(x)&=&\frac{1}{x^2}\\ f'=0&は&x=1\\ \end{array}$$ よって\(f''(1)\gt0\)より\(f\)は\(x=1\)で極小であり，また \(x\lt1\) で常に\(f'\lt0\) 及び \(x\gt1\) で常に\(f'\gt0\) なので最小でもある．
\(f(1)=0\)なので\(f\geq0\)である． $$\begin{array}{rcl} x-1-\log_{\mathrm{e}}{x}&\geq&0\\ -\log_{\mathrm{e}}{x}&\geq&-x+1=-(x-1)\\ \log_{\mathrm{e}}{x}&\leq&(x-1)\\ \end{array}$$

\(\log_{\mathrm{e}}{x}\leq(x-1)\)の底の変換等の式変形

$$\begin{array}{rcl} \log_{\mathrm{e}}{x}&\leq&(x-1)\\ &\leq&-(1-x)\\ -\log_{\mathrm{e}}{x}&\geq&(1-x)\\ \log_{\mathrm{e}}{\frac{1}{x}}&\geq&(1-x)\\ \frac{\log_2{\frac{1}{x}}}{\log_2{\mathrm{e}}}&\geq&(1-x)\\ \log_2{\frac{1}{x}}&\geq&(\log_2{\mathrm{e}})(1-x)\\ \end{array}$$

\(D_n(P\parallel Q)\)の下限

$$\begin{array}{rcl} \log_2{\frac{1}{x}}&\geq&(\log_2{\mathrm{e}})(1-x)\\ \log_2{\frac{1}{\frac{Q(x^n)}{P(x^n)}}}&\geq&(\log_2{\mathrm{e}})\left\{1-\frac{Q(x^n)}{P(x^n)}\right\}&x=\frac{Q(x^n)}{P(x^n)}として代入\\ \log_2{\frac{P(x^n)}{Q(x^n)}}&\geq&(\log_2{\mathrm{e}})\left\{1-\frac{Q(x^n)}{P(x^n)}\right\}\\ E^{n}_{P}\left[\log_2{\frac{P(X^n)}{Q(X^n)}}\right]&\geq&E^{n}_{P}\left[(\log_2{\mathrm{e}})\left\{1-\frac{Q(X^n)}{P(X^n)}\right\}\right]&両辺期待値を取る\\ D_n(P\parallel Q)&\geq&E^{n}_{P}\left[(\log_2{\mathrm{e}})\left\{1-\frac{Q(X^n)}{P(X^n)}\right\}\right]\\ &\geq&(\log_2{\mathrm{e}})E^{n}_{P}\left[1-\frac{Q(X^n)}{P(X^n)}\right]&E[cX]=cE[X]\\ &\geq&\displaystyle (\log_2{\mathrm{e}})\left[\left\{1-\frac{Q(x^m)}{P(x^m)}\right\}P(x^m)+\left\{1-\frac{Q(x^n_2)}{P(x^n_2)}\right\}P(x^n_2)+\dotsb\right]\\ &\geq&\displaystyle (\log_2{\mathrm{e}})\left[\left\{P(x^m)-Q(x^m)\right\}+\left\{P(x^n_2)-Q(x^n_2)\right\}+\dotsb\right]\\ &\geq&\displaystyle (\log_2{\mathrm{e}})\left[\left\{P(x^m)+P(x^n_2)+\dotsb\right\}-\left\{Q(x^m)+Q(x^n_2)+\dotsb\right\}\right]\\ &\geq&\displaystyle (\log_2{\mathrm{e}})\left\{\sum_{X^n}P(x^n)-\sum_{X^n}Q(x^n)\right\}\\ &\geq&\displaystyle (\log_2{\mathrm{e}})\left\{1-\sum_{X^n}Q(x^n)\right\}&\displaystyle \sum_{X^n}P(x^n)=1\\ &\geq&0&\displaystyle \sum_{X^n}Q(x^n)=1のときは0\\ \end{array}$$