独立なガウス分布に従う誤差を含む回帰モデルがn個の同時確率を考える

独立なガウス分布$N({\mu }_i,{\sigma }_i^2)$に従う誤差${ϵ}_i$を含む回帰モデル$y_i=\beta x_i+\alpha +{ϵ}_i$が$n$個の同時確率を考える

準備

$$\begin{array}{rcl}\overline{x}& =& \frac{1}{n}\sum _{i=1}^n{x}_i\\ \overline{y}& =& \frac{1}{n}\sum _{i=1}^n{y}_i\\ \sum _{i=1}^n(x_i-\overline{x})(y_i-\overline{y})& =& \sum _{i=1}^n(x_i{y}_i-x_i\overline{y}-\overline{x}{y}_i+\overline{x}\overline{y})\\ & =& \sum _{i=1}^n{x}_i{y}_i-\overline{y}\sum _{i=1}^n{x}_i-\overline{x}\sum _{i=1}^n{y}_i+\overline{x}\overline{y}\sum _{i=1}^{n}1\\ & =& \sum _{i=1}^n{x}_i{y}_i-\overline{y}n\overline{x}-\overline{x}n\overline{y}+\overline{x}\overline{y}n\\ & =& \sum _{i=1}^n{x}_i{y}_i-2n\overline{x}\overline{y}+n\overline{x}\overline{y}\\ & =& \sum _{i=1}^n{x}_i{y}_i-n\overline{x}\overline{y}\cdots S_{xy}\mathrm{の}\mathrm{式}\mathrm{展}\mathrm{開}\\ \sum _{i=1}^n(x_i-\overline{x}{)}^2& =& \sum _{i=1}^n(x_i^2-2x_i\overline{x}+{\overline{x}}^2)\\ & =& \sum _{i=1}^n{x}_i^2-2\overline{x}\sum _{i=1}^n{x}_i+{\overline{x}}^2\sum _{i=1}^{n}1\\ & =& \sum _{i=1}^n{x}_i^2-2\overline{x}n\overline{x}+{\overline{x}}^{2}n\\ & =& \sum _{i=1}^n{x}_i^2-2n{\overline{x}}^2+n{\overline{x}}^2\\ & =& \sum _{i=1}^n{x}_i^2-n{\overline{x}}^2\cdots S_{xx}\mathrm{の}\mathrm{式}\mathrm{展}\mathrm{開}\end{array}$$

同時確率を考える

$$\begin{array}{rcl}p({ϵ}_1,{ϵ}_2,\cdots ,{ϵ}_n)& =& \frac{1}{\sqrt{2\pi {{\sigma }_1}^2}}{e}^{-\frac{{({ϵ}_1-{\mu }_1)}^2}{2{{\sigma }_1}^2}}\times \frac{1}{\sqrt{2\pi {{\sigma }_2}^2}}{e}^{-\frac{{({ϵ}_2-{\mu }_2)}^2}{2{{\sigma }_2}^2}}\times \cdots \times \frac{1}{\sqrt{2\pi {{\sigma }_n}^2}}{e}^{-\frac{{({ϵ}_n-{\mu }_n)}^2}{2{{\sigma }_n}^2}}\\ & =& \left(\prod _{i=1}^n\frac{1}{\sqrt{2\pi {{\sigma }_i}^2}}\right)\left(\prod _{i=1}^n{e}^{-\frac{{({ϵ}_i-{\mu }_i)}^2}{2{{\sigma }_i}^2}}\right)\\ & =& \left(\prod _{i=1}^n{\left(2\pi {{\sigma }_i}^2\right)}^{-\frac{1}{2}}\right)\left(\prod _{i=1}^n{e}^{-\frac{{(y_i-\beta x_i-\alpha -{\mu }_i)}^2}{2{{\sigma }_i}^2}}\right)\cdots y_i=\beta x_i+\alpha +{ϵ}_i,{ϵ}_i=y_i-\beta x_i-\alpha \end{array}$$

負の対数によって情報量を求める

$$\begin{array}{r}\\ -\ln \left(p\right)& =& -\ln \left(\left(\prod _{i=1}^n{\left(2\pi {{\sigma }_i}^2\right)}^{-\frac{1}{2}}\right)\left(\prod _{i=1}^n{e}^{-\frac{1}{2{{\sigma }_i}^2}{(y_i-\beta x_i-\alpha -{\mu }_i)}^2}\right)\right)\\ & =& -\ln \left(\prod _{i=1}^n{\left(2\pi {{\sigma }_i}^2\right)}^{-\frac{1}{2}}\right)-\ln \left(\prod _{i=1}^n{e}^{-\frac{1}{2{{\sigma }_i}^2}{(y_i-\beta x_i-\alpha -{\mu }_i)}^2}\right)\\ & =& -\sum _{i=1}^n\ln \left({\left(2\pi {{\sigma }_i}^2\right)}^{-\frac{1}{2}}\right)-\sum _{i=1}^n\ln \left(e^{-\frac{1}{2{{\sigma }_i}^2}{(y_i-\beta x_i-\alpha -{\mu }_i)}^2}\right)\\ & =& -\sum _{i=1}^n\{-\frac{1}{2}\ln \left(2\pi {{\sigma }_i}^2\right)\}-\sum _{i=1}^n(-\frac{1}{2{{\sigma }_i}^2}{(y_i-\beta x_i-\alpha -{\mu }_i)}^2)\\ & =& \sum _{i=1}^n\{\frac{1}{2}\ln \left(2\pi {{\sigma }_i}^2\right)\}+\sum _{i=1}^n\frac{1}{2{{\sigma }_i}^2}{(y_i-\beta x_i-\alpha -{\mu }_i)}^2\\ & =& \sum _{i=1}^n\{\frac{1}{2}\ln \left(2\pi {{\sigma }_i}^2\right)\}+\sum _{i=1}^n\frac{1}{2{{\sigma }_i}^2}(y_i^2+{\beta }^2{x}_i^2+{\alpha }^2+{\mu }_i^2-2\beta x_i{y}_i-2\alpha y_i-2y_i{\mu }_i+2\alpha \beta x_i+2\beta x_i{\mu }_i+2\alpha {\mu }_i)\end{array}$$

回帰モデルの係数で微分する

$\beta$について． $$\begin{array}{rcl}\frac{\partial }{\partial \beta }(-\ln \left(p\right))& =& 0+\sum _{i=1}^n\frac{1}{\cancel{2}{{\sigma }_i}^2}(0+\cancel{2}\beta x_i^2+0+0-\cancel{2}{x}_i{y}_i-0-0+\cancel{2}\alpha x_i+\cancel{2}{x}_i{\mu }_i+0)\\ & =& \sum _{i=1}^n\frac{1}{{{\sigma }_i}^2}(\beta x_i^2-x_i{y}_i+\alpha x_i+x_i{\mu }_i)\\ & =& \beta \sum _{i=1}^n\frac{x_i^2}{{{\sigma }_i}^2}-\sum _{i=1}^n\frac{x_i(y_i-\alpha -{\mu }_i)}{{{\sigma }_i}^2}\end{array}$$ $\alpha$について． $$\begin{array}{rcl}\frac{\partial }{\partial \alpha }(-\ln \left(p\right))& =& 0+\sum _{i=1}^n\frac{1}{\cancel{2}{{\sigma }_i}^2}(0+0+\cancel{2}\alpha +0-0-\cancel{2}{y}_i-0+\cancel{2}\beta x_i+0+\cancel{2}{\mu }_i)\\ & =& \sum _{i=1}^n\frac{1}{{{\sigma }_i}^2}(\alpha -y_i+\beta x_i+{\mu }_i)\\ & =& \alpha \sum _{i=1}^n\frac{1}{{{\sigma }_i}^2}-\sum _{i=1}^n\frac{y_i-\beta x_i-{\mu }_i}{{{\sigma }_i}^2}\end{array}$$

微分が0となる$\hat{\alpha },\hat{\beta }$,つまり極値となるパラメータを求める

$\hat{\beta }$について． $$\begin{array}{r}\\ 0& =& \frac{\partial }{\partial \beta }(-\ln \left(p\right))\\ & =& \hat{\beta }\sum _{i=1}^n\frac{x_i^2}{{{\sigma }_i}^2}-\sum _{i=1}^n\frac{x_i(y_i-\hat{\alpha }-{\mu }_i)}{{{\sigma }_i}^2}\\ \hat{\beta }\sum _{i=1}^n\frac{x_i^2}{{{\sigma }_i}^2}& =& \sum _{i=1}^n\frac{x_i(y_i-\hat{\alpha }-{\mu }_i)}{{{\sigma }_i}^2}\\ \hat{\beta }& =& \frac{\sum _{i=1}^n\frac{x_i(y_i-\hat{\alpha }-{\mu }_i)}{{{\sigma }_i}^2}}{\sum _{i=1}^n\frac{x_i^2}{{{\sigma }_i}^2}}\end{array}$$ $\hat{\alpha }$について． $$\begin{array}{r}\\ 0& =& \frac{\partial }{\partial \alpha }(-\ln \left(p\right))\\ & =& \hat{\alpha }\sum _{i=1}^n\frac{1}{{{\sigma }_i}^2}-\sum _{i=1}^n\frac{y_i-\hat{\beta }{x}_i-{\mu }_i}{{{\sigma }_i}^2}\\ \hat{\alpha }\sum _{i=1}^n\frac{1}{{{\sigma }_i}^2}& =& \sum _{i=1}^n\frac{y_i-\hat{\beta }{x}_i-{\mu }_i}{{{\sigma }_i}^2}\\ \hat{\alpha }& =& \frac{\sum _{i=1}^n\frac{y_i-\hat{\beta }{x}_i-{\mu }_i}{{{\sigma }_i}^2}}{\sum _{i=1}^n\frac{1}{{{\sigma }_i}^2}}\end{array}$$

誤差の発生が全て同一モデル${ϵ}_i\sim N(0,{\sigma }^2)$から発生しているとする

$\hat{\alpha }$について． $$\begin{array}{rcl}\hat{\alpha }& =& \frac{\frac{1}{{\sigma }^2}\sum _{i=1}^n(y_i-\hat{\beta }{x}_i-0)}{\frac{n}{{\sigma }^2}}\cdots {ϵ}_i\sim N(0,{\sigma }^2)\\ & =& \frac{1}{n}\sum _{i=1}^n(y_i-\hat{\beta }{x}_i)\\ & =& \frac{\sum _{i=1}^n{y}_i}{n}-\hat{\beta }\frac{\sum _{i=1}^n{x}_i}{n}\\ & =& \overline{y}-\hat{\beta }\overline{x}\end{array}$$ $\hat{\beta }$について． $$\begin{array}{r}\\ \hat{\beta }& =& \frac{\frac{1}{{\sigma }^2}\sum _{i=1}^n{x}_i(y_i-\hat{\alpha }-0)}{\frac{1}{{\sigma }^2}\sum _{i=1}^n{x}_i^2}\cdots {ϵ}_i\sim N(0,{\sigma }^2)\\ & =& \frac{\sum _{i=1}^n{x}_i{y}_i-\hat{\alpha }\sum _{i=1}^n{x}_i}{\sum _{i=1}^n{x}_i^2}\\ & =& \frac{\sum _{i=1}^n{x}_i{y}_i-\hat{\alpha }n\overline{x}}{\sum _{i=1}^n{x}_i^2}\\ & =& \frac{\sum _{i=1}^n{x}_i{y}_i-(\overline{y}-\hat{\beta }\overline{x})n\overline{x}}{\sum _{i=1}^n{x}_i^2}\cdots \hat{\alpha }=\overline{y}-\hat{\beta }\overline{x},\hat{\alpha }\mathrm{で}\mathrm{の}\mathrm{結}\mathrm{果}\mathrm{を}\mathrm{代}\mathrm{入}\mathrm{す}\mathrm{る}\\ & =& \frac{\sum _{i=1}^n{x}_i{y}_i-n\overline{x}\overline{y}+\hat{\beta }n{\overline{x}}^2}{\sum _{i=1}^n{x}_i^2}\\ \hat{\beta }-\frac{\hat{\beta }n{\overline{x}}^2}{\sum _{i=1}^n{x}_i^2}& =& \frac{\sum _{i=1}^n{x}_i{y}_i-n\overline{x}\overline{y}}{\sum _{i=1}^n{x}_i^2}\\ \hat{\beta }\left(\frac{\sum _{i=1}^n{x}_i^2-n{\overline{x}}^2}{\cancel{\sum _{i=1}^n{x}_i^2}}\right)& =& \frac{\sum _{i=1}^n{x}_i{y}_i-n\overline{x}\overline{y}}{\cancel{\sum _{i=1}^n{x}_i^2}}\\ \hat{\beta }& =& \frac{\sum _{i=1}^n{x}_i{y}_i-n\overline{x}\overline{y}}{\sum _{i=1}^n{x}_i^2-n{\overline{x}}^2}\\ & =& \frac{\sum _{i=1}^n(x_i-\overline{x})(y_i-\overline{y})}{\sum _{i=1}^n(x_i-\overline{x}{)}^2}\cdots \mathrm{準}\mathrm{備}\mathrm{よ}\mathrm{り}\end{array}$$

まとめ

$$\{\begin{array}{l}\begin{array}{rcl}\hat{\beta }& =& \frac{\sum _{i=1}^n{x}_i{y}_i-n\overline{x}\overline{y}}{\sum _{i=1}^n{x}_i^2-n{\overline{x}}^2}=\frac{\sum _{i=1}^n(x_i-\overline{x})(y_i-\overline{y})}{\sum _{i=1}^n(x_i-\overline{x}{)}^2}\\ \hat{\alpha }& =& \overline{y}-\hat{\beta }\overline{x}\end{array}\end{array}$$