単回帰モデルの最小二乗推定量の分布

単回帰モデルの最小二乗推定量$\hat{\alpha },\hat{\beta }$の分布

単回帰モデル

$$\begin{array}{rcl}{y}_i& =& \alpha +\beta x_i+{ϵ}_i(i=1,\cdots ,n)\dots {ϵ}_i\stackrel{iid}{\sim }\mathrm{N}(0,{\sigma }^2)\\ \mathrm{E}\left[y_i\right]& =& \mathrm{E}[\alpha +\beta x_i+{ϵ}_i]\\ & =& \alpha +\beta x_i+\mathrm{E}\left[{ϵ}_i\right]\cdots \mathrm{E}[X+t]=\mathrm{E}\left[X\right]+t\\ & =& \alpha +\beta x_i+0\cdots {ϵ}_i\stackrel{iid}{\sim }\mathrm{N}(0,{\sigma }^2)\\ & =& \alpha +\beta x_i\\ \mathrm{V}\left[y_i\right]& =& \mathrm{V}[\alpha +\beta x_i+{ϵ}_i]\\ & =& \mathrm{V}\left[{ϵ}_i\right]\cdots \mathrm{V}[X+t]=\mathrm{V}\left[X\right]\\ & =& {\sigma }^2\cdots {ϵ}_i\stackrel{iid}{\sim }\mathrm{N}(0,{\sigma }^2)\\ y_i& \sim & \mathrm{N}(\alpha +\beta x_i,{\sigma }^2)\end{array}$$ $y_i$は$\mathrm{N}(\alpha +\beta x_i,{\sigma }^2)$に従う確率変数である．

$\hat{\beta }$を$\sum _{i=1}^n{c}_i{y}_i$の形で表す

推定量が$\sum _{i=1}^n{c}_i{x}_i(x_i:\mathrm{標}\mathrm{本},c_i:\mathrm{定}\mathrm{数})$の形で表現できるとき，この推定量を線形推定量(linear estimate)という．
(よく知られる線形推定量の例として平均$\overline{x}$があり，$\overline{x}=\sum _{i=1}^n\frac{1}{n}{x}_i$で表現される) $$\begin{array}{rcl}\hat{\beta }& =& \frac{S_{xy}}{S_{xx}}\cdots \hat{\beta }=\frac{S_{xy}}{S_{xx}},S_{xx}=\sum _{i=1}^n{(x_i-\overline{x})}^2,\overline{x}=\frac{1}{n}\sum _{i=1}^n{x}_i\\ & =& \frac{1}{S_{xx}}\sum _{i=1}^n(x_i-\overline{x})(y_i-\overline{y})\\ & =& \frac{1}{S_{xx}}\sum _{i=1}^n(x_i-\overline{x})y_i\cdots \sum _{i=1}^n(x_i-\overline{x})(y_i-\overline{y})=S_{xy}=\sum _{i=1}^n(x_i-\overline{x})y_i\\ & =& \sum _{i=1}^n\frac{x_i-\overline{x}}{S_{xx}}{y}_i\\ & =& \sum _{i=1}^n{c}_i{y}_i\cdots c_i=\frac{x_i-\overline{x}}{S_{xx}}\end{array}$$

$\hat{\beta }$の期待値を$\sum _{i=1}^n{c}_i{y}_i$から求めてみる

$$\begin{array}{rcl}\mathrm{E}\left[\sum _{i=1}^n{c}_i{y}_i\right]& =& \mathrm{E}\left[\sum _{i=1}^n\frac{x_i-\overline{x}}{S_{xx}}{y}_i\right]\\ & =& \mathrm{E}\left[\frac{S_{xy}}{S_{xx}}\right]\cdots \mathrm{上}\mathrm{記},\frac{S_{xy}}{S_{xx}}=\sum _{i=1}^n\frac{x_i-\overline{x}}{S_{xx}}{y}_i\\ & =& \mathrm{E}\left[\hat{\beta }\right]\cdots \hat{\beta }=\frac{S_{xy}}{S_{xx}}\\ & =& \beta \cdots \mathrm{E}\left[\hat{\beta }\right]=\beta \end{array}$$

$\hat{\beta }$の分散を$\sum _{i=1}^n{c}_i{y}_i$から求めてみる

$$\begin{array}{rcl}\mathrm{V}\left[\sum _{i=1}^n{c}_i{y}_i\right]& =& \mathrm{V}\left[\sum _{i=1}^n\frac{x_i-\overline{x}}{S_{xx}}{y}_i\right]\\ & =& \sum _{i=1}^n\mathrm{V}\left[\frac{x_i-\overline{x}}{S_{xx}}{y}_i\right]\cdots y_i\mathrm{は}\mathrm{互}\mathrm{い}\mathrm{に}\mathrm{独}\mathrm{立}\mathrm{Cov}[y_i,y_j]=0,\mathrm{互}\mathrm{い}\mathrm{に}\mathrm{独}\mathrm{立}\mathrm{の}\mathrm{場}\mathrm{合}\mathrm{V}[X+Y]=\mathrm{V}\left[X\right]+\mathrm{V}\left[Y\right]\\ & =& \sum _{i=1}^n{\left(\frac{x_i-\overline{x}}{S_{xx}}\right)}^2\mathrm{V}\left[y_i\right]\cdots \mathrm{V}\left[cX\right]=c^2\mathrm{V}\left[X\right]\\ & =& \sum _{i=1}^n{\left(\frac{x_i-\overline{x}}{S_{xx}}\right)}^2{\sigma }^2\\ & =& \frac{{\sigma }^2}{S_{xx}^2}\sum _{i=1}^n{(x_i-\overline{x})}^2\\ & =& \frac{{\sigma }^2}{S_{xx}^2}{S}_{xx}\cdots S_{xx}=\sum _{i=1}^n{(x_i-\overline{x})}^2\\ & =& \frac{{\sigma }^2}{S_{xx}}\cdots \mathrm{V}\left[\frac{S_{xy}}{S_{xx}}\right]=\frac{{\sigma }^2}{S_{xx}}\mathrm{と}\mathrm{同}\mathrm{じ}\mathrm{結}\mathrm{果}\end{array}$$

$\hat{\beta }$の分布

以上のように，$\hat{\beta }$は線形推定量であり，正規分布に従う$y_i$の定数倍の和で表すことができた．よって$\hat{\beta }$は同様に正規分布に従い，その期待値と分散はそれぞれ上記で求めたとおりである $\cdots Z=c_{1}X+c_{2}Y(X\sim \mathrm{N}({\mu }_1,{\sigma }_1^2),Y\sim \mathrm{N}({\mu }_2,{\sigma }_2^2),Z\sim \mathrm{N}(c_1{\mu }_1+c_2{\mu }_2,c_1^2{\sigma }_1^2+c_2^2{\sigma }_2^2))$． $$\begin{array}{rcl}\hat{\beta }& \sim & \mathrm{N}(\beta ,\frac{{\sigma }^2}{S_{xx}})\end{array}$$

$\hat{\alpha }$を$\sum _{i=1}^n{c}_i{y}_i$の形で表す

$$\begin{array}{rcl}\hat{\alpha }& =& \overline{y}-\hat{\beta }\overline{x}\cdots \hat{\alpha }=\overline{y}-\hat{\beta }\overline{x}\\ & =& \sum _{i=1}^n\frac{1}{n}{y}_i-\frac{S_{xy}}{S_{xx}}\overline{x}\\ & =& \sum _{i=1}^n\frac{1}{n}{y}_i-\left(\sum _{i=1}^n\frac{x_i-\overline{x}}{S_{xx}}{y}_i\right)\overline{x}\cdots \mathrm{上}\mathrm{記},\frac{S_{xy}}{S_{xx}}=\sum _{i=1}^n\frac{x_i-\overline{x}}{S_{xx}}{y}_i\\ & =& \sum _{i=1}^n\frac{1}{n}{y}_i-\overline{x}\sum _{i=1}^n\frac{x_i-\overline{x}}{S_{xx}}{y}_i\\ & =& \sum _{i=1}^n\frac{1}{n}{y}_i-\sum _{i=1}^n\frac{\overline{x}(x_i-\overline{x})}{S_{xx}}{y}_i\\ & =& \sum _{i=1}^n(\frac{1}{n}-\frac{\overline{x}(x_i-\overline{x})}{S_{xx}})y_i\\ & =& \sum _{i=1}^n{c}_i{y}_i\cdots c_i=\frac{1}{n}-\frac{\overline{x}(x_i-\overline{x})}{S_{xx}}\end{array}$$

$\hat{\alpha }$の期待値を$\sum _{i=1}^n{c}_i{y}_i$から求めてみる

$$\begin{array}{rcl}\mathrm{E}\left[\sum _{i=1}^n{c}_i{y}_i\right]& =& \mathrm{E}\left[\sum _{i=1}^n(\frac{1}{n}-\frac{\overline{x}(x_i-\overline{x})}{S_{xx}})y_i\right]\\ & =& \mathrm{E}\left[\sum _{i=1}^n(\frac{1}{n}{y}_i-\frac{\overline{x}(x_i-\overline{x})}{S_{xx}}{y}_i)\right]\\ & =& \mathrm{E}[\sum _{i=1}^n\frac{1}{n}{y}_i-\sum _{i=1}^n\frac{\overline{x}(x_i-\overline{x})}{S_{xx}}{y}_i]\\ & =& \mathrm{E}[\sum _{i=1}^n\frac{1}{n}{y}_i-\sum _{i=1}^n\frac{\overline{x}(x_i-\overline{x})}{S_{xx}}{y}_i]\\ & =& \mathrm{E}\left[\sum _{i=1}^n\frac{1}{n}{y}_i\right]-\mathrm{E}\left[\sum _{i=1}^n\frac{\overline{x}(x_i-\overline{x})}{S_{xx}}{y}_i\right]\cdots \mathrm{E}[X+Y]=\mathrm{E}\left[X\right]+\mathrm{E}\left[Y\right]\\ & =& \mathrm{E}\left[\frac{1}{n}\sum _{i=1}^n{y}_i\right]-\mathrm{E}\left[\overline{x}\sum _{i=1}^n\frac{(x_i-\overline{x})}{S_{xx}}{y}_i\right]\\ & =& \frac{1}{n}\mathrm{E}\left[\sum _{i=1}^n{y}_i\right]-\overline{x}\mathrm{E}\left[\sum _{i=1}^n\frac{(x_i-\overline{x})}{S_{xx}}{y}_i\right]\cdots \mathrm{E}\left[cX\right]=c\mathrm{E}\left[X\right]\\ & =& \frac{1}{n}\sum _{i=1}^n\mathrm{E}\left[y_i\right]-\overline{x}\mathrm{E}\left[\frac{S_{xy}}{S_{xx}}\right]\cdots \mathrm{上}\mathrm{記},\frac{S_{xy}}{S_{xx}}=\sum _{i=1}^n\frac{x_i-\overline{x}}{S_{xx}}{y}_i\\ & =& \frac{1}{n}\sum _{i=1}^n(\alpha +\beta x_i)-\overline{x}\mathrm{E}\left[\frac{S_{xy}}{S_{xx}}\right]\cdots \mathrm{E}\left[y_i\right]=\alpha +\beta x_i\\ & =& \frac{1}{n}(\alpha \sum _{i=1}^{n}1+\beta \sum _{i=1}^n{x}_i)-\overline{x}\mathrm{E}\left[\hat{\beta }\right]\cdots \hat{\beta }=\frac{S_{xy}}{S_{xx}}\\ & =& \frac{1}{n}(n\alpha +\beta n\overline{x})-\overline{x}\beta \cdots \mathrm{E}\left[\hat{\beta }\right]=\beta \\ & =& \frac{1}{n}n(\alpha +\beta \overline{x})-\overline{x}\beta \\ & =& (\alpha +\beta \overline{x})-\overline{x}\beta \\ & =& \alpha \cdots \mathrm{E}\left[\hat{\alpha }\right]=\alpha \end{array}$$

$\hat{\alpha }$の分散を$\sum _{i=1}^n{c}_i{y}_i$から求めてみる

$$\begin{array}{rcl}\mathrm{V}\left[\sum _{i=1}^n{c}_i{y}_i\right]& =& \mathrm{V}\left[\sum _{i=1}^n(\frac{1}{n}-\frac{\overline{x}(x_i-\overline{x})}{S_{xx}})y_i\right]\\ & =& \sum _{i=1}^n\mathrm{V}\left[(\frac{1}{n}-\frac{\overline{x}(x_i-\overline{x})}{S_{xx}})y_i\right]\cdots \mathrm{V}\left[\frac{S_{xy}}{S_{xx}}\right]=\frac{{\sigma }^2}{S_{xx}}\mathrm{と}\mathrm{同}\mathrm{じ}\mathrm{結}\mathrm{果}\\ & =& \sum _{i=1}^n{(\frac{1}{n}-\frac{\overline{x}(x_i-\overline{x})}{S_{xx}})}^2\mathrm{V}\left[y_i\right]\cdots \mathrm{V}\left[cX\right]=c^2\mathrm{V}\left[X\right]\\ & =& \sum _{i=1}^n{(\frac{1}{n}-\frac{\overline{x}(x_i-\overline{x})}{S_{xx}})}^2{\sigma }^2\\ & =& {\sigma }^2\sum _{i=1}^n{(\frac{1}{n}-\frac{\overline{x}(x_i-\overline{x})}{S_{xx}})}^2\\ & =& {\sigma }^2\sum _{i=1}^n(\frac{1}{n^2}-2\frac{1}{n}\frac{\overline{x}(x_i-\overline{x})}{S_{xx}}+{\left(\frac{\overline{x}(x_i-\overline{x})}{S_{xx}}\right)}^2)\\ & =& {\sigma }^2(\sum _{i=1}^n\frac{1}{n^2}-\sum _{i=1}^{n}2\frac{1}{n}\frac{\overline{x}(x_i-\overline{x})}{S_{xx}}+\sum _{i=1}^n{\left(\frac{\overline{x}(x_i-\overline{x})}{S_{xx}}\right)}^2)\\ & =& {\sigma }^2(\frac{1}{n^2}\sum _{i=1}^{n}1-\frac{2\overline{x}}{n{S}_{xx}}\sum _{i=1}^n(x_i-\overline{x})+\frac{{\overline{x}}^2}{S_{xx}^2}\sum _{i=1}^n{(x_i-\overline{x})}^2)\\ & =& {\sigma }^2(\frac{1}{n^2}n-\frac{2\overline{x}}{n{S}_{xx}}\cdot 0+\frac{{\overline{x}}^2}{S_{xx}^2}{S}_{xx})\\ & =& {\sigma }^2(\frac{1}{n^2}\cdot n-\frac{2\overline{x}}{n{S}_{xx}}\cdot 0+\frac{{\overline{x}}^2}{S_{xx}^2}\cdot S_{xx})\\ & =& {\sigma }^2(\frac{1}{n}+\frac{{\overline{x}}^2}{S_{xx}})\cdots \mathrm{V}[\overline{y}-\hat{\beta }\overline{x}]=(\frac{1}{n}+\frac{{\overline{x}}^2}{S_{xx}}){\sigma }^2\mathrm{と}\mathrm{同}\mathrm{じ}\mathrm{結}\mathrm{果}\end{array}$$

$\hat{\alpha }$の分布

以上のように，$\hat{\alpha }$は線形推定量であり，正規分布に従う$y_i$の定数倍の和で表すことができた．よって$\hat{\alpha }$は同様に正規分布に従い，その期待値と分散はそれぞれ上記で求めたとおりである $\cdots Z=c_{1}X+c_{2}Y(X\sim \mathrm{N}({\mu }_1,{\sigma }_1^2),Y\sim \mathrm{N}({\mu }_2,{\sigma }_2^2),Z\sim \mathrm{N}(c_1{\mu }_1+c_2{\mu }_2,c_1^2{\sigma }_1^2+c_2^2{\sigma }_2^2))$． $$\begin{array}{rcl}\hat{\alpha }& \sim & \mathrm{N}(\alpha ,{\sigma }^2(\frac{1}{n}+\frac{{\overline{x}}^2}{S_{xx}}))\end{array}$$