Sample vs Population Covariance by Marcus Statistics 🔍 # Sample vs Population Covariance The distinction between $$\text{Population form: } \frac{1}{n}\sum_{i=1}^n (X_i-\bar X)(Y_i-\bar Y) \qquad\text{and}\qquad \text{Sample form: } \frac{1}{n-1}\sum_{i=1}^n (X_i-\bar X)(Y_i-\bar Y)$$ where $\bar X = \frac{1}{n}\sum_{i=1}^n X_i$ and $\bar Y = \frac{1}{n}\sum_{i=1}^n Y_i$ are the sample means of the observations, is usually explained with the phrase: <div align="center"><em>"We lose one degree of freedom because we estimated the mean."</em></div> The statement is correct, but it doesn't explain what is lost, why estimating the means causes the loss, or why the correction is exactly $n-1$. The real reason is the expectation identity $$\mathbb{E}\left[\sum_{i=1}^n (X_i-\bar X)(Y_i-\bar Y)\right] = (n-1)\operatorname{Cov}(X,Y)$$ which immediately implies $$\mathbb{E}\left[\frac{1}{n-1}\sum_{i=1}^n (X_i-\bar X)(Y_i-\bar Y)\right] = \operatorname{Cov}(X,Y)$$ The denominator $n-1$ appears because it makes the sample covariance an **unbiased estimator** of the population covariance. In this lesson we derive this result from the definition of covariance. ## Population covariance Let $(X,Y)$ be a pair of random variables whose variances and covariance are finite, well-defined numbers (this rules out only exotic heavy-tailed distributions). Write: - $\mu_X = \mathbb{E}[X]$ - $\mu_Y = \mathbb{E}[Y]$ The population covariance is $$\sigma_{XY} = \operatorname{Cov}(X,Y) = \mathbb{E}\left[(X-\mu_X)(Y-\mu_Y)\right]$$ the expected product of deviations from the population means. Unlike variance, it can be negative (deviations of opposite sign) or zero (no systematic linear association). Expanding the product and using linearity gives the equivalent moment formula $$\operatorname{Cov}(X,Y) = \mathbb{E}[XY] - \mu_X\mu_Y$$ If the entire population of $N$ pairs is known, the covariance $$\sigma_{xy} = \frac{1}{N}\sum_{i=1}^N (x_i-\mu_x)(y_i-\mu_y)$$ divides by $N$: there is no estimation problem, since the true means are known. The situation changes when we observe only a sample. ## Sampling Covariance Let $(X_1,Y_1),\ldots,(X_n,Y_n)$ be i.i.d. copies of $(X,Y)$, so that pairs are independent across observations while $X_i$ and $Y_i$ within a pair may be dependent. This dependence between variables is what covariance measures. Define the sample means $$\bar X=\frac{1}{n}\sum_{i=1}^n X_i \qquad \bar Y=\frac{1}{n}\sum_{i=1}^n Y_i$$ Since $\mu_X,\mu_Y$ are unknown, a natural estimator is $$\widehat{\sigma}_{XY,n} = \frac{1}{n}\sum_{i=1}^n (X_i-\bar X)(Y_i-\bar Y)$$ which uses the average of observed cross-products around the sample means. Note that this is not unbiased for $\sigma_{XY}$. ## Derivation of Sample Covariance $S_{XY} = \frac{1}{n-1}\sum_{i=1}^n(X_i-\bar X)(Y_i-\bar Y)$ Let $(X_1,Y_1),\ldots,(X_n,Y_n)$ be i.i.d. with $$\mathbb{E}[X_i]=\mu_X \qquad \mathbb{E}[Y_i]=\mu_Y \qquad \operatorname{Cov}(X_i,Y_i)=\sigma_{XY}$$ We want to derive $$\mathbb{E}\left[\sum_{i=1}^n(X_i-\bar X)(Y_i-\bar Y)\right] = (n-1)\sigma_{XY}$$ directly from the definition $$\operatorname{Cov}(X,Y)=\mathbb{E}\left[(X-\mu_X)(Y-\mu_Y)\right]$$ **Outline of the Proof** The strategy is to compare what we *can* compute (deviations from the sample means $\bar X, \bar Y$) with what covariance is actually *defined* through (deviations from the population means $\mu_X, \mu_Y$). Rewriting each sample deviation in terms of population deviations splits the cross-product sum into two pieces: - Primary piece - Each term contributes one $\sigma_{XY}$, for a total of $n\sigma_{XY}$ - Correction piece, $n(\bar X-\mu_X)(\bar Y-\mu_Y)$ - Measures how much the estimated means differ from the true means The heart of the proof is computing the expectation of that correction. Expanding it into a double sum, independence across observations kills every $i \neq j$ term, and the $n$ surviving diagonal terms yield $\operatorname{Cov}(\bar X,\bar Y) = \sigma_{XY}/n$, so the correction removes exactly one $\sigma_{XY}$ in expectation. Subtracting one covariance unit from $n\sigma_{XY}$ leaves $(n-1)\sigma_{XY}$. Dividing by $n-1$ instead of $n$ corrects for this. ### 1. Find the expectation of $\bar X$ and $\bar Y$ By linearity of expectation, $$\mathbb{E}[\bar X] = \mathbb{E}\left[\frac1n\sum_{i=1}^nX_i\right] = \frac1n\sum_{i=1}^n\mathbb{E}[X_i] = \frac1n(n\mu_X) = \mu_X$$ Similarly, $$\mathbb{E}[\bar Y] = \mu_Y$$ ### 2. Rewrite the deviations around the population means $$X_i-\bar X = (X_i-\mu_X)-(\bar X-\mu_X)$$ $$Y_i-\bar Y = (Y_i-\mu_Y)-(\bar Y-\mu_Y)$$ ### 3. Expand the product $$(X_i-\bar X)(Y_i-\bar Y) = (X_i-\mu_X)(Y_i-\mu_Y) - \left[(X_i-\mu_X)(\bar Y-\mu_Y) + (\bar X-\mu_X)(Y_i-\mu_Y)\right] + (\bar X-\mu_X)(\bar Y-\mu_Y)$$ The two bracketed cross terms cannot be combined further at this stage. They only simplify once we sum over $i$. ### 4. Sum over $i$ First note that $$\sum_{i=1}^n(X_i-\mu_X) = \sum_{i=1}^nX_i-n\mu_X = n\bar X-n\mu_X = n(\bar X-\mu_X)$$ and similarly $$\sum_{i=1}^n(Y_i-\mu_Y) = n(\bar Y-\mu_Y)$$ Summing the expansion in Step 3 over $i$ therefore gives $$\sum_{i=1}^n(X_i-\bar X)(Y_i-\bar Y) = \sum_{i=1}^n(X_i-\mu_X)(Y_i-\mu_Y) - n(\bar X-\mu_X)(\bar Y-\mu_Y) - n(\bar X-\mu_X)(\bar Y-\mu_Y) + n(\bar X-\mu_X)(\bar Y-\mu_Y)$$ The last negative copy and the positive copy cancel each other: $$\sum_{i=1}^n(X_i-\bar X)(Y_i-\bar Y) = \sum_{i=1}^n(X_i-\mu_X)(Y_i-\mu_Y) - n(\bar X-\mu_X)(\bar Y-\mu_Y) \; \cancel{{}- n(\bar X-\mu_X)(\bar Y-\mu_Y) + n(\bar X-\mu_X)(\bar Y-\mu_Y)}$$ leaving $$\boxed{ \sum_{i=1}^n(X_i-\bar X)(Y_i-\bar Y) = \sum_{i=1}^n(X_i-\mu_X)(Y_i-\mu_Y) - n(\bar X-\mu_X)(\bar Y-\mu_Y) }$$ ### 5. Take expectations of both sides Both sides of the boxed identity are random variables: they depend on the particular sample $(X_1,Y_1),\ldots,(X_n,Y_n)$ that was drawn. The expectation $\mathbb{E}[\cdot]$ is taken with respect to the joint distribution of these $n$ i.i.d. pairs. We average over all possible datasets of size $n$ that the sampling process could produce. $$\mathbb{E}\left[\sum_{i=1}^n(X_i-\bar X)(Y_i-\bar Y)\right] = \mathbb{E}\left[\sum_{i=1}^n(X_i-\mu_X)(Y_i-\mu_Y)\right] - n\,\mathbb{E}\left[(\bar X-\mu_X)(\bar Y-\mu_Y)\right]$$ ### 6. Evaluate the first term: $\mathbb{E}\left[\sum_{i=1}^n(X_i-\mu_X)(Y_i-\mu_Y)\right]$ By linearity of expectation and the definition of covariance, $$\mathbb{E}\left[\sum_{i=1}^n(X_i-\mu_X)(Y_i-\mu_Y)\right] = \sum_{i=1}^n\mathbb{E}\left[(X_i-\mu_X)(Y_i-\mu_Y)\right] = \sum_{i=1}^n\sigma_{XY} = n\sigma_{XY}$$ ### 7. Evaluate the second term: $\mathbb{E}\left[(\bar X-\mu_X)(\bar Y-\mu_Y)\right] = \operatorname{Cov}(\bar X,\bar Y)$ Rewrite each centered sample mean as an average of centered observations: $$\bar X-\mu_X = \frac1n\sum_{i=1}^nX_i - \frac1n\sum_{i=1}^n\mu_X = \frac1n\sum_{i=1}^n(X_i-\mu_X)$$ $$\bar Y-\mu_Y = \frac1n\sum_{j=1}^n(Y_j-\mu_Y)$$ Multiplying, the two factors of $1/n$ combine into $1/n^2$, and the product of sums becomes a double sum: $$(\bar X-\mu_X)(\bar Y-\mu_Y) = \frac{1}{n^2}\sum_{i=1}^n\sum_{j=1}^n(X_i-\mu_X)(Y_j-\mu_Y)$$ Taking expectations and pulling out the constant $1/n^2$, $$\mathbb{E}\left[(\bar X-\mu_X)(\bar Y-\mu_Y)\right] = \frac{1}{n^2}\sum_{i=1}^n\sum_{j=1}^n\mathbb{E}\left[(X_i-\mu_X)(Y_j-\mu_Y)\right]$$ The double sum runs over all $n^2$ index pairs $(i,j)$. Split it according to whether the indices are equal ($i=j$, the $n$ diagonal terms) or not ($i\neq j$, the $n(n-1)$ off-diagonal terms): $$\mathbb{E}\left[(\bar X-\mu_X)(\bar Y-\mu_Y)\right] = \frac{1}{n^2}\left[\;\underbrace{\sum_{i=1}^n\mathbb{E}\left[(X_i-\mu_X)(Y_i-\mu_Y)\right]}_{i=j} \;+\; \underbrace{\sum_{i=1}^n\sum_{\substack{j=1 \\ j\neq i}}^n\mathbb{E}\left[(X_i-\mu_X)(Y_j-\mu_Y)\right]}_{i\neq j}\;\right]$$ We evaluate the two parts separately. ### 8. Consider the off-diagonal terms For $i\neq j$, the pairs $(X_i,Y_i)$ and $(X_j,Y_j)$ are independent, and independence implies $\mathbb{E}[XY]=\mathbb{E}[X]\,\mathbb{E}[Y]$. Therefore, $$\mathbb{E}\left[(X_i-\mu_X)(Y_j-\mu_Y)\right] = \mathbb{E}[X_i-\mu_X]\,\mathbb{E}[Y_j-\mu_Y]$$ However $$\mathbb{E}[X_i-\mu_X] = \mathbb{E}[X_i]-\mu_X = \mu_X-\mu_X = 0$$ so $$\mathbb{E}\left[(X_i-\mu_X)(Y_j-\mu_Y)\right] = 0\cdot 0 = 0$$ All $n(n-1)$ off-diagonal terms vanish. ### 9. The diagonal terms survive For $i=j$, the definition of covariance gives $$\mathbb{E}\left[(X_i-\mu_X)(Y_i-\mu_Y)\right] = \sigma_{XY}$$ There are $n$ such terms. Recall the split from Step 7: $$\mathbb{E}\left[(\bar X-\mu_X)(\bar Y-\mu_Y)\right] = \frac{1}{n^2}\left[\;\underbrace{\sum_{i=1}^n\mathbb{E}\left[(X_i-\mu_X)(Y_i-\mu_Y)\right]}_{i=j} \;+\; \underbrace{\sum_{i=1}^n\sum_{\substack{j=1 \\ j\neq i}}^n\mathbb{E}\left[(X_i-\mu_X)(Y_j-\mu_Y)\right]}_{i\neq j}\;\right]$$ The diagonal part is $n\sigma_{XY}$ and the off-diagonal part is zero, so $$\mathbb{E}\left[(\bar X-\mu_X)(\bar Y-\mu_Y)\right] = \frac{1}{n^2}\left[\,n\sigma_{XY} + 0\,\right] = \frac{1}{n^2}(n\sigma_{XY})$$ Therefore, $$\boxed{ \operatorname{Cov}(\bar X,\bar Y) = \frac{\sigma_{XY}}{n} }$$ ### 10. Combine the two terms Recall from Step 5: $$\mathbb{E}\left[\sum_{i=1}^n(X_i-\bar X)(Y_i-\bar Y)\right] = \underbrace{\mathbb{E}\left[\sum_{i=1}^n(X_i-\mu_X)(Y_i-\mu_Y)\right]}_{=\,n\sigma_{XY} \text{ (Step 6)}} - \; n\,\underbrace{\mathbb{E}\left[(\bar X-\mu_X)(\bar Y-\mu_Y)\right]}_{=\,\sigma_{XY}/n \text{ (Step 9)}}$$ Substituting the two evaluated terms, $$\mathbb{E}\left[\sum_{i=1}^n(X_i-\bar X)(Y_i-\bar Y)\right] = n\sigma_{XY} - n\left(\frac{\sigma_{XY}}{n}\right) = n\sigma_{XY}-\sigma_{XY}$$ Therefore, $$\boxed{ \mathbb{E}\left[\sum_{i=1}^n(X_i-\bar X)(Y_i-\bar Y)\right] = (n-1)\sigma_{XY} }$$ ### 11. The unbiased estimator Dividing by $n$ gives $$\mathbb{E}\left[\frac1n\sum_{i=1}^n(X_i-\bar X)(Y_i-\bar Y)\right] = \frac{n-1}{n}\sigma_{XY}$$ which is biased toward zero. Dividing instead by $n-1$, the constant $\frac{1}{n-1}$ pulls out of the expectation and cancels the factor $(n-1)$ from Step 10: $$\mathbb{E}\left[\frac{1}{n-1}\sum_{i=1}^n(X_i-\bar X)(Y_i-\bar Y)\right] = \frac{1}{n-1}\,\mathbb{E}\left[\sum_{i=1}^n(X_i-\bar X)(Y_i-\bar Y)\right] = \frac{1}{n-1}(n-1)\sigma_{XY} = \sigma_{XY}$$ Therefore, $$\boxed{ S_{XY} = \frac{1}{n-1}\sum_{i=1}^n(X_i-\bar X)(Y_i-\bar Y) \qquad \mathbb{E}[S_{XY}] = \sigma_{XY}}$$ That is the mathematical reason for the denominator $n-1$. ## Final takeaway If the population means were known, dividing by $n$ would give an unbiased estimator. When they are unknown and replaced by $\bar X$ and $\bar Y$, the sample-dependent centering removes covariance from the empirical cross-product: $$\mathbb{E}\left[\sum_{i=1}^n(X_i-\bar X)(Y_i-\bar Y)\right] = n\sigma_{XY} - \sigma_{XY} = (n-1)\sigma_{XY}$$ So the denominator $n-1$ is not justified merely by the phrase "one degree of freedom is lost." The substantive explanation is: $$\boxed{ \text{Estimating the means removes exactly one covariance unit in expectation.} }$$ The degrees-of-freedom interpretation is the geometric reflection of this expectation calculation, not a replacement for it. # Sample vs Population Covariance The distinction between $$\text{Population form: } \frac{1}{n}\sum_{i=1}^n (X_i-\bar X)(Y_i-\bar Y) \qquad\text{and}\qquad \text{Sample form: } \frac{1}{n-1}\sum_{i=1}^n (X_i-\bar X)(Y_i-\bar Y)$$ where $\bar X = \frac{1}{n}\sum_{i=1}^n X_i$ and $\bar Y = \frac{1}{n}\sum_{i=1}^n Y_i$ are the sample means of the observations, is usually explained with the phrase: <div align="center"><em>"We lose one degree of freedom because we estimated the mean."</em></div> The statement is correct, but it doesn't explain what is lost, why estimating the means causes the loss, or why the correction is exactly $n-1$. The real reason is the expectation identity $$\mathbb{E}\left[\sum_{i=1}^n (X_i-\bar X)(Y_i-\bar Y)\right] = (n-1)\operatorname{Cov}(X,Y)$$ which immediately implies $$\mathbb{E}\left[\frac{1}{n-1}\sum_{i=1}^n (X_i-\bar X)(Y_i-\bar Y)\right] = \operatorname{Cov}(X,Y)$$ The denominator $n-1$ appears because it makes the sample covariance an **unbiased estimator** of the population covariance. In this lesson we derive this result from the definition of covariance. ## Population covariance Let $(X,Y)$ be a pair of random variables whose variances and covariance are finite, well-defined numbers (this rules out only exotic heavy-tailed distributions). Write: - $\mu_X = \mathbb{E}[X]$ - $\mu_Y = \mathbb{E}[Y]$ The population covariance is $$\sigma_{XY} = \operatorname{Cov}(X,Y) = \mathbb{E}\left[(X-\mu_X)(Y-\mu_Y)\right]$$ the expected product of deviations from the population means. Unlike variance, it can be negative (deviations of opposite sign) or zero (no systematic linear association). Expanding the product and using linearity gives the equivalent moment formula $$\operatorname{Cov}(X,Y) = \mathbb{E}[XY] - \mu_X\mu_Y$$ If the entire population of $N$ pairs is known, the covariance $$\sigma_{xy} = \frac{1}{N}\sum_{i=1}^N (x_i-\mu_x)(y_i-\mu_y)$$ divides by $N$: there is no estimation problem, since the true means are known. The situation changes when we observe only a sample. ## Sampling Covariance Let $(X_1,Y_1),\ldots,(X_n,Y_n)$ be i.i.d. copies of $(X,Y)$, so that pairs are independent across observations while $X_i$ and $Y_i$ within a pair may be dependent. This dependence between variables is what covariance measures. Define the sample means $$\bar X=\frac{1}{n}\sum_{i=1}^n X_i \qquad \bar Y=\frac{1}{n}\sum_{i=1}^n Y_i$$ Since $\mu_X,\mu_Y$ are unknown, a natural estimator is $$\widehat{\sigma}_{XY,n} = \frac{1}{n}\sum_{i=1}^n (X_i-\bar X)(Y_i-\bar Y)$$ which uses the average of observed cross-products around the sample means. Note that this is not unbiased for $\sigma_{XY}$. ## Derivation of Sample Covariance $S_{XY} = \frac{1}{n-1}\sum_{i=1}^n(X_i-\bar X)(Y_i-\bar Y)$ Let $(X_1,Y_1),\ldots,(X_n,Y_n)$ be i.i.d. with $$\mathbb{E}[X_i]=\mu_X \qquad \mathbb{E}[Y_i]=\mu_Y \qquad \operatorname{Cov}(X_i,Y_i)=\sigma_{XY}$$ We want to derive $$\mathbb{E}\left[\sum_{i=1}^n(X_i-\bar X)(Y_i-\bar Y)\right] = (n-1)\sigma_{XY}$$ directly from the definition $$\operatorname{Cov}(X,Y)=\mathbb{E}\left[(X-\mu_X)(Y-\mu_Y)\right]$$ **Outline of the Proof** The strategy is to compare what we *can* compute (deviations from the sample means $\bar X, \bar Y$) with what covariance is actually *defined* through (deviations from the population means $\mu_X, \mu_Y$). Rewriting each sample deviation in terms of population deviations splits the cross-product sum into two pieces: - Primary piece - Each term contributes one $\sigma_{XY}$, for a total of $n\sigma_{XY}$ - Correction piece, $n(\bar X-\mu_X)(\bar Y-\mu_Y)$ - Measures how much the estimated means differ from the true means The heart of the proof is computing the expectation of that correction. Expanding it into a double sum, independence across observations kills every $i \neq j$ term, and the $n$ surviving diagonal terms yield $\operatorname{Cov}(\bar X,\bar Y) = \sigma_{XY}/n$, so the correction removes exactly one $\sigma_{XY}$ in expectation. Subtracting one covariance unit from $n\sigma_{XY}$ leaves $(n-1)\sigma_{XY}$. Dividing by $n-1$ instead of $n$ corrects for this. ### 1. Find the expectation of $\bar X$ and $\bar Y$ By linearity of expectation, $$\mathbb{E}[\bar X] = \mathbb{E}\left[\frac1n\sum_{i=1}^nX_i\right] = \frac1n\sum_{i=1}^n\mathbb{E}[X_i] = \frac1n(n\mu_X) = \mu_X$$ Similarly, $$\mathbb{E}[\bar Y] = \mu_Y$$ ### 2. Rewrite the deviations around the population means $$X_i-\bar X = (X_i-\mu_X)-(\bar X-\mu_X)$$ $$Y_i-\bar Y = (Y_i-\mu_Y)-(\bar Y-\mu_Y)$$ ### 3. Expand the product $$(X_i-\bar X)(Y_i-\bar Y) = (X_i-\mu_X)(Y_i-\mu_Y) - \left[(X_i-\mu_X)(\bar Y-\mu_Y) + (\bar X-\mu_X)(Y_i-\mu_Y)\right] + (\bar X-\mu_X)(\bar Y-\mu_Y)$$ The two bracketed cross terms cannot be combined further at this stage. They only simplify once we sum over $i$. ### 4. Sum over $i$ First note that $$\sum_{i=1}^n(X_i-\mu_X) = \sum_{i=1}^nX_i-n\mu_X = n\bar X-n\mu_X = n(\bar X-\mu_X)$$ and similarly $$\sum_{i=1}^n(Y_i-\mu_Y) = n(\bar Y-\mu_Y)$$ Summing the expansion in Step 3 over $i$ therefore gives $$\sum_{i=1}^n(X_i-\bar X)(Y_i-\bar Y) = \sum_{i=1}^n(X_i-\mu_X)(Y_i-\mu_Y) - n(\bar X-\mu_X)(\bar Y-\mu_Y) - n(\bar X-\mu_X)(\bar Y-\mu_Y) + n(\bar X-\mu_X)(\bar Y-\mu_Y)$$ The last negative copy and the positive copy cancel each other: $$\sum_{i=1}^n(X_i-\bar X)(Y_i-\bar Y) = \sum_{i=1}^n(X_i-\mu_X)(Y_i-\mu_Y) - n(\bar X-\mu_X)(\bar Y-\mu_Y) \; \cancel{{}- n(\bar X-\mu_X)(\bar Y-\mu_Y) + n(\bar X-\mu_X)(\bar Y-\mu_Y)}$$ leaving $$\boxed{ \sum_{i=1}^n(X_i-\bar X)(Y_i-\bar Y) = \sum_{i=1}^n(X_i-\mu_X)(Y_i-\mu_Y) - n(\bar X-\mu_X)(\bar Y-\mu_Y) }$$ ### 5. Take expectations of both sides Both sides of the boxed identity are random variables: they depend on the particular sample $(X_1,Y_1),\ldots,(X_n,Y_n)$ that was drawn. The expectation $\mathbb{E}[\cdot]$ is taken with respect to the joint distribution of these $n$ i.i.d. pairs. We average over all possible datasets of size $n$ that the sampling process could produce. $$\mathbb{E}\left[\sum_{i=1}^n(X_i-\bar X)(Y_i-\bar Y)\right] = \mathbb{E}\left[\sum_{i=1}^n(X_i-\mu_X)(Y_i-\mu_Y)\right] - n\,\mathbb{E}\left[(\bar X-\mu_X)(\bar Y-\mu_Y)\right]$$ ### 6. Evaluate the first term: $\mathbb{E}\left[\sum_{i=1}^n(X_i-\mu_X)(Y_i-\mu_Y)\right]$ By linearity of expectation and the definition of covariance, $$\mathbb{E}\left[\sum_{i=1}^n(X_i-\mu_X)(Y_i-\mu_Y)\right] = \sum_{i=1}^n\mathbb{E}\left[(X_i-\mu_X)(Y_i-\mu_Y)\right] = \sum_{i=1}^n\sigma_{XY} = n\sigma_{XY}$$ ### 7. Evaluate the second term: $\mathbb{E}\left[(\bar X-\mu_X)(\bar Y-\mu_Y)\right] = \operatorname{Cov}(\bar X,\bar Y)$ Rewrite each centered sample mean as an average of centered observations: $$\bar X-\mu_X = \frac1n\sum_{i=1}^nX_i - \frac1n\sum_{i=1}^n\mu_X = \frac1n\sum_{i=1}^n(X_i-\mu_X)$$ $$\bar Y-\mu_Y = \frac1n\sum_{j=1}^n(Y_j-\mu_Y)$$ Multiplying, the two factors of $1/n$ combine into $1/n^2$, and the product of sums becomes a double sum: $$(\bar X-\mu_X)(\bar Y-\mu_Y) = \frac{1}{n^2}\sum_{i=1}^n\sum_{j=1}^n(X_i-\mu_X)(Y_j-\mu_Y)$$ Taking expectations and pulling out the constant $1/n^2$, $$\mathbb{E}\left[(\bar X-\mu_X)(\bar Y-\mu_Y)\right] = \frac{1}{n^2}\sum_{i=1}^n\sum_{j=1}^n\mathbb{E}\left[(X_i-\mu_X)(Y_j-\mu_Y)\right]$$ The double sum runs over all $n^2$ index pairs $(i,j)$. Split it according to whether the indices are equal ($i=j$, the $n$ diagonal terms) or not ($i\neq j$, the $n(n-1)$ off-diagonal terms): $$\mathbb{E}\left[(\bar X-\mu_X)(\bar Y-\mu_Y)\right] = \frac{1}{n^2}\left[\;\underbrace{\sum_{i=1}^n\mathbb{E}\left[(X_i-\mu_X)(Y_i-\mu_Y)\right]}_{i=j} \;+\; \underbrace{\sum_{i=1}^n\sum_{\substack{j=1 \\ j\neq i}}^n\mathbb{E}\left[(X_i-\mu_X)(Y_j-\mu_Y)\right]}_{i\neq j}\;\right]$$ We evaluate the two parts separately. ### 8. Consider the off-diagonal terms For $i\neq j$, the pairs $(X_i,Y_i)$ and $(X_j,Y_j)$ are independent, and independence implies $\mathbb{E}[XY]=\mathbb{E}[X]\,\mathbb{E}[Y]$. Therefore, $$\mathbb{E}\left[(X_i-\mu_X)(Y_j-\mu_Y)\right] = \mathbb{E}[X_i-\mu_X]\,\mathbb{E}[Y_j-\mu_Y]$$ However $$\mathbb{E}[X_i-\mu_X] = \mathbb{E}[X_i]-\mu_X = \mu_X-\mu_X = 0$$ so $$\mathbb{E}\left[(X_i-\mu_X)(Y_j-\mu_Y)\right] = 0\cdot 0 = 0$$ All $n(n-1)$ off-diagonal terms vanish. ### 9. The diagonal terms survive For $i=j$, the definition of covariance gives $$\mathbb{E}\left[(X_i-\mu_X)(Y_i-\mu_Y)\right] = \sigma_{XY}$$ There are $n$ such terms. Recall the split from Step 7: $$\mathbb{E}\left[(\bar X-\mu_X)(\bar Y-\mu_Y)\right] = \frac{1}{n^2}\left[\;\underbrace{\sum_{i=1}^n\mathbb{E}\left[(X_i-\mu_X)(Y_i-\mu_Y)\right]}_{i=j} \;+\; \underbrace{\sum_{i=1}^n\sum_{\substack{j=1 \\ j\neq i}}^n\mathbb{E}\left[(X_i-\mu_X)(Y_j-\mu_Y)\right]}_{i\neq j}\;\right]$$ The diagonal part is $n\sigma_{XY}$ and the off-diagonal part is zero, so $$\mathbb{E}\left[(\bar X-\mu_X)(\bar Y-\mu_Y)\right] = \frac{1}{n^2}\left[\,n\sigma_{XY} + 0\,\right] = \frac{1}{n^2}(n\sigma_{XY})$$ Therefore, $$\boxed{ \operatorname{Cov}(\bar X,\bar Y) = \frac{\sigma_{XY}}{n} }$$ ### 10. Combine the two terms Recall from Step 5: $$\mathbb{E}\left[\sum_{i=1}^n(X_i-\bar X)(Y_i-\bar Y)\right] = \underbrace{\mathbb{E}\left[\sum_{i=1}^n(X_i-\mu_X)(Y_i-\mu_Y)\right]}_{=\,n\sigma_{XY} \text{ (Step 6)}} - \; n\,\underbrace{\mathbb{E}\left[(\bar X-\mu_X)(\bar Y-\mu_Y)\right]}_{=\,\sigma_{XY}/n \text{ (Step 9)}}$$ Substituting the two evaluated terms, $$\mathbb{E}\left[\sum_{i=1}^n(X_i-\bar X)(Y_i-\bar Y)\right] = n\sigma_{XY} - n\left(\frac{\sigma_{XY}}{n}\right) = n\sigma_{XY}-\sigma_{XY}$$ Therefore, $$\boxed{ \mathbb{E}\left[\sum_{i=1}^n(X_i-\bar X)(Y_i-\bar Y)\right] = (n-1)\sigma_{XY} }$$ ### 11. The unbiased estimator Dividing by $n$ gives $$\mathbb{E}\left[\frac1n\sum_{i=1}^n(X_i-\bar X)(Y_i-\bar Y)\right] = \frac{n-1}{n}\sigma_{XY}$$ which is biased toward zero. Dividing instead by $n-1$, the constant $\frac{1}{n-1}$ pulls out of the expectation and cancels the factor $(n-1)$ from Step 10: $$\mathbb{E}\left[\frac{1}{n-1}\sum_{i=1}^n(X_i-\bar X)(Y_i-\bar Y)\right] = \frac{1}{n-1}\,\mathbb{E}\left[\sum_{i=1}^n(X_i-\bar X)(Y_i-\bar Y)\right] = \frac{1}{n-1}(n-1)\sigma_{XY} = \sigma_{XY}$$ Therefore, $$\boxed{ S_{XY} = \frac{1}{n-1}\sum_{i=1}^n(X_i-\bar X)(Y_i-\bar Y) \qquad \mathbb{E}[S_{XY}] = \sigma_{XY}}$$ That is the mathematical reason for the denominator $n-1$. ## Final takeaway If the population means were known, dividing by $n$ would give an unbiased estimator. When they are unknown and replaced by $\bar X$ and $\bar Y$, the sample-dependent centering removes covariance from the empirical cross-product: $$\mathbb{E}\left[\sum_{i=1}^n(X_i-\bar X)(Y_i-\bar Y)\right] = n\sigma_{XY} - \sigma_{XY} = (n-1)\sigma_{XY}$$ So the denominator $n-1$ is not justified merely by the phrase "one degree of freedom is lost." The substantive explanation is: $$\boxed{ \text{Estimating the means removes exactly one covariance unit in expectation.} }$$ The degrees-of-freedom interpretation is the geometric reflection of this expectation calculation, not a replacement for it. Comments (0) Please log in to comment. No comments yet. Be the first to comment! ← Back to Lessons
Comments (0)
Please log in to comment.
No comments yet. Be the first to comment!