Ch.6: Correlation and regression

Chapter 6 of the book "Probability for the Enthusiastic Beginner" by David Morin (2016) The concept of correlation A model for correlation The correlation coefficient, r Improving the prediction for Y Calculating ρ(x, y) The standard-deviation box The regression lines Two regression examples Least-squares fitting Problems

139 flashcards · Mathematics · by lakesare

Part of Textbooks › Probability for the Enthusiastic Beginner (David Morin) · 7 courses

1. [Correlation and regression] What does a regression line give?

   The expected value of YYY for a given value of XXX (or the expected value of XXX for a given YYY ). It is the same line as the best-fit (least-squares) line, which is often taken as its definition.

2. [The concept of correlation] What does the degree to which two random variables XXX and YYY are correlated describe?

   How much knowledge of XXX helps predict YYY (or vice versa). The larger the correlation, the more one variable helps predict the other.

3. [The concept of correlation] What are the three degrees of correlation, with an example of each?

   Perfect correlation XXX determines YYY exactly. Example: mass XXX in kilograms and mass YYY in grams: Y=1000XY = 1000XY=1000X . Some correlation XXX helps predict YYY , but not exactly. Example: a man's height and his shoe size. Zero correlation XXX does not help predict YYY . Example: the alphabetical position of the second letter of a person's last name ( A=1A = 1A=1 , B=2B = 2B=2 , ...) and his or her cholesterol level.

4. [The concept of correlation] Why do the points of a scatter plot of two perfectly correlated variables lie on a line?

   YYY is a fixed linear function of XXX , so each xxx gives exactly one yyy . Example: mass in grams vs. mass in kilograms: all points lie on y=1000xy = 1000xy=1000x .

5. [The concept of correlation] Why can't a man's height and his shoe size be perfectly correlated?

   Men of the same height have different shoe sizes. Also, shoe sizes take discrete values, while heights are continuous.

6. [The concept of correlation] In a sample of 26 male students, the average shoe size is 10.4, and the average shoe size of the 6-footers is 11.4. What is the better guess for the shoe size of a 6-footer, and why?

   11.4: height and shoe size are positively correlated, so using the height improves the guess. Shoe size vs. height (inches) for the 26 students.

7. [The concept of correlation] Why might even the second letter of a last name and cholesterol level have a small correlation?

   Some names are common in certain ethnicities, and ethnicities differ slightly in average cholesterol (genes, diet). Small correlations can appear where at first one would expect none.

8. [The concept of correlation] What is a positive and what is a negative correlation?

   Positive: an increase in XXX goes with an increase in YYY on average. Negative: an increase in XXX goes with a decrease in YYY on average. The blob of points in the scatter plot slopes upward (positive) or downward (negative).Example: height and shoe size are positively correlated; vitamin C intake and the incidence of scurvy are negatively correlated.

9. [The concept of correlation] Correlation does not necessarily imply causation.

   Vitamin C intake and scurvy: correlation with causation. Height and shoe size: correlation without causation.

10. [The concept of correlation] Height and shoe size are correlated, but neither causes the other. What explains the correlation?

    A third thing, genetics (and diet), affects both. Example: the number of people putting on sunglasses and the number applying sunscreen on a given day are positively correlated; both are caused by sunshine.

11. [The concept of correlation] Give two examples of a nonlinear relation between two variables.

    Area of a square vs. its side length (area === side 2^22 ), and job income vs. age (zero at age 3, a maximum in between, zero again at 100). The chapter deals only with linear correlation.

12. [A model for correlation] In the model for correlation, how is YYY built from XXX ?

    Y=mX+Z\boldsymbol{Y = mX + Z}Y=mX+Z , with mmm a number and ZZZ a random variable independent of XXX . ZZZ collects the effect of all other variables that affect YYY . Usually XXX and ZZZ are taken with mean 0 and standard deviations σx\sigma_xσx​ , σz\sigma_zσz​ .

13. [A model for correlation] The model for correlation Y=mX+ZY = mX + ZY=mX+Z ( ZZZ independent of XXX ) is usually drawn with Gaussian XXX and ZZZ . Is the Gaussian assumption needed for the results about rrr ?

    No, the results hold for any distributions; only the elliptical shape of the blob of points needs Gaussians. Many real-life variables are approximately Gaussian anyway, by the central limit theorem.

14. [A model for correlation] Y=mX+ZY = mX + ZY=mX+Z for random variables XXX , ZZZ and a number mmm . How are the means related?

    μy=mμx+μz\boldsymbol{\mu_y = m\mu_x + \mu_z}μy​=mμx​+μz​ The expectation value of a sum is the sum of the expectation values. So μx=μz=0\mu_x = \mu_z = 0μx​=μz​=0 gives μy=0\mu_y = 0μy​=0 .

15. [A model for correlation] Model: Y=mX+ZY = mX + ZY=mX+Z , where XXX and ZZZ are independent random variables with means 0 and standard deviations σx\sigma_xσx​ and σz\sigma_zσz​ , and mmm is a number.How do you generate one data point (x,y)(x, y)(x,y) from the model?

    Pick a value xxx of XXX and a value zzz of ZZZ , and set y=mx+zy = mx + zy=mx+z . Repeat with new values of XXX and ZZZ for each further point.

16. [A model for correlation] YYY is the measured weight of an object. How can YYY be written in the form Y=mX+ZY = mX + ZY=mX+Z ( ZZZ independent of XXX )?

    XXX = the true weight, ZZZ = the measurement error, m=1m = 1m=1 .

17. [A model for correlation] In practice, are you given the XXX and ZZZ distributions of Y=mX+ZY = mX + ZY=mX+Z ?

    No, usually only data points, from which you compute the correlation coefficient rrr . The model is still the key to deriving the formula for rrr , even without knowing XXX and ZZZ .

18. [A model for correlation] Model: Y=mX+ZY = mX + ZY=mX+Z , where XXX and ZZZ are independent random variables with means 0 and standard deviations σx\sigma_xσx​ and σz\sigma_zσz​ , and mmm is a number.What happens when σz=0\sigma_z = 0σz​=0 ?

    Perfect correlation: Z=0Z = 0Z=0 , so Y=mXY = mXY=mX and all points lie on the line y=mxy = mxy=mx . m=0.5m = 0.5m=0.5 , σx=1\sigma_x = 1σx​=1 , 100 points.

19. [A model for correlation] Model: Y=mX+ZY = mX + ZY=mX+Z , where XXX and ZZZ are independent random variables with means 0 and standard deviations σx\sigma_xσx​ and σz\sigma_zσz​ , and mmm is a number.What does the scatter plot look like when σz\sigma_zσz​ is small but nonzero?

    Strong but not perfect correlation: the points lie close to the line y=mxy = mxy=mx . m=0.5m = 0.5m=0.5 , σx=1\sigma_x = 1σx​=1 , σz=0.1\sigma_z = 0.1σz​=0.1 , 200 points. XXX helps predict YYY a lot, but not exactly.

20. [A model for correlation] Model: Y=mX+ZY = mX + ZY=mX+Z , where XXX and ZZZ are independent random variables with means 0 and standard deviations σx\sigma_xσx​ and σz\sigma_zσz​ , and mmm is a number.What happens when m=0m = 0m=0 ?

    Zero correlation: Y=ZY = ZY=Z , which is independent of XXX . σx=2\sigma_x = 2σx​=2 , σz=1\sigma_z = 1σz​=1 , 2000 points. For every xxx , the YYY values are distributed around 0 with standard deviation σz\sigma_zσz​ .

21. [A model for correlation] Model: Y=mX+ZY = mX + ZY=mX+Z , where XXX and ZZZ are independent Gaussian random variables with means 0 and standard deviations σx\sigma_xσx​ and σz\sigma_zσz​ , and mmm is a number.With m=0m = 0m=0 , how do the YYY values in two thin vertical strips at different xxx compare?

    Same distribution (the one of ZZZ ), except for an overall scaling factor. The factor is the density of XXX at that xxx : the right strip has fewer points because of the factor e−x2/2σx2e^{-x^2/2\sigma_x^2}e−x2/2σx2​ .

22. [A model for correlation] Model: Y=mX+ZY = mX + ZY=mX+Z , where XXX and ZZZ are independent random variables with means 0 and standard deviations σx\sigma_xσx​ and σz\sigma_zσz​ , and mmm is a number.What does the scatter plot look like when mmm is small but nonzero?

    Weak correlation: near the origin it looks uncorrelated; only for larger ∣x∣|x|∣x∣ is there a visible bias in YYY . m=0.2m = 0.2m=0.2 , σx=2\sigma_x = 2σx​=2 , σz=1\sigma_z = 1σz​=1 : more points above the XXX axis on the right, more below on the left.

23. [A model for correlation] Y=mX+ZY = mX + ZY=mX+Z with XXX , ZZZ independent. How does the blob of points move if XXX has mean μx\mu_xμx​ and ZZZ has mean μz\mu_zμz​ , instead of 0?

    Right by μx\mu_xμx​ and up by mμx+μzm\mu_x + \mu_zmμx​+μz​ . μy=mμx+μz\mu_y = m\mu_x + \mu_zμy​=mμx​+μz​ .

24. [A model for correlation] Model: Y=mX+ZY = mX + ZY=mX+Z , where XXX and ZZZ are independent random variables with means 0 and standard deviations σx\sigma_xσx​ and σz\sigma_zσz​ , and mmm is a number.The model treats XXX as the independent variable. Can a scatter plot also be read the other way?

    Yes: treat XXX as a "function" of YYY and ask how much YYY determines XXX . This leads to a second regression line.

25. [A model for correlation] Model: Y=mX+ZY = mX + ZY=mX+Z , where XXX and ZZZ are independent random variables with means 0 and standard deviations σx\sigma_xσx​ and σz\sigma_zσz​ , and mmm is a number.In a strongly correlated scatter plot, the vertical spread of YYY looks smaller in the tails. Is the spread really smaller there?

    No. For every xxx , the YYY values are centered at mxmxmx with the same standard deviation σz\sigma_zσz​ . The spread seems larger in the middle only because there are more points there.

26. [The correlation coefficient, r] Model: Y=mX+ZY = mX + ZY=mX+Z , where XXX and ZZZ are independent random variables with means 0 and standard deviations σx\sigma_xσx​ and σz\sigma_zσz​ , and mmm is a number.What is the standard deviation σy\sigma_yσy​ of YYY ?

    σy=m2σx2+σz2\boldsymbol{\sigma_y = \sqrt{m^2\sigma_x^2 + \sigma_z^2}}σy​=m2σx2​+σz2​​ mXmXmX has standard deviation mσxm\sigma_xmσx​ , and standard deviations of independent variables add in quadrature (true for any distributions).Example: m=0.2m = 0.2m=0.2 , σx=2\sigma_x = 2σx​=2 , σz=1\sigma_z = 1σz​=1 : σy=0.16+1≈1.08\sigma_y = \sqrt{0.16 + 1} \approx 1.08σy​=0.16+1​≈1.08 .

27. [The correlation coefficient, r] Model: Y=mX+ZY = mX + ZY=mX+Z , where XXX and ZZZ are independent random variables with means 0 and standard deviations σx\sigma_xσx​ and σz\sigma_zσz​ , and mmm is a number.What is σy\sigma_yσy​ in the two extreme cases σz=0\sigma_z = 0σz​=0 and m=0m = 0m=0 ?

    σz=0\sigma_z = 0σz​=0 : σy=mσx\boldsymbol{\sigma_y = m\sigma_x}σy​=mσx​ (all of it from XXX ). m=0\ m = 0 m=0 : σy=σz\boldsymbol{\sigma_y = \sigma_z}σy​=σz​ (all of it from ZZZ ). In general σy=m2σx2+σz2\sigma_y = \sqrt{m^2\sigma_x^2 + \sigma_z^2}σy​=m2σx2​+σz2​​ .

28. [The correlation coefficient, r] Model: Y=mX+ZY = mX + ZY=mX+Z , where XXX and ZZZ are independent random variables with means 0 and standard deviations σx\sigma_xσx​ and σz\sigma_zσz​ , and mmm is a number.How is the correlation coefficient rrr defined in terms of mmm , σx\sigma_xσx​ , σz\sigma_zσz​ , and what does it mean?

    r=mσxσy=mσxm2σx2+σz2\boldsymbol{r = \dfrac{m\sigma_x}{\sigma_y} = \dfrac{m\sigma_x}{\sqrt{m^2\sigma_x^2 + \sigma_z^2}}}r=σy​mσx​​=m2σx2​+σz2​​mσx​​ rrr is the fraction of σy\sigma_yσy​ that can be attributed to XXX (the part of σy\sigma_yσy​ from XXX is mσxm\sigma_xmσx​ ).Example: m=0.2m = 0.2m=0.2 , σx=2\sigma_x = 2σx​=2 , σz=1\sigma_z = 1σz​=1 : r=0.4/1.08≈0.37r = 0.4 / 1.08 \approx 0.37r=0.4/1.08≈0.37 .

29. [The correlation coefficient, r] Model: Y=mX+ZY = mX + ZY=mX+Z , where XXX and ZZZ are independent random variables with means 0 and standard deviations σx\sigma_xσx​ and σz\sigma_zσz​ , and mmm is a number.What does r2r^2r2 mean, for the correlation coefficient r=mσx/σyr = m\sigma_x/\sigma_yr=mσx​/σy​ ?

    The fraction of the variance of YYY that can be attributed to XXX . r2=m2σx2m2σx2+σz2r^2 = \dfrac{m^2\sigma_x^2}{m^2\sigma_x^2 + \sigma_z^2}r2=m2σx2​+σz2​m2σx2​​

30. [The correlation coefficient, r] What is the covariance Cov(X,Y)\mathrm{Cov}(X, Y)Cov(X,Y) of two random variables?

    Cov(X,Y)=E[(X−μx)(Y−μy)]\boldsymbol{\mathrm{Cov}(X, Y) = E\big[(X - \mu_x)(Y - \mu_y)\big]}Cov(X,Y)=E[(X−μx​)(Y−μy​)] It measures how much XXX and YYY linearly depend on each other. For μx=μy=0\mu_x = \mu_y = 0μx​=μy​=0 : Cov(X,Y)=E(XY)\mathrm{Cov}(X, Y) = E(XY)Cov(X,Y)=E(XY) .

31. [The correlation coefficient, r] XXX has standard deviation σx\sigma_xσx​ . What is Cov(X,Y)\mathrm{Cov}(X, Y)Cov(X,Y) for Y=XY = XY=X , for YYY independent of XXX , and for Y=−XY = -XY=−X ?

    Y=XY = XY=X : σx2\boldsymbol{\sigma_x^2}σx2​ ; \ independent: 0\boldsymbol{0}0 ; Y=−X\ Y = -X Y=−X : −σx2\boldsymbol{-\sigma_x^2}−σx2​ Cov(X,X)=E[(X−μx)2]=Var(X)\mathrm{Cov}(X, X) = E[(X - \mu_x)^2] = \mathrm{Var}(X)Cov(X,X)=E[(X−μx​)2]=Var(X) , so covariance generalizes variance.

32. [The correlation coefficient, r] Why is Cov(X,Y)=E[(X−μx)(Y−μy)]=0\mathrm{Cov}(X, Y) = E\big[(X - \mu_x)(Y - \mu_y)\big] = 0Cov(X,Y)=E[(X−μx​)(Y−μy​)]=0 for independent XXX and YYY ?

    The expectation of the product is the product E(X−μx)⋅E(Y−μy)E(X - \mu_x) \cdot E(Y - \mu_y)E(X−μx​)⋅E(Y−μy​) , and both factors are 0. E(X−μx)=E(X)−μx=0E(X - \mu_x) = E(X) - \mu_x = 0E(X−μx​)=E(X)−μx​=0 , likewise for YYY .

33. [The correlation coefficient, r] Expanding the product, what is an alternative formula for Cov(X,Y)=E[(X−μx)(Y−μy)]\mathrm{Cov}(X, Y) = E\big[(X - \mu_x)(Y - \mu_y)\big]Cov(X,Y)=E[(X−μx​)(Y−μy​)] ?

    Cov(X,Y)=E(XY)−μxμy\boldsymbol{\mathrm{Cov}(X, Y) = E(XY) - \mu_x\mu_y}Cov(X,Y)=E(XY)−μx​μy​ E[XY−μyX−μxY+μxμy]=E(XY)−μyμx−μxμy+μxμyE[XY - \mu_y X - \mu_x Y + \mu_x\mu_y] = E(XY) - \mu_y\mu_x - \mu_x\mu_y + \mu_x\mu_yE[XY−μy​X−μx​Y+μx​μy​]=E(XY)−μy​μx​−μx​μy​+μx​μy​ . For Y=XY = XY=X it is Var(X)=E(X2)−μx2\mathrm{Var}(X) = E(X^2) - \mu_x^2Var(X)=E(X2)−μx2​ .

34. [The correlation coefficient, r] What is the formula for the correlation coefficient rrr in terms of the covariance?

    r=Cov(X,Y)σxσy\boldsymbol{r = \dfrac{\mathrm{Cov}(X, Y)}{\sigma_x \sigma_y}}r=σx​σy​Cov(X,Y)​ It is equivalent to r=mσx/σyr = m\sigma_x/\sigma_yr=mσx​/σy​ in the model Y=mX+ZY = mX + ZY=mX+Z , but needs no mmm , so it works for plain data.

35. [The correlation coefficient, r] Model: Y=mX+ZY = mX + ZY=mX+Z , where XXX and ZZZ are independent random variables with means 0 and standard deviations σx\sigma_xσx​ and σz\sigma_zσz​ , and mmm is a number.How does Cov(X,Y)σxσy\dfrac{\mathrm{Cov}(X, Y)}{\sigma_x\sigma_y}σx​σy​Cov(X,Y)​ reduce to mσxσy\dfrac{m\sigma_x}{\sigma_y}σy​mσx​​ ?

    Cov(X,mX+Z)=mE(X2)+E(XZ)=mσx2+0\mathrm{Cov}(X, mX + Z) = mE(X^2) + E(XZ) = \boldsymbol{m\sigma_x^2 + 0}Cov(X,mX+Z)=mE(X2)+E(XZ)=mσx2​+0 , and mσx2σxσy=mσxσy\dfrac{m\sigma_x^2}{\sigma_x\sigma_y} = \dfrac{m\sigma_x}{\sigma_y}σx​σy​mσx2​​=σy​mσx​​ . E(XZ)=E(X)E(Z)=0E(XZ) = E(X)E(Z) = 0E(XZ)=E(X)E(Z)=0 by independence. The covariance keeps the part of YYY that comes from XXX and drops the part from ZZZ .

36. [The correlation coefficient, r] What is the covariance of nnn data points (xi,yi)(x_i, y_i)(xi​,yi​) ?

    Cov(x,y)=1n∑(xi−xˉ)(yi−yˉ)\boldsymbol{\mathrm{Cov}(x, y) = \dfrac{1}{n}\sum (x_i - \bar{x})(y_i - \bar{y})}Cov(x,y)=n1​∑(xi​−xˉ)(yi​−yˉ​) The average of the products replaces the expectation value. If xˉ=yˉ=0\bar{x} = \bar{y} = 0xˉ=yˉ​=0 , it is 1n∑xiyi\frac{1}{n}\sum x_i y_in1​∑xi​yi​ .

37. [The correlation coefficient, r] What is the correlation coefficient rrr of nnn data points (xi,yi)(x_i, y_i)(xi​,yi​) ?

    r=∑(xi−xˉ)(yi−yˉ)∑(xi−xˉ)2 ∑(yi−yˉ)2\boldsymbol{r = \dfrac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum (x_i - \bar{x})^2}\,\sqrt{\sum (y_i - \bar{y})^2}}}r=∑(xi​−xˉ)2​∑(yi​−yˉ​)2​∑(xi​−xˉ)(yi​−yˉ​)​ This is Cov(x,y)/(s~xs~y)\mathrm{Cov}(x, y)/(\tilde{s}_x\tilde{s}_y)Cov(x,y)/(s~x​s~y​) , with s~x,s~y\tilde{s}_x, \tilde{s}_ys~x​,s~y​ the standard deviations of the points (factor 1n\frac{1}{n}n1​ inside). The nnn 's cancel.

38. [The correlation coefficient, r] If the nnn data points are a sample from a larger population, should rrr be computed with the sample versions (factor 1n−1\frac{1}{n-1}n−11​ ) of the standard deviations and covariance?

    It makes no difference: the n−1n - 1n−1 factors cancel in r=Cov/(sxsy)r = \mathrm{Cov}/(s_x s_y)r=Cov/(sx​sy​) , just as the nnn 's do. So Excel's COVARIANCE.S with STDEV.S gives the same rrr as COVARIANCE.P with STDEV.P.

39. [The correlation coefficient, r] Why define rrr first as mσx/σym\sigma_x/\sigma_ymσx​/σy​ (in Y=mX+ZY = mX + ZY=mX+Z ) rather than as Cov(X,Y)/(σxσy)\mathrm{Cov}(X, Y)/(\sigma_x\sigma_y)Cov(X,Y)/(σx​σy​) ?

    mσx/σym\sigma_x/\sigma_ymσx​/σy​ shows what rrr means: the fraction of σy\sigma_yσy​ that comes from XXX . If most of σy\sigma_yσy​ comes from XXX and not ZZZ , the correlation is high; if most comes from ZZZ , it is low.

40. [The correlation coefficient, r] r=Cov(X,Y)σxσyr = \dfrac{\mathrm{Cov}(X, Y)}{\sigma_x \sigma_y}r=σx​σy​Cov(X,Y)​ is the correlation coefficient of XXX and YYY .How does rrr change if the means of XXX and YYY change?

    It does not change. None of mmm , σx\sigma_xσx​ , σy\sigma_yσy​ , σz\sigma_zσz​ , Cov(X,Y)\mathrm{Cov}(X, Y)Cov(X,Y) depends on the means; changing them only shifts the blob of points.

41. [The correlation coefficient, r] r=Cov(X,Y)σxσyr = \dfrac{\mathrm{Cov}(X, Y)}{\sigma_x \sigma_y}r=σx​σy​Cov(X,Y)​ is the correlation coefficient of XXX and YYY .How does rrr change under X′=aXX' = aXX′=aX , Y′=bYY' = bYY′=bY with a,b>0a, b > 0a,b>0 ?

    It does not change. Cov(X′,Y′)=ab Cov(X,Y)\mathrm{Cov}(X', Y') = ab\,\mathrm{Cov}(X, Y)Cov(X′,Y′)=abCov(X,Y) and σx′σy′=ab σxσy\sigma_{x'}\sigma_{y'} = ab\,\sigma_x\sigma_yσx′​σy′​=abσx​σy​ . Stretching the axes doesn't change how well XXX predicts YYY .

42. [The correlation coefficient, r] r=Cov(X,Y)σxσyr = \dfrac{\mathrm{Cov}(X, Y)}{\sigma_x \sigma_y}r=σx​σy​Cov(X,Y)​ is the correlation coefficient of XXX and YYY .Is rrr symmetric in XXX and YYY , and why is that surprising?

    Yes, since Cov(X,Y)=Cov(Y,X)\mathrm{Cov}(X, Y) = \mathrm{Cov}(Y, X)Cov(X,Y)=Cov(Y,X) . Surprising because the model Y=mX+ZY = mX + ZY=mX+Z is not symmetric: ZZZ is independent of XXX , not of YYY .

43. [The correlation coefficient, r] Model: Y=mX+ZY = mX + ZY=mX+Z , where XXX and ZZZ are independent random variables with means 0 and standard deviations σx\sigma_xσx​ and σz\sigma_zσz​ , and mmm is a number.What is mmm in terms of rrr , and in terms of the covariance?

    m=rσyσx=Cov(X,Y)σx2\boldsymbol{m = r\dfrac{\sigma_y}{\sigma_x} = \dfrac{\mathrm{Cov}(X, Y)}{\sigma_x^2}}m=rσx​σy​​=σx2​Cov(X,Y)​ mmm is the slope of the regression line.

44. [The correlation coefficient, r] All of these scatter plots have σx=2\sigma_x = 2σx​=2 , σy=1\sigma_y = 1σy​=1 . What do they show about the size of rrr needed for a visible correlation?

    It takes a sizeable rrr : r=0.3r = 0.3r=0.3 looks roughly like r=0r = 0r=0 , and at r=0.7r = 0.7r=0.7 the YYY values for a given XXX still spread widely. So " r=0.7r = 0.7r=0.7 , so I can be fairly certain of YYY given XXX " is wrong.

45. [The correlation coefficient, r] Is r=0.7r = 0.7r=0.7 a "high" correlation?

    It depends on the field. Example: a social scientist finding r=0.7r = 0.7r=0.7 between a characteristic and months of unemployment: a very significant result.Example: a physicist finding r=0.7r = 0.7r=0.7 between fall distance ddd and time squared t2t^2t2 : something is badly wrong, since d=(g/2)t2d = (g/2)t^2d=(g/2)t2 exactly.

46. [The correlation coefficient, r] Model: Y=mX+ZY = mX + ZY=mX+Z , where XXX and ZZZ are independent random variables with means 0 and standard deviations σx\sigma_xσx​ and σz\sigma_zσz​ , and mmm is a number.What does a scatter plot with negative rrr look like, and why?

    The blob of points slopes downward. r=mσx/σy<0r = m\sigma_x/\sigma_y < 0r=mσx​/σy​<0 means m<0m < 0m<0 , so an increase in XXX gives a decrease in YYY on average. ( r=−0.7r = -0.7r=−0.7 shown.)

47. [The correlation coefficient, r] Model: Y=mX+ZY = mX + ZY=mX+Z , where XXX and ZZZ are independent random variables with means 0 and standard deviations σx\sigma_xσx​ and σz\sigma_zσz​ , and mmm is a number.Given σx\sigma_xσx​ , σy\sigma_yσy​ , rrr , what are σz\sigma_zσz​ and mmm ?

    σz=σy1−r2\boldsymbol{\sigma_z = \sigma_y\sqrt{1 - r^2}}σz​=σy​1−r2​ and m=rσyσx\boldsymbol{m = \dfrac{r\sigma_y}{\sigma_x}}m=σx​rσy​​ σz2=σy2−m2σx2\sigma_z^2 = \sigma_y^2 - m^2\sigma_x^2σz2​=σy2​−m2σx2​ and m2σx2=r2σy2m^2\sigma_x^2 = r^2\sigma_y^2m2σx2​=r2σy2​ , so σz2=(1−r2)σy2\sigma_z^2 = (1 - r^2)\sigma_y^2σz2​=(1−r2)σy2​ .Example: r=0.3r = 0.3r=0.3 , σx=2\sigma_x = 2σx​=2 , σy=1\sigma_y = 1σy​=1 : σz=0.91≈0.95\sigma_z = \sqrt{0.91} \approx 0.95σz​=0.91​≈0.95 , m=0.15m = 0.15m=0.15 .

48. [The correlation coefficient, r] What are the steps to find rrr and the slope mmm from a list of data points (xi,yi)(x_i, y_i)(xi​,yi​) ?

    Means xˉ\bar{x}xˉ and yˉ\bar{y}yˉ​ Standard deviations s~x=1n∑(xi−xˉ)2\tilde{s}_x = \sqrt{\frac{1}{n}\sum (x_i - \bar{x})^2}s~x​=n1​∑(xi​−xˉ)2​ , likewise s~y\tilde{s}_ys~y​ Covariance Cov(x,y)=1n∑(xi−xˉ)(yi−yˉ)\mathrm{Cov}(x, y) = \frac{1}{n}\sum (x_i - \bar{x})(y_i - \bar{y})Cov(x,y)=n1​∑(xi​−xˉ)(yi​−yˉ​) Correlation coefficient r=Cov(x,y)/(s~xs~y)r = \mathrm{Cov}(x, y)/(\tilde{s}_x\tilde{s}_y)r=Cov(x,y)/(s~x​s~y​) Slope m=rs~y/s~xm = r\tilde{s}_y/\tilde{s}_xm=rs~y​/s~x​ Example: 20 points wi…

49. [Improving the prediction for Y] How is the badness of a prediction ypy_pyp​ for a random variable YYY measured?

    σp2=E[(Y−yp)2]\boldsymbol{\sigma_p^2 = E\big[(Y - y_p)^2\big]}σp2​=E[(Y−yp​)2] , the variance of YYY measured relative to ypy_pyp​ . For nnn values yiy_iyi​ : s~p2=1n∑(yi−yp)2\tilde{s}_p^2 = \frac{1}{n}\sum (y_i - y_p)^2s~p2​=n1​∑(yi​−yp​)2 , which tends to σp2\sigma_p^2σp2​ as n→∞n \to \inftyn→∞ .

50. [Improving the prediction for Y] Which prediction ypy_pyp​ for YYY minimizes the badness σp2=E[(Y−yp)2]\sigma_p^2 = E\big[(Y - y_p)^2\big]σp2​=E[(Y−yp​)2] ?

    The mean, yp=μyy_p = \mu_yyp​=μy​ . σp2=E[Y2]−2E[Y]yp+yp2\sigma_p^2 = E[Y^2] - 2E[Y]y_p + y_p^2σp2​=E[Y2]−2E[Y]yp​+yp2​ ; setting the derivative −2E[Y]+2yp-2E[Y] + 2y_p−2E[Y]+2yp​ to zero gives yp=E[Y]y_p = E[Y]yp​=E[Y] .

51. [Improving the prediction for Y] With badness E[(Y−yp)2]E\big[(Y - y_p)^2\big]E[(Y−yp​)2] and no knowledge of XXX , how bad is the best prediction for YYY ?

    σy2\boldsymbol{\sigma_y^2}σy2​ , the variance of YYY . The best prediction is μy\mu_yμy​ , and E[(Y−μy)2]=σy2E[(Y - \mu_y)^2] = \sigma_y^2E[(Y−μy​)2]=σy2​ .

52. [Improving the prediction for Y] Model: Y=mX+ZY = mX + ZY=mX+Z , where XXX and ZZZ are independent random variables with means 0 and standard deviations σx\sigma_xσx​ and σz\sigma_zσz​ , and mmm is a number.Given X=x0X = x_0X=x0​ , what is the best prediction for YYY , and why?

    yp=mx0\boldsymbol{y_p = mx_0}yp​=mx0​ , the mean of the YYY values with X=x0X = x_0X=x0​ . Y=mx0+ZY = mx_0 + ZY=mx0​+Z there, and μz=0\mu_z = 0μz​=0 . The best prediction is always the mean, now of the YYY values in the thin strip at x0x_0x0​ . Upper white dot: prediction mx0mx_0mx0​ using x0x_0x0​ . Lower white dot: prediction 0 without XXX .

53. [Improving the prediction for Y] Model: Y=mX+ZY = mX + ZY=mX+Z , where XXX and ZZZ are independent random variables with means 0 and standard deviations σx\sigma_xσx​ and σz\sigma_zσz​ , and mmm is a number.Given X=x0X = x_0X=x0​ and the prediction mx0mx_0mx0​ for YYY , how bad is the prediction (badness E[(Y−yp)2]E[(Y - y_p)^2]E[(Y−yp​)2] )?

    σz2\boldsymbol{\sigma_z^2}σz2​ With X=x0X = x_0X=x0​ , Y−mx0=ZY - mx_0 = ZY−mx0​=Z , so the badness is E[Z2]=σz2E[Z^2] = \sigma_z^2E[Z2]=σz2​ .

54. [Improving the prediction for Y] Model: Y=mX+ZY = mX + ZY=mX+Z , where XXX and ZZZ are independent random variables with means 0 and standard deviations σx\sigma_xσx​ and σz\sigma_zσz​ , and mmm is a number.By which factor does knowing XXX reduce the badness (variance) of the best prediction for YYY ?

    σz2σy2=1−r2\boldsymbol{\dfrac{\sigma_z^2}{\sigma_y^2} = 1 - r^2}σy2​σz2​​=1−r2 , with rrr the correlation coefficient. Without XXX the badness is σy2\sigma_y^2σy2​ , with XXX it is σz2\sigma_z^2σz2​ , and σz=σy1−r2\sigma_z = \sigma_y\sqrt{1 - r^2}σz​=σy​1−r2​ .Example: r=0.5r = 0.5r=0.5 gives 0.750.750.75 : only a slight improvement.

55. [Improving the prediction for Y] Knowing XXX reduces the variance of the best prediction for YYY by the factor 1−r21 - r^21−r2 ( rrr the correlation coefficient).Why do r=1r = 1r=1 and r=0r = 0r=0 give sensible factors?

    r=1r = 1r=1 : factor 0, the prediction is perfect. r=0\ r = 0 r=0 : factor 1, XXX does not help at all.

56. [Improving the prediction for Y] Knowing XXX reduces the variance of the best prediction for YYY by the factor 1−r21 - r^21−r2 ( rrr the correlation coefficient).Do r=0.5r = 0.5r=0.5 and r=−0.5r = -0.5r=−0.5 give different improvements?

    No, both give 1−r2=0.751 - r^2 = 0.751−r2=0.75 . The two blobs of points differ only in tilting up or down.

57. [Calculating ρ(x, y)] What is the joint probability density ρ(x,y)\rho(x, y)ρ(x,y) of two random variables XXX , YYY ?

    A probability per unit area: ρ(x,y) dx dy\rho(x, y)\,dx\,dyρ(x,y)dxdy is the probability that (X,Y)(X, Y)(X,Y) lies in the small area dx dydx\,dydxdy at (x,y)(x, y)(x,y) . Its double integral over the whole plane is 1.

58. [Calculating ρ(x, y)] Model: Y=mX+ZY = mX + ZY=mX+Z , where XXX and ZZZ are independent Gaussian random variables with means 0 and standard deviations σx\sigma_xσx​ and σz\sigma_zσz​ , and mmm is a number.What shape do the curves of constant ρ(x,y)\rho(x, y)ρ(x,y) have?

    Ellipses. So scatter plots of such XXX , YYY are elliptical blobs. ( YYY is Gaussian too.) For non-Gaussian XXX or ZZZ they are not ellipses.

59. [Calculating ρ(x, y)] Model: Y=mX+ZY = mX + ZY=mX+Z , where XXX and ZZZ are independent random variables with means 0 and standard deviations σx\sigma_xσx​ and σz\sigma_zσz​ , and mmm is a number.How does the derivation of the joint density ρ(x,y)\rho(x, y)ρ(x,y) start?

    ρ(x,y) dx dy=(ρX(x) dx)⋅(ρZ(y−mx) dy)\boldsymbol{\rho(x, y)\,dx\,dy = \big(\rho_X(x)\,dx\big) \cdot \big(\rho_Z(y - mx)\,dy\big)}ρ(x,y)dxdy=(ρX​(x)dx)⋅(ρZ​(y−mx)dy) The point (x,y)(x, y)(x,y) needs X=xX = xX=x and Z=y−mxZ = y - mxZ=y−mx ; these are independent, so their probabilities multiply. dz=dydz = dydz=dy for fixed xxx , since ZZZ has coefficient 1.

60. [Calculating ρ(x, y)] Model: Y=mX+ZY = mX + ZY=mX+Z , where XXX and ZZZ are independent Gaussian random variables with means 0 and standard deviations σx\sigma_xσx​ and σz\sigma_zσz​ , and mmm is a number.What is ρ(x,y)\rho(x, y)ρ(x,y) in terms of mmm , σx\sigma_xσx​ , σz\sigma_zσz​ ?

    ρ(x,y)=12πσxσzexp⁡(−x22σx2−(y−mx)22σz2)\boldsymbol{\rho(x, y) = \dfrac{1}{2\pi\sigma_x\sigma_z}\exp\left(-\dfrac{x^2}{2\sigma_x^2} - \dfrac{(y - mx)^2}{2\sigma_z^2}\right)}ρ(x,y)=2πσx​σz​1​exp(−2σx2​x2​−2σz2​(y−mx)2​) The product of the Gaussian densities of XXX at xxx and of ZZZ at y−mxy - mxy−mx .

61. [Calculating ρ(x, y)] XXX , YYY are jointly Gaussian with means 0, standard deviations σx\sigma_xσx​ , σy\sigma_yσy​ , and correlation coefficient rrr .What is the joint density ρ(x,y)\rho(x, y)ρ(x,y) ?

    ρ(x,y)=12πσxσy1−r2exp⁡(−12(1−r2)(x2σx2+y2σy2−2rxyσxσy))\boldsymbol{\rho(x, y) = \dfrac{1}{2\pi\sigma_x\sigma_y\sqrt{1 - r^2}}\exp\left(-\dfrac{1}{2(1 - r^2)}\left(\dfrac{x^2}{\sigma_x^2} + \dfrac{y^2}{\sigma_y^2} - \dfrac{2rxy}{\sigma_x\sigma_y}\right)\right)}ρ(x,y)=2πσx​σy​1−r2​1​exp(−2(1−r2)1​(σx2​x2​+σy2​y2​−σx​σy​2rxy​))

62. [Calculating ρ(x, y)] ρ(x,y)=12πσxσzexp⁡(−x22σx2−(y−mx)22σz2)\rho(x, y) = \dfrac{1}{2\pi\sigma_x\sigma_z}\exp\left(-\dfrac{x^2}{2\sigma_x^2} - \dfrac{(y - mx)^2}{2\sigma_z^2}\right)ρ(x,y)=2πσx​σz​1​exp(−2σx2​x2​−2σz2​(y−mx)2​) is the joint density for Y=mX+ZY = mX + ZY=mX+Z with independent Gaussian XXX , ZZZ (means 0).How do you rewrite ρ(x,y)\rho(x, y)ρ(x,y) in terms of σx\sigma_xσx​ , σy\sigma_yσy​ and rrr ?

    Substitute σz=σy1−r2\sigma_z = \sigma_y\sqrt{1 - r^2}σz​=σy​1−r2​ and m=rσy/σxm = r\sigma_y/\sigma_xm=rσy​/σx​ , then simplify the exponent. Over the common factor −12(1−r2)-\frac{1}{2(1 - r^2)}−2(1−r2)1​ , the exponent has (1−r2)x2σx2\frac{(1 - r^2)x^2}{\sigma_x^2}σx2​(1−r2)x2​ and r2x2σx2\frac{r^2x^2}{\sigma_x^2}σx2​r2x2​ from (y−mx)2(y - mx)^2(y−mx)2 ; the r2x2/σx2r^2x^2/\sigma_x^2r2x2/σx2​ terms cancel, leaving x2σx2+y2σy2−2rxyσxσy\frac{x^2}{\sigma_x^2} + \frac{y^2}{\sigma_y^2} - \frac{2rxy…

63. [Calculating ρ(x, y)] ρ(x,y)=12πσxσy1−r2exp⁡(−12(1−r2)(x2σx2+y2σy2−2rxyσxσy))\rho(x, y) = \dfrac{1}{2\pi\sigma_x\sigma_y\sqrt{1 - r^2}}\exp\left(-\dfrac{1}{2(1 - r^2)}\left(\dfrac{x^2}{\sigma_x^2} + \dfrac{y^2}{\sigma_y^2} - \dfrac{2rxy}{\sigma_x\sigma_y}\right)\right)ρ(x,y)=2πσx​σy​1−r2​1​exp(−2(1−r2)1​(σx2​x2​+σy2​y2​−σx​σy​2rxy​)) Why is the double integral of ρ(x,y)\rho(x, y)ρ(x,y) over the plane equal to 1?

    Completing the square in yyy turns it back into a product of a Gaussian in xxx and a Gaussian in y−mxy - mxy−mx , and each of these integrates to 1. Integrate over yyy first.

64. [Calculating ρ(x, y)] ρ(x,y)=12πσxσy1−r2exp⁡(−12(1−r2)(x2σx2+y2σy2−2rxyσxσy))\rho(x, y) = \dfrac{1}{2\pi\sigma_x\sigma_y\sqrt{1 - r^2}}\exp\left(-\dfrac{1}{2(1 - r^2)}\left(\dfrac{x^2}{\sigma_x^2} + \dfrac{y^2}{\sigma_y^2} - \dfrac{2rxy}{\sigma_x\sigma_y}\right)\right)ρ(x,y)=2πσx​σy​1−r2​1​exp(−2(1−r2)1​(σx2​x2​+σy2​y2​−σx​σy​2rxy​)) Why are the curves of constant ρ(x,y)\rho(x, y)ρ(x,y) ellipses, and when are they tilted?

    The exponent has the form Ax2+By2+CxyAx^2 + By^2 + CxyAx2+By2+Cxy , and Ax2+By2+Cxy=DAx^2 + By^2 + Cxy = DAx2+By2+Cxy=D is an ellipse. It is tilted when C≠0C \neq 0C=0 , i.e. when r≠0r \neq 0r=0 . C∝rC \propto rC∝r . With C=0C = 0C=0 the axes of the ellipse are parallel to the coordinate axes.

65. [Calculating ρ(x, y)] ρ(x,y)=12πσxσy1−r2exp⁡(−12(1−r2)(x2σx2+y2σy2−2rxyσxσy))\rho(x, y) = \dfrac{1}{2\pi\sigma_x\sigma_y\sqrt{1 - r^2}}\exp\left(-\dfrac{1}{2(1 - r^2)}\left(\dfrac{x^2}{\sigma_x^2} + \dfrac{y^2}{\sigma_y^2} - \dfrac{2rxy}{\sigma_x\sigma_y}\right)\right)ρ(x,y)=2πσx​σy​1−r2​1​exp(−2(1−r2)1​(σx2​x2​+σy2​y2​−σx​σy​2rxy​)) What symmetry does ρ(x,y)\rho(x, y)ρ(x,y) have, and why is it not obvious from Y=mX+ZY = mX + ZY=mX+Z ?

    Symmetric in xxx and yyy : swapping x,σxx, \sigma_xx,σx​ with y,σyy, \sigma_yy,σy​ leaves ρ\rhoρ unchanged. Not obvious because ZZZ is independent of XXX , not of YYY , which makes the model look asymmetric. ( rrr is symmetric in xxx and yyy .)

66. [Calculating ρ(x, y)] Model: Y=mX+ZY = mX + ZY=mX+Z , where XXX and ZZZ are independent Gaussian random variables with means 0 and standard deviations σx\sigma_xσx​ and σz\sigma_zσz​ , and mmm is a number.How is Y=mX+ZY = mX + ZY=mX+Z written in terms of σx\sigma_xσx​ , σy\sigma_yσy​ and the correlation coefficient rrr ?

    Y=rσyσxX+Z\boldsymbol{Y = \dfrac{r\sigma_y}{\sigma_x}X + Z}Y=σx​rσy​​X+Z , where ZZZ is independent of XXX with standard deviation σy1−r2\boldsymbol{\sigma_y\sqrt{1 - r^2}}σy​1−r2​ . m=rσy/σxm = r\sigma_y/\sigma_xm=rσy​/σx​ and σz=σy1−r2\sigma_z = \sigma_y\sqrt{1 - r^2}σz​=σy​1−r2​ .

67. [Calculating ρ(x, y)] Model: Y=mX+ZY = mX + ZY=mX+Z , where XXX and ZZZ are independent Gaussian random variables with means 0 and standard deviations σx\sigma_xσx​ and σz\sigma_zσz​ , and mmm is a number.Equivalently, Y=rσyσxX+ZY = \dfrac{r\sigma_y}{\sigma_x}X + ZY=σx​rσy​​X+Z with ZZZ independent of XXX and of standard deviation σy1−r2\sigma_y\sqrt{1 - r^2}σy​1−r2​ .How can XXX be written in terms of YYY , and why?

    X=rσxσyY+Z′\boldsymbol{X = \dfrac{r\sigma_x}{\sigma_y}Y + Z&#x27;}X=σy​rσx​​Y+Z′ , where Z′Z&#x27;Z′ is independent of YYY with standard deviation σx1−r2\boldsymbol{\sigma_x\sqrt{1 - r^2}}σx​1−r2​ . The joint density ρ(x,y)\rho(x, y)ρ(x,y) is symmetric in xxx and yyy (swapping x,σxx, \sigma_xx,σx​ with y,σyy, \sigma_yy,σy​ ), so swapping xxx and yyy in the relation gives the same blob of points.

68. [Calculating ρ(x, y)] Model: Y=mX+ZY = mX + ZY=mX+Z , where XXX and ZZZ are independent Gaussian random variables with means 0 and standard deviations σx\sigma_xσx​ and σz\sigma_zσz​ , and mmm is a number.How much information does the joint density ρ(x,y)\rho(x, y)ρ(x,y) contain?

    All of it: exactly the same as " Y=mX+ZY = mX + ZY=mX+Z with Gaussian XXX , ZZZ of means 0 and standard deviations σx\sigma_xσx​ , σz\sigma_zσz​ ". Everything about the setup can be computed from ρ(x,y)\rho(x, y)ρ(x,y) .

69. [The standard-deviation box] Y=mX+ZY = mX + ZY=mX+Z , where XXX and ZZZ are independent Gaussians with mean 0 and standard deviations σx\sigma_xσx​ , σz\sigma_zσz​ ; YYY has standard deviation σy\sigma_yσy​ , and r=mσx/σyr = m\sigma_x/\sigma_yr=mσx​/σy​ is the correlation coefficient of XXX and YYY .On an ellipse of constant joint density ρ(x,y)\rho(x, y)ρ(x,y) , how are the two points with a given xxx placed relative to the line Y=mXY = mXY=mX ?

    Symmetrically: one the same distance above the line as the other is below. For a given XXX , Y=mX+ZY = mX + ZY=mX+Z and ZZZ is a Gaussian with mean 0, so YYY is symmetric around mXmXmX . The line Y=mXY = mXY=mX bisects every vertical span between two points of the ellipse.

70. [The standard-deviation box] Y=mX+ZY = mX + ZY=mX+Z , where XXX and ZZZ are independent Gaussians with mean 0 and standard deviations σx\sigma_xσx​ , σz\sigma_zσz​ ; YYY has standard deviation σy\sigma_yσy​ , and r=mσx/σyr = m\sigma_x/\sigma_yr=mσx​/σy​ is the correlation coefficient of XXX and YYY .At which points does the line Y=mXY = mXY=mX meet an ellipse of constant joint density ρ(x,y)\rho(x, y)ρ(x,y) ?

    At the ellipse's leftmost and rightmost points, where the ellipse's tangent is vertical. At an intersection point with a non-vertical tangent, the vertical distance to the ellipse would be zero on one side of the line and nonzero on the other, but the line bisects every vertical span.Example: σx=1.5σy\sigma_x = 1.5\sigma_yσx​=1.5σy​ and r=0.56r = 0.56r=0.56 give m=rσy/σx=0.56/1.5≈0.37m = r\sigma_y/\sigma_x = 0.56/1.5 \approx 0.37m=rσy​/σx​=0.56/1.5≈0.37 , the slope of the line through the leftm…

71. [The standard-deviation box] X,YX, YX,Y are zero-mean Gaussians with joint density ρ(x,y)\rho(x, y)ρ(x,y) . What is the e−1/2e^{-1/2}e−1/2 ellipse, and why this factor?

    The set of points with ρ(x,y)=e−1/2ρ(0,0)\rho(x, y) = e^{-1/2}\rho(0, 0)ρ(x,y)=e−1/2ρ(0,0) . It parallels one standard deviation of a single Gaussian: ρ(x)=e−x2/2σ2/2πσ\rho(x) = e^{-x^2/2\sigma^2}/\sqrt{2\pi}\sigmaρ(x)=e−x2/2σ2/2π​σ gives ρ(σ)=e−1/2ρ(0)\rho(\sigma) = e^{-1/2}\rho(0)ρ(σ)=e−1/2ρ(0) .

72. [The standard-deviation box] Y=mX+ZY = mX + ZY=mX+Z , where XXX and ZZZ are independent Gaussians with mean 0 and standard deviations σx\sigma_xσx​ , σz\sigma_zσz​ ; YYY has standard deviation σy\sigma_yσy​ , and r=mσx/σyr = m\sigma_x/\sigma_yr=mσx​/σy​ is the correlation coefficient of XXX and YYY . ρ(x,y)=12πσxσzexp⁡ ⁣(−x22σx2−(y−mx)22σz2)\rho(x, y) = \dfrac{1}{2\pi\sigma_x\sigma_z} \exp\!\Big(-\dfrac{x^2}{2\sigma_x^2} - \dfrac{(y - mx)^2}{2\sigma_z^2}\Big)ρ(x,y)=2πσx​σz​1​exp(−2σx2​x2​−2σz2​…

    x=σx\boldsymbol{x = \sigma_x}x=σx​ (and x=−σxx = -\sigma_xx=−σx​ at the leftmost point) The rightmost point lies on the line y=mxy = mxy=mx , so the second term of the exponent is 0. The exponent must be −1/2-1/2−1/2 , so x2/2σx2=1/2x^2/2\sigma_x^2 = 1/2x2/2σx2​=1/2 .

73. [The standard-deviation box] Y=mX+ZY = mX + ZY=mX+Z , where XXX and ZZZ are independent Gaussians with mean 0 and standard deviations σx\sigma_xσx​ , σz\sigma_zσz​ ; YYY has standard deviation σy\sigma_yσy​ , and r=mσx/σyr = m\sigma_x/\sigma_yr=mσx​/σy​ is the correlation coefficient of XXX and YYY .The ellipse ρ(x,y)=e−1/2ρ(0,0)\rho(x, y) = e^{-1/2}\rho(0, 0)ρ(x,y)=e−1/2ρ(0,0) reaches x=±σxx = \pm\sigma_xx=±σx​ . Why does it reach y=±σyy = \pm\sigma_yy=±σy​ at its highest and lowest points?

    ρ(x,y)\rho(x, y)ρ(x,y) is symmetric in xxx and yyy (with σx↔σy\sigma_x \leftrightarrow \sigma_yσx​↔σy​ ), so the same reasoning holds with xxx and yyy switched. ρ(x,y)=12πσxσy1−r2exp⁡ ⁣(−12(1−r2)(x2σx2+y2σy2−2rxyσxσy))\rho(x, y) = \dfrac{1}{2\pi\sigma_x\sigma_y\sqrt{1 - r^2}} \exp\!\Big(-\dfrac{1}{2(1 - r^2)}\Big(\dfrac{x^2}{\sigma_x^2} + \dfrac{y^2}{\sigma_y^2} - \dfrac{2rxy}{\sigma_x\sigma_y}\Big)\Big)ρ(x,y)=2πσx​σy​1−r2​1​exp(−2(1−r2)1​(σx2​x2​+σy2​y2​−σx​σy​2rxy​))

74. [The standard-deviation box] X,YX, YX,Y are zero-mean Gaussians with standard deviations σx,σy\sigma_x, \sigma_yσx​,σy​ . What is the standard-deviation box?

    The rectangle bounded by the lines x=±σxx = \pm\sigma_xx=±σx​ and y=±σyy = \pm\sigma_yy=±σy​ : the bounding box of the ellipse ρ(x,y)=e−1/2ρ(0,0)\rho(x, y) = e^{-1/2}\rho(0, 0)ρ(x,y)=e−1/2ρ(0,0) .

75. [The standard-deviation box] The standard-deviation box is the rectangle ∣x∣≤σx|x| \le \sigma_x∣x∣≤σx​ , ∣y∣≤σy|y| \le \sigma_y∣y∣≤σy​ ; it bounds the ellipse ρ(x,y)=e−1/2ρ(0,0)\rho(x, y) = e^{-1/2}\rho(0, 0)ρ(x,y)=e−1/2ρ(0,0) of the joint density of two zero-mean Gaussians X,YX, YX,Y with correlation coefficient rrr .Does the standard-deviation box depend on the correlation coefficient rrr ?

    No. It depends only on σx\sigma_xσx​ and σy\sigma_yσy​ ; rrr determines the shape of the ellipse inside the box. Same box ( σx=1.5σy\sigma_x = 1.5\sigma_yσx​=1.5σy​ ): thin ellipse r≈0.93r \approx 0.93r≈0.93 , wide ellipse r≈0.19r \approx 0.19r≈0.19 . The larger rrr gives the smaller σz=σy1−r2\sigma_z = \sigma_y\sqrt{1 - r^2}σz​=σy​1−r2​ .

76. [The regression lines] The standard-deviation box is the rectangle ∣x∣≤σx|x| \le \sigma_x∣x∣≤σx​ , ∣y∣≤σy|y| \le \sigma_y∣y∣≤σy​ ; it bounds the ellipse ρ(x,y)=e−1/2ρ(0,0)\rho(x, y) = e^{-1/2}\rho(0, 0)ρ(x,y)=e−1/2ρ(0,0) of the joint density of two zero-mean Gaussians X,YX, YX,Y with correlation coefficient rrr .Which four lines through the origin can be drawn for this ellipse, and how important is each?

    Lower regression line Through the ellipse's leftmost and rightmost points: Y=mXY = mXY=mX . Very important. Slope rσy/σxr\sigma_y/\sigma_xrσy​/σx​ . Upper regression line Through the ellipse's highest and lowest points. Very important. Slope σy/(rσx)\sigma_y/(r\sigma_x)σy​/(rσx​) . Standard-deviation line Through the corners of the standard-deviation box. Somewhat important. Slope σy/σx\sigma_y/\sigma_xσy​/σx​ . Major axis of the ellipse The ellipse's symmetry axis. Has little to do with probab…

77. [The regression lines] X,YX, YX,Y are zero-mean Gaussians with correlation coefficient rrr . How does the slope of the lower regression line relate to the slope σy/σx\sigma_y/\sigma_xσy​/σx​ of the standard-deviation line?

    It is rrr times as large: rσy/σx\boldsymbol{r\sigma_y/\sigma_x}rσy​/σx​ . The lower regression line is Y=mXY = mXY=mX , and Y=rσyσxX+ZY = \frac{r\sigma_y}{\sigma_x}X + ZY=σx​rσy​​X+Z with ZZZ independent of XXX and σz=σy1−r2\sigma_z = \sigma_y\sqrt{1 - r^2}σz​=σy​1−r2​ .Example: σx=σy\sigma_x = \sigma_yσx​=σy​ , r=0.6r = 0.6r=0.6 : standard-deviation line slope 1, lower regression line slope 0.6.

78. [The regression lines] X,YX, YX,Y are zero-mean Gaussians with correlation coefficient rrr . Besides Y=rσyσxX+ZY = \frac{r\sigma_y}{\sigma_x}X + ZY=σx​rσy​​X+Z , one can write X=rσxσyY+Z′X = \frac{r\sigma_x}{\sigma_y}Y + Z&#x27;X=σy​rσx​​Y+Z′ , with Z′Z&#x27;Z′ independent of YYY and σz′=σx1−r2\sigma_{z&#x27;} = \sigma_x\sqrt{1 - r^2}σz′​=σx​1−r2​ .What is the slope of the upper regression line, drawn in the usual xxx - yyy plane?

    σy/(rσx)\boldsymbol{\sigma_y/(r\sigma_x)}σy​/(rσx​) As XXX in terms of YYY the line is X=rσxσyYX = \frac{r\sigma_x}{\sigma_y}YX=σy​rσx​​Y : rrr times the slope σx/σy\sigma_x/\sigma_yσx​/σy​ of the standard-deviation line, read with YYY as the horizontal axis. Solving for YYY gives Y=σyrσxXY = \frac{\sigma_y}{r\sigma_x}XY=rσx​σy​​X .Example: σx=σy\sigma_x = \sigma_yσx​=σy​ , r=0.5r = 0.5r=0.5 : upper regression line slope 2.

79. [The regression lines] X,YX, YX,Y are zero-mean Gaussians with correlation coefficient rrr . How is the slope of the standard-deviation line related to the slopes of the two regression lines (as drawn in the xxx - yyy plane)?

    It is their geometric mean. rσyσx⋅σyrσx=σyσx\displaystyle\sqrt{\frac{r\sigma_y}{\sigma_x} \cdot \frac{\sigma_y}{r\sigma_x}} = \frac{\sigma_y}{\sigma_x}σx​rσy​​⋅rσx​σy​​​=σx​σy​​ Example: slopes 1 and 2 give a standard-deviation line of slope 2\sqrt{2}2​ .

80. [The regression lines] The standard-deviation box is the rectangle ∣x∣≤σx|x| \le \sigma_x∣x∣≤σx​ , ∣y∣≤σy|y| \le \sigma_y∣y∣≤σy​ ; it bounds the ellipse ρ(x,y)=e−1/2ρ(0,0)\rho(x, y) = e^{-1/2}\rho(0, 0)ρ(x,y)=e−1/2ρ(0,0) of the joint density of two zero-mean Gaussians X,YX, YX,Y with correlation coefficient rrr .Where does the ellipse touch the standard-deviation box, and how does this give rrr ?

    At (rσx,σy)(r\sigma_x, \sigma_y)(rσx​,σy​) and (σx,rσy)(\sigma_x, r\sigma_y)(σx​,rσy​) : the same fraction rrr of the way from each axis to the box's upper-right corner. Checks: r=1r = 1r=1 gives a thin ellipse touching the corners; r=0r = 0r=0 gives an ellipse with axes along the coordinate axes, touching the box at the middle of its sides.

81. [The regression lines] The standard-deviation box is the rectangle ∣x∣≤σx|x| \le \sigma_x∣x∣≤σx​ , ∣y∣≤σy|y| \le \sigma_y∣y∣≤σy​ ; it bounds the ellipse ρ(x,y)=e−1/2ρ(0,0)\rho(x, y) = e^{-1/2}\rho(0, 0)ρ(x,y)=e−1/2ρ(0,0) of the joint density of two zero-mean Gaussians X,YX, YX,Y with correlation coefficient rrr .Why does the ellipse touch the top side of the box at x=rσxx = r\sigma_xx=rσx​ ?

    The highest point lies on the upper regression line Y=σyrσxXY = \frac{\sigma_y}{r\sigma_x}XY=rσx​σy​​X , at height σy\sigma_yσy​ , so its xxx -value is rσxr\sigma_xrσx​ . Likewise the rightmost point lies on the lower regression line Y=rσyσxXY = \frac{r\sigma_y}{\sigma_x}XY=σx​rσy​​X at x=σxx = \sigma_xx=σx​ , so its yyy -value is rσyr\sigma_yrσy​ .

82. [The regression lines] The standard-deviation box is the rectangle ∣x∣≤σx|x| \le \sigma_x∣x∣≤σx​ , ∣y∣≤σy|y| \le \sigma_y∣y∣≤σy​ ; it bounds the ellipse ρ(x,y)=e−1/2ρ(0,0)\rho(x, y) = e^{-1/2}\rho(0, 0)ρ(x,y)=e−1/2ρ(0,0) of the joint density of two zero-mean Gaussians X,YX, YX,Y with correlation coefficient rrr .Where does the ellipse cross the coordinate axes?

    At y=±σy1−r2y = \pm\sigma_y\sqrt{1 - r^2}y=±σy​1−r2​ and x=±σx1−r2x = \pm\sigma_x\sqrt{1 - r^2}x=±σx​1−r2​ : the fraction 1−r2\sqrt{1 - r^2}1−r2​ of the way from the origin to the box's sides. At x=0x = 0x=0 the exponent of ρ(x,y)\rho(x, y)ρ(x,y) is −y22(1−r2)σy2-\dfrac{y^2}{2(1 - r^2)\sigma_y^2}−2(1−r2)σy2​y2​ ; setting it to −1/2-1/2−1/2 gives y=σy1−r2y = \sigma_y\sqrt{1 - r^2}y=σy​1−r2​ , which is σz\sigma_zσz​ in Y=mX+ZY = mX + ZY=mX+Z .Checks: r=1r = 1r=1 gives intercepts 0 (a thin ellipse…

83. [The regression lines] Why are the regression lines called the "lines of averages"?

    The lower one bisects every vertical span between two points of a constant- ρ\rhoρ ellipse, the upper one every horizontal span. The lower line gives the average YYY for each XXX , the upper line the average XXX for each YYY . Reason: in Y=rσyσxX+ZY = \frac{r\sigma_y}{\sigma_x}X + ZY=σx​rσy​​X+Z and X=rσxσyY+Z′X = \frac{r\sigma_x}{\sigma_y}Y + Z&#x27;X=σy​rσx​​Y+Z′ , the Gaussians ZZZ and Z′Z&#x27;Z′ are symmetric with mean 0.

84. [The regression lines] In a scatter plot of many points from two correlated zero-mean Gaussians X,YX, YX,Y , around which points are the points of a vertical strip and of a horizontal strip symmetrically distributed?

    Vertical strip: around its intersection with the lower regression line. Horizontal strip: around its intersection with the upper regression line. Intersections shown as white dots.

85. [The regression lines] Y=mX+ZY = mX + ZY=mX+Z , where XXX and ZZZ are independent Gaussians with mean 0 and standard deviations σx\sigma_xσx​ , σz\sigma_zσz​ ; YYY has standard deviation σy\sigma_yσy​ , and r=mσx/σyr = m\sigma_x/\sigma_yr=mσx​/σy​ is the correlation coefficient of XXX and YYY .In a scatter plot of (X,Y)(X, Y)(X,Y) , the points seem to spread less around the lower regression line near the plot's left and right edges. Why?

    There are fewer points there. The distribution of YYY around the line is the same Gaussian ZZZ for every vertical strip. ZZZ is independent of XXX , so the probability of a given ZZZ value does not depend on XXX .

86. [The regression lines] When is the lower regression line relevant, and when the upper one?

    Lower: when XXX is the independent variable. Upper: when YYY is. The lower line gives the average YYY for each XXX (from Y=rσyσxX+ZY = \frac{r\sigma_y}{\sigma_x}X + ZY=σx​rσy​​X+Z ); the upper line gives the average XXX for each YYY (from X=rσxσyY+Z′X = \frac{r\sigma_x}{\sigma_y}Y + Z&#x27;X=σy​rσx​​Y+Z′ ).

87. [The regression lines] Why can't a single line pass through the middle of both all vertical strips and all horizontal strips of a scatter plot of two correlated Gaussians?

    For r=0r = 0r=0 the lower regression line is the xxx -axis and the upper one is the yyy -axis: perpendicular, as different as two lines can be. The two regression lines coincide only for r=±1r = \pm1r=±1 ; for r≈1r \approx 1r≈1 they are nearly the same.

88. [The regression lines] In a scatter plot of two correlated Gaussians, the lower regression line differs from the symmetry axis of the elliptical cloud of points and looks "unbalanced". Is it wrong?

    No. What matters is that it cuts every vertical strip of points in half, which the symmetry axis does not.

89. [The regression lines] Given real data points (xi,yi)(x_i, y_i)(xi​,yi​) instead of known distributions, what are the slopes of the two regression lines?

    rs~y/s~x\boldsymbol{r\tilde{s}_y/\tilde{s}_x}rs~y​/s~x​ and s~y/(rs~x)\boldsymbol{\tilde{s}_y/(r\tilde{s}_x)}s~y​/(rs~x​) , with the data's standard deviations s~x,s~y\tilde{s}_x, \tilde{s}_ys~x​,s~y​ and correlation coefficient rrr in place of σx,σy\sigma_x, \sigma_yσx​,σy​ .

90. [The regression lines] Two correlated Gaussians X,YX, YX,Y have means μx,μy\mu_x, \mu_yμx​,μy​ that are not 0. Where do the two regression lines intersect?

    At (μx,μy)(\mu_x, \mu_y)(μx​,μy​) , the middle of the cloud of points. Equivalently, use X′=X−μxX&#x27; = X - \mu_xX′=X−μx​ and Y′=Y−μyY&#x27; = Y - \mu_yY′=Y−μy​ : the point (μx,μy)(\mu_x, \mu_y)(μx​,μy​) becomes the origin of the X′X&#x27;X′ - Y′Y&#x27;Y′ plane.

91. [Two regression examples] Model of test scores: Y=X+ZY = X + ZY=X+Z , where YYY is a person's score, XXX the person's innate ability (average score over many tests), ZZZ an independent random contribution with mean 0. All measured relative to the mean 100.Why does the random contribution ZZZ have mean 0?

    Because innate ability x0x_0x0​ is defined as the person's average score: averaging Y=x0+ZY = x_0 + ZY=x0​+Z over many tests gives μy=x0+μz=x0\mu_y = x_0 + \mu_z = x_0μy​=x0​+μz​=x0​ , so μz=0\mu_z = 0μz​=0 .

92. [Two regression examples] IQ test scores have mean 100 and standard deviation 15. A score is innate ability plus an independent random contribution with mean 0.The people who scored 130 take another equivalent test. Is their average score higher than, lower than or equal to 130?

    Lower than 130. With the model's parameters ( σz=σx\sigma_z = \sigma_xσz​=σx​ , so r=1/2r = 1/\sqrt{2}r=1/2​ ) the average is 115.

93. [Two regression examples] Model of test scores: Y=X+ZY = X + ZY=X+Z , where YYY is a person's score, XXX the person's innate ability (average score over many tests), ZZZ an independent random contribution with mean 0. All measured relative to the mean 100.The score has σy=15\sigma_y = 15σy​=15 and the model assumes σz=σx\sigma_z = \sigma_xσz​=σx​ . What are σx\sigma_xσx​ , the correlation coefficient rrr , and the slopes of the two regression lines?

    σx=15/2≈10.6\boldsymbol{\sigma_x = 15/\sqrt{2} \approx 10.6}σx​=15/2​≈10.6 , r=1/2≈0.71\boldsymbol{r = 1/\sqrt{2} \approx 0.71}r=1/2​≈0.71 , slopes 1 and 2 σy2=σx2+σz2=2σx2\sigma_y^2 = \sigma_x^2 + \sigma_z^2 = 2\sigma_x^2σy2​=σx2​+σz2​=2σx2​ , so r=mσx/σy=1/2r = m\sigma_x/\sigma_y = 1/\sqrt{2}r=mσx​/σy​=1/2​ . Lower line: slope m=1m = 1m=1 ; upper line: σy/(rσx)=2/(1/2)=2\sigma_y/(r\sigma_x) = \sqrt{2}/(1/\sqrt{2}) = 2σy​/(rσx​)=2​/(1/2​)=2 .

94. [Two regression examples] Model of test scores: Y=X+ZY = X + ZY=X+Z , where YYY is a person's score, XXX the person's innate ability (average score over many tests), ZZZ an independent random contribution with mean 0. All measured relative to the mean 100.Here the lower regression line is Y=XY = XY=X and the upper one is Y=2XY = 2XY=2X .How do you find the average second-test score of the people who scored 130 (30 above the mean)?

    Upper line at y=30y = 30y=30 gives their average ability xavg=15x_{\text{avg}} = 15xavg​=15 ; the lower line at x=15x = 15x=15 gives the second-test average 151515 , i.e. 115. On the second test the ZZZ values average to 0, so yavg=m xavgy_{\text{avg}} = m\,x_{\text{avg}}yavg​=mxavg​ : the height of the lower regression line at xavgx_{\text{avg}}xavg​ (white dot).

95. [Two regression examples] Why, intuitively, do people who scored 130 (above the mean of 100) score lower on average when retaking an equivalent test?

    More people have ability 130−a130 - a130−a than 130+a130 + a130+a , so a score of 130 more often comes from lower ability plus good luck than from higher ability plus bad luck. This holds because a Gaussian peaks at its mean and falls off on either side. Example: ability 120 with luck +10+10+10 is more common than ability 140 with luck −10-10−10 .

96. [Two regression examples] Model of test scores: Y=X+ZY = X + ZY=X+Z , where YYY is a person's score, XXX the person's innate ability (average score over many tests), ZZZ an independent random contribution with mean 0. All measured relative to the mean 100. σz=σx\sigma_z = \sigma_xσz​=σx​ , so people scoring 130 average 115 on a second test.What do the people who scored 70 average on a second test?

    85: higher, toward the mean. Here more of the 70-scorers have ability above 70 and were unlucky than the other way around.

97. [Two regression examples] The effect that the average second-test score of a group is closer to the mean than its first-test score is called regression toward the mean.

    Example: in an IQ model with r2=1/2r^2 = 1/2r2=1/2 , people scoring 130 average 115 on a second test, people scoring 70 average 85.

98. [Two regression examples] Theorem 6.1 People score y1y_1y1​ points above the mean on a test. On average, how many points above the mean do they score when they take the test (or an equivalent one) again?

    yavg=r2y1\boldsymbol{y_{\text{avg}} = r^2 y_1}yavg​=r2y1​ , where rrr is the correlation coefficient between score and innate ability. Example: r=1/2r = 1/\sqrt{2}r=1/2​ , y1=30y_1 = 30y1​=30 gives yavg=15y_{\text{avg}} = 15yavg​=15 .

99. [Two regression examples] Model: score Y=mX+ZY = mX + ZY=mX+Z with innate ability XXX and independent random ZZZ of mean 0; correlation coefficient rrr .How does the proof of yavg=r2y1y_{\text{avg}} = r^2 y_1yavg​=r2y1​ (the average retest score of people who scored y1y_1y1​ ) go?

    Upper regression line: their average ability is xavg=rσxσyy1x_{\text{avg}} = \frac{r\sigma_x}{\sigma_y}y_1xavg​=σy​rσx​​y1​ . Lower regression line: yavg=m xavgy_{\text{avg}} = m\,x_{\text{avg}}yavg​=mxavg​ . yavg=rσyσx⋅rσxσy y1=r2y1y_{\text{avg}} = \dfrac{r\sigma_y}{\sigma_x} \cdot \dfrac{r\sigma_x}{\sigma_y}\,y_1 = r^2 y_1yavg​=σx​rσy​​⋅σy​rσx​​y1​=r2y1​ , using m=rσy/σxm = r\sigma_y/\sigma_xm=rσy​/σx​ .

100. [Two regression examples] State yavg=r2y1y_{\text{avg}} = r^2 y_1yavg​=r2y1​ for a general data set from Y=mX+ZY = mX + ZY=mX+Z , without tests.

     Among the points with Y=y1Y = y_1Y=y1​ , let xavgx_{\text{avg}}xavg​ be their average XXX . Then the average YYY at X=xavgX = x_{\text{avg}}X=xavg​ is r2y1r^2 y_1r2y1​ .