Ch.3: Expectation values

Chapter 3 of the book "Probability for the Enthusiastic Beginner" by David Morin (2016) Expectation value Variance Standard deviation Standard deviation of the mean Sample variance Problems

123 flashcards · Mathematics · by lakesare

Part of Textbooks › Probability for the Enthusiastic Beginner (David Morin) · 7 courses

1. [Expectation value] What is a random variable?

   A variable that takes numerical values, each with a certain probability. Written with an uppercase letter XXX ; its possible values with lowercase letters x1,x2,…x_1, x_2, \dotsx1​,x2​,… Example: XXX = number of Heads in two coin tosses, with values x1=0x_1 = 0x1​=0 , x2=1x_2 = 1x2​=1 , x3=2x_3 = 2x3​=2 .

2. [Expectation value] Why are the outcomes Heads and Tails of a coin toss not a random variable, and how do you make one from them?

   The outcomes of a random variable must be numbers; assign numbers to them, e.g. 1 to Heads and 0 to Tails.

3. [Expectation value] What is the probability distribution of a random variable XXX ?

   The collection of the probabilities P(xi)P(x_i)P(xi​) of all its values xix_ixi​ , i.e. how the total probability 1 is distributed among the outcomes. Example: XXX = number of Heads in two tosses: P(0)=14P(0) = \frac{1}{4}P(0)=41​ , P(1)=12P(1) = \frac{1}{2}P(1)=21​ , P(2)=14P(2) = \frac{1}{4}P(2)=41​ , since HH, HT, TH, TT are equally likely.

4. [Expectation value] What is the expectation value (expected value) of a random variable XXX , and how is it written?

   The average value of XXX expected in a large number of trials; written E(X)≡μX≡μE(X) \equiv \mu_X \equiv \muE(X)≡μX​≡μ . The subscript XXX in μX\mu_XμX​ is dropped when only one random variable is involved.Example: a die roll has E(X)=(1+2+3+4+5+6)/6=3.5E(X) = (1 + 2 + 3 + 4 + 5 + 6)/6 = 3.5E(X)=(1+2+3+4+5+6)/6=3.5 .

5. [Expectation value] How do the words average, expectation value and mean differ in use?

   average For trials that have already taken place. Example: the average number of points per game a player scored last season was 14. expectation value For trials yet to be carried out. Example: the expectation value of the number of Heads in 10 coin tosses is 5. mean For either past or future trials.

6. [Expectation value] The expectation value of a die roll is 3.5, yet no roll gives 3.5. What does E(X)=3.5E(X) = 3.5E(X)=3.5 say?

   Over many rolls the average is about 3.5: e.g. 1000 rolls add up to about 3500. A total of exactly 3500 is very unlikely, but that doesn't matter for the expectation value.

7. [Expectation value] Is the expected value of XXX the value that is most likely to occur?

   No. It is the average of the outcomes, which need not be likely or even possible. Example: four equally likely outcomes 1,2,2,71, 2, 2, 71,2,2,7 : the most likely value is 2, the expected value is (1+2+2+7)/4=3(1 + 2 + 2 + 7)/4 = 3(1+2+2+7)/4=3 , which never occurs.

8. [Expectation value] You draw a card from a deck and look only at its suit. Why does "the expected value of the suit" make no sense, and how can you make it make sense?

   A suit is not a number, so the suits can't be averaged. Assign suit values 1 to 4: their expected value is 2.5. With numbers assigned, the suit value is a random variable.

9. [Expectation value] A box has three balls: two labeled 1 and one labeled 4. You pick one; XXX is its number. How does imagining nnn trials give E(X)E(X)E(X) ?

   About 23n\frac{2}{3}n32​n picks give 1 and 13n\frac{1}{3}n31​n give 4, so E(X)=23n⋅1+13n⋅4n=23⋅1+13⋅4=2E(X) = \frac{\frac{2}{3}n \cdot 1 + \frac{1}{3}n \cdot 4}{n} = \frac{2}{3} \cdot 1 + \frac{1}{3} \cdot 4 = 2E(X)=n32​n⋅1+31​n⋅4​=32​⋅1+31​⋅4=2 . The nnn cancels, as it should: the expected average can't depend on the number of hypothetical trials.

10. [Expectation value] XXX has outcomes x1,…,xmx_1, \dots, x_mx1​,…,xm​ with probabilities p1,…,pmp_1, \dots, p_mp1​,…,pm​ . E(X)= ?E(X) = \; ?E(X)=?

    E(X)=p1x1+p2x2+⋯+pmxm\boldsymbol{E(X) = p_1x_1 + p_2x_2 + \dots + p_mx_m}E(X)=p1​x1​+p2​x2​+⋯+pm​xm​ Called the weighted average of the outcomes: each outcome is multiplied by its probability, so outcomes that occur more often influence the average more. For a continuous random variable the sum becomes an integral.Example: outcomes 1 and 4 with probabilities 23,13\frac{2}{3}, \frac{1}{3}32​,31​ : E(X)=23⋅1+13⋅4=2E(X) = \frac{2}{3} \cdot 1 + \frac{1}{3} \cdot 4 = 2E(X)=32​⋅1+31​⋅4=2 .

11. [Expectation value] You flip a coin four times. What is the expected number of Heads, computed with E(X)=∑pixiE(X) = \sum p_ix_iE(X)=∑pi​xi​ ?

    2\boldsymbol{2}2 : 116⋅0+416⋅1+616⋅2+416⋅3+116⋅4=3216=2\frac{1}{16} \cdot 0 + \frac{4}{16} \cdot 1 + \frac{6}{16} \cdot 2 + \frac{4}{16} \cdot 3 + \frac{1}{16} \cdot 4 = \frac{32}{16} = 2161​⋅0+164​⋅1+166​⋅2+164​⋅3+161​⋅4=1632​=2 The probabilities of 0 to 4 Heads are (4k)/16\binom{4}{k}/16(k4​)/16 . Without work: half of the coins are Heads on average.

12. [Expectation value] You flip a coin until you get Heads. What is the expected total number of flips?

    2\boldsymbol{2}2 kkk flips (Tails k−1k - 1k−1 times, then Heads) has probability 12k\frac{1}{2^k}2k1​ , so the expected number is 12⋅1+14⋅2+18⋅3+116⋅4+⋯=2\frac{1}{2} \cdot 1 + \frac{1}{4} \cdot 2 + \frac{1}{8} \cdot 3 + \frac{1}{16} \cdot 4 + \dots = 221​⋅1+41​⋅2+81​⋅3+161​⋅4+⋯=2 .

13. [Expectation value] Theorem 3.1. For two random variables XXX and YYY : E(X+Y)= ?E(X + Y) = \; ?E(X+Y)=? Does this need XXX and YYY to be independent?

    E(X+Y)=E(X)+E(Y)\boldsymbol{E(X + Y) = E(X) + E(Y)}E(X+Y)=E(X)+E(Y) , and no, independence is not needed.

14. [Expectation value] E(X+Y)=E(X)+E(Y)E(X + Y) = E(X) + E(Y)E(X+Y)=E(X)+E(Y) for any random variables XXX and YYY .How does the proof go?

    Over n→∞n \to \inftyn→∞ trials with values xi,yix_i, y_ixi​,yi​ : E(X+Y)=1n∑(xi+yi)=1n∑xi+1n∑yi=E(X)+E(Y)E(X + Y) = \frac{1}{n}\sum (x_i + y_i) = \frac{1}{n}\sum x_i + \frac{1}{n}\sum y_i = E(X) + E(Y)E(X+Y)=n1​∑(xi​+yi​)=n1​∑xi​+n1​∑yi​=E(X)+E(Y) . XXX contributes E(X)E(X)E(X) to the average and YYY contributes E(Y)E(Y)E(Y) . No step uses independence.

15. [Expectation value] When are two random variables XXX and YYY independent?

    When P(x∣y)=P(x)P(x|y) = P(x)P(x∣y)=P(x) for all values xxx of XXX and yyy of YYY , or equivalently P(x,y)=P(x)P(y)P(x, y) = P(x)P(y)P(x,y)=P(x)P(y) for all x,yx, yx,y . The value of one variable doesn't affect the probability distribution of the other; formally P(X=x and Y=y)=P(X=x)⋅P(Y=y)P(X = x \text{ and } Y = y) = P(X = x) \cdot P(Y = y)P(X=x and Y=y)=P(X=x)⋅P(Y=y) . Then any event about XXX is independent of any event about YYY .Example: the results of rolling two dice. If the left die sh…

16. [Expectation value] XXX is 1 or 2 with probability 12\frac{1}{2}21​ each, and an independent YYY is 1, 2 or 3 with probability 13\frac{1}{3}31​ each. What is the distribution of X+YX + YX+Y and its expectation value?

    P(2)=16, P(3)=26, P(4)=26, P(5)=16\boldsymbol{P(2) = \frac{1}{6}, \ P(3) = \frac{2}{6}, \ P(4) = \frac{2}{6}, \ P(5) = \frac{1}{6}}P(2)=61​, P(3)=62​, P(4)=62​, P(5)=61​ , so E(X+Y)=216=3.5\boldsymbol{E(X + Y) = \frac{21}{6} = 3.5}E(X+Y)=621​=3.5 Each of the 6 pairs (x,y)(x, y)(x,y) has probability 12⋅13=16\frac{1}{2} \cdot \frac{1}{3} = \frac{1}{6}21​⋅31​=61​ by independence. And indeed E(X)+E(Y)=1.5+2=3.5E(X) + E(Y) = 1.5 + 2 = 3.5E(X)+E(Y)=1.5+2=3.5 .

17. [Expectation value] XXX and YYY are dependent, and you aren't told how. Can you find E(X+Y)E(X + Y)E(X+Y) from the distributions of XXX and of YYY ?

    Yes: E(X+Y)=E(X)+E(Y)E(X + Y) = E(X) + E(Y)E(X+Y)=E(X)+E(Y) still holds. What you can't do is find the probabilities of the values of X+YX + YX+Y and use E=∑pixiE = \sum p_ix_iE=∑pi​xi​ directly: they depend on the dependence.

18. [Expectation value] For random variables X1,…,XnX_1, \dots, X_nX1​,…,Xn​ and constants a1,…,an,ca_1, \dots, a_n, ca1​,…,an​,c : E(a1X1+⋯+anXn+c)= ?E(a_1X_1 + \dots + a_nX_n + c) = \; ?E(a1​X1​+⋯+an​Xn​+c)=?

    a1E(X1)+⋯+anE(Xn)+c\boldsymbol{a_1E(X_1) + \dots + a_nE(X_n) + c}a1​E(X1​)+⋯+an​E(Xn​)+c Same reasoning as for E(X+Y)=E(X)+E(Y)E(X + Y) = E(X) + E(Y)E(X+Y)=E(X)+E(Y) ; independence is again not needed.Example: XXX a die roll: E(2X+1)=2⋅3.5+1=8E(2X + 1) = 2 \cdot 3.5 + 1 = 8E(2X+1)=2⋅3.5+1=8 .

19. [Expectation value] When are random variables X1,…,XnX_1, \dots, X_nX1​,…,Xn​ identically distributed, and what is then E(X1+⋯+Xn)E(X_1 + \dots + X_n)E(X1​+⋯+Xn​) ?

    When they all have the same probability distribution; then E(X1+⋯+Xn)=nE(X)\boldsymbol{E(X_1 + \dots + X_n) = nE(X)}E(X1​+⋯+Xn​)=nE(X) , with XXX any one of the XiX_iXi​ . They arise from nnn trials of the same process, e.g. nnn rolls of a die.Example: XXX = 1 for Heads, 0 for Tails, E(X)=12E(X) = \frac{1}{2}E(X)=21​ . The expected number of Heads in 100 flips is 100⋅12=50100 \cdot \frac{1}{2} = 50100⋅21​=50 .

20. [Expectation value] X1,…,XnX_1, \dots, X_nX1​,…,Xn​ are identically distributed, each distributed like XXX . Is the random variable X1+⋯+XnX_1 + \dots + X_nX1​+⋯+Xn​ the same as nXnXnX ?

    No. nXnXnX picks one value and multiplies it by nnn ; X1+⋯+XnX_1 + \dots + X_nX1​+⋯+Xn​ picks nnn values (generally different) and adds them. They have the same expectation value nE(X)nE(X)nE(X) , but different spreads around it.Example: ten dice: 10X10X10X is a multiple of 10 from 10 to 60, while X1+⋯+X10X_1 + \dots + X_{10}X1​+⋯+X10​ can be any integer from 10 to 60.

21. [Expectation value] Is E(1/X)=1/E(X)E(1/X) = 1/E(X)E(1/X)=1/E(X) ? Is E(X2)=(E(X))2E(X^2) = (E(X))^2E(X2)=(E(X))2 ?

    No, neither. E(aX+bY+c)=aE(X)+bE(Y)+cE(aX + bY + c) = aE(X) + bE(Y) + cE(aX+bY+c)=aE(X)+bE(Y)+c holds only for linear combinations. Example: XXX a die roll: E(1/X)=16(1+12+⋯+16)≈0.41E(1/X) = \frac{1}{6}(1 + \frac{1}{2} + \dots + \frac{1}{6}) \approx 0.41E(1/X)=61​(1+21​+⋯+61​)≈0.41 , but 1/E(X)=1/3.5≈0.291/E(X) = 1/3.5 \approx 0.291/E(X)=1/3.5≈0.29 . E(X2)=16(12+⋯+62)=916≈15.17E(X^2) = \frac{1}{6}(1^2 + \dots + 6^2) = \frac{91}{6} \approx 15.17E(X2)=61​(12+⋯+62)=691​≈15.17 , but (E(X))2=3.52=12…

22. [Expectation value] Theorem 3.2. For random variables XXX and YYY : when does E(XY)=E(X)⋅E(Y)E(XY) = E(X) \cdot E(Y)E(XY)=E(X)⋅E(Y) hold?

    When XXX and YYY are independent. Unlike E(X+Y)=E(X)+E(Y)E(X + Y) = E(X) + E(Y)E(X+Y)=E(X)+E(Y) , which holds always.

23. [Expectation value] E(XY)=E(X)⋅E(Y)E(XY) = E(X) \cdot E(Y)E(XY)=E(X)⋅E(Y) for independent random variables XXX and YYY .How does the proof go, and where is independence used?

    E(XY)=∑i∑jP(xi,yj) xiyj=∑i∑jP(xi)P(yj) xiyj=(∑iP(xi)xi)(∑jP(yj)yj)E(XY) = \sum_i \sum_j P(x_i, y_j) \, x_iy_j = \sum_i \sum_j P(x_i)P(y_j) \, x_iy_j = \big(\sum_i P(x_i)x_i\big)\big(\sum_j P(y_j)y_j\big)E(XY)=∑i​∑j​P(xi​,yj​)xi​yj​=∑i​∑j​P(xi​)P(yj​)xi​yj​=(∑i​P(xi​)xi​)(∑j​P(yj​)yj​) . Independence gives P(xi,yj)=P(xi)P(yj)P(x_i, y_j) = P(x_i)P(y_j)P(xi​,yj​)=P(xi​)P(yj​) , which lets the double sum split into a product of two sums.

24. [Expectation value] XXX is a coin flip with Heads = 2, Tails = 1, and YYY a second coin flip with Heads = 4, Tails = 3. What is E(XY)E(XY)E(XY) if the coins are independent, and if the second coin always shows the same side as the first?

    Independent: E(XY)=214=E(X)E(Y)E(XY) = \frac{21}{4} = E(X)E(Y)E(XY)=421​=E(X)E(Y) . Same side: E(XY)=112≠E(X)E(Y)E(XY) = \frac{11}{2} \ne E(X)E(Y)E(XY)=211​=E(X)E(Y) . E(X)=32E(X) = \frac{3}{2}E(X)=23​ , E(Y)=72E(Y) = \frac{7}{2}E(Y)=27​ .Independent: four equally likely products 8,6,4,38, 6, 4, 38,6,4,3 , average 214\frac{21}{4}421​ .Same side: two equally likely products 2⋅4=82 \cdot 4 = 82⋅4=8 and 1⋅3=31 \cdot 3 = 31⋅3=3 , average 112\frac{11}{2}211​ .

25. [Expectation value] In a game, you roll a die and win the number rolled in dollars. What fee per game makes it a fair game?

    3.50 dollars, the expectation value of the winnings. In a fair game neither side wins money on average. Paying more, you lose on average; charging less, the house loses on average. Casinos charge more than the expectation value, which is why they stay in business.

26. [Expectation value] Should you always play a game whose fee is below the expectation value of the winnings, and never one whose fee exceeds it?

    No, it depends on the circumstances. "Fee = expectation value" is the right price only if the goal is a fair game; real-life goals can override that goal.

27. [Variance] What is the variance Var⁡(X)\operatorname{Var}(X)Var(X) of a random variable XXX with mean μ\muμ ?

    Var⁡(X)=E[(X−μ)2]\boldsymbol{\operatorname{Var}(X) = E\big[(X - \mu)^2\big]}Var(X)=E[(X−μ)2] : the expectation value of the square of the difference between XXX and its mean. Also written σX2\sigma_X^2σX2​ or σ2\sigma^2σ2 . "Variance of a random variable" and "variance of a probability distribution" mean the same.Example: fair coin, Heads = 1, Tails = 0, μ=12\mu = \frac{1}{2}μ=21​ : Var⁡(X)=12[(1−12)2+(0−12)2]=14\operatorname{Var}(X) = \frac{1}{2}\big[(1 - \tfrac{1}{2})^2 + (0 - \tfrac{1}{2})^2…

28. [Variance] What are the steps to find the variance of a random variable?

    1. Find the mean. 2. Find all differences from the mean. 3. Square them. 4. Take the expectation value of the squares. E[(X−μ)2]E\big[(X - \mu)^2\big]E[(X−μ)2] is computed with E=∑pixiE = \sum p_ix_iE=∑pi​xi​ , the values xix_ixi​ replaced by (xi−μ)2(x_i - \mu)^2(xi​−μ)2 .

29. [Variance] What is the variance of a die roll?

    ≈2.92\boldsymbol{\approx 2.92}≈2.92 (exactly 3512\frac{35}{12}1235​ ) μ=3.5\mu = 3.5μ=3.5 : 16[6.25+2.25+0.25+0.25+2.25+6.25]=17.56≈2.92\frac{1}{6}\big[6.25 + 2.25 + 0.25 + 0.25 + 2.25 + 6.25\big] = \frac{17.5}{6} \approx 2.9261​[6.25+2.25+0.25+0.25+2.25+6.25]=617.5​≈2.92 .

30. [Variance] A biased coin gives Heads (value 1) with probability ppp and Tails (value 0) with probability q=1−pq = 1 - pq=1−p . What are its mean and variance?

    μ=p\boldsymbol{\mu = p}μ=p , Var⁡(X)=pq\boldsymbol{\operatorname{Var}(X) = pq}Var(X)=pq Var⁡(X)=p(1−p)2+(1−p)(0−p)2=p(1−p)[(1−p)+p]=p(1−p)\operatorname{Var}(X) = p(1 - p)^2 + (1 - p)(0 - p)^2 = p(1 - p)\big[(1 - p) + p\big] = p(1 - p)Var(X)=p(1−p)2+(1−p)(0−p)2=p(1−p)[(1−p)+p]=p(1−p) .Example: p=12p = \frac{1}{2}p=21​ gives the fair coin's 14\frac{1}{4}41​ .

31. [Variance] In Var⁡(X)=E[(X−μ)2]\operatorname{Var}(X) = E\big[(X - \mu)^2\big]Var(X)=E[(X−μ)2] , why must you first square the differences from the mean and then take the expectation value, not the other way round?

    The other order always gives 0: (E(X−μ))2=(E(X)−μ)2=(μ−μ)2=0\big(E(X - \mu)\big)^2 = \big(E(X) - \mu\big)^2 = (\mu - \mu)^2 = 0(E(X−μ))2=(E(X)−μ)2=(μ−μ)2=0 .

32. [Variance] You relabel the faces of a die as 101 to 106. What happens to the mean and to the variance?

    The mean becomes 103.5; the variance stays 2.92. The variance depends only on the spread of the outcomes around the mean: all differences from the mean stay the same. The same holds for the standard deviation, which stays 2.92=1.71\sqrt{2.92} = 1.712.92​=1.71 .

33. [Variance] aaa is a constant. Var⁡(aX)= ?\operatorname{Var}(aX) = \; ?Var(aX)=? Why?

    a2Var⁡(X)\boldsymbol{a^2\operatorname{Var}(X)}a2Var(X) E(aX)=aμE(aX) = a\muE(aX)=aμ , so Var⁡(aX)=E[(aX−aμ)2]=E[a2(X−μ)2]=a2E[(X−μ)2]\operatorname{Var}(aX) = E\big[(aX - a\mu)^2\big] = E\big[a^2(X - \mu)^2\big] = a^2E\big[(X - \mu)^2\big]Var(aX)=E[(aX−aμ)2]=E[a2(X−μ)2]=a2E[(X−μ)2] .Example: fair coin with Heads = 2, Tails = 0: variance 22⋅14=12^2 \cdot \frac{1}{4} = 122⋅41​=1 .

34. [Variance] Theorem 3.3. XXX and YYY are independent random variables. Var⁡(X+Y)= ?\operatorname{Var}(X + Y) = \; ?Var(X+Y)=?

    Var⁡(X)+Var⁡(Y)\boldsymbol{\operatorname{Var}(X) + \operatorname{Var}(Y)}Var(X)+Var(Y) Not obvious, since XXX enters E[(X−μ)2]E\big[(X - \mu)^2\big]E[(X−μ)2] nonlinearly. Fails for dependent XXX and YYY .

35. [Variance] Var⁡(X+Y)=Var⁡(X)+Var⁡(Y)\operatorname{Var}(X + Y) = \operatorname{Var}(X) + \operatorname{Var}(Y)Var(X+Y)=Var(X)+Var(Y) for independent random variables XXX and YYY .How does the proof go?

    Expand E[((X−μX)+(Y−μY))2]E\big[\big((X - \mu_X) + (Y - \mu_Y)\big)^2\big]E[((X−μX​)+(Y−μY​))2] ; the cross term 2E[(X−μX)(Y−μY)]2E\big[(X - \mu_X)(Y - \mu_Y)\big]2E[(X−μX​)(Y−μY​)] is 0. The mean of X+YX + YX+Y is μX+μY\mu_X + \mu_YμX​+μY​ . By independence E[(X−μX)(Y−μY)]=E(X−μX)⋅E(Y−μY)=0⋅0E\big[(X - \mu_X)(Y - \mu_Y)\big] = E(X - \mu_X) \cdot E(Y - \mu_Y) = 0 \cdot 0E[(X−μX​)(Y−μY​)]=E(X−μX​)⋅E(Y−μY​)=0⋅0 .

36. [Variance] XXX and YYY are two coin flips (Heads = 1, Tails = 0), each with variance 14\frac{1}{4}41​ . What is Var⁡(X+Y)\operatorname{Var}(X + Y)Var(X+Y) if the coins are independent, and if the second always shows the same side as the first?

    Independent: 12=Var⁡(X)+Var⁡(Y)\frac{1}{2} = \operatorname{Var}(X) + \operatorname{Var}(Y)21​=Var(X)+Var(Y) . Same side: 1≠Var⁡(X)+Var⁡(Y)1 \ne \operatorname{Var}(X) + \operatorname{Var}(Y)1=Var(X)+Var(Y) . Independent: X+YX + YX+Y is 0, 1, 2 with probabilities 14,12,14\frac{1}{4}, \frac{1}{2}, \frac{1}{4}41​,21​,41​ and mean 1: 14⋅1+12⋅0+14⋅1=12\frac{1}{4} \cdot 1 + \frac{1}{2} \cdot 0 + \frac{1}{4} \cdot 1 = \frac{1}{2}41​⋅1+21​⋅0+41​⋅1=21​ .Same side: X+YX + YX+Y is 0 or 2, each with probab…

37. [Variance] X1,…,XnX_1, \dots, X_nX1​,…,Xn​ are independent. Var⁡(X1+⋯+Xn)= ?\operatorname{Var}(X_1 + \dots + X_n) = \; ?Var(X1​+⋯+Xn​)=? How does it follow from the case of two variables?

    Var⁡(X1)+⋯+Var⁡(Xn)\boldsymbol{\operatorname{Var}(X_1) + \dots + \operatorname{Var}(X_n)}Var(X1​)+⋯+Var(Xn​) Apply Var⁡(X+Y)=Var⁡(X)+Var⁡(Y)\operatorname{Var}(X + Y) = \operatorname{Var}(X) + \operatorname{Var}(Y)Var(X+Y)=Var(X)+Var(Y) with Y=XnY = X_nY=Xn​ and X=X1+⋯+Xn−1X = X_1 + \dots + X_{n-1}X=X1​+⋯+Xn−1​ , then with Y=Xn−1Y = X_{n-1}Y=Xn−1​ , and so on.

38. [Variance] X1,…,XnX_1, \dots, X_nX1​,…,Xn​ are i.i.d., each distributed like XXX . What does i.i.d. mean, and what is Var⁡(X1+⋯+Xn)\operatorname{Var}(X_1 + \dots + X_n)Var(X1​+⋯+Xn​) ?

    Independent and identically distributed; Var⁡(X1+⋯+Xn)=nVar⁡(X)\boldsymbol{\operatorname{Var}(X_1 + \dots + X_n) = n\operatorname{Var}(X)}Var(X1​+⋯+Xn​)=nVar(X) . Example: number of Heads in nnn fair flips: n⋅14=n4n \cdot \frac{1}{4} = \frac{n}{4}n⋅41​=4n​ .

39. [Variance] A biased coin gives Heads with probability ppp and Tails with probability q=1−pq = 1 - pq=1−p . What is the variance of the number of Heads in nnn flips?

    npq\boldsymbol{npq}npq It is the sum of nnn i.i.d. flips (Heads = 1, Tails = 0), each with variance pqpqpq .Example: 100 fair flips: 100⋅12⋅12=25100 \cdot \frac{1}{2} \cdot \frac{1}{2} = 25100⋅21​⋅21​=25 .

40. [Variance] X1,…,XnX_1, \dots, X_nX1​,…,Xn​ are i.i.d., each distributed like XXX . Compare Var⁡(X1+⋯+Xn)\operatorname{Var}(X_1 + \dots + X_n)Var(X1​+⋯+Xn​) with Var⁡(nX)\operatorname{Var}(nX)Var(nX) .

    Var⁡(X1+⋯+Xn)=nVar⁡(X)\boldsymbol{\operatorname{Var}(X_1 + \dots + X_n) = n\operatorname{Var}(X)}Var(X1​+⋯+Xn​)=nVar(X) , but Var⁡(nX)=n2Var⁡(X)\boldsymbol{\operatorname{Var}(nX) = n^2\operatorname{Var}(X)}Var(nX)=n2Var(X) . Identically distributed variables generally take different values, so their sum isn't one value times nnn . Their expectation values agree ( nE(X)nE(X)nE(X) ), their variances don't.

41. [Variance] XXX and YYY are independent. Is Var⁡(XY)=Var⁡(X)Var⁡(Y)\operatorname{Var}(XY) = \operatorname{Var}(X)\operatorname{Var}(Y)Var(XY)=Var(X)Var(Y) ?

    No, not even for independent XXX and YYY . Example: two independent fair coins with Heads = 1, Tails = 0: XYXYXY is 1 with probability 14\frac{1}{4}41​ , so Var⁡(XY)=14⋅34=316\operatorname{Var}(XY) = \frac{1}{4} \cdot \frac{3}{4} = \frac{3}{16}Var(XY)=41​⋅43​=163​ , but Var⁡(X)Var⁡(Y)=116\operatorname{Var}(X)\operatorname{Var}(Y) = \frac{1}{16}Var(X)Var(Y)=161​ .

42. [Variance] Write Var⁡(X)\operatorname{Var}(X)Var(X) in terms of E(X2)E(X^2)E(X2) and μ=E(X)\mu = E(X)μ=E(X) .

    Var⁡(X)=E(X2)−μ2\boldsymbol{\operatorname{Var}(X) = E(X^2) - \mu^2}Var(X)=E(X2)−μ2 E[X2−2μX+μ2]=E(X2)−2μE(X)+μ2=E(X2)−2μ2+μ2E\big[X^2 - 2\mu X + \mu^2\big] = E(X^2) - 2\mu E(X) + \mu^2 = E(X^2) - 2\mu^2 + \mu^2E[X2−2μX+μ2]=E(X2)−2μE(X)+μ2=E(X2)−2μ2+μ2 , using linearity of EEE .Example: die roll: 916−3.52=3512≈2.92\frac{91}{6} - 3.5^2 = \frac{35}{12} \approx 2.92691​−3.52=1235​≈2.92 .

43. [Variance] What is the variance of a set SSS of numbers x1,…,xnx_1, \dots, x_nx1​,…,xn​ ?

    Var⁡(S)=1n∑i=1n(xi−x‾)2\boldsymbol{\operatorname{Var}(S) = \frac{1}{n}\sum_{i=1}^{n} (x_i - \overline{x})^2}Var(S)=n1​∑i=1n​(xi​−x)2 : the average of the squared differences from the mean x‾≡⟨x⟩=1n∑xi\overline{x} \equiv \langle x \rangle = \frac{1}{n}\sum x_ix≡⟨x⟩=n1​∑xi​ . Example: S={2.3,5.6,3.8,4.7}S = \{2.3, 5.6, 3.8, 4.7\}S={2.3,5.6,3.8,4.7} , x‾=4.1\overline{x} = 4.1x=4.1 : 14(3.24+2.25+0.09+0.36)=1.485\frac{1}{4}(3.24 + 2.25 + 0.09 + 0.36) = 1.48541​(3.24+2.25+0.09+0.36)=1.485 .

44. [Variance] Why is the variance of a set of numbers an average of squared differences, while the variance of a random variable is an expectation value of them?

    A set of numbers has no probability distribution and no future trials to expect anything of; you can only average it. So each number is weighted equally, unlike the outcomes in E[(X−μ)2]E\big[(X - \mu)^2\big]E[(X−μ)2] , which are weighted by their probabilities.

45. [Standard deviation] What is the standard deviation σX\sigma_XσX​ of a random variable XXX ?

    σX=Var⁡(X)\boldsymbol{\sigma_X = \sqrt{\operatorname{Var}(X)}}σX​=Var(X)​ , so σ=E[(X−μ)2]=E(X2)−μ2\boldsymbol{\sigma = \sqrt{E\big[(X - \mu)^2\big]} = \sqrt{E(X^2) - \mu^2}}σ=E[(X−μ)2]​=E(X2)−μ2​ . A rough measure of how far the outcomes are spread from the mean. Hence the notation σX2\sigma_X^2σX2​ (or σ2\sigma^2σ2 ) for the variance.Example: die roll: σ=2.92=1.71\sigma = \sqrt{2.92} = 1.71σ=2.92​=1.71 .

46. [Standard deviation] Why is the standard deviation a more appropriate measure of the spread of XXX than the variance?

    σ\sigmaσ has the same units as XXX ; the variance has the units of X2X^2X2 . So σ\sigmaσ can be drawn on the same axis as the values of XXX in a plot of the distribution.

47. [Standard deviation] aaa is a constant. σaX= ?\sigma_{aX} = \; ?σaX​=?

    σaX=∣a∣ σX\boldsymbol{\sigma_{aX} = |a|\,\sigma_X}σaX​=∣a∣σX​ The square root of Var⁡(aX)=a2Var⁡(X)\operatorname{Var}(aX) = a^2\operatorname{Var}(X)Var(aX)=a2Var(X) .

48. [Standard deviation] XXX and YYY are independent. How is σX+Y\sigma_{X+Y}σX+Y​ obtained from σX\sigma_XσX​ and σY\sigma_YσY​ ?

    σX+Y2=σX2+σY2\boldsymbol{\sigma_{X+Y}^2 = \sigma_X^2 + \sigma_Y^2}σX+Y2​=σX2​+σY2​ , i.e. σX+Y=σX2+σY2\sigma_{X+Y} = \sqrt{\sigma_X^2 + \sigma_Y^2}σX+Y​=σX2​+σY2​​ This is Var⁡(X+Y)=Var⁡(X)+Var⁡(Y)\operatorname{Var}(X + Y) = \operatorname{Var}(X) + \operatorname{Var}(Y)Var(X+Y)=Var(X)+Var(Y) . For nnn independent variables: σX1+⋯+Xn=σX12+⋯+σXn2\sigma_{X_1 + \dots + X_n} = \sqrt{\sigma_{X_1}^2 + \dots + \sigma_{X_n}^2}σX1​+⋯+Xn​​=σX1​2​+⋯+σXn​2​​ .Example: σX=3\sigma_X = 3σX​=3 , σY=4\sigma_Y =…

49. [Standard deviation] For independent random variables, standard deviations add in quadrature: σX+Y2=σX2+σY2\sigma_{X+Y}^2 = \sigma_X^2 + \sigma_Y^2σX+Y2​=σX2​+σY2​ .

50. [Standard deviation] X1,…,XnX_1, \dots, X_nX1​,…,Xn​ are i.i.d., each distributed like XXX . σX1+⋯+Xn= ?\sigma_{X_1 + \dots + X_n} = \; ?σX1​+⋯+Xn​​=?

    n σX\boldsymbol{\sqrt{n}\,\sigma_X}n​σX​ σX2+⋯+σX2=nσX2\sqrt{\sigma_X^2 + \dots + \sigma_X^2} = \sqrt{n\sigma_X^2}σX2​+⋯+σX2​​=nσX2​​ . The spread of a sum grows like n\sqrt{n}n​ , not like nnn .

51. [Standard deviation] A biased coin gives Heads with probability ppp and Tails with probability q=1−pq = 1 - pq=1−p . What is the standard deviation of the number of Heads in nnn flips, and in nnn fair flips?

    Biased: σ=npq\boldsymbol{\sigma = \sqrt{npq}}σ=npq​ . Fair: σ=n/4=n2\boldsymbol{\sigma = \sqrt{n/4} = \frac{\sqrt{n}}{2}}σ=n/4​=2n​​ . One flip (Heads = 1, Tails = 0) has σ=pq\sigma = \sqrt{pq}σ=pq​ ; nnn i.i.d. flips have n\sqrt{n}n​ times that.

52. [Standard deviation] What is the standard deviation of the number of Heads in 100 fair coin flips, and how do you get it quickly for 1000 and 10,000 flips?

    100 flips: σ=100/4=5\sigma = \sqrt{100/4} = 5σ=100/4​=5 . Since σ∝n\sigma \propto \sqrt{n}σ∝n​ : 1000 flips about 10⋅5≈15\sqrt{10} \cdot 5 \approx 1510​⋅5≈15 , 10,000 flips 10⋅5=5010 \cdot 5 = 5010⋅5=50 . For 1000 flips it is actually 250≈15.8\sqrt{250} \approx 15.8250​≈15.8 .

53. [Standard deviation] Express E(X2)E(X^2)E(X2) in terms of σ\sigmaσ and μ\muμ .

    E(X2)=σ2+μ2\boldsymbol{E(X^2) = \sigma^2 + \mu^2}E(X2)=σ2+μ2 Rearranged from σ2=E(X2)−μ2\sigma^2 = E(X^2) - \mu^2σ2=E(X2)−μ2 .Example: die roll: 2.92+12.25=15.17=9162.92 + 12.25 = 15.17 = \frac{91}{6}2.92+12.25=15.17=691​ .

54. [Standard deviation] E(X2)=σ2+μ2E(X^2) = \sigma^2 + \mu^2E(X2)=σ2+μ2 Which two limiting cases check this formula, and why?

    μ=0\mu = 0μ=0 : it says σ2=E(X2)\sigma^2 = E(X^2)σ2=E(X2) , which is the definition E[(X−μ)2]E\big[(X - \mu)^2\big]E[(X−μ)2] with μ=0\mu = 0μ=0 . σ=0\sigma = 0σ=0 : it says E(X2)=μ2E(X^2) = \mu^2E(X2)=μ2 . With σ=0\sigma = 0σ=0 there is no spread: the only outcome is μ\muμ , and the expectation value of one number is that number.

55. [Standard deviation] Where are the six outcomes of a die roll relative to the range μ±σ\mu \pm \sigmaμ±σ ?

    2, 3, 4, 5 lie inside, 1 and 6 lie outside 3.5±1.71=[1.79,5.21]3.5 \pm 1.71 = [1.79, 5.21]3.5±1.71=[1.79,5.21] . Die roll: μ=3.5\mu = 3.5μ=3.5 , σ=1.71\sigma = 1.71σ=1.71 .

56. [Standard deviation] For a fair coin flip with Heads = 1 and Tails = 0, where do the outcomes lie relative to μ±σ\mu \pm \sigmaμ±σ ? Why?

    Both exactly at μ±σ\mu \pm \sigmaμ±σ : μ=12\mu = \frac{1}{2}μ=21​ and σ=12\sigma = \frac{1}{2}σ=21​ . Both outcomes are at distance 12\frac{1}{2}21​ from the mean, so the root-mean-square distance is 12\frac{1}{2}21​ .

57. [Standard deviation] A biased coin gives Heads = 1 with probability 110\frac{1}{10}101​ and Tails = 0 with probability 910\frac{9}{10}109​ . What are μ\muμ and σ\sigmaσ , and where do the outcomes lie relative to μ±σ\mu \pm \sigmaμ±σ ?

    μ=0.1\boldsymbol{\mu = 0.1}μ=0.1 , σ=110⋅910=0.3\boldsymbol{\sigma = \sqrt{\frac{1}{10} \cdot \frac{9}{10}} = 0.3}σ=101​⋅109​​=0.3 : 0 lies inside, 1 lies far outside.

58. [Standard deviation] For the number of Heads in 20 fair coin flips, σ=202≈2.24\sigma = \frac{\sqrt{20}}{2} \approx 2.24σ=220​​≈2.24 around μ=10\mu = 10μ=10 . Only 5 of the 21 outcomes lie within μ±σ\mu \pm \sigmaμ±σ . Why don't the 16 outcomes outside make σ\sigmaσ larger?

    The outcomes far from the mean have small probabilities, so they have little weight in σ=E[(X−μ)2]\sigma = \sqrt{E\big[(X - \mu)^2\big]}σ=E[(X−μ)2]​ . Inside 10±2.2410 \pm 2.2410±2.24 : 8, 9, 10, 11, 12.

59. [Standard deviation of the mean] A fair coin is tossed nnn times. The total number of Heads has standard deviation σtot=n2\sigma_{\text{tot}} = \frac{\sqrt{n}}{2}σtot​=2n​​ . What is the standard deviation σavg\sigma_{\text{avg}}σavg​ of the average number of Heads per toss?

    σavg=σtotn=12n\boldsymbol{\sigma_{\text{avg}} = \frac{\sigma_{\text{tot}}}{n} = \frac{1}{2\sqrt{n}}}σavg​=nσtot​​=2n​1​ The average is Xavg=Xtot/nX_{\text{avg}} = X_{\text{tot}}/nXavg​=Xtot​/n , and σaX=∣a∣ σX\sigma_{aX} = |a|\,\sigma_XσaX​=∣a∣σX​ for a constant aaa .Example: n=20n = 20n=20 gives σavg=1220≈0.11\sigma_{\text{avg}} = \frac{1}{2\sqrt{20}} \approx 0.11σavg​=220​1​≈0.11 ; n=2000n = 2000n=2000 gives 122000≈0.011\frac{1}{2\sqrt{2000}} \approx 0.01122000​1​≈0.011 .

60. [Standard deviation of the mean] How do you get the probability distribution of the average number of Heads per toss in nnn tosses from the distribution of the total number of Heads?

    Keep the same probabilities and divide every value on the xxx axis by nnn . Example: in 20 tosses, 12 Heads (probability about 12%) becomes an average of 12/20=0.612/20 = 0.612/20=0.6 Heads per toss, with the same probability.

61. [Standard deviation of the mean] A fair coin is tossed 20 times, and then 2000 times. What happens to σtot\sigma_{\text{tot}}σtot​ (standard deviation of the total number of Heads) and to σavg\sigma_{\text{avg}}σavg​ (standard deviation of the average number of Heads per toss)?

    σtot\sigma_{\text{tot}}σtot​ becomes 10 times larger, σavg\sigma_{\text{avg}}σavg​ becomes 10 times smaller. nnn grows by a factor of 100; σtot=n2\sigma_{\text{tot}} = \frac{\sqrt{n}}{2}σtot​=2n​​ is proportional to n\sqrt{n}n​ and σavg=12n\sigma_{\text{avg}} = \frac{1}{2\sqrt{n}}σavg​=2n​1​ to 1/n1/\sqrt{n}1/n​ . Distribution of the average number of Heads per toss: 20 tosses ( σavg=0.11\sigma_{\text{avg}} = 0.11σavg​=0.11 ) and 2000 tosses ( σavg=0.011\sigma_{\text{avg}} = 0.011σavg​=0.011 ).

62. [Standard deviation of the mean] In nnn tosses of a fair coin, which deviation from the expected value grows with nnn and which shrinks: the absolute deviation of the number of Heads, or the fractional deviation (of the average number of Heads per toss)?

    Absolute deviation grows (like n\sqrt{n}n​ ), fractional deviation shrinks (like 1/n1/\sqrt{n}1/n​ ). The absolute deviation σtot\sigma_{\text{tot}}σtot​ grows only like n\sqrt{n}n​ , so dividing it by nnn gives σavg∝1/n\sigma_{\text{avg}} \propto 1/\sqrt{n}σavg​∝1/n​ , which decreases.

63. [Standard deviation of the mean] The expected average number of Heads per toss of a fair coin is 0.50.50.5 for every number nnn of tosses. Does the distribution of the average depend on nnn ?

    Yes: its width shrinks like 1/n1/\sqrt{n}1/n​ . Example: in 20 tosses an average of 0.6 or more is quite possible; in 2000 tosses it is extremely unlikely.

64. [Standard deviation of the mean] Theorem 3.4 (standard deviation of the mean) A random variable XXX has standard deviation σ\sigmaσ (any distribution). X‾\overline{X}X is the average of nnn independent trials of XXX . Then σX‾= ?\sigma_{\overline{X}} = \; ?σX​=?

    σX‾=σn\boldsymbol{\sigma_{\overline{X}} = \frac{\sigma}{\sqrt{n}}}σX​=n​σ​ σX‾\sigma_{\overline{X}}σX​ depends on nnn , although the notation does not show it.Example: a single roll of a die, with X=1X = 1X=1 for a 6 and X=0X = 0X=0 otherwise, has σ=16⋅56≈0.37\sigma = \sqrt{\frac{1}{6} \cdot \frac{5}{6}} \approx 0.37σ=61​⋅65​​≈0.37 ; the average over 10,00010{,}00010,000 rolls has σX‾≈0.37/100=0.0037\sigma_{\overline{X}} \approx 0.37/100 = 0.0037σX​≈0.37/100=0.0037 .

65. [Standard deviation of the mean] Theorem: if X‾=X1+⋯+Xnn\overline{X} = \frac{X_1 + \dots + X_n}{n}X=nX1​+⋯+Xn​​ for independent, identically distributed XiX_iXi​ , each with standard deviation σ\sigmaσ , then σX‾=σn\sigma_{\overline{X}} = \frac{\sigma}{\sqrt{n}}σX​=n​σ​ .How does the proof go?

    The sum X1+⋯+XnX_1 + \dots + X_nX1​+⋯+Xn​ has standard deviation n σ\sqrt{n}\,\sigman​σ ; dividing by nnn gives n σn=σn\frac{\sqrt{n}\,\sigma}{n} = \frac{\sigma}{\sqrt{n}}nn​σ​=n​σ​ . n σ\sqrt{n}\,\sigman​σ : variances of independent variables add, so Var⁡(X1+⋯+Xn)=nσ2\operatorname{Var}(X_1 + \dots + X_n) = n\sigma^2Var(X1​+⋯+Xn​)=nσ2 . Dividing by nnn : σaY=∣a∣ σY\sigma_{aY} = |a|\,\sigma_YσaY​=∣a∣σY​ .

66. [Standard deviation of the mean] X1,…,XnX_1, \dots, X_nX1​,…,Xn​ are independent random variables with possibly different standard deviations σX1,…,σXn\sigma_{X_1}, \dots, \sigma_{X_n}σX1​​,…,σXn​​ . What is the standard deviation of their average X‾=X1+⋯+Xnn\overline{X} = \frac{X_1 + \dots + X_n}{n}X=nX1​+⋯+Xn​​ ?

    σX‾=σX12+⋯+σXn2n\boldsymbol{\sigma_{\overline{X}} = \frac{\sqrt{\sigma_{X_1}^2 + \dots + \sigma_{X_n}^2}}{n}}σX​=nσX1​2​+⋯+σXn​2​​​ If all σXi\sigma_{X_i}σXi​​ equal σ\sigmaσ (even with different distributions), this is nσ2n=σn\frac{\sqrt{n\sigma^2}}{n} = \frac{\sigma}{\sqrt{n}}nnσ2​​=n​σ​ .Example: σX1=3\sigma_{X_1} = 3σX1​​=3 , σX2=4\sigma_{X_2} = 4σX2​​=4 gives σX‾=9+162=2.5\sigma_{\overline{X}} = \frac{\sqrt{9 + 16}}{2} = 2.5σX​=29+16​​=2.5 .

67. [Standard deviation of the mean] What does the law of large numbers say, as Morin states it informally?

    With a very large number of trials, the observed average is very likely very close to the theoretically predicted average. The standard deviation of the mean, σ/n\sigma/\sqrt{n}σ/n​ , going to 0 as n→∞n \to \inftyn→∞ is consistent with this law.

68. [Standard deviation of the mean] For a Gaussian (bell-curve) distribution with mean μ\muμ and standard deviation σ\sigmaσ , what is the probability of lying within 111 , 222 and 333 standard deviations of the mean?

    68% ( μ±σ\mu \pm \sigmaμ±σ ), 95% ( μ±2σ\mu \pm 2\sigmaμ±2σ ), 99.7% ( μ±3σ\mu \pm 3\sigmaμ±3σ ) Many distributions, such as the number of Heads in many coin tosses, are approximately Gaussian. Being more than 9 standard deviations above the mean has probability about 10−1910^{-19}10−19 .

69. [Standard deviation of the mean] A fair coin is tossed 2000 times; the average number of Heads per toss has mean 0.50.50.5 and standard deviation σavg=0.011\sigma_{\text{avg}} = 0.011σavg​=0.011 . Is an average of 0.60.60.6 or more plausible?

    No: 0.60.60.6 is about 9 standard deviations above the mean. 0.1/0.011≈90.1 / 0.011 \approx 90.1/0.011≈9 . For a roughly Gaussian distribution, the probability of 9 or more standard deviations above the mean is about 10−1910^{-19}10−19 .

70. [Standard deviation of the mean] 10,00010{,}00010,000 dice are rolled. What are the expected value and the standard deviation of the total number of 6's?

    Expected value ≈1667\approx 1667≈1667 , standard deviation σtot≈37\sigma_{\text{tot}} \approx 37σtot​≈37 . Count a 6 as 1 and any other result as 0: a biased coin with p=16p = \frac{1}{6}p=61​ , q=56q = \frac{5}{6}q=65​ . Then 10,0006≈1667\frac{10{,}000}{6} \approx 1667610,000​≈1667 and σtot=npq=10,000⋅16⋅56≈37\sigma_{\text{tot}} = \sqrt{npq} = \sqrt{10{,}000 \cdot \frac{1}{6} \cdot \frac{5}{6}} \approx 37σtot​=npq​=10,000⋅61​⋅65​​≈37 .

71. [Standard deviation of the mean] 10,00010{,}00010,000 dice are rolled. What are the expected value and the standard deviation of the average number of 6's per roll?

    Expected value 16\frac{1}{6}61​ , standard deviation σavg≈0.0037\sigma_{\text{avg}} \approx 0.0037σavg​≈0.0037 . Either divide the total's σtot=37\sigma_{\text{tot}} = 37σtot​=37 by n=10,000n = 10{,}000n=10,000 , or divide a single roll's σsingle=pq=16⋅56≈0.37\sigma_{\text{single}} = \sqrt{pq} = \sqrt{\frac{1}{6} \cdot \frac{5}{6}} \approx 0.37σsingle​=pq​=61​⋅65​​≈0.37 by n=100\sqrt{n} = 100n​=100 .

72. [Standard deviation of the mean] nnn independent identical trials, each with standard deviation σsingle\sigma_{\text{single}}σsingle​ . How do you get σtot\sigma_{\text{tot}}σtot​ (of the sum) and σavg\sigma_{\text{avg}}σavg​ (of the average) from σsingle\sigma_{\text{single}}σsingle​ , and σavg\sigma_{\text{avg}}σavg​ from σtot\sigma_{\text{tot}}σtot​ ?

    σtot=n σsingle\boldsymbol{\sigma_{\text{tot}} = \sqrt{n}\,\sigma_{\text{single}}}σtot​=n​σsingle​ , σavg=σsingle/n\ \boldsymbol{\sigma_{\text{avg}} = \sigma_{\text{single}} / \sqrt{n}} σavg​=σsingle​/n​ , σavg=σtot/n\ \boldsymbol{\sigma_{\text{avg}} = \sigma_{\text{tot}} / n} σavg​=σtot​/n Example: 10,00010{,}00010,000 dice rolls, counting 6's: σsingle=0.37\sigma_{\text{single}} = 0.37σsingle​=0.37 , σtot=37\sigma_{\text{tot}} = 37σtot​=37 , σavg=0.0037\sigma_{\text{avg}} = 0.0037σavg​=0.0037 .

73. [Standard deviation of the mean] 10,00010{,}00010,000 dice are rolled; the number of 6's has mean 166716671667 and standard deviation 373737 . Is there a reasonable chance of getting a 6 on at least 20% of the rolls?

    No: 20% is 2000 sixes, about 9 standard deviations above the mean. 2000−1667=333=9⋅372000 - 1667 = 333 = 9 \cdot 372000−1667=333=9⋅37 . Even 18% (1800 sixes, 3.6 σ3.6\,\sigma3.6σ above the mean) has probability only about 0.016%. Distribution of the number of 6's in 10,00010{,}00010,000 rolls.

74. [Standard deviation of the mean] 10,00010{,}00010,000 dice are rolled; the fraction of rolls showing a 6 has mean 16.7% and standard deviation 0.37%. How do percentages show that 20% sixes is essentially impossible?

    20%−16.7%=3.3%20\% - 16.7\% = 3.3\%20%−16.7%=3.3% , which is 9 standard deviations of 0.37%0.37\%0.37% .

75. [Standard deviation of the mean] The number of 6's in 10,00010{,}00010,000 dice rolls has a distribution that is nearly symmetric around its maximum, although P(6)=16≠P(not 6)=56P(6) = \frac{1}{6} \ne P(\text{not } 6) = \frac{5}{6}P(6)=61​=P(not 6)=65​ . When is the distribution of a binomial count with unequal probabilities symmetric?

    Only approximately, for a large number of trials, and only in the region near the maximum. For few rolls, or over the whole range 0 to 10,00010{,}00010,000 , it is clearly not symmetric. With probabilities 12\frac{1}{2}21​ and 12\frac{1}{2}21​ it is exactly symmetric.

76. [Sample variance] What is the difference between probability and statistics?

    Probability: from a known distribution, predict outcomes. Statistics: from observed outcomes, deduce the distribution. Probability goes from theory to experiment, statistics from experiment to theory.

77. [Sample variance] What is the variance Var⁡(X)=σ2\operatorname{Var}(X) = \sigma^2Var(X)=σ2 of a random variable XXX with mean μ=E[X]\mu = E[X]μ=E[X] , in two forms?

    Var⁡(X)=E[(X−μ)2]=E[X2]−μ2\boldsymbol{\operatorname{Var}(X) = E\big[(X - \mu)^2\big] = E[X^2] - \mu^2}Var(X)=E[(X−μ)2]=E[X2]−μ2 The second form uses that μ\muμ is a constant, so E[2Xμ]=2μE[X]=2μ2E[2X\mu] = 2\mu E[X] = 2\mu^2E[2Xμ]=2μE[X]=2μ2 .Example: a fair coin with Heads =1= 1=1 , Tails =0= 0=0 : E[X2]−μ2=12−14=14E[X^2] - \mu^2 = \frac{1}{2} - \frac{1}{4} = \frac{1}{4}E[X2]−μ2=21​−41​=41​ .

78. [Sample variance] x1,…,xnx_1, \dots, x_nx1​,…,xn​ are nnn numbers picked at random from an unknown distribution, with average x‾=1n∑xi\overline{x} = \frac{1}{n}\sum x_ix=n1​∑xi​ . How is s~2\tilde{s}^2s~2 , the variance of the set of nnn numbers, defined?

    s~2=1n∑i=1n(xi−x‾)2\boldsymbol{\tilde{s}^2 = \frac{1}{n} \sum_{i=1}^{n} (x_i - \overline{x})^2}s~2=n1​∑i=1n​(xi​−x)2 Example: for the numbers 1,31, 31,3 : x‾=2\overline{x} = 2x=2 , s~2=1+12=1\tilde{s}^2 = \frac{1 + 1}{2} = 1s~2=21+1​=1 .

79. [Sample variance] x1,…,xnx_1, \dots, x_nx1​,…,xn​ are nnn numbers picked at random from a distribution with unknown mean μ\muμ . Why does the variance of the set, s~2=1n∑(xi−x‾)2\tilde{s}^2 = \frac{1}{n}\sum (x_i - \overline{x})^2s~2=n1​∑(xi​−x)2 , use the average x‾\overline{x}x instead of μ\muμ ?

    μ\muμ is not known; only the numbers xix_ixi​ are. For large nnn , x‾\overline{x}x is likely close to μ\muμ , but unlikely to equal it exactly.

80. [Sample variance] In E[s~2]=1n∑E[(Xi−X‾)2]E[\tilde{s}^2] = \frac{1}{n}\sum E\big[(X_i - \overline{X})^2\big]E[s~2]=n1​∑E[(Xi​−X)2] , why are the uppercase XiX_iXi​ used instead of the lowercase xix_ixi​ from s~2=1n∑(xi−x‾)2\tilde{s}^2 = \frac{1}{n}\sum (x_i - \overline{x})^2s~2=n1​∑(xi​−x)2 ?

    xix_ixi​ is a specific number; XiX_iXi​ is a random variable. An expectation value averages over many trials, i.e. over many values of XiX_iXi​ . The expectation value of a specific number is that number: E[173.92]=173.92E[173.92] = 173.92E[173.92]=173.92 .

81. [Sample variance] X1,…,XnX_1, \dots, X_nX1​,…,Xn​ are independent, identically distributed, with variance σ2\sigma^2σ2 , and s~2=1n∑(Xi−X‾)2\tilde{s}^2 = \frac{1}{n}\sum (X_i - \overline{X})^2s~2=n1​∑(Xi​−X)2 , where X‾=1n∑Xi\overline{X} = \frac{1}{n}\sum X_iX=n1​∑Xi​ . For n=2n = 2n=2 , what is E[s~2]E[\tilde{s}^2]E[s~2] ?

    E[s~2]=σ22\boldsymbol{E[\tilde{s}^2] = \frac{\sigma^2}{2}}E[s~2]=2σ2​ Both terms Xi−X1+X22X_i - \frac{X_1 + X_2}{2}Xi​−2X1​+X2​​ equal ±X1−X22\pm\frac{X_1 - X_2}{2}±2X1​−X2​​ , so E[s~2]=14E[(X1−X2)2]=14(2(σ2+μ2)−2μ2)E[\tilde{s}^2] = \frac{1}{4}E\big[(X_1 - X_2)^2\big] = \frac{1}{4}\big(2(\sigma^2 + \mu^2) - 2\mu^2\big)E[s~2]=41​E[(X1​−X2​)2]=41​(2(σ2+μ2)−2μ2) . E[Xi2]=σ2+μ2E[X_i^2] = \sigma^2 + \mu^2E[Xi2​]=σ2+μ2 , and independence gives E[X1X2]=E[X1]E[X2]=μ2E[X_1X_2] = E[X_1]E[X_2] = \mu^2E[X…

82. [Sample variance] s~2=1n∑(Xi−X‾)2\tilde{s}^2 = \frac{1}{n}\sum (X_i - \overline{X})^2s~2=n1​∑(Xi​−X)2 , where X‾=1n∑Xi\overline{X} = \frac{1}{n}\sum X_iX=n1​∑Xi​ . Why can E[s~2]E[\tilde{s}^2]E[s~2] not depend on the mean μ\muμ of the distribution?

    Shifting all XiX_iXi​ by the same amount shifts X‾\overline{X}X by that amount too, so every Xi−X‾X_i - \overline{X}Xi​−X is unchanged.

83. [Sample variance] Theorem 3.5 X1,…,XnX_1, \dots, X_nX1​,…,Xn​ are independent, identically distributed, with variance σ2\sigma^2σ2 ; X‾=1n∑Xi\overline{X} = \frac{1}{n}\sum X_iX=n1​∑Xi​ . Then E[s~2]=E[1n∑i=1n(Xi−X‾)2]= ?E[\tilde{s}^2] = E\Big[\frac{1}{n}\sum_{i=1}^{n} (X_i - \overline{X})^2\Big] = \; ?E[s~2]=E[n1​∑i=1n​(Xi​−X)2]=?

    E[s~2]=n−1n σ2\boldsymbol{E[\tilde{s}^2] = \frac{n-1}{n}\,\sigma^2}E[s~2]=nn−1​σ2 So s~2\tilde{s}^2s~2 underestimates σ2\sigma^2σ2 on average, for every finite nnn .Example: n=2n = 2n=2 gives σ22\frac{\sigma^2}{2}2σ2​ , off by a factor of 12\frac{1}{2}21​ .

84. [Sample variance] X1,…,XnX_1, \dots, X_nX1​,…,Xn​ are independent, identically distributed, with mean μ\muμ and variance σ2\sigma^2σ2 ; X‾=1n∑Xi\overline{X} = \frac{1}{n}\sum X_iX=n1​∑Xi​ and s~2=1n∑(Xi−X‾)2\tilde{s}^2 = \frac{1}{n}\sum (X_i - \overline{X})^2s~2=n1​∑(Xi​−X)2 .Proving E[s~2]=n−1nσ2E[\tilde{s}^2] = \frac{n-1}{n}\sigma^2E[s~2]=nn−1​σ2 , step 1: expanding the square, E[s~2]E[\tilde{s}^2]E[s~2] equals what simpler expression?

    E[s~2]=E[X2]−E[X‾2]\boldsymbol{E[\tilde{s}^2] = E[X^2] - E\big[\overline{X}^2\big]}E[s~2]=E[X2]−E[X2] 1nE[∑Xi2−2(∑Xi)X‾+nX‾2]\frac{1}{n}E\Big[\sum X_i^2 - 2\big(\sum X_i\big)\overline{X} + n\overline{X}^2\Big]n1​E[∑Xi2​−2(∑Xi​)X+nX2] , and ∑Xi=nX‾\sum X_i = n\overline{X}∑Xi​=nX makes the middle term −2nX‾2-2n\overline{X}^2−2nX2 . Each E[Xi2]E[X_i^2]E[Xi2​] equals a common E[X2]E[X^2]E[X2] .Unlike Var⁡(X)=E[X2]−μ2\operatorname{Var}(X) = E[X^2] - \mu^2Var(X)=E[X2]−μ2 , here X‾\overline{X}X is a r…

85. [Sample variance] X1,…,XnX_1, \dots, X_nX1​,…,Xn​ are independent, identically distributed, with mean μ\muμ and variance σ2\sigma^2σ2 ; X‾=1n∑Xi\overline{X} = \frac{1}{n}\sum X_iX=n1​∑Xi​ .What is E[X‾2]E\big[\overline{X}^2\big]E[X2] , and what is the quick way to see it?

    E[X‾2]=σ2n+μ2\boldsymbol{E\big[\overline{X}^2\big] = \frac{\sigma^2}{n} + \mu^2}E[X2]=nσ2​+μ2 For any random variable, E[Y2]=σY2+μY2E[Y^2] = \sigma_Y^2 + \mu_Y^2E[Y2]=σY2​+μY2​ . Here X‾\overline{X}X has mean μ\muμ and standard deviation σn\frac{\sigma}{\sqrt{n}}n​σ​ .

86. [Sample variance] X1,…,XnX_1, \dots, X_nX1​,…,Xn​ are independent, identically distributed, with mean μ\muμ and variance σ2\sigma^2σ2 ; X‾=1n∑Xi\overline{X} = \frac{1}{n}\sum X_iX=n1​∑Xi​ .How do you get E[X‾2]=σ2n+μ2E\big[\overline{X}^2\big] = \frac{\sigma^2}{n} + \mu^2E[X2]=nσ2​+μ2 by expanding (X1+⋯+Xn)2(X_1 + \dots + X_n)^2(X1​+⋯+Xn​)2 ?

    nnn square terms with E[Xi2]=σ2+μ2E[X_i^2] = \sigma^2 + \mu^2E[Xi2​]=σ2+μ2 , and (n2)\binom{n}{2}(2n​) cross terms 2XiXj2X_iX_j2Xi​Xj​ with E[XiXj]=μ2E[X_iX_j] = \mu^2E[Xi​Xj​]=μ2 : E[X‾2]=1n2(n(σ2+μ2)+n(n−1)μ2)=σ2n+μ2E\big[\overline{X}^2\big] = \frac{1}{n^2}\Big(n(\sigma^2 + \mu^2) + n(n-1)\mu^2\Big) = \frac{\sigma^2}{n} + \mu^2E[X2]=n21​(n(σ2+μ2)+n(n−1)μ2)=nσ2​+μ2 E[XiXj]=E[Xi]E[Xj]E[X_iX_j] = E[X_i]E[X_j]E[Xi​Xj​]=E[Xi​]E[Xj​] needs independence, which is why E[Xi2]=E[Xi⋅Xi]≠μ2E[X_i^2] = E[X…

87. [Sample variance] X1,…,XnX_1, \dots, X_nX1​,…,Xn​ are independent, identically distributed, with mean μ\muμ and variance σ2\sigma^2σ2 ; X‾=1n∑Xi\overline{X} = \frac{1}{n}\sum X_iX=n1​∑Xi​ and s~2=1n∑(Xi−X‾)2\tilde{s}^2 = \frac{1}{n}\sum (X_i - \overline{X})^2s~2=n1​∑(Xi​−X)2 . Known: E[s~2]=E[X2]−E[X‾2]E[\tilde{s}^2] = E[X^2] - E\big[\overline{X}^2\big]E[s~2]=E[X2]−E[X2] .How does this finish the proof of E[s~2]=n−1nσ2E[\tilde{s}^2] = \frac{n-1}{n}\sigma^2E[s~2]=nn−1​σ2 ?

    E[s~2]=(σ2+μ2)−(σ2n+μ2)=n−1n σ2\boldsymbol{E[\tilde{s}^2] = (\sigma^2 + \mu^2) - \big(\frac{\sigma^2}{n} + \mu^2\big) = \frac{n-1}{n}\,\sigma^2}E[s~2]=(σ2+μ2)−(nσ2​+μ2)=nn−1​σ2 μ2\mu^2μ2 cancels, as it must, since s~2\tilde{s}^2s~2 does not change when all XiX_iXi​ are shifted by a constant.

88. [Sample variance] x1,…,xnx_1, \dots, x_nx1​,…,xn​ are numbers drawn from a distribution with mean μ\muμ , with average x‾\overline{x}x . Intuitively, why is (x1−x‾)2(x_1 - \overline{x})^2(x1​−x)2 smaller on average than (x1−μ)2(x_1 - \mu)^2(x1​−μ)2 ?

    x‾\overline{x}x contains x1x_1x1​ , so x‾\overline{x}x is shifted toward x1x_1x1​ . The other n−1n - 1n−1 numbers average μ\muμ , so the average of all nnn lies a little closer to x1x_1x1​ . The effect is largest for small nnn . This is why E[s~2]=n−1nσ2<σ2E[\tilde{s}^2] = \frac{n-1}{n}\sigma^2 < \sigma^2E[s~2]=nn−1​σ2<σ2 .

89. [Sample variance] For independent identically distributed XiX_iXi​ with mean μ\muμ and variance σ2\sigma^2σ2 : E[s~2]=E[X2]−E[X‾2]E[\tilde{s}^2] = E[X^2] - E\big[\overline{X}^2\big]E[s~2]=E[X2]−E[X2] , compared with σ2=E[X2]−μ2\sigma^2 = E[X^2] - \mu^2σ2=E[X2]−μ2 .Why does this show E[s~2]<σ2E[\tilde{s}^2] < \sigma^2E[s~2]<σ2 ?

    E[X‾2]E\big[\overline{X}^2\big]E[X2] is larger than μ2\mu^2μ2 , by σ2n\frac{\sigma^2}{n}nσ2​ , so more is subtracted. X‾\overline{X}X usually differs slightly from μ\muμ , and the square of that difference adds to E[X‾2]E\big[\overline{X}^2\big]E[X2] .

90. [Sample variance] x1,…,xnx_1, \dots, x_nx1​,…,xn​ are numbers drawn from a distribution, with average x‾\overline{x}x . How is the sample variance s2s^2s2 defined?

    s2=1n−1∑i=1n(xi−x‾)2=nn−1 s~2\boldsymbol{s^2 = \frac{1}{n-1} \sum_{i=1}^{n} (x_i - \overline{x})^2 = \frac{n}{n-1}\,\tilde{s}^2}s2=n−11​∑i=1n​(xi​−x)2=n−1n​s~2 Here s~2=1n∑(xi−x‾)2\tilde{s}^2 = \frac{1}{n}\sum (x_i - \overline{x})^2s~2=n1​∑(xi​−x)2 is the variance of the set.Example: for the numbers 1,31, 31,3 : x‾=2\overline{x} = 2x=2 , s2=1+11=2s^2 = \frac{1 + 1}{1} = 2s2=11+1​=2 (while s~2=1\tilde{s}^2 = 1s~2=1 ).

91. [Sample variance] X1,…,XnX_1, \dots, X_nX1​,…,Xn​ are independent draws from a distribution with variance σ2\sigma^2σ2 . What is the expectation value of the sample variance s2=1n−1∑(Xi−X‾)2s^2 = \frac{1}{n-1}\sum (X_i - \overline{X})^2s2=n−11​∑(Xi​−X)2 ?

    E[s2]=σ2\boldsymbol{E[s^2] = \sigma^2}E[s2]=σ2 , for every nnn . E[s2]=nn−1E[s~2]=nn−1⋅n−1nσ2E[s^2] = \frac{n}{n-1}E[\tilde{s}^2] = \frac{n}{n-1} \cdot \frac{n-1}{n}\sigma^2E[s2]=n−1n​E[s~2]=n−1n​⋅nn−1​σ2 .

92. [Sample variance] When is a quantity computed from data an unbiased estimator of σ2\sigma^2σ2 , and which of s~2=1n∑(xi−x‾)2\tilde{s}^2 = \frac{1}{n}\sum (x_i - \overline{x})^2s~2=n1​∑(xi​−x)2 and s2=1n−1∑(xi−x‾)2s^2 = \frac{1}{n-1}\sum (x_i - \overline{x})^2s2=n−11​∑(xi​−x)2 is one?

    When its expectation value equals σ2\sigma^2σ2 . s2s^2s2 is unbiased; s~2\tilde{s}^2s~2 is biased. E[s2]=σ2E[s^2] = \sigma^2E[s2]=σ2 , while E[s~2]=n−1nσ2E[\tilde{s}^2] = \frac{n-1}{n}\sigma^2E[s~2]=nn−1​σ2 depends on nnn and is smaller than σ2\sigma^2σ2 .

93. [Sample variance] What do σ2\sigma^2σ2 , s~2\tilde{s}^2s~2 and s2s^2s2 denote?

    σ2\sigma^2σ2 The distribution variance or population variance. s~2=1n∑(xi−x‾)2\tilde{s}^2 = \frac{1}{n}\sum (x_i - \overline{x})^2s~2=n1​∑(xi​−x)2 The variance of a set of nnn numbers; a biased estimator of σ2\sigma^2σ2 . s2=1n−1∑(xi−x‾)2s^2 = \frac{1}{n-1}\sum (x_i - \overline{x})^2s2=n−11​∑(xi​−x)2 The sample variance of a set of nnn numbers; an unbiased estimator of σ2\sigma^2σ2 . Example: for the numbers 1,31, 31,3 : s~2=1\tilde{s}^2 = 1s~2=1 , s2=2s^2 = 2s2=2 .

94. [Sample variance] Someone asks for the "variance of a sample of nnn numbers". Which do you compute: s~2=1n∑(xi−x‾)2\tilde{s}^2 = \frac{1}{n}\sum (x_i - \overline{x})^2s~2=n1​∑(xi​−x)2 or the sample variance s2=1n−1∑(xi−x‾)2s^2 = \frac{1}{n-1}\sum (x_i - \overline{x})^2s2=n−11​∑(xi​−x)2 ?

    s~2\tilde{s}^2s~2 , with nnn in the denominator: the request is for the variance of a set of numbers. Only a request for the "sample variance" means s2s^2s2 . Writing the symbol removes the ambiguity.

95. [Sample variance] You have nnn numbers. When should you use s~2=1n∑(xi−x‾)2\tilde{s}^2 = \frac{1}{n}\sum (x_i - \overline{x})^2s~2=n1​∑(xi​−x)2 , and when the sample variance s2=1n−1∑(xi−x‾)2s^2 = \frac{1}{n-1}\sum (x_i - \overline{x})^2s2=n−11​∑(xi​−x)2 ?

    s~2\tilde{s}^2s~2 if you care only about these nnn numbers; s2s^2s2 if you want to estimate the variance of the distribution (population) they come from. For large nnn the difference between nnn and n−1n - 1n−1 hardly matters.

96. [Sample variance] You want the variance σ2\sigma^2σ2 of an unknown distribution by picking numbers from it. What are the two main options?

    One huge set As n→∞n \to \inftyn→∞ , s~2\tilde{s}^2s~2 of a single set approaches σ2\sigma^2σ2 : n−1n→1\frac{n-1}{n} \to 1nn−1​→1 , and the spread of s~2\tilde{s}^2s~2 around its expected value goes to 0. Many sets of normal size Compute s~2\tilde{s}^2s~2 for each of NNN sets of nnn numbers ( nnn small, even 2), average the NNN values, and multiply by nn−1\frac{n}{n-1}n−1n​ . The average of NNN values has spread proportional to 1/N1/\sqrt{N}1/N​ , so for large NNN it is very close to E[s~2]=n−1…

97. [Sample variance] N=105N = 10^5N=105 sets of n=100n = 100n=100 numbers are drawn from a Gaussian with σ=1\sigma = 1σ=1 , and the sample variance s2s^2s2 of each set is computed. What is the average of the NNN values of s2s^2s2 , roughly?

    About 1 (the run gave 1.000621.000621.00062 ), consistent with E[s2]=σ2E[s^2] = \sigma^2E[s2]=σ2 . Histogram of the 10510^5105 values of s2s^2s2 .

98. [Sample variance] For a Gaussian distribution with standard deviation σ\sigmaσ and large nnn , Var⁡(s2)≈2σ4n\operatorname{Var}(s^2) \approx \frac{2\sigma^4}{n}Var(s2)≈n2σ4​ . With σ=1\sigma = 1σ=1 and n=100n = 100n=100 , what is the standard deviation of the sample variance s2s^2s2 ?

    About 0.14 Var⁡(s2)≈2100=0.02\operatorname{Var}(s^2) \approx \frac{2}{100} = 0.02Var(s2)≈1002​=0.02 , and 0.02≈0.14\sqrt{0.02} \approx 0.140.02​≈0.14 .

99. [Sample variance] NNN sets of nnn numbers are drawn, and the sample variance s2s^2s2 of each set is computed. If NNN grows (from 10510^5105 to 10710^7107 ), what happens to the spread of the histogram of the s2s^2s2 values, and to their average?

    The spread stays the same; the average gets closer to σ2\sigma^2σ2 . Var⁡(s2)\operatorname{Var}(s^2)Var(s2) depends only on nnn . The average of NNN values has standard deviation σs2/N\sigma_{s^2}/\sqrt{N}σs2​/N​ .Example: n=100n = 100n=100 , σ=1\sigma = 1σ=1 : σs2≈0.14\sigma_{s^2} \approx 0.14σs2​≈0.14 , and N=105N = 10^5N=105 gives 0.14/105≈4.4⋅10−40.14/\sqrt{10^5} \approx 4.4 \cdot 10^{-4}0.14/105​≈4.4⋅10−4 , so an average of 1.000621.000621.00062 is about 1.5 standard deviations from 1.

100. [Sample variance] s2=1n−1∑(xi−x‾)2s^2 = \frac{1}{n-1}\sum (x_i - \overline{x})^2s2=n−11​∑(xi​−x)2 is an unbiased estimator of σ2\sigma^2σ2 . Is the sample standard deviation s=s2s = \sqrt{s^2}s=s2​ an unbiased estimator of σ\sigmaσ ?

     No: the average of many values of sss is smaller than σ\sigmaσ . To estimate σ\sigmaσ , average many values of s2s^2s2 and then take the square root.