Pishro-Nik: Introduction to Probability

Hossein Pishro-Nik: Introduction to Probability, Statistics, and Random Processes

70 flashcards · Mathematics · by lakesare

1. What's a null set?

   Just an empty set, ∅.

2. What's a universal set?

   S = { all things that we could possibly consider in this context } In the language of probability theory, the universal set is called the sample space.

3. When are sets A = B?

   A = B iff A ⊂ B and B ⊂ A. Two sets are equal if they have the exact same elements. Thus, A = B if and only if A ⊂ B and B ⊂ A.

4. What does this notation denote?

   A union of n sets: A1 ∪ A2 ...∪... An.

5. What does this notation denote?

   An intersection of n sets: A ∩ B.

6. What does this notation denote?

   A complement of a set A: S - A The complement of a set A is the set of all elements that are in the universal set S but are not in A. I

7. When are A and B sets disjoint?

   Two sets A and B are mutually exclusive or disjoint if they do not have any shared elements: A ∩ B = ∅.

8. When are multiple sets disjoint?

   Several sets are called disjoint if they are pairwise disjoint, i.e., no two of them share a common elements: A ∩ B = ∅, B ∩ C = ∅, A ∩ C = ∅.

9. Name 2 conditions for non-empty sets A, B, and C to be a partition of the set S.

   These sets are pairwise disjoint A ∩ B = ∅, B ∩ C = ∅, A ∩ C = ∅Their union is S A ∪ B ∪ C = S

10. Name De Morgan's law in context of set theory.

11. This is the distributive property in elementary algebra. How is it described?

    One says that: multiplication distributes over addition.

12. This is the distributive property in set theory. How is it described?

    Intersection distributes over union, AND Union distributes over intersection. Both ∩ and ∪ play the role of * in: a * (b + c).

13. How many elements are in the Cartesian product A × B?

    6. By the multiplication principle - we choose out of 3 elements (from A), and then out of 2 elements (from B).

14. What does this denote?

    A Cartesian product between two sets of real numbers R.

15. What's the inclusion-exclusion principle?

    Inclusion-exclusion principle is about cardinality of the unions of sets.

16. Name 1 example of countable and uncountable sets.

    countable - N, Z, Q And any of their subsets are countable.uncountable - R Any set containing an interval on the real line such as [a, b], (a, b], [a, b), or (a, b), where a < b is uncountable.

17. What's a probability space?

18. When do we have discrete VS continuous probability space?

    Probability space: Discrete - when our sample space S is countable Continuous - when our sample space S is uncountable The basic idea is that in discrete probability models we can compute the probability of events by adding all the corresponding outcomes, while in continuous probability models we need to use integration instead of summation.

19. What's a trial VS random experiment?

    Random experiment - when we observe something uncertain. E.g., before tossing a coin you do not know the result.Trial - when we repeat a random experiment several times. Thus, a trial is a particular performance of a random experiment. E.g., in the example of tossing a coin, each trial will result in either heads or tails.

20. What's an outcome VS sample space of a random experiment?

    Outcome - a result of a random experiment. Sample space - the set of all possible outcomes. Thus in the context of a random experiment, the sample space is our universal set.

21. Name 3 axioms of probability.

    Axiom 1: Probability cannot be negative. The smallest value for P(A) is zero and if P(A) = 0, then the event A will never happen.Axiom 2: Probability of the whole sample space is equal to one, i.e., 100 percent. The reason for this is that the sample space S contains all possible outcomes of our random experiment.Axiom 3: If some events are disjoint (i.e., there is no overlap between them), then the probability of their union must be the summations of their probabilities.

22. What does the notation P(A, B) mean?

23. Given the axioms of probability, solve this example.

    It is worth noting that we often write P(1) instead of P({1}) to simplify the notation, but we should emphasize that probability is defined for sets (events) not for individual outcomes.

24. What does this mean?

    This is the 3rd axiom of probability, when our sample space is countable. Then to find a probability of event, all we need to do is sum the probabilities of each outcome in the event!

25. What 2 assumptions are needed to state that: Probability of the event A = n of outcomes in A / n of outcomes in S

    That S is a countable sample space That all outcomes in S are equally likely

26. To find the probability of an event, all we have to do is sum the probabilities of individual outcomes in that event. Name an example where the outcomes have: different probabilities same probabilities "Sample space with equally likely outcomes"

    Example where outcomes have different probabilities: Example where outcomes have equal probabilities:

27. Why is this formula true?

    When we know that B has occurred, every outcome that is outside B should be discarded. Thus, our sample space S is reduced to the set B.

28. What is the probability that both children are girls: Given that the first child is a girl? Given that we know one of them is a girl? We ask the father: "Do you have at least one daughter?" He responds "Yes!".

    "First child is a girl" is a stronger claim! Given that the first child is a girl - 1/2 Given that we know one of them is a girl - 1/3

29. [Chain rule for conditional probability] Name 2 steps to derive: from:

    [Apply P(B∩C) = P(B) * P(C|B)] P(A ∩ B ∩ C) = P (A ∩ (B ∩ C)) = P(A) * P(B ∩ C | A) [Condition both on A] P(B ∩ C) = P(B) * P(B|C) P(B ∩ C|A) = P(B|A) * P(B|C,A)

30. [Chain rule for conditional probability] Apply the chain rule for conditional probability to solve:

    As we will see later on, another way to solve this problem is to use counting arguments.

31. When are events A and B said to be independent?

32. [Chain rule for conditional probability] Give the chain rule for 3 events.

33. [Independent events] How would you approach solving this?

    Two events are independent iff P(A∩B) = P(A) * P(B), so we want to determine whether this equality holds.

34. [Independent events] When are events A, B and C said to be independent?

    All 4 conditions must hold for 3 events to be independent: This might look like a difficult definition, but we can usually argue that the events are independent in a much easier way. For example, we might be able to justify independence by looking at the way the random experiment is performed. A simple example of an independent event is when you toss a coin repeatedly.

35. [Random variables] What's a random variable X?

    Random variable X is a function that maps outcomes → real numbers X: S → R E.g. if we're interested in the number of heads, X can be 0, 1, 2, 3, 4 or 5.

36. What's meant by the "value of a function"?

    It's the output of the function. The value of a function, given the value(s) assigned to its argument(s), is the quantity assumed by the function for these argument values.

37. [Random variables] What's RX?

    It's the range of the X function.

38. [Random variables] Find the range of the random variable X.

39. [Random variables] When is the random variable said to be discrete VS continuous?

    A function X is discrete if its range is a countable set. Random variable is discrete Random variable is continuous

40. What does PMF stand for?

    Probability Mass Function

41. How does PX(1) read? This is the Probability Mass Function (PMF).

    PX(1) reads as 'probability that X outputs 1'. While the above notation is the standard notation for the PMF of X, it might look confusing at first. The subscript X here indicates that this is the PMF of the random variable X. Thus, for example, PX(1) shows the probability that X=1.

42. What's the geometric series sum formula for infinite series (when |x| < 1)? Geometric series sum formula for finite series.

43. Find the sample space: S the range of X: RX probability mass function (PMS): PX

44. Visualize a probability Mass Function (PMF) PX.

    y-axis: PMF of X, depending on particular values of X x-axis: values in the range of X

45. What do PDF vs CDF stand for?

    PDF - Probability Density Function CDF - Cumulative Distribution Function

46. What's a z-score?

    Z-score is how many standard deviations below or above the population mean a raw score is. Z-scores are a way to compare results to a "normal" population. Z-score is also called a standard score.

47. What is Kullback-Lieber divergence used for? Suppose that we're space-scientists visiting a distant, new planet and we've discovered a species of biting worms that we'd like to study. We've found that these worms have 10 teeth, but because of all the chomping away, many of them end up missing teeth.

    With KL divergence we can calculate exactly how much information is lost when we approximate one distribution with another. Kullback-Lieber divergence allow us to determine which approximating distribution is better.

48. What's KL in KL divergence?

    Kullback-Lieber divergence Also called relative entropy

49. What does DKL(p||q) stand for?

    Kullback-Lieber divergence between observed probability distribution p, and approximating probability distribution q As we can see the information lost by using the Binomial approximation is greater than using the uniform approximation. If we have to choose one to represent our observations, we're better off sticking with the Uniform approximation.

50. Why Kullback-Lieber divergence can't be used as a distance?

    Because it's not symmetric:

51. What's a Jensen–Shannon divergence?

    It's a divergence based on the Kullback–Leibler divergence, but symmetric, and it always has a finite value.

52. What's a Cohen's kappa?

    Cohen's kappa is a coefficient that measures the agreement between two raters who each classify N items into C mutually exclusive categories. po - relative agreement among raters, pe - hypothetical probability of chance agreement.

53. [Notation] What are the names of these functions?

    X - Random variable function PX - Probability Mass Function

54. [Notation] How do you read { X = xk }?

    { X = xk } = event that X evaluates to xk = { outcome : X(outcome) = xk }

55. [Notation] How do you read X?

    X = random variable X = function: outcomes → Rx

56. [Notation] How do you read P(X = xk)?

    P(X = xk) = probability of the event that X evaluates to xk = P({ X = xk }) { X = xk } = event that X evaluates to xk = { outcome : X(outcome) = xk }

57. [Notation] How do you read RX?

    Rx = range of the random variable X function

58. [Notation] How do you read PX?

    PX = probability of each of the x event = Probability Mass Function = function: Rx → probabilities

59. [Notation] Name one of the meanings for "distribution of the random variable X"?

    PX (Probability Mass Function)

60. [Notation - probability distributions] How do you read "X ∼ Bernoulli(p)"?

    Random variable X is a Bernoulli random variable with parameter p.

61. [Notation - probability distributions] How do you read "X ∼ Binomial(n, p)"?

    Random variable X is a Binomial random variable with parameters n and p.

62. [Notation - probability distributions] How do you read "X ∼ Poisson(λ)"?

    Random variable X is a Poisson random variable with parameter λ.

63. [Notation] How do you read "PXY"?

    PXY = probability that X = x AND Y = y = joint Probability Mass Function of X and Y = P(X = x and Y = y)

64. [Joint PMFs] What form does RXY have?

    RXY = RX × RY = { tuples (x, y) }

65. [Joint PMFs] What's the implication of joint PMFs containing all information about the distributions of X and Y? Joint PMF

    We can derive PX and PY just by knowing PXY.

66. [Joint PMFs] What's the marginal PMF of X? Joint PMF

    PX

67. [Joint PMFs] How could we find PX (marginal PMF of X)? Joint PMF

    PX(x) = PXY(x, y1) + ...PXY(x, yn)

68. Conceptually - what's a random variable?

    A random variable is a way to subdivide outcomes into events. Random variable is a function that takes in an outcome, and returns an integer.

69. What are the names of the functions in green VS red?

    in green - random variable function (X) in red - probability mass function (PX)

70. Where is a prior and posterior in the Bayes Formula?