Practical Statistics for Data Scientists (part 1)
114 flashcards · Computer Science · by charlie42
What's a trimmed mean
The average of all values after dropping a fixed number of extreme values.
Example of the use of trimmed mean
In international diving the top score and bottom score from five judges are dropped, and the final score is the average of the scores from the three remaining judges.
Common percentage to trim to calculate trimmed mean
10%
What does robust mean
Not sensitive to extreme values.
How do you read x̄
x-bar
What x̄ represents
Mean of a sample from a population
Difference between N and n in statistics
N refers to a population, n to a sample from a population
Two examples when we want to use weighted mean
if we are taking the average from multiple sensors and one of the sensors is less accurate, then we might downweight the data from that sensor We can give a higher weight to the values from the groups that were underrepresented
Python package that contains trimmed mean function
scipy.stats
Key ideas about estimates of location (2)
The basic metric for location is the mean, but it can be sensitive to extreme values (outlier). Other metrics (median, trimmed mean) are less sensitive to outliers and unusual distributions and hence are more robust.
What are deviations
The difference between the observed values and the estimate of location
Deviations of {1, 4, 4}
1 – 3 = –2, 4 – 3 = 1, 4 – 3 = 1
Why standard deviation is more used than variance
It is on the same scale as the original data
Formula for variance and standard deviation
Why standard deviation is sensitive to outliers
It involves squaring the data
What MAD stands for
Median absolute deviation from the median
Formula for MAD
Alternative to MAD for estimating variability robustly
Trimmed standard deviation
What are order statistics
Statistics based on sorted (ranked) data
Most basic order statistic
Range
What's a Pth percentile
the Pth percentile is a value such that at least P percent of the values take on this value or less and at least (100 – P) percent of the values take on this value or more.
First step to calculate a percentile
Sort the data
What's interquartile range
Difference between the 25th percentile and the 75th percentile It's a common measurement of variability
Why calculating exact percentiles can be computationally expensive
You need to sort the data first
How to calculate IQR of a column in python
state['Population'].quantile(0.75) - state['Population'].quantile(0.25)
Why the standard deviation is often much larger than MAD?
The standard deviation is sensitive to outliers
Key ideas about estimates of variability
Variance and standard deviation are the most widespread and routinely reported statistics of variability. Both are sensitive to outliers. More robust metrics include mean absolute deviation, median absolute deviation from the median, and percentiles (quantiles)
What's a histogram
A plot of the frequency table
What are 25th, 50th, and 75th percentiles are called
Quartiles
What 10th, 20th, …, 90th percentiles are called
Deciles
1 quartile = ? quantile = ? percentile
1 quartile = 0.25 quantile = 25 percentile
What does the box show? (2)
Half the states fall between about 2 million and about 7 million The top and bottom of the box are the 75th and 25th percentiles, respectively
What's kurtosis
Propensity of the data to have extreme values
How is kurtosis generally measured
Visually
What is the line called?
Density plot
The total area under the density curve = ?
1
What does area under the curve between two points on the x-axis correspond to?
The proportion of the distribution lying between those two points
What's mode
The most commonly occurring category or value in a data set.
If the odds that a team will win are 2 to 1, its probability of winning is ...
2/(2+1) = 2/3
What's an expected value of a categorical variable
Expected value is the sum of values times their probability of occurrence
What does this formula calculate
Pearson’s correlation coefficient
What are Spearman’s rho or Kendall’s tau
More robust correlation coefficients, based on the rank of data
Better alternative to a scatter plot for large datasets
Hexagonal binning plot
3 alternatives to scatter plots
Heat maps, hexagonal binning, and contour plots for a visual representation of a two-dimensional density
Tool to explore relationships between two categorical variables
Contingency table (pivot_table in python)
Plots to explore a relationship between a categorical and a continuous variable
Boxplots and violin plots
What's a sample statistic
A metric calculated for a sample of data drawn from a larger population.
What's a data distribution
The frequency distribution of individual values in a data set.
What's a sampling distribution
The frequency distribution of a sample statistic over many samples or resamples.
What's central limit theorem
The tendency of the sampling distribution to take on a normal shape as sample size rises.
What's standard error
The variability (standard deviation) of a sample statistic over many samples (not to be confused with standard deviation, which by itself, refers to variability of individual data values).
Standard error formula
What's a bootstrap sample
A sample taken with replacement from an observed data set.
What bootstrapping is used for
Assessing variability of a sample statistic
Algorithm for a bootstrap resampling of the mean
How to get a bootstrap sample in python
sample = resample(loans_income)
An x% confidence interval around a sample estimate should, on average, ...
contain similar sample estimates x% of the time (when a similar sampling procedure is followed)
The algorithm for a bootstrap confidence interval, given a sample of size n, and a sample statistic of interest
What's Error
The difference between a data point and a predicted or average value.
What does it mean to standardize data
Subtract the mean and divide by the standard deviation
What's a z-score
The result of standardizing an individual data point.
What a Standard normal
A normal distribution with mean = 0 and standard deviation = 1.
QQ-Plot
A plot to visualize how close a sample distribution is to a specified distribution, e.g., the normal distribution.
Describe the distribution of this data, explain
The data is normal in the middle but has much longer tails (values in lower quantiles are lower than normal, and values in high quantiles are higher than normal)
What does t-distribution look like?
The t-distribution is a normally shaped distribution, except that it is a bit thicker and longer on the tails. As the number of degrees of freedom grows, the t-distribution approaches the normal distribution with mean 0 and variance 1. For this reason ν is also known as the normality parameter.
What t-distribution can be used for?
Estimating confidence interval of a sample statistic if bootstraping is not possible (sample statistic needs to be normally distributed). If s is the sample standard deviation, a 90% confidence interval around the sample mean is given by: The t-distribution has been used as a reference for the distribution of a sample mean, the difference between two sample means, regression parameters, and other statistics.
Why is t-distribution accurate for describing the behavior of sample statistics?
The t-distribution’s accuracy in depicting the behavior of a sample statistic requires that the distribution of that statistic for that sample be shaped like a normal distribution. It turns out that sample statistics are often normally distributed, even when the underlying population data is not (a fact which led to widespread application of the t-distribution) (central limit theorem)
What's a Binomial trial?
A trial with two outcomes.
Synonym of Binomial trial
Bernoulli trial
What's binomial distribution?
Distribution of number of successes in x trials.
Two parameters of a binomial distribution (and their symbols)
Number of trials (n) and specified probability (p) of success in each trial
What does x stand for in binomial distribution?
Number of successes in a given number of trials
What does n stand for in binomial distribution?
Number of trials
What does p stand for in binomial distribution?
Probability (p) of success in each trial
What does this R function calculates? dbinom(x=2, size=5, p=0.1)
The probability of observing exactly x = 2 successes in size = 5 trials, where the probability of success for each trial is p = 0.1
Mean of a binomial distribution with n=10 and p=0.5?
n * p = 5 (expected number of successes in 10 trials)
Visualize plot of a binomial distribution with n=3000 and p=0.3 (don't forget the axis labels)
When binomial distribution is close to normal distribution?
With large n, and provided p is not too close to 0 or 1
Formula for variance of binomial distribution
n*p*q
What are possible outcomes of a binomial trial?
A binomial trial is an experiment with two possible outcomes: one with probability p and the other with probability 1 – p.
What's alpha?
The probability threshold of “unusualness” that chance results must surpass for actual outcomes to be deemed statistically significant.
What are type 1 and type 2 errors?
Type 1 error (more common): Mistakenly concluding an effect is real (when it is due to chance). Type 2 error: Mistakenly concluding an effect is due to chance (when it is real).
Resampling procedure to confirm that difference in conversion ratio is significant
Put 200+182 1s and 23,539+22,406 0s in a bag Repeat n times: Randomly draw (without replacement) 200+23,539 (size of A) and the rest (size of B) Find difference in conversion ratio between these two sets (they should be more or less equal) Calculate proportion of resample pairs with difference higher than the original observed difference (p-value)
When type 2 error usually happens?
When you have a small sample size. Your calculated p-value is high, so you conclude that "effect is not proven", but with a bigger sample size you would obtain a p-value under significance threshold.
What's false discovery rate?
Across multiple tests, the rate of making a Type 1 error
Distribution that approximates resampling procedure for means
t-distribution
If you know the mean for a sample of 10 values, how many degrees of freedom are there?
9: once you know 9 of the sample values, the 10th can be calculated and is not free to vary
Why degrees of freedom are relevant in logistic regression?
When creating dummy variables for categorical variables, including the last one (e.g. Sunday is redundant if including Mo-Sa binary columns) will cause some models to fail (due to a multicollinearity error)
Difference between two-sample mean comparison and ANOVA
Instead of comparing difference between two sample means to differences between resample means, we compare variance between multiple sample means to variances between resample means
Alternative to permutation test for comparing the mean of two groups
t-test
Decompose variance for the top-left value
1. Start with grand average: 173.75. 2. Add treatment (group) effect: –1.75 (172 – 173.75). 3. Add residual: –8 (164 – 172). 4. Equals: 164.
The chi-square distribution is typically concerned with... (2)
counts of subjects or items falling into categories
The chi-square statistic measures
the extent of departure from what you would expect in a null model.
The F-statistic is the ratio of
the variability among the group means to the variability within each group (also called residual variability).
What does Poisson distribution models?
Number of events per unit of time (or space)
The only parameter of the Poisson distribution
λ (lambda): mean number of events that occurs in a specified interval of time (or space). The variance for a Poisson distribution is also λ
What will this R code generate? rpois(100, lambda=2) Give an example with customer service calls
100 random numbers from a Poisson distribution with λ = 2. For example, if incoming customer service calls average two per minute, this code will simulate 100 minutes, returning the number of calls in each of those 100 minutes.
What will this R code generate? rexp(n=100, rate=0.2) Give an example with customer service calls
100 random numbers from an exponential distribution where the mean number of events per time period is 0.2. So you could use it to simulate 100 intervals, in minutes, between service calls, where the average rate of incoming calls is 0.2 per minute.
How to obtain the λ parameter for a Poisson distribution?
Use software to model the data and obtain best-fitting distribution
What does "r × c" mean?
“rows by columns"