MLAT Recaps
37 flashcards · Computer Science · by Mathias777-crypto
What does the probability space consist of (3)?
Sample space Ω\OmegaΩ Event space F\mathcal FF Probability function PPP
Give an example for the probability space!
Tossing die: Sample space: {1, 2, 3, 4, 5, 6} Event space: {2}, {1, 3, 5}, {2, 4, 6} Probability function: P({2}) = 1/6, P({1, 3, 5}) = P({2, 4, 6)} = 1/2
What's the definition of a random variable as asked for in a job interview at Google? How to describe that in a mathematical statement?
A function that maps from sample space to a measurable space. X:Ω→EX : \Omega \rightarrow EX:Ω→E
Give an example which illustrates what a random variable is and its mathematical representation!
Tossing coin: E.g. the outcomes heads or tails are mapped to the measurable values -1 and 1 {H,T}→{−1,1}\{ H, T \} \rightarrow \{ -1, 1 \}{H,T}→{−1,1}
What are the two types of random variables?
Discrete and continuous
What's the simplest example of a discrete distribution and how is it defined mathematically (formula, bracket notation)? What is its expectation value by calculation?
Bernoulli distribution f(k,p)=pk(1−p)1−kf(k, p) = p^k (1 - p)^{1-k}f(k,p)=pk(1−p)1−k f(k,p)={p if k=1; (p−1) if k=0f(k, p) = \{ p \ \text{if} \ k = 1; \ (p-1) \ \text{if} \ k=0f(k,p)={p if k=1; (p−1) if k=0 E(X)=∑ixipi=1p+0(p−1)=p\mathbb{E} (X) = \sum_i x_i p_i = 1 p + 0 (p - 1) = pE(X)=∑ixipi=1p+0(p−1)=p
What are the two formulas for the variance of a random variable and how to proceed to calculate it?
Var(X)Var(X)Var(X) === E[(X−E(X))2]\mathbb{E} [( X - \mathbb{E} (X))^2]E[(X−E(X))2] === E(X2)−E2(X)\mathbb{E}(X^2) - \mathbb{E}^2 (X)E(X2)−E2(X) === ∑i(xi−E(X))2p(xi)\sum_i (x_i - \mathbb{E}(X))^2 p(x_i)∑i(xi−E(X))2p(xi)
How does a Poisson distribution roughly look like? What situation does it model?
It shows the probability, that within a certain time interval, an event happens k times, when on average, the event happens λ\lambdaλ times.
How to denote a distribution in notation? Give an example (Poisson)!
}, \text{parameter(s)})"> f(<outcome variable>,parameter(s))f( \text{<outcome variable>}, \text{parameter(s)})f(<outcome variable>,parameter(s)) f(k;λ)f( k; \lambda )f(k;λ)
Which shape does a covariance matrix have? Which symbol?
n×nn \times nn×n where n is the number of variables in a random variable vector Σ\SigmaΣ
How does data with no covariance generally look like?
All component variance directions are parallel to their axis.
What does the Central Limit Theorem state?
The mean values of n sufficiently large sample "bags" of any distribution are normally distributed when n goes to inifinity. They are standard normally distributed if e.g. decreased by the mean and divided by the standard deviation of the sample distribution.
What are dependent and independent random variables? What follows concerning their joint probability?
A dependent random variable's distribution changes depending on the outcome of another random variable. An independent random variable's distribution is independent of other variables outcomes. The joint probabilities of two independent random variables can be calculated by multiplying their marginal distributions, which is not possible with dependent random variables.
What's the formula for conditional probability? How does it make sense?
P(A∣B)=P(A∩B)P(B)P(A \vert B) = \frac{P(A \cap B)}{P(B)} P(A∣B)=P(B)P(A∩B) It is the proportion of the probability of A and B happening to the probability of either A and B or not A and B happening.
Wie lautet der Satz von Bayes? Wie werden die einzelnen Bestandteile genannt? Wie können diese im Lichte von Machine Learning verstanden werden?
p(A∣B)=p(B∣A)p(A)p(B)p(A \vert B) = \frac{p(B \vert A) p(A)}{p(B)}p(A∣B)=p(B)p(B∣A)p(A) posterior=likelihood×priorevidence\text{posterior} = \frac{ \text{likelihood} \times \text{prior}}{\text{evidence}}posterior=evidencelikelihood×prior A = parameters B = data p(A) ... Estimate of the parameters p(B) ... Tricky p(B|A) ... Likelihood of the data given the parameters p(A|B) ... Probability of the parameters given the data The parameters are updated according to the likelihood of the data given…
How does the example for Bayes' Theorem with the drug users and the drug test at the airport look like?
The question is: What is the probability, that someone who was tested positive for drug use is indeed a drug user? This can be calculated with only the basic drug users ratio and the true positive and the true negative rate given. By Bayes Theorem the desired probability (posterior) is the drug user ratio (prior) times the true positive rate (likelihood), divided by the overall ratio of positve tests (evidence). The latter figure can be calculated by adding the drug user ratio times the true po…
What does marginalization mean? How to mathematically describe the marginalization of the outcome xix_ixi when the outcomes of the other random variable are also called xxx ? In the drug usage example on Bayes' Theorem, does there a marginalization take place and if yes, which? How, again, does that look like mathematically?
It's the act of summing over all the joint probabilities of one variable's outcome. p(xi)=∑jp(xi,xj)p(x_i) = \sum_j p(x_i, x_j)p(xi)=∑jp(xi,xj) Yes, to find the positive-test rate, the margin of positive tests must be found, thus, it must be marginalized over the users and non-users. p(Pos)=∑userp(Pos,user)p(Pos) = \sum_{user} p(Pos, user)p(Pos)=∑userp(Pos,user)
When can you in fact not do a marginalization? What follows from that concerning Bayes Theorem?
With continuous distributions; Bayes Theorem is hard to use with continuous distributions.
What the hell is the law of total probability?
p(A)=∑np(A∣Bn)p(Bn)p(A) = \sum_n p(A \vert B_n) p(B_n)p(A)=∑np(A∣Bn)p(Bn)
What is a vector space? Give examples (2)!
A vector set is a set of vectors with addition and scalar multiplication defined, that fulfill 10 axioms like closure of vector addition or commutativity of vector addition. Examples for a vector space are: the set of all 2 by 3 matrices and the set of functions that map real numbers to real numbers.
The heck is a normed vector space? Give an example for an axiom that must be fulfilled!
It's a vector space on which a norm is defined. A norm is about a distance. Again, certain axioms must be fulfilled so that it is a normed vector space, for instance: ∣∣u+v∣∣≤∣∣u∣∣+∣∣v∣∣||u + v|| \leq ||u|| + ||v||∣∣u+v∣∣≤∣∣u∣∣+∣∣v∣∣
What is an inner product space?
It is a vector space with the inner product operation.
What is PCA in a nutshell?
In PCA, the covariance matrix C of X is decomposed into the product its eigenvectors U, its eigenvalues and the transpose of its eigenvectors. C=ULUTC = U L U^TC=ULUT Then, X is transformed into a lower-dimensional representation Y by being multiplied with the eigenvectors corresponding to the largest eigenvalues. Y=XU ∣U∈Rmx2 Y = X U \ | U \in \mathbb{R}^{m x 2} Y=XU ∣U∈Rmx2
What does Estimation Theory comprise?
Tools and notions how to estimate the parameters of an assumed distribution of given data
What is deep generative modelling? Give an example!
To learn to sample from an unknown data distribution; SORA videos are random samples from a data distribution of all the movies in the world (however, this distribution is incredibly complex). It is not necessary to know exactly about the distribution, though, because one learns to sample from it without.
What is the outline of MLAT starting with deep generative modelling? What are the two interludes?
PCA -> pPCA -> VAE -> HVAE -> Diffusion models Interludes: Statistical Learning Theory, Bayesian Deep Learning
How do we get the best model in Supervised Machine Learning Techniques?
We train a model by a loss function and try to make the generalization error as small as possible which is again measured by a loss function.
What's the formal declaration of a loss function? What can the model be?
L(y,g(x;w))L \left( y, g \left( x; w \right) \right)L(y,g(x;w)) A gaussian distribution, a neural network, etc.
Name 4 loss functions!
Zero-one loss Cross entropy loss Binary cross entropy loss Quadratic loss
What's the formula for quadratic loss?
Lq(y,g(x;w))=(y−g(x;w))2L_q \left( y, g(x;w) \right) = \left( y - g(x;w) \right)^2Lq(y,g(x;w))=(y−g(x;w))2
Which loss function will we focus on in deep generative modelling and why?
Quadratic loss, because we will try to reconstruct an input image on a pixel basis.
What is the empirical risk? What is its mathematical formulation?
The empirical risk is the sum of the losses of all data samples divided by the number of data samples. Remp(g(.;w),Z)=1l∑i=1lL(yi,g(xi;w))R_{emp} \left( g(.;w), Z \right) = \frac{1}{l} \sum_{i=1}^l L \left( y_i, g(x_i; w) \right)Remp(g(.;w),Z)=l1∑i=1lL(yi,g(xi;w))
How is the empirical risk related to the generalization error (also express mathematically) ? What does this relation imply?
If the number of samples goes to infinity, the empirical risk approximates the generalization error. liml→∞(Remp(g(.;w),Z))=R(g(.;w))\lim_{l \to \infty} \left( R_{emp} \left( g (.;w), Z \right) \right) = R \left( g(.; w) \right)liml→∞(Remp(g(.;w),Z))=R(g(.;w)) It implies that the more training data samples there are the better will the model perform on unseen data.
What is a common plot when it comes to the bias-variance trade-off? What does it show?
It shows training and the test error curves versus model complexity. If complexity is low, both training and test error are high; this is due to the fact, that the model cannot account for the variance in the data therefor the model variance is low and the bias is high. With increasing complexity, both training and test error decrease because the model can account for the variance in the data; so the variance rises while the bias drops. However, if the model complexity increases even further, t…
What is roughly meant by a model's bias and variance?
Bias: The ability of a model to fit the true relation Variance: The amount of how much a model's performance varies across data sets.
Which model is the goal concerning the bias-variance trade-off?
A model with low bias and low variance.
In how far is the common bias-variance trade-off picture outdated? What's the phenomenon called?
Large language models have shown, that with rising complexity, there comes a point when the test error shrinks again. It's called the double descent curve.