Machine learning
31 flashcards · Computer Science · by Daniel Vente
What is overfitting?
When a model becomes overly complicated and learns the noise included in a data set
Define the L_1 loss function as a function of theta
Define the L_2 loss function as a function of theta
Describe the gradient decent algorithm and explain its parameters
For all I set θj:=θj−α∂L(θ)∂θj\theta_j := \theta_j-\alpha\frac{\partial L(\theta)}{ \partial\theta_j}θj:=θj−α∂θj ∂L(θ) Until the error rate is converged. Here α\alphaα is the learning rate. We continually make steps in the parameter space in the negative of the gradient. i.e. the direction of steepest decent
explain the necessity of choosing a correct learning rate in gradient decent
If it is too small the convergence will be to slow, if it is too large, the decent can overshoot and start diverging
Produce the normal equation for linear regression and explain when it can be usefull
(XTX)−1XTy(X^TX)^{-1}X^Ty(XTX)−1XTy this is an analytical solution to the least squares loss function for linear regression. This can be faster than gradient decent if you don't have a lot of data and is parameter less
Whta is the cost function for logistic regression?
What is regularization?
An automatic way to reduce the scale of features to avoid overfitting. This is done by also minimising the norm of the parameter vector. Often done w.r.t. the L_1 or L_2 norm
Produce a cost function with regularisation
12m[∑i=1m(h(x(i))−y(i))2+λ∑j=1nθj2]\frac{1}{2m}\left[\sum^m_{i=1}(h(x^{(i)})-y^{(i)})^2+\lambda\sum^n_{j=1}\theta^2_j\right]2m 1[∑i=1m(h(x(i))−y(i))2+λ∑j=1nθj2]
explain the term data-leakage and why it is bad
Data-leakage is when data (either samples or parameters) from the test set are used in training, giving the model an overly optimistic rating when testing.
Define precision
TPTP+FP\frac{TP}{TP+FP}TP+FP TP with TP and FP being true positives and false positives respectiveally. i.e. what ratio of positive tests is correct positive?
define Recall
TPTP+FN\frac{TP}{TP+FN}TP+FN TP with TP and FN being true positives and false negatives respectiveally. i.e. what percentage of true positives was tested positively?
define the F1 score and explain when it can be useful
The harmonic mean of precision and recall. can be useful when a single number is necessary to compare performance automatically and neither recall nor precision is considered more important than the other.
Explain what a Receiver Operating Characteristic is
A plot of the true positive rates vs the false positive rate.
describe and give advantages and disadvangaes of using a confusion matrix
A confusion matric is a matrix that displays the actual classes of samples against the predicted ones. This can give a detailed information about where the model performs well and where it does not for both true and false classifications, but becomes hard to interpret when there are a lot of features and doesn't always give good information about how to improve the model or which one is best.
describe the process and advantages of using $k$-fold cross validation
split the training data into k (approximatedly) equally sized folds. every fold is used for testing/validation once and the rest is used for training. The final resulst is some average of the individual results. This is a good practice for model selection. Especially when there is too little data to set aside a decent testset.
Explain the difference between normalisation and standardisation
Normalisation compresses the data into the same range [0,1] traditionally. also called feature scaling Standardidation shapes the data into a standard gausian distribution by removing the mean and std
explain how polynomial regression works
more features are generated by multiplying the features together. then linear regression is done on the data including the newly generated features.
Explain the difference between ridge and lasso regression
Ridge uses the L_2 norm for regression while lasso uses L_1
Define a spline of order M
a piece wise polynomial built from M+K basis functions:
give the formula for a maximum Likelihood classifier
Y^=argmaxCP(X∣C)\widehat Y = argmax_C \mathbb{P}(X|C)Y =argmaxCP(X∣C)
give the formula for a maximum a posteriori classifier
Describe how a local density estimation classifier works and give several examples
Estimate the probability density at a position from the number of samples inside a certain volume. examples: decision tree tophat kernel KNN histogram
Define a Parzen density estimator and give several ewxamples of possible kernels
P(X)=1N∑n=1N1V⋅k(X−xnh)\mathbb{P}(X) = \frac1N \sum_{n=1}^N\frac1V\cdot k \left(\frac{X-x_n}{h}\right)P(X)=N 1∑n=1NV 1⋅k(h X−xn) where h is the bandwidth. Examples of kernels include: tophat gaussian exponential linear cosine
describe a hard and soft margin classifier
Hard margin it will attempt to find a decision boundary so that the two categories are separated by the widest possible margin. Only works for linearly separable categories. Soft margin adds slack variables to allow some miss-classification but punish it.
fomulate the hinge loss
Name a problem with using the hinge loss and a possible solution
It is non-differentiable at the hinge points. This can be solved by picking any of the subgradients i.e. subgradient-decent
Describe how Principle component analysis works
PCA finds the dimensions of maximum variance, and these can be used to perform dimensionality reduction. PCA is sensitive to feature scaling so normalisation is needed.
Define deep learning
any learning algorithm that attempts to learn intermediate representations
describe stochastic gradient decent
Gradient decent with a random subset of the training data at every iteration instead of the whole set
What is the legrangian relaxation for SVMs?
Lp=12∥β∥2+C∑i=1Nξi−∑i=1Nαi[yi(xiTβ+β0)−(1−ξi)]−∑i=1NμiξiL_p = \frac12\lVert\beta\rVert^2 + C\sum^N_{i=1}\xi_i-\sum^N_{i=1}\alpha_i\left[y_i(x_i^T\beta+\beta_0)-(1-\xi_i)\right]-\sum^N_{i=1}\mu_i\xi_iLp=2 1∥β∥2+C∑i=1Nξi−∑i=1Nαi[yi(xiTβ+β0)−(1−ξi)]−∑i=1Nμiξi