Machine Learning Supervised

118 flashcards · Computer Science · by Sarah Zoe Koch

1. What is the generalization error (GE)?

   The GE is the expected loss on future data for a given model

2. Give the definition (formula) of the GE:

3. In practice, we want to minimize the empirical risk RempR_{emp}Remp​ on our data set (Empirical Risk Minimization. Give the definition of RempR_{emp}Remp​ :

4. For what do we need our data to be independent and identically distributed (i.i.d.)?

   For estimating the generalization error via the empirical risk

5. T/F: A good estimation of the generalization error via the empirical risk generally requires a large number of samples

   True

6. T/F: A good estimation of the generalization error via the empirical risk generally requires use of a differentiable loss function

   False

7. T/F: A good estimation of the generalization error via the empirical risk generally requires that data points are uniformly distributed

   False

8. What are ROC curves used for?

   shows the performance of a binary classifier model (can be used for multi class classification) at varying threshold values

9. What does the ROC curve plot?

   The ROC curve is the plot of the true positive rate (TPR) against the false positive rate (FPR) at each threshold setting

10. T/F: Cross validation can help to optimize model hyperparameters

    True

11. T/F: Having a small training error but a large test error indicates the problem of underfitting

    False, this indicates the problem of overfitting

12. How does the model variance behave with increasing model class complexity?

    The model variance typically increases

13. T/F: Bias typically decreases with increasing model class complexity

    True

14. T/F: Bias typically increases with increasing model class complexity

    False, it decreases

15. How does the unavoidable error behave when training a model poorly?

    It stays the same

16. How does the unavoidable error behave when training a model "perfectly"?

    It stays the same

17. Give a short description of Bayes Optimal Classifier:

    BOC is a probabilistic model that makes the most probable prediction for a new sample

18. T/F: Supervised machine learning uses explicit knowledge to design models deductively

    False, this is True for explicit models. Supervised machine learning uses previously observed data to create a model inductively

19. Considering KNNs. Is it Underfitting or Overfitting: k=1

    Overfitting

20. What do KNNs use a distance measure for?

    For measuring closeness

21. T/F: KNN can also be used for regression

    True

22. T/F: KNN can work with unlabeled data

    False

23. T/F: KNN can work with labeled data

    True

24. T/F: Support vector regression only uses the sign of the discriminant function.

    False. The support vector regression uses the magnitude and not only the sign of the discriminant function

25. Which value does the Lagrange parameter αi{\alpha}_iαi​ need for a sample fxif{x}_ifxi​ to be a support vector?

    the corresponding lagrange parameter αi{\alpha}_iαi​ needs to be strictly larger than 0

26. T/F: Kernels allow to transform data into a lower-dimensional space where separability can be easier achieved

    False

27. T/F: The mathematical problem (constrained convex optimization) has a global solution

    True

28. T/F: The mathematical problem (constrained convex optimization) does not have a global solution

    False

29. Give the form of the Gaussian Kernel:

30. Give the form of the Polynomial Kernel:

31. Give the form of the Sigmoid Kernel:

32. T/F: SVMs work effectively if the number of dimensions (features) ddd is much smaller than the number of samples lll

    False. It works most effectively when the number of dimensions is greater than the number of samples

33. T/F: SVMs work effectively if the number of dimensions (features) ddd is bigger than the number of samples lll

    True

34. Multi-class classification with SVMs: T/F: Training pairwise classifiers is usually more time costly due to smaller training sets

    False, it is less time costly

35. Multi-class classification with SVMs: T/F: Training pairwise classifiers is usually less time costly due to bigger training sets

    False, due to smaller training sets

36. Multi-class classification with SVMs: T/F: Training pairwise classifiers is usually more time costly due to smaller training sets

    True

37. Multi-class classification with SVMs: How many classifiers (SVMs) are trained in the 1 vs 1 approach?

    M(M−1)2\frac{M(M-1)}{2}2M(M−1)​

38. Multi-class classification with SVMs: T/F: In the 1 vs all approach there are M−1M-1M−1 classifiers (SVMs)

    False

39. Multi-class classification with SVMs: T/F: In the one-versus-all approach, the final classification is done by majority vote

    False

40. Multi-class classification with SVMs: T/F: In the one-versus-one approach, the final classification is done by choosing the class which produces the largest value of all classifiers

    False

41. Multi-class classification with SVMs: T/F: In SVM-based multi-class classification, a multi-class problem is divided into many binary classifications tasks

    True

42. Multi-class classification with SVMs: T/F: In SVM-based multi-class classification, a multi-class problem is divided into two binary classifications tasks

    False, they are divided into two binary classification tasks

43. Multi-class classification with SVMs: T/F: In SVM-based multi-class classification, a single-class problem is divided into many binary classifications tasks

    False, a multi-class problem is divided into many binary classification tasks

44. Give the explanation of a decision tree:

    = is a predictive model to go from observations about an item (represented in the branches) to conclusions about the item's target value (represented in the leaves)

45. What do the leaves represent in a decision tree?

    the item's target value

46. What do the branches represent in a decision tree?

    the observations of an item

47. Why does Decision Tree Learning part the training data hierarchically?

    Such that leaf nodes are homogeneous in terms of the target class

48. T/F: Classification Trees are mainly designed for continuous data.

    False, for categorical data

49. T/F: Classification Trees are mainly designed for categorical data.

    True

50. T/F: Regression Trees can be applied to numerical features.

    True

51. T/F: Regression Trees can be applied to classification features.

    False

52. Decision Trees: What do we use the Stopping criterion and Pruning for?

    They are relevant for adjusting the complexity of decision trees

53. Decision Trees: What does the Stopping Criterion do?

    It tells when to stop further growing of the tree

54. Decision Trees: What does Pruning do?

    Tells us whether/how to collapse unnecessarily deep sub-trees

55. Decision Trees: Binary splits: T/F: The number of leaf nodes in a tree cannot be larger than 2S2^S2S with SSS being the number of splits

    True

56. Decision trees: What does a split maximize and what is the result of this? What is the criterion for the produced subsets?

    A split maximizes the information gain if the produced subsets are completely homogeneous

57. Decision Trees: T/F: Decision tree learning algorithms recursively split the training set into subsets

    True

58. Decision Trees: T/F: Decision tree learning algorithms recursively split the test set into subsets

    False, it splits the training set

59. Give an explanation of the Gini Impurity:

    it gives the probability of incorrectly classifying randomly chosen element in a dataset if it were randomly labeled according to the class distribution in the dataset

60. What is the maximal and minimal Gini Impurity of a data set with MMM classes ?

    max: (1−1M)(1-\frac{1}{M})(1−M1​) if classes are uniformly distributed in the set min: 000 if all samples of the set belong to one single class

61. When is the Gini Impurity equal to 0 ?

    If all samples of the set belong to one single class

62. T/F: Decision trees can be applied to categorical, but not to numerical features

    False, They can be applied to both. Regression Trees for numerical features

63. T/F: The decision tree complexity is adjusted via the stopping criterion and via pruning

    True

64. T/F: Information gain is a common splitting criterion for regression.

    False, for Classification

65. What is a common splitting criterion for Regression?

    Variance Reduction

66. Besides Information Gain, what is a common splitting criterion for Classification?

    Gini Impurity

67. Decision Trees: non-binary splits: T/F: The number of leaf nodes in a tree cannot be larger than 2S2^S2S with SSS being the number of splits

    False, this is only True for binary splits

68. What is a common measure of variable importance in random forests?

    The mean decrease in Gini Impurity

69. T/F: A random forest in general has smaller bias than the individual trees

    False

70. T/F: In AdaBoost, the less accurate the weak classifiers are, the higher the influence they are given for the classification output

    False

71. T/F: In a random forest, each individual decision tree is built upon a random bootstrap subsample of the original data set

    True

72. T/F: Gradient Boosted Trees are generally easier to tune compared to Random Forests

    False

73. What is forward stagewise additive modelling?

    A robust boosting method which can be applied to different loss functions

74. T/F: In Boosting, the main idea is that those samples which led to a small error (i.e. were classified well) in an iteration get a larger weight in the next iteration.

    False

75. T/F: In general, ensemble methods aim at reducing the bias and increasing the variance.

    False

76. In Logistic Regression: Can the loss function be minimized by gradient descent?

    Yes

77. Shortly, explain what gradient descent is:

    Gradient descent is an optimization algorithm for finding a local minimum of a differentiable function. It is used to find the values of the parameters which minimize the loss function

78. T/F: Logistic Regression is a non-convex problem as can be seen by the positive-semidefiniteness of the Hesse matrix

    False

79. Which form does the sigmoid function have?

    σ(x)=11+e−x\sigma(x) = \frac{1}{1+e^{-x}}σ(x)=1+e−x1​

80. The sigmoid function has the form: 1−x−11-x^{-1}1−x−1

    False, it has the form 11+e−x\frac{1}{1+e^{-x}}1+e−x1​

81. T/F: Assume that yyy data are generated around a function g(x)g(x)g(x) with Gaussian noise. Maximizing the negative log-likelihood of observing the data is equivalent to minimizing the squared error between yyy and g(x)g(x)g(x) .

    False

82. T/F: The momentum term in gradient descent is particularly helpful in flat regions of the loss function

    True

83. T/F: The momentum term in gradient descent is particularly helpful in hilly regions of the loss function

    False, in flat regions

84. T/F: Logistic regression has no closed-form solution

    True

85. T/F: Logistic regression has a closed-form solution

    False

86. Which function is also known as the Boltzmann distribution?

    The softmax function

87. For which classification tasks is the softmax function suitable?

    For multi-class classification tasks

88. If g(xi;w,ϕ)=0g(x_{i}; w, \phi) = 0g(xi​;w,ϕ)=0 and yi=1y_{i} = 1yi​=1 what is the update rule for the weight vector?

    w:=w+ηxiw := w+\eta x_{i}w:=w+ηxi​

89. T/F: If the data set is not linearly separable, then there exists a stopping criterion such that with probability 1 the perceptron learning algorithm will terminate and solve the learning task

    False, only if the data set is linearly seperable

90. T/F: If the data set is linearly separable, then there exists a stopping criterion such that with probability 1 the perceptron learning algorithm will terminate and solve the learning task

    True

91. T/F: If g(xi;w,ϕ)=1g(x_{i}; w, \phi) = 1 g(xi​;w,ϕ)=1 and yi=0y_i = 0yi​=0 then the update rule for the threshold is: ϕ=ϕ−η\phi = \phi-\etaϕ=ϕ−η

    False

92. T/F: A perceptron can solve the XOR problem, but only in certain instances (i.e. not with probability 1) when the algorithm terminates.

    False

93. T/F: The final solution (values of the and ) of the perceptron learning algorithm is always unique

    False

94. T/F: The final solution (values of the and ) of the perceptron learning algorithm is not always unique

    True

95. T/F: The perceptron is a simple linear threshold unit

    True

96. What is Double descent?

    As model complexity increases, the test error decreases, increases and decreases again (overfitting in the end)

97. T/F: Double descent is the phenomenon that the test risk first decreases with increasing number of parameters, then increases up to the interpolation threshold, and then decreases again. The final decrease takes place in the so-called over-parametrized regime, where there are more data points than parameters

    False

98. T/F: Multi-layer perceptrons are equivalent to fully connected feed-forward neural networks

    True

99. T/F: Multi-layer perceptrons are almost equivalent to fully connected feed-forward neural networks

    False, they are equivalent

100. What is the gradient descent update rule?

     wnew=wold−ηδwRw_{new} = w_{old} - \eta \delta_{w}Rwnew​=wold​−ηδw​R